[RFC,00/19] Integration of Kernel Test Framework (KTF) into the kernel tree
mbox series

Message ID cover.92d76bb4f6dcedc971d0b72a49e8e459a98bca54.1565676440.git-series.knut.omang@oracle.com
Headers show
Series
  • Integration of Kernel Test Framework (KTF) into the kernel tree
Related show

Message

Knut Omang Aug. 13, 2019, 6:09 a.m. UTC
KTF has already been available for a while as a separate git repository with
means to facilitate use with any kernel version.

KTF can be used both for "pure" unit testing and for more pragmatic
approaches to component testing. Apart from some useful features that
KTF uses from the kernel toolbox (such as kallsyms, kprobes),
KTF does not depend on any special environment such as UML or on a
large set of mocked up APIs to be useful. KTF basically allows test
code to be inserted and managed as separate kernel modules, while
providing the tests convenient access to almost the full range of kernel
APIs, both exposed and private. And once a KTF test set exists, it
should be fairly easy to compile and run it against older versions of
the kernel as well.

This series proposes a non-intrusive integration of KTF into the kernel
hopefully presented in digestable pieces.
For convenience, the patch set is also available on top of v5.2
at https://github.com/knuto/linux/pull/new/ktf_v1 .

The high level structure of the KTF code is as follows:

External dependencies for the user land side:
* libnl3 for netlink communication
* googletest for test runner, reporting and test selection support.

Kernel components:
* Simple core test and test suite related abstractions:
  core data structures ktf_case, ktf_test, an assertion macro "infrastructure"
  with ASSERT_* and EXPECT_* macros and helper functions
* Bookkeeping data structures:
  - ktf_map - a (key, value) mapping container used to implement management
    of instances of the higher level abstraction needs, such as ktf_handle and ktf_context.
  - ktf_handle: A global environment that a test runs within.
  - ktf_context: a test suite specific abstraction to allow a test to execute within
	one or more contexts. An example use of context can be a device
	instance: A test can be instantiated to run on all available such
	devices, according to test suite defined criteria.
* A generic netlink protocol (ktf_nl, ktf_unlproto) which defines operations to
  allow a user space part of a test to query the kernel for test information,
  invoke tests and get feedback reports about results.
* An alternative debugfs interface to allow examining and executing kernel-only tests
  without a user level program.
* Support for overriding and modifying behaviour of kernel calls.

User mode components:
* A test executor based on and integrated with Googletest.
  Googletest is one of several mature user land unit test suites for
  C/C++. The choice allowed us to focus on kernel specific
  functionality rather than having to reinvent too many wheels.
* Tools to aid in creating new test modules (suites):
  To facilitate a developer friendly way of testing internals of a module or the
  kernel itself, one of the important features of KTF, we often need to access
  symbols deliberately not exposed from a module.
  KTF contains a script used to create definitions based on kallsyms
  lookup for easy access to symbols not exposed by a module or the kernel.
  The user just provides a simple text file with a list of the symbols by
  module.

This series is an attempt to address feedback from several people
that having the functionality readily available within the kernel repository
is desired.

An in-tree KTF allows test suites to be provided with the kernel, which makes
it easy to use KTF based suites as part of the kernel selftests and driver
test suites. Having the ability to still build and run the latest versions of
test suites against older kernels should be of great value to stable maintainers,
distributions and driver maintainers, who would want to have an easy path,
with minimal backporting efforts to make sure that criterias implemented by
new test logic is also valid for these kernels.

Our definite goal moving forward is to try to satisfy both needs in a
transparent way. The plan is to let the standalone KTF repository follow the
in-kernel one, and to allow test suites to be maintained similarly,
and to support maintenance by proper tooling.

Mode of integration into the kernel
===================================

One feature of KTF is that it allows tests to work, look and feel similar
whether they execute entirely in user mode, entirely in kernel mode,
or half and half (hybrid tests). KTF consist of both user space
and kernel code. Unlike e.g. kselftest, KTF in the Github version
does not attempt to address the test runner aspects of testing.

Due to the need for building modules, KTF requires access to kernel module
build facilities (obj-m). But KTF also has nontrivial needs for user
land building, and we think it is good to keep the build structure in a way that
allows KTF to be built both in-tree and out-of-tree without
necessarily having to reconfigure the kernel.

This first version of kernel integration of KTF solves this challenge
by co-locating everything associated with KTF under ktf/ as in the
github version, but use the little used hostprogs-y and hostlibs-y
features to build the user space side. The first patch in the series is
fixes to make it work in a natural way to suit our needs.

Positioning for natural building within the kernel tree
=======================================================

Currently we find significant amount of C level tests within the following paths:

tools/testing/selftests/  (kselftests, almost entirely user space so far)
lib/  			  (various kernel level mostly unit tests)

and in the making::

kunit/			  (kernel only (UML))

So all kernel code are currently located directly within the kernel
build paths, accessed from the top level Makefile, to allow everything
to be controlled by config and from the main build targets for the
kernel. But this also poses challenges, in that .config has to be
modified to build tests. And once a .config is changed, we no longer
in principle logically operate on the same kernel.

A better approach seems to be to follow the principle
taken by kselftest: To have all the logic associated with the test
inside the test tree, and make it available for building separately
from the kernel itself. This require us to have a means to build
kernel modules from within the test tree, separately from the main
kernel paths. Currently this seems to only by supported via the M=
option used to build out-of-tree modules. This was also easy to get to work
for the kernel parts, based on the Github version of KTF, where we
already do this. With the additional need to compile user land code,
using the corresponding hostprogs-y and hostNNlibs-y seemed natural,
but this has been challenging: The build macros does not really
support hostprogs-y etc as "first class citizens" so some amount of
hacking is in there in this first draft version.
Using hostprogs-y etc is also a different approach that what is used
for C code in kselftest today, but we imagine that there's room for
unification here to get the best of both worlds, with the help of
the wider Kbuild community.

As an initial proposal, we have positioned ktf as an additional
kselftest target, under tools/testing/selftests/ktf, and the recommended:

make TARGETS="ktf" kselftest

way of building and running should work, even from normal user accounts,
if the user running it has sudo rights (for the kernel module insertion
and removal). This will run the selftests for KTF itself, and should
be a good starting point for adding more test cases. We also have had
activities going to take some of the existing test suites under lib/
and convert them into KTF based test suites, and will get back to this
later.

A trimmed down output from the above make target would look like this:

  ...
  CC [M]  tools/testing/selftests/ktf/kernel/ktf_override.o
  LD [M]  tools/testing/selftests/ktf/kernel/ktf.o
  HOSTCC  -fPIC tools/testing/selftests/ktf/lib/ktf_unlproto.o
  HOSTCXX -fPIC tools/testing/selftests/ktf/lib/ktf_int.o
  KTFSYMS tools/testing/selftests/ktf/selftest/ktf_syms.h
  CC [M]  tools/testing/selftests/ktf/selftest/self.o
  LD [M]  tools/testing/selftests/ktf/selftest/selftest.o
  HOSTCXX tools/testing/selftests/ktf/user/ktftest.o
  HOSTCXX tools/testing/selftests/ktf/user/hybrid.o
  HOSTLD  tools/testing/selftests/ktf/user/ktftest
  Building modules, stage 2.
  MODPOST 7 modules
  LD [M]  tools/testing/selftests/ktf/kernel/ktf.ko
  LD [M]  tools/testing/selftests/ktf/selftest/selftest.ko
  running tests
make BUILD=/net/abi/local/abi/build/kernel/ktf/tools/testing/selftests -f scripts/runtests.mk run_tests
TAP version 13
1..1
...
ok 1 selftests: ktf: runtests.sh

We're looking forward to feedback on this, and also to more discussion
around unit testing at the testing & fuzzing workshop at LPC!

Alan Maguire (3):
  ktf: Implementation of ktf support for overriding function entry and return.
  ktf: A simple debugfs interface to test results
  ktf: Simple coverage support

Knut Omang (16):
  kbuild: Fixes to rules for host-cshlib and host-cxxshlib
  ktf: Introduce the main part of the kernel side of ktf
  ktf: Introduce a generic netlink protocol for test result communication
  ktf: An implementation of a generic associative array container
  ktf: Configurable context support for network info setup
  ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
  ktf: Add documentation for Kernel Test Framework (KTF)
  ktf: Add a small test suite with a few tests to test KTF itself
  ktf: Main part of user land library for executing tests
  ktf: Integration logic for running ktf tests from googletest
  ktf: Internal debugging facilities
  ktf: Some simple examples
  ktf: Some user applications to run tests
  ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
  kselftests: Enable building ktf
  Documentation/dev-tools: Add index entry for KTF documentation

 Documentation/dev-tools/index.rst                   |    1 +-
 Documentation/dev-tools/ktf/concepts.rst            |  242 +++-
 Documentation/dev-tools/ktf/debugging.rst           |  248 +++-
 Documentation/dev-tools/ktf/examples.rst            |   26 +-
 Documentation/dev-tools/ktf/features.rst            |  307 ++++-
 Documentation/dev-tools/ktf/implementation.rst      |   70 +-
 Documentation/dev-tools/ktf/index.rst               |   14 +-
 Documentation/dev-tools/ktf/installation.rst        |   73 +-
 Documentation/dev-tools/ktf/introduction.rst        |  134 ++-
 Documentation/dev-tools/ktf/progref.rst             |  144 ++-
 scripts/Makefile.host                               |   17 +-
 tools/testing/selftests/Makefile                    |    1 +-
 tools/testing/selftests/ktf/Makefile                |   21 +-
 tools/testing/selftests/ktf/examples/Makefile       |   17 +-
 tools/testing/selftests/ktf/examples/h2.c           |   45 +-
 tools/testing/selftests/ktf/examples/h3.c           |   84 +-
 tools/testing/selftests/ktf/examples/h4.c           |   62 +-
 tools/testing/selftests/ktf/examples/hello.c        |   38 +-
 tools/testing/selftests/ktf/examples/kgdemo.c       |   61 +-
 tools/testing/selftests/ktf/kernel/Makefile         |   15 +-
 tools/testing/selftests/ktf/kernel/ktf.h            |  604 +++++++-
 tools/testing/selftests/ktf/kernel/ktf_context.c    |  409 +++++-
 tools/testing/selftests/ktf/kernel/ktf_cov.c        |  690 ++++++++-
 tools/testing/selftests/ktf/kernel/ktf_cov.h        |   94 +-
 tools/testing/selftests/ktf/kernel/ktf_debugfs.c    |  356 ++++-
 tools/testing/selftests/ktf/kernel/ktf_debugfs.h    |   34 +-
 tools/testing/selftests/ktf/kernel/ktf_map.c        |  261 +++-
 tools/testing/selftests/ktf/kernel/ktf_map.h        |  154 ++-
 tools/testing/selftests/ktf/kernel/ktf_netctx.c     |  132 ++-
 tools/testing/selftests/ktf/kernel/ktf_netctx.h     |   64 +-
 tools/testing/selftests/ktf/kernel/ktf_nl.c         |  516 ++++++-
 tools/testing/selftests/ktf/kernel/ktf_nl.h         |   15 +-
 tools/testing/selftests/ktf/kernel/ktf_override.c   |   45 +-
 tools/testing/selftests/ktf/kernel/ktf_override.h   |   15 +-
 tools/testing/selftests/ktf/kernel/ktf_test.c       |  397 +++++-
 tools/testing/selftests/ktf/kernel/ktf_test.h       |  381 ++++-
 tools/testing/selftests/ktf/kernel/ktf_unlproto.h   |  105 +-
 tools/testing/selftests/ktf/lib/Makefile            |   21 +-
 tools/testing/selftests/ktf/lib/ktf.h               |  114 +-
 tools/testing/selftests/ktf/lib/ktf_debug.cc        |   20 +-
 tools/testing/selftests/ktf/lib/ktf_debug.h         |   59 +-
 tools/testing/selftests/ktf/lib/ktf_int.cc          | 1031 ++++++++++++-
 tools/testing/selftests/ktf/lib/ktf_int.h           |   84 +-
 tools/testing/selftests/ktf/lib/ktf_run.cc          |  177 ++-
 tools/testing/selftests/ktf/lib/ktf_unlproto.c      |   21 +-
 tools/testing/selftests/ktf/scripts/ktf_syms.mk     |   16 +-
 tools/testing/selftests/ktf/scripts/resolve         |  188 ++-
 tools/testing/selftests/ktf/scripts/runtests.mk     |    3 +-
 tools/testing/selftests/ktf/scripts/runtests.sh     |  100 +-
 tools/testing/selftests/ktf/scripts/top_make.mk     |   14 +-
 tools/testing/selftests/ktf/selftest/Makefile       |   17 +-
 tools/testing/selftests/ktf/selftest/context.c      |  149 ++-
 tools/testing/selftests/ktf/selftest/context.h      |   15 +-
 tools/testing/selftests/ktf/selftest/context_self.h |   34 +-
 tools/testing/selftests/ktf/selftest/hybrid.c       |   35 +-
 tools/testing/selftests/ktf/selftest/hybrid.h       |   24 +-
 tools/testing/selftests/ktf/selftest/hybrid_self.h  |   27 +-
 tools/testing/selftests/ktf/selftest/ktf_syms.txt   |   17 +-
 tools/testing/selftests/ktf/selftest/self.c         |  661 ++++++++-
 tools/testing/selftests/ktf/user/Makefile           |   26 +-
 tools/testing/selftests/ktf/user/hybrid.cc          |   39 +-
 tools/testing/selftests/ktf/user/ktfcov.cc          |   68 +-
 tools/testing/selftests/ktf/user/ktfrun.cc          |   20 +-
 tools/testing/selftests/ktf/user/ktftest.cc         |   46 +-
 64 files changed, 8909 insertions(+), 9 deletions(-)
 create mode 100644 Documentation/dev-tools/ktf/concepts.rst
 create mode 100644 Documentation/dev-tools/ktf/debugging.rst
 create mode 100644 Documentation/dev-tools/ktf/examples.rst
 create mode 100644 Documentation/dev-tools/ktf/features.rst
 create mode 100644 Documentation/dev-tools/ktf/implementation.rst
 create mode 100644 Documentation/dev-tools/ktf/index.rst
 create mode 100644 Documentation/dev-tools/ktf/installation.rst
 create mode 100644 Documentation/dev-tools/ktf/introduction.rst
 create mode 100644 Documentation/dev-tools/ktf/progref.rst
 create mode 100644 tools/testing/selftests/ktf/Makefile
 create mode 100644 tools/testing/selftests/ktf/examples/Makefile
 create mode 100644 tools/testing/selftests/ktf/examples/h2.c
 create mode 100644 tools/testing/selftests/ktf/examples/h3.c
 create mode 100644 tools/testing/selftests/ktf/examples/h4.c
 create mode 100644 tools/testing/selftests/ktf/examples/hello.c
 create mode 100644 tools/testing/selftests/ktf/examples/kgdemo.c
 create mode 100644 tools/testing/selftests/ktf/kernel/Makefile
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_context.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_cov.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_cov.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_debugfs.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_debugfs.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_map.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_map.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_netctx.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_netctx.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_nl.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_nl.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_override.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_override.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_test.c
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_test.h
 create mode 100644 tools/testing/selftests/ktf/kernel/ktf_unlproto.h
 create mode 100644 tools/testing/selftests/ktf/lib/Makefile
 create mode 100644 tools/testing/selftests/ktf/lib/ktf.h
 create mode 100644 tools/testing/selftests/ktf/lib/ktf_debug.cc
 create mode 100644 tools/testing/selftests/ktf/lib/ktf_debug.h
 create mode 100644 tools/testing/selftests/ktf/lib/ktf_int.cc
 create mode 100644 tools/testing/selftests/ktf/lib/ktf_int.h
 create mode 100644 tools/testing/selftests/ktf/lib/ktf_run.cc
 create mode 100644 tools/testing/selftests/ktf/lib/ktf_unlproto.c
 create mode 100644 tools/testing/selftests/ktf/scripts/ktf_syms.mk
 create mode 100755 tools/testing/selftests/ktf/scripts/resolve
 create mode 100644 tools/testing/selftests/ktf/scripts/runtests.mk
 create mode 100755 tools/testing/selftests/ktf/scripts/runtests.sh
 create mode 100644 tools/testing/selftests/ktf/scripts/top_make.mk
 create mode 100644 tools/testing/selftests/ktf/selftest/Makefile
 create mode 100644 tools/testing/selftests/ktf/selftest/context.c
 create mode 100644 tools/testing/selftests/ktf/selftest/context.h
 create mode 100644 tools/testing/selftests/ktf/selftest/context_self.h
 create mode 100644 tools/testing/selftests/ktf/selftest/hybrid.c
 create mode 100644 tools/testing/selftests/ktf/selftest/hybrid.h
 create mode 100644 tools/testing/selftests/ktf/selftest/hybrid_self.h
 create mode 100644 tools/testing/selftests/ktf/selftest/ktf_syms.txt
 create mode 100644 tools/testing/selftests/ktf/selftest/self.c
 create mode 100644 tools/testing/selftests/ktf/user/Makefile
 create mode 100644 tools/testing/selftests/ktf/user/hybrid.cc
 create mode 100644 tools/testing/selftests/ktf/user/ktfcov.cc
 create mode 100644 tools/testing/selftests/ktf/user/ktfrun.cc
 create mode 100644 tools/testing/selftests/ktf/user/ktftest.cc

base-commit: 0ecfebd2b52404ae0c54a878c872bb93363ada36

Comments

Brendan Higgins Aug. 13, 2019, 8:10 a.m. UTC | #1
On Mon, Aug 12, 2019 at 11:11 PM Knut Omang <knut.omang@oracle.com> wrote:
>
> KTF has already been available for a while as a separate git repository with
> means to facilitate use with any kernel version.
>
> KTF can be used both for "pure" unit testing and for more pragmatic
> approaches to component testing. Apart from some useful features that
> KTF uses from the kernel toolbox (such as kallsyms, kprobes),
> KTF does not depend on any special environment such as UML or on a
> large set of mocked up APIs to be useful. KTF basically allows test
> code to be inserted and managed as separate kernel modules, while
> providing the tests convenient access to almost the full range of kernel
> APIs, both exposed and private. And once a KTF test set exists, it
> should be fairly easy to compile and run it against older versions of
> the kernel as well.
>
> This series proposes a non-intrusive integration of KTF into the kernel
> hopefully presented in digestable pieces.
> For convenience, the patch set is also available on top of v5.2
> at https://github.com/knuto/linux/pull/new/ktf_v1 .
>
> The high level structure of the KTF code is as follows:
>
> External dependencies for the user land side:
> * libnl3 for netlink communication
> * googletest for test runner, reporting and test selection support.

So I take it that KTF depends on a fully booted kernel with an up and
running userspace?

> Kernel components:
> * Simple core test and test suite related abstractions:
>   core data structures ktf_case, ktf_test, an assertion macro "infrastructure"
>   with ASSERT_* and EXPECT_* macros and helper functions
> * Bookkeeping data structures:
>   - ktf_map - a (key, value) mapping container used to implement management
>     of instances of the higher level abstraction needs, such as ktf_handle and ktf_context.
>   - ktf_handle: A global environment that a test runs within.
>   - ktf_context: a test suite specific abstraction to allow a test to execute within
>         one or more contexts. An example use of context can be a device
>         instance: A test can be instantiated to run on all available such
>         devices, according to test suite defined criteria.
> * A generic netlink protocol (ktf_nl, ktf_unlproto) which defines operations to
>   allow a user space part of a test to query the kernel for test information,
>   invoke tests and get feedback reports about results.
> * An alternative debugfs interface to allow examining and executing kernel-only tests
>   without a user level program.
> * Support for overriding and modifying behaviour of kernel calls.
>
> User mode components:
> * A test executor based on and integrated with Googletest.
>   Googletest is one of several mature user land unit test suites for
>   C/C++. The choice allowed us to focus on kernel specific
>   functionality rather than having to reinvent too many wheels.
> * Tools to aid in creating new test modules (suites):
>   To facilitate a developer friendly way of testing internals of a module or the
>   kernel itself, one of the important features of KTF, we often need to access
>   symbols deliberately not exposed from a module.
>   KTF contains a script used to create definitions based on kallsyms
>   lookup for easy access to symbols not exposed by a module or the kernel.
>   The user just provides a simple text file with a list of the symbols by
>   module.
>
> This series is an attempt to address feedback from several people
> that having the functionality readily available within the kernel repository
> is desired.
>
> An in-tree KTF allows test suites to be provided with the kernel, which makes
> it easy to use KTF based suites as part of the kernel selftests and driver
> test suites. Having the ability to still build and run the latest versions of
> test suites against older kernels should be of great value to stable maintainers,
> distributions and driver maintainers, who would want to have an easy path,
> with minimal backporting efforts to make sure that criterias implemented by
> new test logic is also valid for these kernels.
>
> Our definite goal moving forward is to try to satisfy both needs in a
> transparent way. The plan is to let the standalone KTF repository follow the
> in-kernel one, and to allow test suites to be maintained similarly,
> and to support maintenance by proper tooling.
>
> Mode of integration into the kernel
> ===================================
>
> One feature of KTF is that it allows tests to work, look and feel similar
> whether they execute entirely in user mode, entirely in kernel mode,
> or half and half (hybrid tests). KTF consist of both user space
> and kernel code. Unlike e.g. kselftest, KTF in the Github version
> does not attempt to address the test runner aspects of testing.
>
> Due to the need for building modules, KTF requires access to kernel module
> build facilities (obj-m). But KTF also has nontrivial needs for user
> land building, and we think it is good to keep the build structure in a way that
> allows KTF to be built both in-tree and out-of-tree without
> necessarily having to reconfigure the kernel.
>
> This first version of kernel integration of KTF solves this challenge
> by co-locating everything associated with KTF under ktf/ as in the
> github version, but use the little used hostprogs-y and hostlibs-y
> features to build the user space side. The first patch in the series is
> fixes to make it work in a natural way to suit our needs.
>
> Positioning for natural building within the kernel tree
> =======================================================
>
> Currently we find significant amount of C level tests within the following paths:
>
> tools/testing/selftests/  (kselftests, almost entirely user space so far)
> lib/                      (various kernel level mostly unit tests)
>
> and in the making::
>
> kunit/                    (kernel only (UML))
>
> So all kernel code are currently located directly within the kernel
> build paths, accessed from the top level Makefile, to allow everything
> to be controlled by config and from the main build targets for the
> kernel. But this also poses challenges, in that .config has to be
> modified to build tests. And once a .config is changed, we no longer
> in principle logically operate on the same kernel.
>
> A better approach seems to be to follow the principle
> taken by kselftest: To have all the logic associated with the test
> inside the test tree, and make it available for building separately
> from the kernel itself. This require us to have a means to build
> kernel modules from within the test tree, separately from the main
> kernel paths. Currently this seems to only by supported via the M=
> option used to build out-of-tree modules. This was also easy to get to work
> for the kernel parts, based on the Github version of KTF, where we
> already do this. With the additional need to compile user land code,
> using the corresponding hostprogs-y and hostNNlibs-y seemed natural,
> but this has been challenging: The build macros does not really
> support hostprogs-y etc as "first class citizens" so some amount of
> hacking is in there in this first draft version.
> Using hostprogs-y etc is also a different approach that what is used
> for C code in kselftest today, but we imagine that there's room for
> unification here to get the best of both worlds, with the help of
> the wider Kbuild community.
>
> As an initial proposal, we have positioned ktf as an additional
> kselftest target, under tools/testing/selftests/ktf, and the recommended:
>
> make TARGETS="ktf" kselftest
>
> way of building and running should work, even from normal user accounts,
> if the user running it has sudo rights (for the kernel module insertion
> and removal). This will run the selftests for KTF itself, and should
> be a good starting point for adding more test cases. We also have had
> activities going to take some of the existing test suites under lib/
> and convert them into KTF based test suites, and will get back to this
> later.
>
> A trimmed down output from the above make target would look like this:
>
>   ...
>   CC [M]  tools/testing/selftests/ktf/kernel/ktf_override.o
>   LD [M]  tools/testing/selftests/ktf/kernel/ktf.o
>   HOSTCC  -fPIC tools/testing/selftests/ktf/lib/ktf_unlproto.o
>   HOSTCXX -fPIC tools/testing/selftests/ktf/lib/ktf_int.o
>   KTFSYMS tools/testing/selftests/ktf/selftest/ktf_syms.h
>   CC [M]  tools/testing/selftests/ktf/selftest/self.o
>   LD [M]  tools/testing/selftests/ktf/selftest/selftest.o
>   HOSTCXX tools/testing/selftests/ktf/user/ktftest.o
>   HOSTCXX tools/testing/selftests/ktf/user/hybrid.o
>   HOSTLD  tools/testing/selftests/ktf/user/ktftest
>   Building modules, stage 2.
>   MODPOST 7 modules
>   LD [M]  tools/testing/selftests/ktf/kernel/ktf.ko
>   LD [M]  tools/testing/selftests/ktf/selftest/selftest.ko
>   running tests
> make BUILD=/net/abi/local/abi/build/kernel/ktf/tools/testing/selftests -f scripts/runtests.mk run_tests
> TAP version 13
> 1..1
> ...
> ok 1 selftests: ktf: runtests.sh
>
> We're looking forward to feedback on this, and also to more discussion
> around unit testing at the testing & fuzzing workshop at LPC!

Sounds good! Glad to have this on the lists!

> Alan Maguire (3):
>   ktf: Implementation of ktf support for overriding function entry and return.
>   ktf: A simple debugfs interface to test results
>   ktf: Simple coverage support
>
> Knut Omang (16):
>   kbuild: Fixes to rules for host-cshlib and host-cxxshlib
>   ktf: Introduce the main part of the kernel side of ktf
>   ktf: Introduce a generic netlink protocol for test result communication
>   ktf: An implementation of a generic associative array container
>   ktf: Configurable context support for network info setup
>   ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
>   ktf: Add documentation for Kernel Test Framework (KTF)
>   ktf: Add a small test suite with a few tests to test KTF itself
>   ktf: Main part of user land library for executing tests
>   ktf: Integration logic for running ktf tests from googletest
>   ktf: Internal debugging facilities
>   ktf: Some simple examples
>   ktf: Some user applications to run tests
>   ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
>   kselftests: Enable building ktf
>   Documentation/dev-tools: Add index entry for KTF documentation
>
>  Documentation/dev-tools/index.rst                   |    1 +-
>  Documentation/dev-tools/ktf/concepts.rst            |  242 +++-
>  Documentation/dev-tools/ktf/debugging.rst           |  248 +++-
>  Documentation/dev-tools/ktf/examples.rst            |   26 +-
>  Documentation/dev-tools/ktf/features.rst            |  307 ++++-
>  Documentation/dev-tools/ktf/implementation.rst      |   70 +-
>  Documentation/dev-tools/ktf/index.rst               |   14 +-
>  Documentation/dev-tools/ktf/installation.rst        |   73 +-
>  Documentation/dev-tools/ktf/introduction.rst        |  134 ++-
>  Documentation/dev-tools/ktf/progref.rst             |  144 ++-
>  scripts/Makefile.host                               |   17 +-
>  tools/testing/selftests/Makefile                    |    1 +-
>  tools/testing/selftests/ktf/Makefile                |   21 +-
>  tools/testing/selftests/ktf/examples/Makefile       |   17 +-
>  tools/testing/selftests/ktf/examples/h2.c           |   45 +-
>  tools/testing/selftests/ktf/examples/h3.c           |   84 +-
>  tools/testing/selftests/ktf/examples/h4.c           |   62 +-
>  tools/testing/selftests/ktf/examples/hello.c        |   38 +-
>  tools/testing/selftests/ktf/examples/kgdemo.c       |   61 +-
>  tools/testing/selftests/ktf/kernel/Makefile         |   15 +-
>  tools/testing/selftests/ktf/kernel/ktf.h            |  604 +++++++-
>  tools/testing/selftests/ktf/kernel/ktf_context.c    |  409 +++++-
>  tools/testing/selftests/ktf/kernel/ktf_cov.c        |  690 ++++++++-
>  tools/testing/selftests/ktf/kernel/ktf_cov.h        |   94 +-
>  tools/testing/selftests/ktf/kernel/ktf_debugfs.c    |  356 ++++-
>  tools/testing/selftests/ktf/kernel/ktf_debugfs.h    |   34 +-
>  tools/testing/selftests/ktf/kernel/ktf_map.c        |  261 +++-
>  tools/testing/selftests/ktf/kernel/ktf_map.h        |  154 ++-
>  tools/testing/selftests/ktf/kernel/ktf_netctx.c     |  132 ++-
>  tools/testing/selftests/ktf/kernel/ktf_netctx.h     |   64 +-
>  tools/testing/selftests/ktf/kernel/ktf_nl.c         |  516 ++++++-
>  tools/testing/selftests/ktf/kernel/ktf_nl.h         |   15 +-
>  tools/testing/selftests/ktf/kernel/ktf_override.c   |   45 +-
>  tools/testing/selftests/ktf/kernel/ktf_override.h   |   15 +-
>  tools/testing/selftests/ktf/kernel/ktf_test.c       |  397 +++++-
>  tools/testing/selftests/ktf/kernel/ktf_test.h       |  381 ++++-
>  tools/testing/selftests/ktf/kernel/ktf_unlproto.h   |  105 +-
>  tools/testing/selftests/ktf/lib/Makefile            |   21 +-
>  tools/testing/selftests/ktf/lib/ktf.h               |  114 +-
>  tools/testing/selftests/ktf/lib/ktf_debug.cc        |   20 +-
>  tools/testing/selftests/ktf/lib/ktf_debug.h         |   59 +-
>  tools/testing/selftests/ktf/lib/ktf_int.cc          | 1031 ++++++++++++-
>  tools/testing/selftests/ktf/lib/ktf_int.h           |   84 +-
>  tools/testing/selftests/ktf/lib/ktf_run.cc          |  177 ++-
>  tools/testing/selftests/ktf/lib/ktf_unlproto.c      |   21 +-
>  tools/testing/selftests/ktf/scripts/ktf_syms.mk     |   16 +-
>  tools/testing/selftests/ktf/scripts/resolve         |  188 ++-
>  tools/testing/selftests/ktf/scripts/runtests.mk     |    3 +-
>  tools/testing/selftests/ktf/scripts/runtests.sh     |  100 +-
>  tools/testing/selftests/ktf/scripts/top_make.mk     |   14 +-
>  tools/testing/selftests/ktf/selftest/Makefile       |   17 +-
>  tools/testing/selftests/ktf/selftest/context.c      |  149 ++-
>  tools/testing/selftests/ktf/selftest/context.h      |   15 +-
>  tools/testing/selftests/ktf/selftest/context_self.h |   34 +-
>  tools/testing/selftests/ktf/selftest/hybrid.c       |   35 +-
>  tools/testing/selftests/ktf/selftest/hybrid.h       |   24 +-
>  tools/testing/selftests/ktf/selftest/hybrid_self.h  |   27 +-
>  tools/testing/selftests/ktf/selftest/ktf_syms.txt   |   17 +-
>  tools/testing/selftests/ktf/selftest/self.c         |  661 ++++++++-
>  tools/testing/selftests/ktf/user/Makefile           |   26 +-
>  tools/testing/selftests/ktf/user/hybrid.cc          |   39 +-
>  tools/testing/selftests/ktf/user/ktfcov.cc          |   68 +-
>  tools/testing/selftests/ktf/user/ktfrun.cc          |   20 +-
>  tools/testing/selftests/ktf/user/ktftest.cc         |   46 +-
>  64 files changed, 8909 insertions(+), 9 deletions(-)
>  create mode 100644 Documentation/dev-tools/ktf/concepts.rst
>  create mode 100644 Documentation/dev-tools/ktf/debugging.rst
>  create mode 100644 Documentation/dev-tools/ktf/examples.rst
>  create mode 100644 Documentation/dev-tools/ktf/features.rst
>  create mode 100644 Documentation/dev-tools/ktf/implementation.rst
>  create mode 100644 Documentation/dev-tools/ktf/index.rst
>  create mode 100644 Documentation/dev-tools/ktf/installation.rst
>  create mode 100644 Documentation/dev-tools/ktf/introduction.rst
>  create mode 100644 Documentation/dev-tools/ktf/progref.rst
>  create mode 100644 tools/testing/selftests/ktf/Makefile
>  create mode 100644 tools/testing/selftests/ktf/examples/Makefile
>  create mode 100644 tools/testing/selftests/ktf/examples/h2.c
>  create mode 100644 tools/testing/selftests/ktf/examples/h3.c
>  create mode 100644 tools/testing/selftests/ktf/examples/h4.c
>  create mode 100644 tools/testing/selftests/ktf/examples/hello.c
>  create mode 100644 tools/testing/selftests/ktf/examples/kgdemo.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/Makefile
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_context.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_cov.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_cov.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_debugfs.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_debugfs.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_map.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_map.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_netctx.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_netctx.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_nl.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_nl.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_override.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_override.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_test.c
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_test.h
>  create mode 100644 tools/testing/selftests/ktf/kernel/ktf_unlproto.h
>  create mode 100644 tools/testing/selftests/ktf/lib/Makefile
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf.h
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf_debug.cc
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf_debug.h
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf_int.cc
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf_int.h
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf_run.cc
>  create mode 100644 tools/testing/selftests/ktf/lib/ktf_unlproto.c
>  create mode 100644 tools/testing/selftests/ktf/scripts/ktf_syms.mk
>  create mode 100755 tools/testing/selftests/ktf/scripts/resolve
>  create mode 100644 tools/testing/selftests/ktf/scripts/runtests.mk
>  create mode 100755 tools/testing/selftests/ktf/scripts/runtests.sh
>  create mode 100644 tools/testing/selftests/ktf/scripts/top_make.mk
>  create mode 100644 tools/testing/selftests/ktf/selftest/Makefile
>  create mode 100644 tools/testing/selftests/ktf/selftest/context.c
>  create mode 100644 tools/testing/selftests/ktf/selftest/context.h
>  create mode 100644 tools/testing/selftests/ktf/selftest/context_self.h
>  create mode 100644 tools/testing/selftests/ktf/selftest/hybrid.c
>  create mode 100644 tools/testing/selftests/ktf/selftest/hybrid.h
>  create mode 100644 tools/testing/selftests/ktf/selftest/hybrid_self.h
>  create mode 100644 tools/testing/selftests/ktf/selftest/ktf_syms.txt
>  create mode 100644 tools/testing/selftests/ktf/selftest/self.c
>  create mode 100644 tools/testing/selftests/ktf/user/Makefile
>  create mode 100644 tools/testing/selftests/ktf/user/hybrid.cc
>  create mode 100644 tools/testing/selftests/ktf/user/ktfcov.cc
>  create mode 100644 tools/testing/selftests/ktf/user/ktfrun.cc
>  create mode 100644 tools/testing/selftests/ktf/user/ktftest.cc
>
> base-commit: 0ecfebd2b52404ae0c54a878c872bb93363ada36
> --
> git-series 0.9.1
Brendan Higgins Aug. 13, 2019, 8:17 a.m. UTC | #2
On Mon, Aug 12, 2019 at 11:11 PM Knut Omang <knut.omang@oracle.com> wrote:
[...]
> Alan Maguire (3):
>   ktf: Implementation of ktf support for overriding function entry and return.
>   ktf: A simple debugfs interface to test results
>   ktf: Simple coverage support
>
> Knut Omang (16):
>   kbuild: Fixes to rules for host-cshlib and host-cxxshlib
>   ktf: Introduce the main part of the kernel side of ktf
>   ktf: Introduce a generic netlink protocol for test result communication
>   ktf: An implementation of a generic associative array container
>   ktf: Configurable context support for network info setup
>   ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
>   ktf: Add documentation for Kernel Test Framework (KTF)
>   ktf: Add a small test suite with a few tests to test KTF itself
>   ktf: Main part of user land library for executing tests
>   ktf: Integration logic for running ktf tests from googletest
>   ktf: Internal debugging facilities
>   ktf: Some simple examples
>   ktf: Some user applications to run tests
>   ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
>   kselftests: Enable building ktf
>   Documentation/dev-tools: Add index entry for KTF documentation
>
>  Documentation/dev-tools/index.rst                   |    1 +-
>  Documentation/dev-tools/ktf/concepts.rst            |  242 +++-
>  Documentation/dev-tools/ktf/debugging.rst           |  248 +++-
>  Documentation/dev-tools/ktf/examples.rst            |   26 +-
>  Documentation/dev-tools/ktf/features.rst            |  307 ++++-
>  Documentation/dev-tools/ktf/implementation.rst      |   70 +-
>  Documentation/dev-tools/ktf/index.rst               |   14 +-
>  Documentation/dev-tools/ktf/installation.rst        |   73 +-
>  Documentation/dev-tools/ktf/introduction.rst        |  134 ++-
>  Documentation/dev-tools/ktf/progref.rst             |  144 ++-
>  scripts/Makefile.host                               |   17 +-
>  tools/testing/selftests/Makefile                    |    1 +-
>  tools/testing/selftests/ktf/Makefile                |   21 +-
>  tools/testing/selftests/ktf/examples/Makefile       |   17 +-
>  tools/testing/selftests/ktf/examples/h2.c           |   45 +-
>  tools/testing/selftests/ktf/examples/h3.c           |   84 +-
>  tools/testing/selftests/ktf/examples/h4.c           |   62 +-
>  tools/testing/selftests/ktf/examples/hello.c        |   38 +-
>  tools/testing/selftests/ktf/examples/kgdemo.c       |   61 +-
>  tools/testing/selftests/ktf/kernel/Makefile         |   15 +-
>  tools/testing/selftests/ktf/kernel/ktf.h            |  604 +++++++-
>  tools/testing/selftests/ktf/kernel/ktf_context.c    |  409 +++++-
>  tools/testing/selftests/ktf/kernel/ktf_cov.c        |  690 ++++++++-
>  tools/testing/selftests/ktf/kernel/ktf_cov.h        |   94 +-
>  tools/testing/selftests/ktf/kernel/ktf_debugfs.c    |  356 ++++-
>  tools/testing/selftests/ktf/kernel/ktf_debugfs.h    |   34 +-
>  tools/testing/selftests/ktf/kernel/ktf_map.c        |  261 +++-
>  tools/testing/selftests/ktf/kernel/ktf_map.h        |  154 ++-
>  tools/testing/selftests/ktf/kernel/ktf_netctx.c     |  132 ++-
>  tools/testing/selftests/ktf/kernel/ktf_netctx.h     |   64 +-
>  tools/testing/selftests/ktf/kernel/ktf_nl.c         |  516 ++++++-
>  tools/testing/selftests/ktf/kernel/ktf_nl.h         |   15 +-
>  tools/testing/selftests/ktf/kernel/ktf_override.c   |   45 +-
>  tools/testing/selftests/ktf/kernel/ktf_override.h   |   15 +-
>  tools/testing/selftests/ktf/kernel/ktf_test.c       |  397 +++++-
>  tools/testing/selftests/ktf/kernel/ktf_test.h       |  381 ++++-
>  tools/testing/selftests/ktf/kernel/ktf_unlproto.h   |  105 +-
>  tools/testing/selftests/ktf/lib/Makefile            |   21 +-
>  tools/testing/selftests/ktf/lib/ktf.h               |  114 +-
>  tools/testing/selftests/ktf/lib/ktf_debug.cc        |   20 +-
>  tools/testing/selftests/ktf/lib/ktf_debug.h         |   59 +-
>  tools/testing/selftests/ktf/lib/ktf_int.cc          | 1031 ++++++++++++-
>  tools/testing/selftests/ktf/lib/ktf_int.h           |   84 +-
>  tools/testing/selftests/ktf/lib/ktf_run.cc          |  177 ++-
>  tools/testing/selftests/ktf/lib/ktf_unlproto.c      |   21 +-
>  tools/testing/selftests/ktf/scripts/ktf_syms.mk     |   16 +-
>  tools/testing/selftests/ktf/scripts/resolve         |  188 ++-
>  tools/testing/selftests/ktf/scripts/runtests.mk     |    3 +-
>  tools/testing/selftests/ktf/scripts/runtests.sh     |  100 +-
>  tools/testing/selftests/ktf/scripts/top_make.mk     |   14 +-
>  tools/testing/selftests/ktf/selftest/Makefile       |   17 +-
>  tools/testing/selftests/ktf/selftest/context.c      |  149 ++-
>  tools/testing/selftests/ktf/selftest/context.h      |   15 +-
>  tools/testing/selftests/ktf/selftest/context_self.h |   34 +-
>  tools/testing/selftests/ktf/selftest/hybrid.c       |   35 +-
>  tools/testing/selftests/ktf/selftest/hybrid.h       |   24 +-
>  tools/testing/selftests/ktf/selftest/hybrid_self.h  |   27 +-
>  tools/testing/selftests/ktf/selftest/ktf_syms.txt   |   17 +-
>  tools/testing/selftests/ktf/selftest/self.c         |  661 ++++++++-
>  tools/testing/selftests/ktf/user/Makefile           |   26 +-
>  tools/testing/selftests/ktf/user/hybrid.cc          |   39 +-
>  tools/testing/selftests/ktf/user/ktfcov.cc          |   68 +-
>  tools/testing/selftests/ktf/user/ktfrun.cc          |   20 +-
>  tools/testing/selftests/ktf/user/ktftest.cc         |   46 +-
>  64 files changed, 8909 insertions(+), 9 deletions(-)

It also looks like all your test code lives outside of the kernel
source dir. I take it you intend for tests to live in
tools/testing/selftests/ktf/ ?

[...]
gregkh@linuxfoundation.org Aug. 13, 2019, 8:23 a.m. UTC | #3
On Tue, Aug 13, 2019 at 08:09:15AM +0200, Knut Omang wrote:
> and in the making::
> 
> kunit/			  (kernel only (UML))

You are going to have to integrate this with kunit, to come up with a
superset of both in the end.

And I do not think that kunit is only UML, it's just that seems to be
what Brendan tests with, but should work with other arches as well.

thanks,

greg k-h
Knut Omang Aug. 13, 2019, 9:51 a.m. UTC | #4
On Tue, 2019-08-13 at 10:23 +0200, Greg Kroah-Hartman wrote:
> On Tue, Aug 13, 2019 at 08:09:15AM +0200, Knut Omang wrote:
> > and in the making::
> > 
> > kunit/			  (kernel only (UML))
> 
> You are going to have to integrate this with kunit, to come up with a
> superset of both in the end.

Yes, I agree - getting to a unified approach has been my intention since I first brought this
up at LPC'17.

> And I do not think that kunit is only UML, it's just that seems to be
> what Brendan tests with, but should work with other arches as well.

If I get Brendan right, it is UML only now but can be extended to also support 
kernels running on real hardware. Still it is kernel only, while KTF also has the 
hybrid mode, where a test can have code and assertions both in user mode and kernel mode.
This is made easier and more streamlined by letting all reporting happen from user mode.

Thanks!
Knut

> thanks,
> 
> greg k-h
Knut Omang Aug. 13, 2019, 11:29 a.m. UTC | #5
On Tue, 2019-08-13 at 01:17 -0700, Brendan Higgins wrote:
> On Mon, Aug 12, 2019 at 11:11 PM Knut Omang <knut.omang@oracle.com> wrote:
> [...]
> > Alan Maguire (3):
> >   ktf: Implementation of ktf support for overriding function entry and return.
> >   ktf: A simple debugfs interface to test results
> >   ktf: Simple coverage support
> > 
> > Knut Omang (16):
> >   kbuild: Fixes to rules for host-cshlib and host-cxxshlib
> >   ktf: Introduce the main part of the kernel side of ktf
> >   ktf: Introduce a generic netlink protocol for test result communication
> >   ktf: An implementation of a generic associative array container
> >   ktf: Configurable context support for network info setup
> >   ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
> >   ktf: Add documentation for Kernel Test Framework (KTF)
> >   ktf: Add a small test suite with a few tests to test KTF itself
> >   ktf: Main part of user land library for executing tests
> >   ktf: Integration logic for running ktf tests from googletest
> >   ktf: Internal debugging facilities
> >   ktf: Some simple examples
> >   ktf: Some user applications to run tests
> >   ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
> >   kselftests: Enable building ktf
> >   Documentation/dev-tools: Add index entry for KTF documentation
> > 
> >  Documentation/dev-tools/index.rst                   |    1 +-
> >  Documentation/dev-tools/ktf/concepts.rst            |  242 +++-
> >  Documentation/dev-tools/ktf/debugging.rst           |  248 +++-
> >  Documentation/dev-tools/ktf/examples.rst            |   26 +-
> >  Documentation/dev-tools/ktf/features.rst            |  307 ++++-
> >  Documentation/dev-tools/ktf/implementation.rst      |   70 +-
> >  Documentation/dev-tools/ktf/index.rst               |   14 +-
> >  Documentation/dev-tools/ktf/installation.rst        |   73 +-
> >  Documentation/dev-tools/ktf/introduction.rst        |  134 ++-
> >  Documentation/dev-tools/ktf/progref.rst             |  144 ++-
> >  scripts/Makefile.host                               |   17 +-
> >  tools/testing/selftests/Makefile                    |    1 +-
> >  tools/testing/selftests/ktf/Makefile                |   21 +-
> >  tools/testing/selftests/ktf/examples/Makefile       |   17 +-
> >  tools/testing/selftests/ktf/examples/h2.c           |   45 +-
> >  tools/testing/selftests/ktf/examples/h3.c           |   84 +-
> >  tools/testing/selftests/ktf/examples/h4.c           |   62 +-
> >  tools/testing/selftests/ktf/examples/hello.c        |   38 +-
> >  tools/testing/selftests/ktf/examples/kgdemo.c       |   61 +-
> >  tools/testing/selftests/ktf/kernel/Makefile         |   15 +-
> >  tools/testing/selftests/ktf/kernel/ktf.h            |  604 +++++++-
> >  tools/testing/selftests/ktf/kernel/ktf_context.c    |  409 +++++-
> >  tools/testing/selftests/ktf/kernel/ktf_cov.c        |  690 ++++++++-
> >  tools/testing/selftests/ktf/kernel/ktf_cov.h        |   94 +-
> >  tools/testing/selftests/ktf/kernel/ktf_debugfs.c    |  356 ++++-
> >  tools/testing/selftests/ktf/kernel/ktf_debugfs.h    |   34 +-
> >  tools/testing/selftests/ktf/kernel/ktf_map.c        |  261 +++-
> >  tools/testing/selftests/ktf/kernel/ktf_map.h        |  154 ++-
> >  tools/testing/selftests/ktf/kernel/ktf_netctx.c     |  132 ++-
> >  tools/testing/selftests/ktf/kernel/ktf_netctx.h     |   64 +-
> >  tools/testing/selftests/ktf/kernel/ktf_nl.c         |  516 ++++++-
> >  tools/testing/selftests/ktf/kernel/ktf_nl.h         |   15 +-
> >  tools/testing/selftests/ktf/kernel/ktf_override.c   |   45 +-
> >  tools/testing/selftests/ktf/kernel/ktf_override.h   |   15 +-
> >  tools/testing/selftests/ktf/kernel/ktf_test.c       |  397 +++++-
> >  tools/testing/selftests/ktf/kernel/ktf_test.h       |  381 ++++-
> >  tools/testing/selftests/ktf/kernel/ktf_unlproto.h   |  105 +-
> >  tools/testing/selftests/ktf/lib/Makefile            |   21 +-
> >  tools/testing/selftests/ktf/lib/ktf.h               |  114 +-
> >  tools/testing/selftests/ktf/lib/ktf_debug.cc        |   20 +-
> >  tools/testing/selftests/ktf/lib/ktf_debug.h         |   59 +-
> >  tools/testing/selftests/ktf/lib/ktf_int.cc          | 1031 ++++++++++++-
> >  tools/testing/selftests/ktf/lib/ktf_int.h           |   84 +-
> >  tools/testing/selftests/ktf/lib/ktf_run.cc          |  177 ++-
> >  tools/testing/selftests/ktf/lib/ktf_unlproto.c      |   21 +-
> >  tools/testing/selftests/ktf/scripts/ktf_syms.mk     |   16 +-
> >  tools/testing/selftests/ktf/scripts/resolve         |  188 ++-
> >  tools/testing/selftests/ktf/scripts/runtests.mk     |    3 +-
> >  tools/testing/selftests/ktf/scripts/runtests.sh     |  100 +-
> >  tools/testing/selftests/ktf/scripts/top_make.mk     |   14 +-
> >  tools/testing/selftests/ktf/selftest/Makefile       |   17 +-
> >  tools/testing/selftests/ktf/selftest/context.c      |  149 ++-
> >  tools/testing/selftests/ktf/selftest/context.h      |   15 +-
> >  tools/testing/selftests/ktf/selftest/context_self.h |   34 +-
> >  tools/testing/selftests/ktf/selftest/hybrid.c       |   35 +-
> >  tools/testing/selftests/ktf/selftest/hybrid.h       |   24 +-
> >  tools/testing/selftests/ktf/selftest/hybrid_self.h  |   27 +-
> >  tools/testing/selftests/ktf/selftest/ktf_syms.txt   |   17 +-
> >  tools/testing/selftests/ktf/selftest/self.c         |  661 ++++++++-
> >  tools/testing/selftests/ktf/user/Makefile           |   26 +-
> >  tools/testing/selftests/ktf/user/hybrid.cc          |   39 +-
> >  tools/testing/selftests/ktf/user/ktfcov.cc          |   68 +-
> >  tools/testing/selftests/ktf/user/ktfrun.cc          |   20 +-
> >  tools/testing/selftests/ktf/user/ktftest.cc         |   46 +-
> >  64 files changed, 8909 insertions(+), 9 deletions(-)
> 
> It also looks like all your test code lives outside of the kernel
> source dir. I take it you intend for tests to live in
> tools/testing/selftests/ktf/ ?

Good point,
with this RFC it would be just to add another directory under ktf/
but this was just to find a simple way to integrate it below selftests,
without interfering with the current structure. 

I imagine a tighter integration/unification between normal Kbuild targets and selftests 
targets that also took kernel module building into consideration would be a better solution.

I think this is a good topic for discussion.
As I indicate above, I think it is problematic that test code has to be explicitly 
configured in as we configure code features of the kernel, which changes the "logical" 
kernel we build.

So some more "native" support for test modules are desired, IMHO.

Knut

> [...]
Brendan Higgins Aug. 13, 2019, 5:02 p.m. UTC | #6
On Tue, Aug 13, 2019 at 2:51 AM Knut Omang <knut.omang@oracle.com> wrote:
>
> On Tue, 2019-08-13 at 10:23 +0200, Greg Kroah-Hartman wrote:
> > On Tue, Aug 13, 2019 at 08:09:15AM +0200, Knut Omang wrote:
> > > and in the making::
> > >
> > > kunit/                        (kernel only (UML))
> >
> > You are going to have to integrate this with kunit, to come up with a
> > superset of both in the end.
>
> Yes, I agree - getting to a unified approach has been my intention since I first brought this
> up at LPC'17.
>
> > And I do not think that kunit is only UML, it's just that seems to be
> > what Brendan tests with, but should work with other arches as well.
>
> If I get Brendan right, it is UML only now but can be extended to also support
> kernels running on real hardware. Still it is kernel only, while KTF also has the
> hybrid mode, where a test can have code and assertions both in user mode and kernel mode.
> This is made easier and more streamlined by letting all reporting happen from user mode.

Nope, the KUnit patchset currently under review *does* support any
architecture; we have tested it on x86, ARM, and UML, but it should
work on any architecture.

I added support for that a while ago due to popular demand.
Brendan Higgins Aug. 13, 2019, 5:50 p.m. UTC | #7
On Tue, Aug 13, 2019 at 4:29 AM Knut Omang <knut.omang@oracle.com> wrote:
>
> On Tue, 2019-08-13 at 01:17 -0700, Brendan Higgins wrote:
> > On Mon, Aug 12, 2019 at 11:11 PM Knut Omang <knut.omang@oracle.com> wrote:
> > [...]
> > > Alan Maguire (3):
> > >   ktf: Implementation of ktf support for overriding function entry and return.
> > >   ktf: A simple debugfs interface to test results
> > >   ktf: Simple coverage support
> > >
> > > Knut Omang (16):
> > >   kbuild: Fixes to rules for host-cshlib and host-cxxshlib
> > >   ktf: Introduce the main part of the kernel side of ktf
> > >   ktf: Introduce a generic netlink protocol for test result communication
> > >   ktf: An implementation of a generic associative array container
> > >   ktf: Configurable context support for network info setup
> > >   ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
> > >   ktf: Add documentation for Kernel Test Framework (KTF)
> > >   ktf: Add a small test suite with a few tests to test KTF itself
> > >   ktf: Main part of user land library for executing tests
> > >   ktf: Integration logic for running ktf tests from googletest
> > >   ktf: Internal debugging facilities
> > >   ktf: Some simple examples
> > >   ktf: Some user applications to run tests
> > >   ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
> > >   kselftests: Enable building ktf
> > >   Documentation/dev-tools: Add index entry for KTF documentation
> > >
> > >  Documentation/dev-tools/index.rst                   |    1 +-
> > >  Documentation/dev-tools/ktf/concepts.rst            |  242 +++-
> > >  Documentation/dev-tools/ktf/debugging.rst           |  248 +++-
> > >  Documentation/dev-tools/ktf/examples.rst            |   26 +-
> > >  Documentation/dev-tools/ktf/features.rst            |  307 ++++-
> > >  Documentation/dev-tools/ktf/implementation.rst      |   70 +-
> > >  Documentation/dev-tools/ktf/index.rst               |   14 +-
> > >  Documentation/dev-tools/ktf/installation.rst        |   73 +-
> > >  Documentation/dev-tools/ktf/introduction.rst        |  134 ++-
> > >  Documentation/dev-tools/ktf/progref.rst             |  144 ++-
> > >  scripts/Makefile.host                               |   17 +-
> > >  tools/testing/selftests/Makefile                    |    1 +-
> > >  tools/testing/selftests/ktf/Makefile                |   21 +-
> > >  tools/testing/selftests/ktf/examples/Makefile       |   17 +-
> > >  tools/testing/selftests/ktf/examples/h2.c           |   45 +-
> > >  tools/testing/selftests/ktf/examples/h3.c           |   84 +-
> > >  tools/testing/selftests/ktf/examples/h4.c           |   62 +-
> > >  tools/testing/selftests/ktf/examples/hello.c        |   38 +-
> > >  tools/testing/selftests/ktf/examples/kgdemo.c       |   61 +-
> > >  tools/testing/selftests/ktf/kernel/Makefile         |   15 +-
> > >  tools/testing/selftests/ktf/kernel/ktf.h            |  604 +++++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_context.c    |  409 +++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_cov.c        |  690 ++++++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_cov.h        |   94 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_debugfs.c    |  356 ++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_debugfs.h    |   34 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_map.c        |  261 +++-
> > >  tools/testing/selftests/ktf/kernel/ktf_map.h        |  154 ++-
> > >  tools/testing/selftests/ktf/kernel/ktf_netctx.c     |  132 ++-
> > >  tools/testing/selftests/ktf/kernel/ktf_netctx.h     |   64 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_nl.c         |  516 ++++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_nl.h         |   15 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_override.c   |   45 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_override.h   |   15 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_test.c       |  397 +++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_test.h       |  381 ++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_unlproto.h   |  105 +-
> > >  tools/testing/selftests/ktf/lib/Makefile            |   21 +-
> > >  tools/testing/selftests/ktf/lib/ktf.h               |  114 +-
> > >  tools/testing/selftests/ktf/lib/ktf_debug.cc        |   20 +-
> > >  tools/testing/selftests/ktf/lib/ktf_debug.h         |   59 +-
> > >  tools/testing/selftests/ktf/lib/ktf_int.cc          | 1031 ++++++++++++-
> > >  tools/testing/selftests/ktf/lib/ktf_int.h           |   84 +-
> > >  tools/testing/selftests/ktf/lib/ktf_run.cc          |  177 ++-
> > >  tools/testing/selftests/ktf/lib/ktf_unlproto.c      |   21 +-
> > >  tools/testing/selftests/ktf/scripts/ktf_syms.mk     |   16 +-
> > >  tools/testing/selftests/ktf/scripts/resolve         |  188 ++-
> > >  tools/testing/selftests/ktf/scripts/runtests.mk     |    3 +-
> > >  tools/testing/selftests/ktf/scripts/runtests.sh     |  100 +-
> > >  tools/testing/selftests/ktf/scripts/top_make.mk     |   14 +-
> > >  tools/testing/selftests/ktf/selftest/Makefile       |   17 +-
> > >  tools/testing/selftests/ktf/selftest/context.c      |  149 ++-
> > >  tools/testing/selftests/ktf/selftest/context.h      |   15 +-
> > >  tools/testing/selftests/ktf/selftest/context_self.h |   34 +-
> > >  tools/testing/selftests/ktf/selftest/hybrid.c       |   35 +-
> > >  tools/testing/selftests/ktf/selftest/hybrid.h       |   24 +-
> > >  tools/testing/selftests/ktf/selftest/hybrid_self.h  |   27 +-
> > >  tools/testing/selftests/ktf/selftest/ktf_syms.txt   |   17 +-
> > >  tools/testing/selftests/ktf/selftest/self.c         |  661 ++++++++-
> > >  tools/testing/selftests/ktf/user/Makefile           |   26 +-
> > >  tools/testing/selftests/ktf/user/hybrid.cc          |   39 +-
> > >  tools/testing/selftests/ktf/user/ktfcov.cc          |   68 +-
> > >  tools/testing/selftests/ktf/user/ktfrun.cc          |   20 +-
> > >  tools/testing/selftests/ktf/user/ktftest.cc         |   46 +-
> > >  64 files changed, 8909 insertions(+), 9 deletions(-)
> >
> > It also looks like all your test code lives outside of the kernel
> > source dir. I take it you intend for tests to live in
> > tools/testing/selftests/ktf/ ?
>
> Good point,
> with this RFC it would be just to add another directory under ktf/
> but this was just to find a simple way to integrate it below selftests,
> without interfering with the current structure.
>
> I imagine a tighter integration/unification between normal Kbuild targets and selftests
> targets that also took kernel module building into consideration would be a better solution.

I think tests should live alongside the code that they test, so if I
understand what you are saying, then I agree. Where do you think KTF
should go then?

> I think this is a good topic for discussion.
> As I indicate above, I think it is problematic that test code has to be explicitly
> configured in as we configure code features of the kernel, which changes the "logical"
> kernel we build.

Yep, Luis Chamberlain and I have been discussing this for a while (I
should probably try to open up that conversation and add you to it);
it's a very tricky problem. I think that in the long term a good goal
is to have a way to express code dependencies separate from the
configuration system, but that would probably mean some substantial
changes to Kbuild and friends. Even then, we have all these macros
that generate different code (not just on/off) depending on how the
kernel is configured.

> So some more "native" support for test modules are desired, IMHO.

What do you mean by "native"? Just having the tests be more aware of the code?