@@ -4,21 +4,20 @@
Running tests with kunit_tool
=============================
-We can either run KUnit tests using kunit_tool or can run tests
-manually, and then use kunit_tool to parse the results. To run tests
-manually, see: Documentation/dev-tools/kunit/run_manual.rst.
-As long as we can build the kernel, we can run KUnit.
+This documentation describes running tests with kunit_tool script.
+If you'd like to manually run tests, see
+Documentation/dev-tools/kunit/run_manual.rst.
kunit_tool is a Python script which configures and builds a kernel, runs
tests, and formats the test results.
-Run command:
+To run the tests:
.. code-block::
./tools/testing/kunit/kunit.py run
-We should see the following:
+You should see the following output:
.. code-block::
@@ -26,124 +25,112 @@ We should see the following:
Building KUnit kernel...
Starting KUnit kernel...
-We may want to use the following options:
+You can also specify options to the script when running the tests. For
+example, to run tests utilizing all available CPUs within 30 seconds
+time limit:
.. code-block::
./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`
-- ``--timeout`` sets a maximum amount of time for tests to run.
-- ``--jobs`` sets the number of threads to build the kernel.
+For list of all options, see::
-kunit_tool will generate a ``.kunitconfig`` with a default
-configuration, if no other ``.kunitconfig`` file exists
-(in the build directory). In addition, it verifies that the
-generated ``.config`` file contains the ``CONFIG`` options in the
-``.kunitconfig``.
-It is also possible to pass a separate ``.kunitconfig`` fragment to
-kunit_tool. This is useful if we have several different groups of
-tests we want to run independently, or if we want to use pre-defined
-test configs for certain subsystems.
+ ./tools/testing/kunit/kunit.py run --help
+
+Kernel configuration
+====================
+
+kunit_tool will generate kernel configuration named ``.kunitconfig``
+if it doesn't exist in the build directory. For list of selected options by
+default, see ``tools/testing/kunit/configs/default.config`` file.
+
+In addition, it verifies that the synchronized ``.config`` file
+contains options enabled in ``.kunitconfig``. It will errored out if
+you have not enabled option dependencies required by all selected options.
+
+It is also possible to specify different ``.kunitconfig`` to
+kunit_tool. This is useful if you have several different groups of
+tests that you want to run independently, or if you want to use pre-defined
+configurations for certain subsystems.
To use a different ``.kunitconfig`` file (such as one
-provided to test a particular subsystem), pass it as an option:
+provided to test a particular subsystem), specify ``--kunitconfig`` option.
+For example, to run ext4 tests:
.. code-block::
./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig
-To view kunit_tool flags (optional command-line arguments), run:
+Customizing configuration
+-------------------------
-.. code-block::
+A ``.kunitconfig`` is a configuration file generated by
+``make savedefconfig``, used for running enabled set of tests. This file
+contains configuration options with specific enabled tests. The file also
+contains any other coptions required by the tests, for example
+dependencies for features under tests, arch-specific configs and so on.
- ./tools/testing/kunit/kunit.py run --help
-
-Creating a ``.kunitconfig`` file
-================================
-
-If we want to run a specific set of tests (rather than those listed
-in the KUnit ``defconfig``), we can provide Kconfig options in the
-``.kunitconfig`` file. For default .kunitconfig, see:
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/kunit/configs/default.config.
-A ``.kunitconfig`` is a ``minconfig`` (a .config
-generated by running ``make savedefconfig``), used for running a
-specific set of tests. This file contains the regular Kernel configs
-with specific test targets. The ``.kunitconfig`` also
-contains any other config options required by the tests (For example:
-dependencies for features under tests, configs that enable/disable
-certain code blocks, arch configs and so on).
-
-To create a ``.kunitconfig``, using the KUnit ``defconfig``:
+To generate default ``.kunitconfig``:
.. code-block::
cd $PATH_TO_LINUX_REPO
cp tools/testing/kunit/configs/default.config .kunit/.kunitconfig
-We can then add any other Kconfig options. For example:
+You can then customize the configuration. As with configuring regular kernels
+for production deployment, it is recommended to use
+:doc:`one of configuration interfaces </kbuild/kconfig>`. For example,
+to configure through menuconfig::
-.. code-block::
+ make O=.kunit KCONFIG_CONFIG=.kunit/.kunitconfig menuconfig
- CONFIG_LIST_KUNIT_TEST=y
+.. note::
+ ``.config`` will only be resynchronized if it is the superset of
+ ``.kunitconfig``. Removing any options from the latter will not
+ resynchronize the former. In that case, you need to manually
+ update the former by::
-kunit_tool ensures that all config options in ``.kunitconfig`` are
-set in the kernel ``.config`` before running the tests. It warns if we
-have not included the options dependencies.
+ cp .kunitconfig .config
-.. note:: Removing something from the ``.kunitconfig`` will
- not rebuild the ``.config file``. The configuration is only
- updated if the ``.kunitconfig`` is not a subset of ``.config``.
- This means that we can use other tools
- (For example: ``make menuconfig``) to adjust other config options.
- The build dir needs to be set for ``make menuconfig`` to
- work, therefore by default use ``make O=.kunit menuconfig``.
+Step-by-step run
+================
-Configuring, building, and running tests
-========================================
+It is possible to sequentially run testing steps for finer-grained control.
-If we want to make manual changes to the KUnit build process, we
-can run part of the KUnit build process independently.
-When running kunit_tool, from a ``.kunitconfig``, we can generate a
-``.config`` by using the ``config`` argument:
+First, synchronize ``.config`` from ``.kunitconfig`` that have been
+configured before:
.. code-block::
./tools/testing/kunit/kunit.py config
-To build a KUnit kernel from the current ``.config``, we can use the
-``build`` argument:
+Before the tests can be run, the kernel needs to be built first:
.. code-block::
./tools/testing/kunit/kunit.py build
-If we already have built UML kernel with built-in KUnit tests, we
-can run the kernel, and display the test results with the ``exec``
-argument:
+The whole tests can now be executed:
.. code-block::
./tools/testing/kunit/kunit.py exec
-The ``run`` command discussed in section: **Running tests with kunit_tool**,
-is equivalent to running the above three commands in sequence.
-
Parsing test results
====================
-KUnit tests output displays results in TAP (Test Anything Protocol)
-format. When running tests, kunit_tool parses this output and prints
-a summary. To see the raw test results in TAP format, we can pass the
-``--raw_output`` argument:
+KUnit generates test results in TAP (Test Anything Protocol) format. When
+the tests are running, kunit_tool parses the result and prints
+the summary. To see the results in raw TAP format, you can pass
+``--raw_output`` option:
.. code-block::
./tools/testing/kunit/kunit.py run --raw_output
-If we have KUnit results in the raw TAP format, we can parse them and
-print the human-readable summary with the ``parse`` command for
-kunit_tool. This accepts a filename for an argument, or will read from
-standard input.
+Now you can parse the output and print the human-readable summary with the
+``parse`` command. It accepts the filename argument. If none is given, it
+will read from standard input.
.. code-block:: bash
@@ -155,74 +142,63 @@ standard input.
Filtering tests
===============
-By passing a bash style glob filter to the ``exec`` or ``run``
-commands, we can run a subset of the tests built into a kernel . For
-example: if we only want to run KUnit resource tests, use:
+You can also specify tests using :manpage:`glob(7)` pattern. For example,
+to run resource tests (which are prefixed by ``kunit-resource``):
.. code-block::
./tools/testing/kunit/kunit.py run 'kunit-resource*'
-This uses the standard glob format with wildcard characters.
-
.. _kunit-on-qemu:
Running tests on QEMU
=====================
-kunit_tool supports running tests on qemu as well as
-via UML. To run tests on qemu, by default it requires two flags:
+Besides running tests with UML kernel, kunit_wrapper also supports
+running tests on QEMU. Two options are required for this to work:
-- ``--arch``: Selects a configs collection (Kconfig, qemu config options
- and so on), that allow KUnit tests to be run on the specified
- architecture in a minimal way. The architecture argument is same as
- the option name passed to the ``ARCH`` variable used by Kbuild.
- Not all architectures currently support this flag, but we can use
- ``--qemu_config`` to handle it. If ``um`` is passed (or this flag
- is ignored), the tests will run via UML. Non-UML architectures,
- for example: i386, x86_64, arm and so on; run on qemu.
+- ``--arch``: Run tests for the specified architecture. The value will be
+ passed to make as ``ARCH`` environment variable.
+ Not all architectures currently support this flag. For such architectures,
+ use ``--qemu_config`` instead (discussed later). If ``um`` is specifed (or
+ if this option is not given), the tests will run in UML kernel. Otherwise,
+ the tests run on QEMU.
-- ``--cross_compile``: Specifies the Kbuild toolchain. It passes the
- same argument as passed to the ``CROSS_COMPILE`` variable used by
- Kbuild. As a reminder, this will be the prefix for the toolchain
- binaries such as GCC. For example:
+- ``--cross_compile``: Specifies the cross-compiler toolchain prefix.
+ The value will be used for ``CROSS_COMPILE`` environment variable when
+ building the kernel.
- - ``sparc64-linux-gnu`` if we have the sparc toolchain installed on
- our system.
+Examples:
- - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux``
- if we have downloaded the microblaze toolchain from the 0-day
- website to a directory in our home directory called toolchains.
+ - To run tests for native x86_64 architecture:
-This means that for most architectures, running under qemu is as simple as:
-
-.. code-block:: bash
+ .. code-block:: bash
./tools/testing/kunit/kunit.py run --arch=x86_64
-When cross-compiling, we'll likely need to specify a different toolchain, for
-example:
+ - To cross-compile and test for s390:
-.. code-block:: bash
+ .. code-block:: bash
./tools/testing/kunit/kunit.py run \
--arch=s390 \
--cross_compile=s390x-linux-gnu-
-If we want to run KUnit tests on an architecture not supported by
-the ``--arch`` flag, or want to run KUnit tests on qemu using a
-non-default configuration; then we can write our own``QemuConfig``.
-These ``QemuConfigs`` are written in Python. They have an import line
-``from..qemu_config import QemuArchParams`` at the top of the file.
-The file must contain a variable called ``QEMU_ARCH`` that has an
-instance of ``QemuArchParams`` assigned to it. See example in:
-``tools/testing/kunit/qemu_configs/x86_64.py``.
+Custom QEMU configuration
+-------------------------
+If you want to test on an architecture unsupported by the ``--arch`` option,
+or to customize QEMU invocation, you can write ``QemuConfig`` configuration,
+which is written in Python. The file begins with
+:code:`from..qemu_config import QemuArchParams` as import directive.
+The file must contain an object called ``QEMU_ARCH`` which is
+built from ``QemuArchParams`` class.
-Once we have a ``QemuConfig``, we can pass it into kunit_tool,
-using the ``--qemu_config`` flag. When used, this flag replaces the
-``--arch`` flag. For example: using
-``tools/testing/kunit/qemu_configs/x86_64.py``, the invocation appear
-as
+Once ``QemuConfig`` have been configured, you can specify the file to
+kunit_tool using the ``--qemu_config`` option. This overrides ``--arch``
+option.
+
+For example, to specify QEMU configuration for native x86_64 architecture
+with 12 build jobs and 60 seconds timeout:
.. code-block:: bash
@@ -231,93 +207,9 @@ as
--jobs=12 \
--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
-Running command-line arguments
-==============================
+Command options
+===============
-kunit_tool has a number of other command-line arguments which can
-be useful for our test environment. Below are the most commonly used
-command line arguments:
+To see a list of options for a particular command (along with usage), see::
-- ``--help``: Lists all available options. To list common options,
- place ``--help`` before the command. To list options specific to that
- command, place ``--help`` after the command.
-
- .. note:: Different commands (``config``, ``build``, ``run``, etc)
- have different supported options.
-- ``--build_dir``: Specifies kunit_tool build directory. It includes
- the ``.kunitconfig``, ``.config`` files and compiled kernel.
-
-- ``--make_options``: Specifies additional options to pass to make, when
- compiling a kernel (using ``build`` or ``run`` commands). For example:
- to enable compiler warnings, we can pass ``--make_options W=1``.
-
-- ``--alltests``: Enable a predefined set of options in order to build
- as many tests as possible.
-
- .. note:: The list of enabled options can be found in
- ``tools/testing/kunit/configs/all_tests.config``.
-
- If you only want to enable all tests with otherwise satisfied
- dependencies, instead add ``CONFIG_KUNIT_ALL_TESTS=y`` to your
- ``.kunitconfig``.
-
-- ``--kunitconfig``: Specifies the path or the directory of the ``.kunitconfig``
- file. For example:
-
- - ``lib/kunit/.kunitconfig`` can be the path of the file.
-
- - ``lib/kunit`` can be the directory in which the file is located.
-
- This file is used to build and run with a predefined set of tests
- and their dependencies. For example, to run tests for a given subsystem.
-
-- ``--kconfig_add``: Specifies additional configuration options to be
- appended to the ``.kunitconfig`` file. For example:
-
- .. code-block::
-
- ./tools/testing/kunit/kunit.py run --kconfig_add CONFIG_KASAN=y
-
-- ``--arch``: Runs tests on the specified architecture. The architecture
- argument is same as the Kbuild ARCH environment variable.
- For example, i386, x86_64, arm, um, etc. Non-UML architectures run on qemu.
- Default is `um`.
-
-- ``--cross_compile``: Specifies the Kbuild toolchain. It passes the
- same argument as passed to the ``CROSS_COMPILE`` variable used by
- Kbuild. This will be the prefix for the toolchain
- binaries such as GCC. For example:
-
- - ``sparc64-linux-gnu-`` if we have the sparc toolchain installed on
- our system.
-
- - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux``
- if we have downloaded the microblaze toolchain from the 0-day
- website to a specified path in our home directory called toolchains.
-
-- ``--qemu_config``: Specifies the path to a file containing a
- custom qemu architecture definition. This should be a python file
- containing a `QemuArchParams` object.
-
-- ``--qemu_args``: Specifies additional qemu arguments, for example, ``-smp 8``.
-
-- ``--jobs``: Specifies the number of jobs (commands) to run simultaneously.
- By default, this is set to the number of cores on your system.
-
-- ``--timeout``: Specifies the maximum number of seconds allowed for all tests to run.
- This does not include the time taken to build the tests.
-
-- ``--kernel_args``: Specifies additional kernel command-line arguments. May be repeated.
-
-- ``--run_isolated``: If set, boots the kernel for each individual suite/test.
- This is useful for debugging a non-hermetic test, one that
- might pass/fail based on what ran before it.
-
-- ``--raw_output``: If set, generates unformatted output from kernel. Possible options are:
-
- - ``all``: To view the full kernel output, use ``--raw_output=all``.
-
- - ``kunit``: This is the default option and filters to KUnit output. Use ``--raw_output`` or ``--raw_output=kunit``.
-
-- ``--json``: If set, stores the test results in a JSON format and prints to `stdout` or
- saves to a file if a filename is specified.
+ ./tools/testing/kunit/kunit.py <command> --help
Rewrite the documentation for clarity. Major points: * Switch from first-person to second-person point of view * The configuration step is now within "Kernel configuration" section * Replace description of common options with pointer to running --help for commands of the wrapper script Signed-off-by: Bagas Sanjaya <bagasdotme@gmail.com> --- Documentation/dev-tools/kunit/run_wrapper.rst | 302 ++++++------------ 1 file changed, 97 insertions(+), 205 deletions(-)