From patchwork Sat Sep 21 00:18:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Higgins X-Patchwork-Id: 11155323 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3CF9C195A for ; Sat, 21 Sep 2019 00:21:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 11F202086A for ; Sat, 21 Sep 2019 00:21:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mziq7dsM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2392664AbfIUATI (ORCPT ); Fri, 20 Sep 2019 20:19:08 -0400 Received: from mail-pg1-f202.google.com ([209.85.215.202]:39822 "EHLO mail-pg1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390685AbfIUATH (ORCPT ); Fri, 20 Sep 2019 20:19:07 -0400 Received: by mail-pg1-f202.google.com with SMTP id t19so5439143pgh.6 for ; Fri, 20 Sep 2019 17:19:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=b333+3kU3sNm1guAvj5VAE2cIf3fv45+vyYGNDAuAeE=; b=mziq7dsMElwEKnE1Z7ti79XMqDY7c/2K4hTPRgTcmt/uJm/FDpfsCztwZ+w7qdA7QX cWIcYpMvZF7XU/Rai8qNzBCPmLM2rxwmmvuSrQuQMwc4RCMB7u1BKdHPTgV7SOyIkILM /+9JuZoYISDytzawuH6Le9KixbEBfpczjFPyVSf3MN2EM7va18p+Olf0W59MQ2epAu17 odPXBt1H/hj3PfMx0DjDYB5fZFKAYtIWifJ9zBeXZFJHtaqddA5d38/t9wwEVKr1WXUJ ODGP0GLzVruYjTn89kr0c6Ut7CUY4DjD5fA+eVGK4t1QxMX6OoN0Y65y4ZTEKZqdKkMB 5i3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=b333+3kU3sNm1guAvj5VAE2cIf3fv45+vyYGNDAuAeE=; b=OjOT+BmKe2ITHFkiWzcPTjdUjK9Vrd4n21Rphskg3FtDmtLrIYTFRjXIfHIu9okPmJ Dx5NNq4pXCPYLHn9QBPlspq4S7khxH8TV6z0dpc6IYNu5Uj5aGb4dTBEdod9a2pY61k0 98eZmsTQ+YAusss2HeokBPFsN/vm3ZF03GrSEl5twREDFbbYqQM90YsqEPc9f0nbgCLH nIGp1gX8qw8IQa/lS9EACR+cR+sO4uAX5QGNAaw4ldS+mGmeNPfFjQ+Nl1+m4s6UbXVY LAA5eq+WW9lQgjz0uysiPW4BZsU1B4S4F8Goqh9jbknpGTNXCmzm0Ah4fBWnsi5etqeW +O5w== X-Gm-Message-State: APjAAAW2IWSKHD2rpY3nA9amunkPg8YfejZt9XZrQJjt8aKIAA7vSGp1 BoA7z1nYFGpy9Rgw1wN3gBWi9BDB4f5DJZpDxMWIpg== X-Google-Smtp-Source: APXvYqw8GLoMb6Kss29cJO37f73z8953oOCu04U9qJiShjXq1nWZykvaYDEc/VJlamPMqIFZnL5bLBONGJcNWXQKydKvZw== X-Received: by 2002:a65:4002:: with SMTP id f2mr18079221pgp.447.1569025143686; Fri, 20 Sep 2019 17:19:03 -0700 (PDT) Date: Fri, 20 Sep 2019 17:18:36 -0700 Message-Id: <20190921001855.200947-1-brendanhiggins@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.23.0.351.gc4317032e6-goog Subject: [PATCH v17 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework From: Brendan Higgins To: frowand.list@gmail.com, gregkh@linuxfoundation.org, jpoimboe@redhat.com, keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, peterz@infradead.org, robh@kernel.org, sboyd@kernel.org, shuah@kernel.org, tytso@mit.edu, yamada.masahiro@socionext.com Cc: devicetree@vger.kernel.org, dri-devel@lists.freedesktop.org, kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-nvdimm@lists.01.org, linux-um@lists.infradead.org, Alexander.Levin@microsoft.com, Tim.Bird@sony.com, amir73il@gmail.com, dan.carpenter@oracle.com, daniel@ffwll.ch, jdike@addtoit.com, joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com, knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au, pmladek@suse.com, rdunlap@infradead.org, richard@nod.at, rientjes@google.com, rostedt@goodmis.org, wfg@linux.intel.com, torvalds@linux-foundation.org, Brendan Higgins Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org ## TL;DR This revision addresses comments from Linus[1] and Randy[2], by moving top level `kunit/` directory to `lib/kunit/` and likewise moves top level Kconfig entry under lib/Kconfig.debug, so the KUnit submenu now shows up under the "Kernel Hacking" menu. As a consequence of this, I rewrote patch 06/18 (kbuild: enable building KUnit), and now needs to be re-acked/reviewed. ## Background This patch set proposes KUnit, a lightweight unit testing and mocking framework for the Linux kernel. Unlike Autotest and kselftest, KUnit is a true unit testing framework; it does not require installing the kernel on a test machine or in a VM (however, KUnit still allows you to run tests on test machines or in VMs if you want[3]) and does not require tests to be written in userspace running on a host kernel. Additionally, KUnit is fast: From invocation to completion KUnit can run several dozen tests in about a second. Currently, the entire KUnit test suite for KUnit runs in under a second from the initial invocation (build time excluded). KUnit is heavily inspired by JUnit, Python's unittest.mock, and Googletest/Googlemock for C++. KUnit provides facilities for defining unit test cases, grouping related test cases into test suites, providing common infrastructure for running tests, mocking, spying, and much more. ### What's so special about unit testing? A unit test is supposed to test a single unit of code in isolation, hence the name. There should be no dependencies outside the control of the test; this means no external dependencies, which makes tests orders of magnitudes faster. Likewise, since there are no external dependencies, there are no hoops to jump through to run the tests. Additionally, this makes unit tests deterministic: a failing unit test always indicates a problem. Finally, because unit tests necessarily have finer granularity, they are able to test all code paths easily solving the classic problem of difficulty in exercising error handling code. ### Is KUnit trying to replace other testing frameworks for the kernel? No. Most existing tests for the Linux kernel are end-to-end tests, which have their place. A well tested system has lots of unit tests, a reasonable number of integration tests, and some end-to-end tests. KUnit is just trying to address the unit test space which is currently not being addressed. ### More information on KUnit There is a bunch of documentation near the end of this patch set that describes how to use KUnit and best practices for writing unit tests. For convenience I am hosting the compiled docs here[4]. Additionally for convenience, I have applied these patches to a branch[5]. The repo may be cloned with: git clone https://kunit.googlesource.com/linux This patchset is on the kunit/initial/v5.3/v17 branch. ## History since v15 ### v17 - Addressed comments on 06/19 (lib: enable building KUnit in lib/) from Stephen Boyd by moving KUnit submenu ahead of Runtime Testing submenu. ### v16 - Addressed comments from Linus Torvalds by moving all kunit/ paths to lib/kunit/. - Addressed comments by Randy Dunlap by moving KUnit Kconfig under lib/Kconfig.debug so the KUnit submenu shows up under the "Kernel Hacking" menu. [1] https://www.lkml.org/lkml/2019/9/20/696 [2] https://www.lkml.org/lkml/2019/9/20/738 [3] https://google.github.io/kunit-docs/third_party/kernel/docs/usage.html#kunit-on-non-uml-architectures [4] https://google.github.io/kunit-docs/third_party/kernel/docs/ [5] https://kunit.googlesource.com/linux/+/kunit/initial/v5.3/v17 --- Avinash Kondareddy (1): kunit: test: add tests for KUnit managed resources Brendan Higgins (16): kunit: test: add KUnit test runner core kunit: test: add test resource management API kunit: test: add string_stream a std::stream like string builder kunit: test: add assertion printing library kunit: test: add the concept of expectations lib: enable building KUnit in lib/ kunit: test: add initial tests objtool: add kunit_try_catch_throw to the noreturn list kunit: test: add support for test abort kunit: test: add tests for kunit test abort kunit: test: add the concept of assertions kunit: defconfig: add defconfigs for building KUnit tests Documentation: kunit: add documentation for KUnit MAINTAINERS: add entry for KUnit the unit testing framework MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section kunit: fix failure to build without printk Felix Guo (1): kunit: tool: add Python wrappers for running KUnit tests Iurii Zaikin (1): kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec() Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kunit/api/index.rst | 16 + Documentation/dev-tools/kunit/api/test.rst | 11 + Documentation/dev-tools/kunit/faq.rst | 62 + Documentation/dev-tools/kunit/index.rst | 79 + Documentation/dev-tools/kunit/start.rst | 180 ++ Documentation/dev-tools/kunit/usage.rst | 576 +++++++ MAINTAINERS | 13 + arch/um/configs/kunit_defconfig | 3 + include/kunit/assert.h | 356 ++++ include/kunit/string-stream.h | 51 + include/kunit/test.h | 1490 +++++++++++++++++ include/kunit/try-catch.h | 75 + kernel/Makefile | 2 + kernel/sysctl-test.c | 392 +++++ lib/Kconfig.debug | 13 + lib/Makefile | 2 + lib/kunit/Kconfig | 38 + lib/kunit/Makefile | 9 + lib/kunit/assert.c | 141 ++ lib/kunit/example-test.c | 88 + lib/kunit/string-stream-test.c | 52 + lib/kunit/string-stream.c | 217 +++ lib/kunit/test-test.c | 331 ++++ lib/kunit/test.c | 478 ++++++ lib/kunit/try-catch.c | 118 ++ tools/objtool/check.c | 1 + tools/testing/kunit/.gitignore | 3 + tools/testing/kunit/configs/all_tests.config | 3 + tools/testing/kunit/kunit.py | 136 ++ tools/testing/kunit/kunit_config.py | 66 + tools/testing/kunit/kunit_kernel.py | 149 ++ tools/testing/kunit/kunit_parser.py | 310 ++++ tools/testing/kunit/kunit_tool_test.py | 206 +++ .../test_is_test_passed-all_passed.log | 32 + .../test_data/test_is_test_passed-crash.log | 69 + .../test_data/test_is_test_passed-failure.log | 36 + .../test_is_test_passed-no_tests_run.log | 75 + .../test_output_isolated_correctly.log | 106 ++ .../test_data/test_read_from_file.kconfig | 17 + 40 files changed, 6003 insertions(+) create mode 100644 Documentation/dev-tools/kunit/api/index.rst create mode 100644 Documentation/dev-tools/kunit/api/test.rst create mode 100644 Documentation/dev-tools/kunit/faq.rst create mode 100644 Documentation/dev-tools/kunit/index.rst create mode 100644 Documentation/dev-tools/kunit/start.rst create mode 100644 Documentation/dev-tools/kunit/usage.rst create mode 100644 arch/um/configs/kunit_defconfig create mode 100644 include/kunit/assert.h create mode 100644 include/kunit/string-stream.h create mode 100644 include/kunit/test.h create mode 100644 include/kunit/try-catch.h create mode 100644 kernel/sysctl-test.c create mode 100644 lib/kunit/Kconfig create mode 100644 lib/kunit/Makefile create mode 100644 lib/kunit/assert.c create mode 100644 lib/kunit/example-test.c create mode 100644 lib/kunit/string-stream-test.c create mode 100644 lib/kunit/string-stream.c create mode 100644 lib/kunit/test-test.c create mode 100644 lib/kunit/test.c create mode 100644 lib/kunit/try-catch.c create mode 100644 tools/testing/kunit/.gitignore create mode 100644 tools/testing/kunit/configs/all_tests.config create mode 100755 tools/testing/kunit/kunit.py create mode 100644 tools/testing/kunit/kunit_config.py create mode 100644 tools/testing/kunit/kunit_kernel.py create mode 100644 tools/testing/kunit/kunit_parser.py create mode 100755 tools/testing/kunit/kunit_tool_test.py create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-crash.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-failure.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_run.log create mode 100644 tools/testing/kunit/test_data/test_output_isolated_correctly.log create mode 100644 tools/testing/kunit/test_data/test_read_from_file.kconfig