From patchwork Mon Apr 8 19:02:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10889977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EC60922 for ; Mon, 8 Apr 2019 19:02:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1730928787 for ; Mon, 8 Apr 2019 19:02:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0B721287B5; Mon, 8 Apr 2019 19:02:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D0BC28787 for ; Mon, 8 Apr 2019 19:02:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726953AbfDHTCr (ORCPT ); Mon, 8 Apr 2019 15:02:47 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:33462 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726738AbfDHTCq (ORCPT ); Mon, 8 Apr 2019 15:02:46 -0400 Received: by mail-pf1-f194.google.com with SMTP id h5so3547294pfo.0 for ; Mon, 08 Apr 2019 12:02:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=date:from:to:cc:subject:message-id:mime-version:content-disposition; bh=HJKxCELBlhj34ruYHQWtU7c/+3GXqO7hCwSsTAoIDk4=; b=L19TGLZ1+QF8/E3TILu2P1I/goaWvJQ3/tWWT8JawxshYQwCpb6CW5wVLFL3MflqmR ZxJWO7eshfv1X7atJjZuRufW8ZOtTQxW8Tepjve+nRJjzpgb8ekmnz8VC3drsDzvPUe/ cmXGtHHVfTeUUoZIpd3oE9YDZp55EaYC97UXw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition; bh=HJKxCELBlhj34ruYHQWtU7c/+3GXqO7hCwSsTAoIDk4=; b=oAEMu0PH/q3hLu02b/n9Vkurkaz2pP73Xy8eYRXumDK0MjiGPMOfGwjl/YGy7AXCpk 4CIt22nBIBWYSyY0Cf742nhXu0dx2LohkWaVIK2Iutj8s0PJ9Hkh+PZIW4/XWdC6t8wT XwCKoeBhEAh0cgEcPUbUg3BzA0HSu+xIX59K2YauRB7zCCoTwqJ9sYTOd0trb9yR1CKW czHBSxIYG0vQe08OUwRNK2T4+FxaSsCJSBcqCwuEB1Ld11K7Bx6ZiTqbMXgwVKnUcuDy 10CW3bh2lWOquwV52vmgYdLJOHDRXSNUWHzQsoIb0Yg4W78nP6WETExxMwhPS9Kt3tJ6 x9vA== X-Gm-Message-State: APjAAAU0oSOf9zQ03IMZuVlUbvYZA8N8EqrTDIz/iCEq0uQsNpQ4xA3z 9Paoa1ABg9is8eVcWWx0veMmQQ== X-Google-Smtp-Source: APXvYqzOQHBvr/7b6KgOPWgilpZje9z/LfIQQ6fr8oIKaL60678lVUZ4Ooo2K1myfTYn57LEoxzXHA== X-Received: by 2002:aa7:9089:: with SMTP id i9mr32089877pfa.115.1554750165846; Mon, 08 Apr 2019 12:02:45 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id a3sm50529511pfn.182.2019.04.08.12.02.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 08 Apr 2019 12:02:44 -0700 (PDT) Date: Mon, 8 Apr 2019 12:02:43 -0700 From: Kees Cook To: Shuah Khan Cc: linux-kselftest@vger.kernel.org Subject: [RFC][PATCH] selftests/lib.mk: Move test output to diagnostic lines Message-ID: <20190408190243.GA33074@beast> MIME-Version: 1.0 Content-Disposition: inline Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This changes the selftest output is several ways: - total test count is reported at start. - each test's result is on a single line with "# SKIP" as needed per spec. - each test's output is report _before_ the result line as a commented "diagnostic" line. This creates a bit of a kernel-specific TAP output where the diagnostics precede the results. The TAP spec isn't entirely clear about this, though, so I think it's the correct solution so as to not keep interactive runs making sense. If the output _followed_ the result line in the spec-suggested YAML form, each test would dump all of its output at once instead of as it went, making debugging harder. Also, while "sed -u" is used to add the "# " line prefixes, this still doesn't work for all output. For example, the "timer" lists print out text, then do the work, then print a result and a newline. This isn't visible any more. And some tests still show nothing until they finish. I haven't found a way to force the prefixing while keeping the output entirely unbuffered. :( Note that the shell construct needed to both get an exit code from the first command in a pipe and still filter the pipe (to add the "# " prefix) uses a POSIX solution rather than the bash "pipefail" option which is not supported by dash. Signed-off-by: Kees Cook --- tools/testing/selftests/lib.mk | 54 +++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 23 deletions(-) diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 8b0f16409ed7..056dac8f5701 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -5,6 +5,7 @@ CC := $(CROSS_COMPILE)gcc ifeq (0,$(MAKELEVEL)) OUTPUT := $(shell pwd) endif +srcdir = $(notdir $(shell pwd)) # The following are built by lib.mk common compile rules. # TEST_CUSTOM_PROGS should be used by tests that require @@ -32,38 +33,45 @@ endif .ONESHELL: define RUN_TEST_PRINT_RESULT - TEST_HDR_MSG="selftests: "`basename $$PWD`:" $$BASENAME_TEST"; \ - echo $$TEST_HDR_MSG; \ - echo "========================================"; \ + TEST_HDR_MSG="selftests: $(srcdir): $$BASENAME_TEST"; \ if [ ! -x $$TEST ]; then \ - echo "$$TEST_HDR_MSG: Warning: file $$BASENAME_TEST is not executable, correct this.";\ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - else \ - cd `dirname $$TEST` > /dev/null; \ - if [ "X$(summary)" != "X" ]; then \ - (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - else \ - (./$$BASENAME_TEST && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - fi; \ - cd - > /dev/null; \ + if [ "X$(summary)" = "X" ]; then \ + echo "# warning: 'file $$TEST is not executable, correct this.'";\ + fi; \ + echo "not ok $$test_num $$TEST_HDR_MSG"; \ + else \ + cd `dirname $$TEST` > /dev/null; \ + if [ "X$(summary)" != "X" ]; then \ + (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 &&\ + echo "ok $$test_num $$TEST_HDR_MSG") || \ + (if [ $$? -eq $$skip ]; then \ + echo "not ok $$test_num $$TEST_HDR_MSG # SKIP"; \ + else \ + echo "not ok $$test_num $$TEST_HDR_MSG";\ + fi); \ + else \ + echo "# $$TEST_HDR_MSG"; \ + (((((./$$BASENAME_TEST 2>&1; echo $$? >&3) | \ + sed -ue 's/^/# /' >&4) 3>&1) | \ + (read xs; exit $$xs)) 4>&1 && \ + echo "ok $$test_num $$TEST_HDR_MSG") || \ + (if [ $$? -eq $$skip ]; then \ + echo "not ok $$test_num $$TEST_HDR_MSG # SKIP"; \ + else \ + echo "not ok $$test_num $$TEST_HDR_MSG";\ + fi); \ + fi; \ + cd - > /dev/null; \ fi; endef define RUN_TESTS @export KSFT_TAP_LEVEL=`echo 1`; \ test_num=`echo 0`; \ + total=`echo "$(1)" | wc -w`; \ skip=`echo 4`; \ echo "TAP version 13"; \ + echo "1..$$total selftests: $(srcdir)"; \ for TEST in $(1); do \ BASENAME_TEST=`basename $$TEST`; \ test_num=`echo $$test_num+1 | bc`; \