From patchwork Mon Apr 8 20:30:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10890085 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4FDB61669 for ; Mon, 8 Apr 2019 20:30:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E2912842E for ; Mon, 8 Apr 2019 20:30:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1D892284F0; Mon, 8 Apr 2019 20:30:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5AB492842E for ; Mon, 8 Apr 2019 20:30:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726842AbfDHUaQ (ORCPT ); Mon, 8 Apr 2019 16:30:16 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:39470 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726532AbfDHUaP (ORCPT ); Mon, 8 Apr 2019 16:30:15 -0400 Received: by mail-pg1-f195.google.com with SMTP id k3so7945736pga.6 for ; Mon, 08 Apr 2019 13:30:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=date:from:to:cc:subject:message-id:mime-version:content-disposition; bh=mud3jxvSVQoIY6rHELYqOR/+Wgq1vH34LaBIMfX0DM0=; b=N+X5pxrtpkWrIizA6OBHI5MdDRnmr39Petp2UKbojQDLMae19pAiWkQVlk7oHm39Hp FQ1P3cSO5GSbUnTm6aMsQXTWfBcwTgP2kYoVUjq6k84+3hTIb+9BXYR0RYTnzdq93q9E Ay6CeSoWw6/xvjblXt7pi8S2kCoZ1UB5yrSeQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition; bh=mud3jxvSVQoIY6rHELYqOR/+Wgq1vH34LaBIMfX0DM0=; b=CzggfB5yi0b7kAkfbbGdTfRfNnxO7wvHAx5983+vEr6fszijL9Nv2h8OWfXhH2BTPN DhYLVNM/jqpl/RBSk0TXtRHFkJ6uWHYflZaC58ynkkrVR0Y2T8TAc7LDueaUnHOIkq5v Js0noe7OgRqNoNCd7yn2Yf4giZvE9vss2hQ1KA16uQo9ASRvwcuof9n+WUhmSMCAOcPA RjdgEW31AxnmeJgBODroEBBevZLHP88LKsAU4gkz9ELwjlHoD07IiKJobIlqTvJw4eON rLtblOsrdPqxJglLxa/YubeAWJeSOtWc+r8YiknlHdL/3cSRNIMib/1DWj+Y6dR3zl9R sn7A== X-Gm-Message-State: APjAAAX4w8VfUL7Da/uczVsxObAvIYkZJGhFnRsXXbWHrmfSR1ctZOp4 gSJ6TuM2Ddun3OYKI4mB93zHkA== X-Google-Smtp-Source: APXvYqxnqqW4vzr8b2+eQARaRLXp9nwTfXQ866drQ4paW+2/HgjjWZbPUNiy9WsTtCtaY/JY7FiDlw== X-Received: by 2002:a63:7c45:: with SMTP id l5mr30519251pgn.303.1554755415003; Mon, 08 Apr 2019 13:30:15 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id k9sm40467957pga.22.2019.04.08.13.30.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 08 Apr 2019 13:30:13 -0700 (PDT) Date: Mon, 8 Apr 2019 13:30:12 -0700 From: Kees Cook To: Shuah Khan Cc: linux-kselftest@vger.kernel.org Subject: [PATCH] selftests/lib.mk: Move test output to diagnostic lines Message-ID: <20190408203012.GA566@beast> MIME-Version: 1.0 Content-Disposition: inline Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This changes the selftest output is several ways: - total test count is reported at start. - each test's result is on a single line with "# SKIP" as needed per spec. - each test's output is prefixed with "# " as a "diagnostic line". This creates a bit of a kernel-specific TAP dialect where the diagnostics precede the results. The TAP spec isn't entirely clear about this, though, so I think it's the correct solution so as to keep interactive runs making sense. If the output _followed_ the result line in the spec-suggested YAML form, each test would dump all of its output at once instead of as it went, making debugging harder. This does, however, solve the recursive TAP output problem, as sub-tests will simply be prefixed by "# ". Parsing sub-tests becomes a simple problem of just removing the first two characters of a given top-level test's diagnostic output, and parsing the results. Note that the shell construct needed to both get an exit code from the first command in a pipe and still filter the pipe (to add the "# " prefix) uses a POSIX solution rather than the bash "pipefail" option which is not supported by dash. Example: $ tools/testing/selftests $ make --silent -C x86 run_tests TAP version 13 1..15 selftests: x86 # selftests: x86: single_step_syscall_64 # [RUN] Set TF and check nop # [OK] Survived with TF set and 9 traps # [RUN] Set TF and check syscall-less opportunistic sysret # [OK] Survived with TF set and 12 traps # [RUN] Set TF and check a fast syscall # [OK] Survived with TF set and 22 traps # [RUN] Fast syscall with TF cleared # [OK] Nothing unexpected happened ok 1 selftests: x86: single_step_syscall_64 # selftests: x86: sysret_ss_attrs_64 # [RUN] Syscalls followed by SS validation # [OK] We survived ok 2 selftests: x86: sysret_ss_attrs_64 # selftests: x86: syscall_nt_64 # [RUN] Set NT and issue a syscall # [OK] The syscall worked and flags are still set # [RUN] Set NT|TF and issue a syscall # [OK] The syscall worked and flags are still set ok 3 selftests: x86: syscall_nt_64 ...[skipping example output]... ok 15 selftests: x86: sysret_rip_64 $ make --silent -C x86 run_tests | tappy ............F.. ====================================================================== FAIL: selftests: x86: mov_ss_trap_64 ---------------------------------------------------------------------- ---------------------------------------------------------------------- Ran 15 tests in 0.000s FAILED (failures=1) Signed-off-by: Kees Cook --- No longer RFC! :) --- tools/testing/selftests/lib.mk | 59 ++++++++++++++++++------------- tools/testing/selftests/prefix.pl | 23 ++++++++++++ 2 files changed, 57 insertions(+), 25 deletions(-) create mode 100755 tools/testing/selftests/prefix.pl diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 8b0f16409ed7..d3267c08a7b1 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -5,6 +5,8 @@ CC := $(CROSS_COMPILE)gcc ifeq (0,$(MAKELEVEL)) OUTPUT := $(shell pwd) endif +testname = $(notdir $(shell pwd)) +selfdir = $(realpath $(dir $(filter %/lib.mk,$(MAKEFILE_LIST)))) # The following are built by lib.mk common compile rules. # TEST_CUSTOM_PROGS should be used by tests that require @@ -32,38 +34,45 @@ endif .ONESHELL: define RUN_TEST_PRINT_RESULT - TEST_HDR_MSG="selftests: "`basename $$PWD`:" $$BASENAME_TEST"; \ - echo $$TEST_HDR_MSG; \ - echo "========================================"; \ + TEST_HDR_MSG="selftests: $(testname): $$BASENAME_TEST"; \ if [ ! -x $$TEST ]; then \ - echo "$$TEST_HDR_MSG: Warning: file $$BASENAME_TEST is not executable, correct this.";\ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - else \ - cd `dirname $$TEST` > /dev/null; \ - if [ "X$(summary)" != "X" ]; then \ - (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - else \ - (./$$BASENAME_TEST && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - fi; \ - cd - > /dev/null; \ + if [ "X$(summary)" = "X" ]; then \ + echo "# warning: 'file $$TEST is not executable, correct this.'";\ + fi; \ + echo "not ok $$test_num $$TEST_HDR_MSG"; \ + else \ + cd `dirname $$TEST` > /dev/null; \ + if [ "X$(summary)" != "X" ]; then \ + (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 &&\ + echo "ok $$test_num $$TEST_HDR_MSG") || \ + (if [ $$? -eq $$skip ]; then \ + echo "not ok $$test_num $$TEST_HDR_MSG # SKIP"; \ + else \ + echo "not ok $$test_num $$TEST_HDR_MSG";\ + fi); \ + else \ + echo "# $$TEST_HDR_MSG"; \ + (((((stdbuf -i0 -o0 -e0 ./$$BASENAME_TEST 2>&1; echo $$? >&3) | \ + $(selfdir)/prefix.pl >&4) 3>&1) | \ + (read xs; exit $$xs)) 4>&1 && \ + echo "ok $$test_num $$TEST_HDR_MSG") || \ + (if [ $$? -eq $$skip ]; then \ + echo "not ok $$test_num $$TEST_HDR_MSG # SKIP"; \ + else \ + echo "not ok $$test_num $$TEST_HDR_MSG";\ + fi); \ + fi; \ + cd - > /dev/null; \ fi; endef define RUN_TESTS @export KSFT_TAP_LEVEL=`echo 1`; \ - test_num=`echo 0`; \ - skip=`echo 4`; \ + test_num=0; \ + total=`echo "$(1)" | wc -w`; \ + skip=4; \ echo "TAP version 13"; \ + echo "1..$$total selftests: $(testname)"; \ for TEST in $(1); do \ BASENAME_TEST=`basename $$TEST`; \ test_num=`echo $$test_num+1 | bc`; \ diff --git a/tools/testing/selftests/prefix.pl b/tools/testing/selftests/prefix.pl new file mode 100755 index 000000000000..ec7e48118183 --- /dev/null +++ b/tools/testing/selftests/prefix.pl @@ -0,0 +1,23 @@ +#!/usr/bin/perl +# SPDX-License-Identifier: GPL-2.0 +# Prefix all lines with "# ", unbuffered. Command being piped in may need +# to have unbuffering forced with "stdbuf -i0 -o0 -e0 $cmd". +use strict; + +binmode STDIN; +binmode STDOUT; + +STDOUT->autoflush(1); + +my $needed = 1; +while (1) { + my $char; + my $bytes = sysread(STDIN, $char, 1); + exit 0 if ($bytes == 0); + if ($needed) { + print "# "; + $needed = 0; + } + print $char; + $needed = 1 if ($char eq "\n"); +}