From patchwork Tue Apr 9 23:55:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10892875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F14281708 for ; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D9D0C28387 for ; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE29D28701; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59BC8288BB for ; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726680AbfDIX4K (ORCPT ); Tue, 9 Apr 2019 19:56:10 -0400 Received: from mail-pf1-f171.google.com ([209.85.210.171]:34719 "EHLO mail-pf1-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726741AbfDIX4J (ORCPT ); Tue, 9 Apr 2019 19:56:09 -0400 Received: by mail-pf1-f171.google.com with SMTP id b3so250454pfd.1 for ; Tue, 09 Apr 2019 16:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FFT1aVePQMmOXt7Bx8AXONIyJ3gNRAjMM6LapXIRXhg=; b=nCH9VZ1jjr8A0YYdyXW87NLqn/Yg2isK7BH876WqeZimsOgmvOqy2bBRMrC0YQN8El 9Pu1uXKyPVfToPKjyO6uK4HURg50W0U9BzD2WayZVU9iCs682iCOp+9i1bXfAv68Vjbg t2ysjgaofcRJ3s9xisOl2gXiurWq8xdm4ftgY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FFT1aVePQMmOXt7Bx8AXONIyJ3gNRAjMM6LapXIRXhg=; b=cEla6+wLnG6vXaxDn1T0MYVUhBgAj/iJRF698WbqGqlgMC99y3iEnRCRqmJzTuozWQ A5H34LlSJfffTC6lcaU86+yRPQBQ6TtMKu5Q1NGMU15mALuwkNwsA0EsoKVlbN+FDU// psyD9oFYnnoUJBjP+zSHWyhDoOVG3iQm6vvzSBFAgYWSYFGPwYBwe1C6aSWwiQsf3XI9 WZgYkQ4Zigpk8vDEQcxWf+pMOQ9cP69WCosTH14nA3ijwXB7W4en/GxJjYLdXqP5ohYu P9c4VAhnSrzPYDocI09ma0tdboCUPUu8wJvmJ6i9fixbzEHq2ALyaKsHaLW4ngDBmZuW Rx1g== X-Gm-Message-State: APjAAAVdEdvBD4Z3ApV8nH3blWOqqazmKHKuv7c5PAtdjC71J/uMEbno ACkU+JhAGUoUahQ3SR2LCRJPYPxsWtA= X-Google-Smtp-Source: APXvYqyUngpl0+K4riscqnwr7X+TY7gbTYIiZfYt14mRMUMw1iVvp993M755n7Ies6KLu+t7Dvp/qQ== X-Received: by 2002:a63:c302:: with SMTP id c2mr38195144pgd.235.1554854169078; Tue, 09 Apr 2019 16:56:09 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id c1sm9564851pgq.33.2019.04.09.16.56.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Apr 2019 16:56:05 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , linux-kselftest@vger.kernel.org Subject: [PATCH 1/6] selftests: Extract single-test shell logic from lib.mk Date: Tue, 9 Apr 2019 16:55:51 -0700 Message-Id: <20190409235556.3967-2-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409235556.3967-1-keescook@chromium.org> References: <20190409235556.3967-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In order to improve the reusability of the kselftest test running logic, this extracts the single-test logic from lib.mk into kselftest/runner.sh which lib.mk can call directly. No changes in output. As part of the change, this removes the unused "summary" Makefile variable (and tests). However, future merging with the "emit_tests" target needs to be able to redirect output, so a new "logfile" variable is introduced. Signed-off-by: Kees Cook --- tools/testing/selftests/.gitignore | 1 - tools/testing/selftests/kselftest/runner.sh | 31 +++++++++++++++++++ tools/testing/selftests/lib.mk | 33 ++------------------- 3 files changed, 34 insertions(+), 31 deletions(-) create mode 100644 tools/testing/selftests/kselftest/runner.sh diff --git a/tools/testing/selftests/.gitignore b/tools/testing/selftests/.gitignore index 91750352459d..8059ce834247 100644 --- a/tools/testing/selftests/.gitignore +++ b/tools/testing/selftests/.gitignore @@ -1,4 +1,3 @@ -kselftest gpiogpio-event-mon gpiogpio-hammer gpioinclude/ diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh new file mode 100644 index 000000000000..77d5510ac4c5 --- /dev/null +++ b/tools/testing/selftests/kselftest/runner.sh @@ -0,0 +1,31 @@ +#!/bin/sh +# +# Runs a set of tests in a given subdirectory. +export skip_rc=4 +export logfile=/dev/stdout + +run_one() +{ + TEST="$1" + NUM="$2" + + BASENAME_TEST=$(basename $TEST) + + TEST_HDR_MSG="selftests: "`basename $PWD`:" $BASENAME_TEST" + echo "$TEST_HDR_MSG" + echo "========================================" + if [ ! -x "$TEST" ]; then + echo "$TEST_HDR_MSG: Warning: file $TEST is not executable, correct this." + echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + else + cd `dirname $TEST` > /dev/null + (./$BASENAME_TEST >> "$logfile" 2>&1 && + echo "ok 1..$test_num $TEST_HDR_MSG [PASS]") || + (if [ $? -eq $skip_rc ]; then \ + echo "not ok 1..$test_num $TEST_HDR_MSG [SKIP]" + else + echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + fi) + cd - >/dev/null + fi +} diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 8b0f16409ed7..7da79fe0bb78 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -5,6 +5,7 @@ CC := $(CROSS_COMPILE)gcc ifeq (0,$(MAKELEVEL)) OUTPUT := $(shell pwd) endif +selfdir = $(realpath $(dir $(filter %/lib.mk,$(MAKEFILE_LIST)))) # The following are built by lib.mk common compile rules. # TEST_CUSTOM_PROGS should be used by tests that require @@ -31,43 +32,15 @@ all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES) endif .ONESHELL: -define RUN_TEST_PRINT_RESULT - TEST_HDR_MSG="selftests: "`basename $$PWD`:" $$BASENAME_TEST"; \ - echo $$TEST_HDR_MSG; \ - echo "========================================"; \ - if [ ! -x $$TEST ]; then \ - echo "$$TEST_HDR_MSG: Warning: file $$BASENAME_TEST is not executable, correct this.";\ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - else \ - cd `dirname $$TEST` > /dev/null; \ - if [ "X$(summary)" != "X" ]; then \ - (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - else \ - (./$$BASENAME_TEST && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - fi; \ - cd - > /dev/null; \ - fi; -endef - define RUN_TESTS @export KSFT_TAP_LEVEL=`echo 1`; \ test_num=`echo 0`; \ - skip=`echo 4`; \ + . $(selfdir)/kselftest/runner.sh; \ echo "TAP version 13"; \ for TEST in $(1); do \ BASENAME_TEST=`basename $$TEST`; \ test_num=`echo $$test_num+1 | bc`; \ - $(call RUN_TEST_PRINT_RESULT,$(TEST),$(BASENAME_TEST),$(test_num),$(skip)) \ + run_one "$$BASENAME_TEST" "$$test_num"; \ done; endef From patchwork Tue Apr 9 23:55:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10892869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF34D1800 for ; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C756128387 for ; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BB7D728832; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9432428701 for ; Tue, 9 Apr 2019 23:56:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726663AbfDIX4I (ORCPT ); Tue, 9 Apr 2019 19:56:08 -0400 Received: from mail-pg1-f181.google.com ([209.85.215.181]:33325 "EHLO mail-pg1-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726592AbfDIX4I (ORCPT ); Tue, 9 Apr 2019 19:56:08 -0400 Received: by mail-pg1-f181.google.com with SMTP id k19so363496pgh.0 for ; Tue, 09 Apr 2019 16:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dxVzCuzkWLEHkltut7V3wMwkElZVHW5jHqo47nGm1T8=; b=XjhstPacEXqHZ6JBDJ2MzZ7EINAeYcybmpsvuJ/s6ctVX1N3qZbwOGnHT4U1v1cj5n Xs9jh4/WpcpioLOz9DLjt0xB6ot3kZs1JN9+gJ9LS+a1ATJ8kaW44QFJ3Hy6XZAkx3iv NPIngPJkCxxEjDIdfrx+O1lE1Tsfxy1gOwDJI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dxVzCuzkWLEHkltut7V3wMwkElZVHW5jHqo47nGm1T8=; b=o89feHXiHz1P5mnZCGJLx5LkPGQocsJOFjFmaKEIhyucCBw1tX6REpO0XpHWTVyb9r u4jUIyfsaTTdpl2gjEI3Mi1GIoOey6OkS/DlWSfT2LJm8JcrvyUqgXAhViujxC24KH2M YB8FqGxlBTFwzQPliFm40AHSBUU6hm7SsbB8MQi3xB+dESPD4ZDFhQIVW9Q8tB079WAf TZ6O0KaT22YCNQDJcWj3v4NYTO+7lEK9Ht/UoIAx6Ki1bffHr1rgNn+WhytWFNCZkWO9 q482QyqffRfO/O4kvoBrwID6vLmvbI7ylVaXQtbRZW8eEjixwscDmwX3sB43qOxHUUbu 5kXA== X-Gm-Message-State: APjAAAX/aGbdlD/MjQ6SzSNyK469DC5sRZeNWDhLHSp/Hcm8XNUknTOD ggDRIsmpvb3UlTTjVIhTmjWFjQ== X-Google-Smtp-Source: APXvYqz/Qt12MrDWtXM5B6cM3tBfkWi3qnUP5v7N4wsZpeGl7TrItYqNOnyIpPbnd2AwUY1+NQVGJQ== X-Received: by 2002:a63:e20b:: with SMTP id q11mr38355762pgh.263.1554854167439; Tue, 09 Apr 2019 16:56:07 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id 75sm62024011pfr.55.2019.04.09.16.56.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Apr 2019 16:56:05 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , linux-kselftest@vger.kernel.org Subject: [PATCH 2/6] selftests: Use runner.sh for emit targets Date: Tue, 9 Apr 2019 16:55:52 -0700 Message-Id: <20190409235556.3967-3-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409235556.3967-1-keescook@chromium.org> References: <20190409235556.3967-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This reuses the new runner.sh for the emit targets instead of manually running each test via run_kselftest.sh. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 11 +++++------ tools/testing/selftests/lib.mk | 15 ++------------- 2 files changed, 7 insertions(+), 19 deletions(-) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 971fc8428117..45327e921169 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -131,7 +131,8 @@ ALL_SCRIPT := $(INSTALL_PATH)/run_kselftest.sh install: ifdef INSTALL_PATH @# Ask all targets to install their files - mkdir -p $(INSTALL_PATH) + mkdir -p $(INSTALL_PATH)/kselftest + install -m 744 kselftest/runner.sh $(INSTALL_PATH)/kselftest/ @for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ make OUTPUT=$$BUILD_TARGET -C $$TARGET INSTALL_PATH=$(INSTALL_PATH)/$$TARGET install; \ @@ -141,15 +142,13 @@ ifdef INSTALL_PATH echo "#!/bin/sh" > $(ALL_SCRIPT) echo "BASE_DIR=\$$(realpath \$$(dirname \$$0))" >> $(ALL_SCRIPT) echo "cd \$$BASE_DIR" >> $(ALL_SCRIPT) + echo ". ./kselftest/runner.sh" >> $(ALL_SCRIPT) echo "ROOT=\$$PWD" >> $(ALL_SCRIPT) echo "if [ \"\$$1\" = \"--summary\" ]; then" >> $(ALL_SCRIPT) - echo " OUTPUT=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) - echo " cat /dev/null > \$$OUTPUT" >> $(ALL_SCRIPT) - echo "else" >> $(ALL_SCRIPT) - echo " OUTPUT=/dev/stdout" >> $(ALL_SCRIPT) + echo " logfile=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) + echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT) echo "fi" >> $(ALL_SCRIPT) echo "export KSFT_TAP_LEVEL=1" >> $(ALL_SCRIPT) - echo "export skip=4" >> $(ALL_SCRIPT) for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 7da79fe0bb78..8a4fad5d3934 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -78,24 +78,13 @@ else $(error Error: set INSTALL_PATH to use install) endif -define EMIT_TESTS +emit_tests: @test_num=`echo 0`; \ for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ test_num=`echo $$test_num+1 | bc`; \ - TEST_HDR_MSG="selftests: "`basename $$PWD`:" $$BASENAME_TEST"; \ - echo "echo $$TEST_HDR_MSG"; \ - if [ ! -x $$TEST ]; then \ - echo "echo \"$$TEST_HDR_MSG: Warning: file $$BASENAME_TEST is not executable, correct this.\""; \ - echo "echo \"not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]\""; \ - else - echo "(./$$BASENAME_TEST >> \$$OUTPUT 2>&1 && echo \"ok 1..$$test_num $$TEST_HDR_MSG [PASS]\") || (if [ \$$? -eq \$$skip ]; then echo \"not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]\"; else echo \"not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]\"; fi;)"; \ - fi; \ + echo "run_one \"$$BASENAME_TEST\" \"$$test_num\""; \ done; -endef - -emit_tests: - $(EMIT_TESTS) # define if isn't already. It is undefined in make O= case. ifeq ($(RM),) From patchwork Tue Apr 9 23:55:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10892873 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 747A118B7 for ; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5772928387 for ; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BCB828701; Tue, 9 Apr 2019 23:56:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3BC02883B for ; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726755AbfDIX4J (ORCPT ); Tue, 9 Apr 2019 19:56:09 -0400 Received: from mail-pg1-f169.google.com ([209.85.215.169]:43282 "EHLO mail-pg1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726680AbfDIX4J (ORCPT ); Tue, 9 Apr 2019 19:56:09 -0400 Received: by mail-pg1-f169.google.com with SMTP id z9so333802pgu.10 for ; Tue, 09 Apr 2019 16:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BBz3uzW2MkXRWFcDEhL/aWwf5vAxjd4KRec/q6yV/Yk=; b=ME+Jm/IUhSyjts8heL5egIe0w7S0fbc+6ib8//R5yLBxYbA8wyfUmxZLy5QbuM7DEG MdGw3XEd0bGfg/OgzoDQRNYZja0dYeNpjZhSH0YdMpwjA1hBBVeJICxuQG9/+lLPc8Ba GVcKUBPefA30it7UEp+csbkKLwlfD4IEMLrdA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BBz3uzW2MkXRWFcDEhL/aWwf5vAxjd4KRec/q6yV/Yk=; b=t566WU7aiWRVNkJC0K0leaeS2I9UE0IvzoZqcCJlskt0BmSA5t9QCcM9+Y8Drm652J Y0M2bQoKKFBy1Lth7QfkPygy3HRXwmx5UCff4xXRWlwpOoH4/7a+nOt7BcWbbVg0ZH74 OmruyV8v1m8NO1XKLkyvUrtLSx5IUHlJwydBMKrb8oClFqDUjRZTawDe7vCF1FSEg1yw Bfi7rv+Qybr4tcgNspkr1IW4clFKXLR5viZLf0C5/MBGnvgzXNNDpqJlVWU/fN0+90Ua IjMblWno+RX7vpgBaZyKIsichNW44OV3ERG49OB2QCQdJmLGj2gC6H7i+Y/uAX5M3UU9 OqWQ== X-Gm-Message-State: APjAAAUUeNqLMyhH0/DxtweiTmp2H5SS8ftdoPlbVjMB58zbQzuPDULi +GjdO8Bvk8VJl0mMwrYFE+eZMQ== X-Google-Smtp-Source: APXvYqz1Mtoq+4WhLun89pzJ4yACc/wtjhxkrxjJxbbg+ncCv9AjEyE1KOlIg7Fq5duSqRbTqdvJTA== X-Received: by 2002:aa7:8ac8:: with SMTP id b8mr39828381pfd.234.1554854168596; Tue, 09 Apr 2019 16:56:08 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id i15sm17011592pfr.8.2019.04.09.16.56.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Apr 2019 16:56:05 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , linux-kselftest@vger.kernel.org Subject: [PATCH 3/6] selftests: Extract logic for multiple test runs Date: Tue, 9 Apr 2019 16:55:53 -0700 Message-Id: <20190409235556.3967-4-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409235556.3967-1-keescook@chromium.org> References: <20190409235556.3967-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This moves the logic for running multiple tests into a single "run_many" function of runner.sh. Both "run_tests" and "emit_tests" are modified to use it. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 6 ++---- tools/testing/selftests/kselftest/runner.sh | 19 ++++++++++++++++--- tools/testing/selftests/lib.mk | 16 ++++------------ 3 files changed, 22 insertions(+), 19 deletions(-) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 45327e921169..72178d28f9c6 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -148,16 +148,14 @@ ifdef INSTALL_PATH echo " logfile=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT) echo "fi" >> $(ALL_SCRIPT) - echo "export KSFT_TAP_LEVEL=1" >> $(ALL_SCRIPT) for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ - echo "echo ; echo TAP version 13" >> $(ALL_SCRIPT); \ - echo "echo Running tests in $$TARGET" >> $(ALL_SCRIPT); \ - echo "echo ========================================" >> $(ALL_SCRIPT); \ echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \ echo "cd $$TARGET" >> $(ALL_SCRIPT); \ + echo -n "run_many" >> $(ALL_SCRIPT); \ make -s --no-print-directory OUTPUT=$$BUILD_TARGET -C $$TARGET emit_tests >> $(ALL_SCRIPT); \ + echo "" >> $(ALL_SCRIPT); \ echo "cd \$$ROOT" >> $(ALL_SCRIPT); \ done; diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 77d5510ac4c5..51139f42a6ca 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -1,17 +1,19 @@ #!/bin/sh # # Runs a set of tests in a given subdirectory. +export KSFT_TAP_LEVEL=1 export skip_rc=4 export logfile=/dev/stdout run_one() { - TEST="$1" - NUM="$2" + DIR="$1" + TEST="$2" + NUM="$3" BASENAME_TEST=$(basename $TEST) - TEST_HDR_MSG="selftests: "`basename $PWD`:" $BASENAME_TEST" + TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" echo "$TEST_HDR_MSG" echo "========================================" if [ ! -x "$TEST" ]; then @@ -29,3 +31,14 @@ run_one() cd - >/dev/null fi } + +run_many() +{ + echo "TAP version 13" + DIR=$(basename "$PWD") + test_num=0 + for TEST in "$@"; do + test_num=$(( test_num + 1 )) + run_one "$DIR" "$TEST" "$test_num" + done +} diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 8a4fad5d3934..8be13d117101 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -33,15 +33,8 @@ endif .ONESHELL: define RUN_TESTS - @export KSFT_TAP_LEVEL=`echo 1`; \ - test_num=`echo 0`; \ . $(selfdir)/kselftest/runner.sh; \ - echo "TAP version 13"; \ - for TEST in $(1); do \ - BASENAME_TEST=`basename $$TEST`; \ - test_num=`echo $$test_num+1 | bc`; \ - run_one "$$BASENAME_TEST" "$$test_num"; \ - done; + run_many $(1) endef run_tests: all @@ -79,12 +72,11 @@ else endif emit_tests: - @test_num=`echo 0`; \ for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ - test_num=`echo $$test_num+1 | bc`; \ - echo "run_one \"$$BASENAME_TEST\" \"$$test_num\""; \ - done; + echo " \\"; \ + echo -n " \"$$BASENAME_TEST\""; \ + done; \ # define if isn't already. It is undefined in make O= case. ifeq ($(RM),) From patchwork Tue Apr 9 23:55:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10892867 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B47FF1708 for ; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 92EC728387 for ; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 874C62883B; Tue, 9 Apr 2019 23:56:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6421228387 for ; Tue, 9 Apr 2019 23:56:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726640AbfDIX4H (ORCPT ); Tue, 9 Apr 2019 19:56:07 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:41204 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726592AbfDIX4H (ORCPT ); Tue, 9 Apr 2019 19:56:07 -0400 Received: by mail-pg1-f193.google.com with SMTP id f6so339000pgs.8 for ; Tue, 09 Apr 2019 16:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ht0iEC0j4OarhyDSbuUqVPCj03juNb5j0gNHH6SpSmw=; b=Qmd+M1wwbFny3gtH46RqfXESkiPDZMJY5q4jvOKtfl7YV7+OYia7h5SbljyK1U3FKE Y7Xz50Pv3ZBZNQihrQpSF0wWR/PgYu/yp+c0SVMHGMs7tiGk9JVCLd/O1Oibf32GMI5N dIfvuhDOsZtTY8MwdP6dbOYa5DisMpej5Yd8A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ht0iEC0j4OarhyDSbuUqVPCj03juNb5j0gNHH6SpSmw=; b=ScOpeHnL9kU1t4iM8ASB6v+SNx9b5T70nBklcERNT75/Qhe8OnauzLUl20TeUtA2z4 lX/yOh0vGjvQaF5nEtHUcH5GkNnNPwkOxvWZUq4CMlZAGF9re87qOjJu2AI5KM2GJvC5 DqVpAv5vjE5cu+RPhuR5Ip00yBKgqo+LBh/d8/jKKhiy6iCzA9id6tRP4KgZ8OBcsS5q GbITQhMnXavl4lBeNRtE/Ao/6qGSdybuYnjL3kVdHNgTuxNxThEaqWnI2d1kx8xeE9Aw pW6xIjhRl2/kPi7zZE4RquNDFTUsBfUs1q2u5sOCVZEpGqhIlqnUvmwSMmwVkKkrlYzi FZcA== X-Gm-Message-State: APjAAAWC5Ei4gyx3KfI60LuSk0TOrwrBUFpZxVDLX3e3ftZw2X2w2w5b 2AHxNAoMJql/Nk7+ud9L5N9nN6SAgFg= X-Google-Smtp-Source: APXvYqxChk/l3QtKS/+6xBsqBYNdC8ORu+ssMhl+OcCmMs+yPx6HXaDsgJ0VPZ8ueTuCOrUIMcBfmQ== X-Received: by 2002:a63:b305:: with SMTP id i5mr35881672pgf.274.1554854166784; Tue, 09 Apr 2019 16:56:06 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id j67sm56729208pfc.72.2019.04.09.16.56.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Apr 2019 16:56:05 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , linux-kselftest@vger.kernel.org Subject: [PATCH 4/6] selftests/runner: Add plan line and fix result line syntax Date: Tue, 9 Apr 2019 16:55:54 -0700 Message-Id: <20190409235556.3967-5-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409235556.3967-1-keescook@chromium.org> References: <20190409235556.3967-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The TAP version 13 spec requires a "plan" line, which has been missing. Since we always know how many tests we're going to run, emit the count on the plan line. This also fixes the result lines to remove the "1.." prefix which is against spec, and to mark skips with the correct "# SKIP" suffix. Signed-off-by: Kees Cook --- tools/testing/selftests/kselftest/runner.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 51139f42a6ca..7f8d6b127693 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -18,15 +18,15 @@ run_one() echo "========================================" if [ ! -x "$TEST" ]; then echo "$TEST_HDR_MSG: Warning: file $TEST is not executable, correct this." - echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + echo "not ok $test_num $TEST_HDR_MSG" else cd `dirname $TEST` > /dev/null (./$BASENAME_TEST >> "$logfile" 2>&1 && - echo "ok 1..$test_num $TEST_HDR_MSG [PASS]") || + echo "ok $test_num $TEST_HDR_MSG") || (if [ $? -eq $skip_rc ]; then \ - echo "not ok 1..$test_num $TEST_HDR_MSG [SKIP]" + echo "not ok $test_num $TEST_HDR_MSG # SKIP" else - echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + echo "not ok $test_num $TEST_HDR_MSG" fi) cd - >/dev/null fi @@ -37,6 +37,8 @@ run_many() echo "TAP version 13" DIR=$(basename "$PWD") test_num=0 + total=$(echo "$@" | wc -w) + echo "1..$total" for TEST in "$@"; do test_num=$(( test_num + 1 )) run_one "$DIR" "$TEST" "$test_num" From patchwork Tue Apr 9 23:55:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10892877 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 732861800 for ; Tue, 9 Apr 2019 23:56:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C4F528387 for ; Tue, 9 Apr 2019 23:56:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 50E3628727; Tue, 9 Apr 2019 23:56:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0134328387 for ; Tue, 9 Apr 2019 23:56:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726748AbfDIX4K (ORCPT ); Tue, 9 Apr 2019 19:56:10 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:37010 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726741AbfDIX4K (ORCPT ); Tue, 9 Apr 2019 19:56:10 -0400 Received: by mail-pl1-f195.google.com with SMTP id w23so200465ply.4 for ; Tue, 09 Apr 2019 16:56:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Y/RpihLB9WB13L/MPSV54cuBFONpGo1gl+FackKYf7M=; b=fMjZ62MyDP6Fqd1QMdlAhjj5DZJ424E2zEmOfhTxRBPMJRHLoG8qiu/+J1RNQDYG1f BYpJMxiy15aHzUcTTG72jnvosXPIKOmv23sXSMQxRyAcZw3Ar4Bv3c6h4GLYRq3DNifX VkNGnPkj4aSx/nCdLaur0vbIhdBQqEt/lOpoQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Y/RpihLB9WB13L/MPSV54cuBFONpGo1gl+FackKYf7M=; b=b1WT4Na4fmeKiApMAZJZ+Ip7aTE6ERmNpl7JMpHV5QcTjHWcruPfnkuM5GiZ+TWpwc Ic4VQpnij+rxMVQ5qd9UUuLTORzo0HNPF4UdZHeuuZmX0ZWupUK3NiMzsCAvZPjgatcy oSsmUe9EbSByKijyvsXD7qA1/KvxGFrMpIPxe0QwkjUyemWx0MyJxR6TryUDZWfeUwbH Ef/KncBORoqsnlB1M8W6SPKpHcRDjs5fTTiLehISpNbMliElzufAhJ5kJiF0wErYDujl Ls3D39Ow4BOHCzy3DzPR25xjAMYIWyIBxkwxHTMRyOqAvKYxGe+etr+N+9Zx+Y+Oswgw mDFQ== X-Gm-Message-State: APjAAAWx2FAfiCMApnttdTvMTF+8mQueJpNRpRxSk1h5CyQbzCLW18IN SI7N7h3HzXRS+eMopsG8dDqInA== X-Google-Smtp-Source: APXvYqzDsGOxH0rTx1Jjqe62/jhM+aRczFBXPlyL8O/qW6JjWw+PeBJc1NGnh77e6tt5aNhh8FNXDg== X-Received: by 2002:a17:902:599d:: with SMTP id p29mr14340476pli.156.1554854169891; Tue, 09 Apr 2019 16:56:09 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id d3sm50075110pfn.113.2019.04.09.16.56.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Apr 2019 16:56:09 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , linux-kselftest@vger.kernel.org Subject: [PATCH 5/6] selftests/runner: Distinguish between missing and non-executable Date: Tue, 9 Apr 2019 16:55:55 -0700 Message-Id: <20190409235556.3967-6-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409235556.3967-1-keescook@chromium.org> References: <20190409235556.3967-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If a test was missing (e.g. wrong architecture, etc), the test runner would incorrectly claim the test was non-executable. This adds an existence check to report correctly. Signed-off-by: Kees Cook --- tools/testing/selftests/kselftest/runner.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 7f8d6b127693..5793660dbe3f 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -17,7 +17,12 @@ run_one() echo "$TEST_HDR_MSG" echo "========================================" if [ ! -x "$TEST" ]; then - echo "$TEST_HDR_MSG: Warning: file $TEST is not executable, correct this." + echo -n "$TEST_HDR_MSG: Warning: file $TEST is " + if [ ! -e "$TEST" ]; then + echo "missing!" + else + echo "not executable, correct this." + fi echo "not ok $test_num $TEST_HDR_MSG" else cd `dirname $TEST` > /dev/null From patchwork Tue Apr 9 23:55:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10892879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18D801708 for ; Tue, 9 Apr 2019 23:56:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 003E828387 for ; Tue, 9 Apr 2019 23:56:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E8B9628727; Tue, 9 Apr 2019 23:56:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63A8928387 for ; Tue, 9 Apr 2019 23:56:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726756AbfDIX4M (ORCPT ); Tue, 9 Apr 2019 19:56:12 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:37010 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726741AbfDIX4L (ORCPT ); Tue, 9 Apr 2019 19:56:11 -0400 Received: by mail-pl1-f194.google.com with SMTP id w23so200489ply.4 for ; Tue, 09 Apr 2019 16:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L3VkJhqGguQYNQ544jAt4E93KwkMKUR3iV5epN09oo4=; b=FGS7DyCAibmO6BnoFRAXuW6EO5qHF33CfL9ulkM+rkuwlxH3lmfHIqw/NnNECTMi2b jzGNttVsuTEwo3TbfkIhPdN15iQ4XpI2ZlasfTR84WCrGy1bLNdsE/0Tw2r2bsq8tNZ+ XPog2lwyk5oGn4vqqjsvkynybRpilyl7zFRz8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=L3VkJhqGguQYNQ544jAt4E93KwkMKUR3iV5epN09oo4=; b=Tgm8eG3ybvQI/1x98LZ9oh73raSmGPMAZIfEbp2ZqswbxEgON8Lijw51ESZOU4MmHg CYfH06RfubAUFt/Z3DbmvXk8IprWhYFvHR2jqfwRdELpWV5w9K1Wnuxiw7EqC2QENn1a s6qFcQ2s12IU4iNjI4fX2BrJxC7iJC5q6188MIk982Q/7PezjcO615qN0eCXXP1sF7nz vTx/fgvtOmvZp94XdNSRM1xcjZxs7ZpTZgoZFrGxUXzVqh2xJ043VS3NftCm5HxXkOM3 U1n8CmLfLWh6K/MisS8gcfbApyla+5Zi+WU9ybhSHX11BtNHmYBHrKgPU+7KYKa49xA0 Wvvw== X-Gm-Message-State: APjAAAXwM/0s0ekGKAu6EzNds0K2xaZ/Y42jux0LSpRUtKNk0/eYkRsK lLcNbSP4DHPHDKTzFK0Gs3XSvQ== X-Google-Smtp-Source: APXvYqwlLmqoJZ5GRKTAZypnHlB2wXwRwYwg3faz2aZh/pEhyM6PY/ubL2HWq2DHyyeRPPs13bhd2g== X-Received: by 2002:a17:902:b94a:: with SMTP id h10mr40582160pls.82.1554854170438; Tue, 09 Apr 2019 16:56:10 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id g67sm52591223pfg.94.2019.04.09.16.56.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Apr 2019 16:56:09 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , linux-kselftest@vger.kernel.org Subject: [PATCH 6/6] selftests: Move test output to diagnostic lines Date: Tue, 9 Apr 2019 16:55:56 -0700 Message-Id: <20190409235556.3967-7-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409235556.3967-1-keescook@chromium.org> References: <20190409235556.3967-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This changes the selftest output so that each test's output is prefixed with "# " as a TAP "diagnostic line". This creates a bit of a kernel-specific TAP dialect where the diagnostics precede the results. The TAP spec isn't entirely clear about this, though, so I think it's the correct solution so as to keep interactive runs making sense. If the output _followed_ the result line in the spec-suggested YAML form, each test would dump all of its output at once instead of as it went, making debugging harder. This does, however, solve the recursive TAP output problem, as sub-tests will simply be prefixed by "# ". Parsing sub-tests becomes a simple problem of just removing the first two characters of a given top-level test's diagnostic output, and parsing the results. Note that the shell construct needed to both get an exit code from the first command in a pipe and still filter the pipe (to add the "# " prefix) uses a POSIX solution rather than the bash "pipefail" option which is not supported by dash. Since some test environments may have a very minimal set of utilities available, the new prefixing code will fall back to doing line-at-a-time prefixing if perl and/or stdbuf are not available. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/kselftest/prefix.pl | 23 +++++++++++++ tools/testing/selftests/kselftest/runner.sh | 37 ++++++++++++++++++--- tools/testing/selftests/lib.mk | 1 + 4 files changed, 58 insertions(+), 4 deletions(-) create mode 100755 tools/testing/selftests/kselftest/prefix.pl diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 72178d28f9c6..2e92a9727a4d 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -133,6 +133,7 @@ ifdef INSTALL_PATH @# Ask all targets to install their files mkdir -p $(INSTALL_PATH)/kselftest install -m 744 kselftest/runner.sh $(INSTALL_PATH)/kselftest/ + install -m 744 kselftest/prefix.pl $(INSTALL_PATH)/kselftest/ @for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ make OUTPUT=$$BUILD_TARGET -C $$TARGET INSTALL_PATH=$(INSTALL_PATH)/$$TARGET install; \ diff --git a/tools/testing/selftests/kselftest/prefix.pl b/tools/testing/selftests/kselftest/prefix.pl new file mode 100755 index 000000000000..ec7e48118183 --- /dev/null +++ b/tools/testing/selftests/kselftest/prefix.pl @@ -0,0 +1,23 @@ +#!/usr/bin/perl +# SPDX-License-Identifier: GPL-2.0 +# Prefix all lines with "# ", unbuffered. Command being piped in may need +# to have unbuffering forced with "stdbuf -i0 -o0 -e0 $cmd". +use strict; + +binmode STDIN; +binmode STDOUT; + +STDOUT->autoflush(1); + +my $needed = 1; +while (1) { + my $char; + my $bytes = sysread(STDIN, $char, 1); + exit 0 if ($bytes == 0); + if ($needed) { + print "# "; + $needed = 0; + } + print $char; + $needed = 1 if ($char eq "\n"); +} diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 5793660dbe3f..f8b545a6e0e9 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -5,6 +5,34 @@ export KSFT_TAP_LEVEL=1 export skip_rc=4 export logfile=/dev/stdout +# There isn't a shell-agnostic way to find the path of a sourced file, +# so we must rely on BASE_DIR being set to find other tools. +if [ -z "$BASE_DIR" ]; then + echo "Error: BASE_DIR must be set before sourcing." >&2 + exit 1 +fi + +# If Perl is unavailable, we must fall back to line-at-a-time prefixing +# with sed instead of unbuffered output. +tap_prefix() +{ + if [ ! -x /usr/bin/perl ]; then + sed -e 's/^/# /' + else + "$BASE_DIR"/kselftest/prefix.pl + fi +} + +# If stdbuf is unavailable, we must fall back to line-at-a-time piping. +tap_unbuffer() +{ + if ! which stdbuf >/dev/null ; then + "$@" + else + stdbuf -i0 -o0 -e0 "$@" + fi +} + run_one() { DIR="$1" @@ -14,10 +42,9 @@ run_one() BASENAME_TEST=$(basename $TEST) TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" - echo "$TEST_HDR_MSG" - echo "========================================" + echo "# $TEST_HDR_MSG" if [ ! -x "$TEST" ]; then - echo -n "$TEST_HDR_MSG: Warning: file $TEST is " + echo -n "# Warning: file $TEST is " if [ ! -e "$TEST" ]; then echo "missing!" else @@ -26,7 +53,9 @@ run_one() echo "not ok $test_num $TEST_HDR_MSG" else cd `dirname $TEST` > /dev/null - (./$BASENAME_TEST >> "$logfile" 2>&1 && + (((((tap_unbuffer ./$BASENAME_TEST 2>&1; echo $? >&3) | + tap_prefix >&4) 3>&1) | + (read xs; exit $xs)) 4>>"$logfile" && echo "ok $test_num $TEST_HDR_MSG") || (if [ $? -eq $skip_rc ]; then \ echo "not ok $test_num $TEST_HDR_MSG # SKIP" diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 8be13d117101..5c6505bec8be 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -33,6 +33,7 @@ endif .ONESHELL: define RUN_TESTS + BASE_DIR="$(selfdir)"; \ . $(selfdir)/kselftest/runner.sh; \ run_many $(1) endef