| Message ID | 20200914022227.437143-1-liuhangbin@gmail.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 84b785532887acc21ce439f2ff6fe6c532a2bc98 |
| Headers | show |
| Series | [PATCHv5,kselftest,next] selftests/run_kselftest.sh: make each test individually selectable | expand |
On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: > > Currently, after generating run_kselftest.sh, there is no way to choose > which test we could run. All the tests are listed together and we have > to run all every time. This patch enhanced the run_kselftest.sh to make > the tests individually selectable. e.g. > > $ ./run_kselftest.sh -t "bpf size timers" My test run break on linux next ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' ./run_kselftest.sh: line 1331: `-e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' steps do run: # run_kselftest.sh file generated by kselftest Makefile and included in tarball ./run_kselftest.sh 2>&1 | tee "${LOGFILE}" ref: https://github.com/nareshkamboju/test-definitions/blob/master/automated/linux/kselftest/kselftest.sh#L222 full test run log: https://lkft.validation.linaro.org/scheduler/job/1786826#L1391 > > Before the patch: > ================ > > $ cat run_kselftest.sh > \#!/bin/sh > BASE_DIR=$(realpath $(dirname $0)) > cd $BASE_DIR > . ./kselftest/runner.sh > ROOT=$PWD > if [ "$1" = "--summary" ]; then > logfile=$BASE_DIR/output.log > cat /dev/null > $logfile > fi > [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg > cd android > run_many \ > "run.sh" > cd $ROOT > ...<snip>... > [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg > cd zram > run_many \ > "zram.sh" > cd $ROOT > > After the patch: > =============== > > $ cat run_kselftest.sh > \#!/bin/sh > BASE_DIR=$(realpath $(dirname $0)) > . ./kselftest/runner.sh > TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram" > > run_android() > { > [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg > cd android > run_many \ > "run.sh" > cd $ROOT > } > > ...<snip>... > > run_filesystems_binderfs() > { > [ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg > cd filesystems/binderfs > run_many \ > "binderfs_test" > cd $ROOT > } > > ...<snip>... > > run_zram() > { > [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg > cd zram > run_many \ > "zram.sh" > cd $ROOT > } > > usage() > { > cat <<EOF > usage: ${0##*/} OPTS > -s | --summary Only print summary info and put detailed log in output.log > -t | --tests Test name you want to run specifically > -h | --help Show this usage info > EOF > } > > while true; do > case "$1" in > -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;; > -t | --tests ) TESTS=$2; shift 2 ;; > -l | --list ) echo $TESTS; exit 0 ;; > -h | --help ) usage; exit 0 ;; > "" ) break;; > * ) usage; exit 1;; > esac > done > > cd $BASE_DIR > ROOT=$PWD > for folder in $TESTS; do > folder=$(echo $folder | tr -s '/-' '_') > run_$folder > done > > Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote: > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: > > > > Currently, after generating run_kselftest.sh, there is no way to choose > > which test we could run. All the tests are listed together and we have > > to run all every time. This patch enhanced the run_kselftest.sh to make > > the tests individually selectable. e.g. > > > > $ ./run_kselftest.sh -t "bpf size timers" > > My test run break on linux next > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' > ./run_kselftest.sh: line 1331: `-e -s | --summary ) > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' Yes, please revert this patch. The resulting script is completely trashed: BASE_DIR=$(realpath $(dirname $0)) . ./kselftest/runner.sh TESTS="seccomp" run_seccomp() { -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg -e cd seccomp -en run_many \ -ne "seccomp_bpf" \ -ne "seccomp_benchmark" -e cd $ROOT }
On 9/25/20 3:16 PM, Kees Cook wrote: > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote: >> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: >>> >>> Currently, after generating run_kselftest.sh, there is no way to choose >>> which test we could run. All the tests are listed together and we have >>> to run all every time. This patch enhanced the run_kselftest.sh to make >>> the tests individually selectable. e.g. >>> >>> $ ./run_kselftest.sh -t "bpf size timers" >> >> My test run break on linux next >> >> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' >> ./run_kselftest.sh: line 1331: `-e -s | --summary ) >> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' > > Yes, please revert this patch. The resulting script is completely > trashed: > > BASE_DIR=$(realpath $(dirname $0)) > . ./kselftest/runner.sh > TESTS="seccomp" > > run_seccomp() > { > -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg > -e cd seccomp > -en run_many > \ > -ne "seccomp_bpf" > \ > -ne "seccomp_benchmark" > > -e cd $ROOT > } > > > Will do. thanks, -- Shuah
On 9/25/20 3:16 PM, Kees Cook wrote: > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote: >> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: >>> >>> Currently, after generating run_kselftest.sh, there is no way to choose >>> which test we could run. All the tests are listed together and we have >>> to run all every time. This patch enhanced the run_kselftest.sh to make >>> the tests individually selectable. e.g. >>> >>> $ ./run_kselftest.sh -t "bpf size timers" >> >> My test run break on linux next >> >> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' >> ./run_kselftest.sh: line 1331: `-e -s | --summary ) >> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' > > Yes, please revert this patch. The resulting script is completely > trashed: > Thank you both. Now reverted. thanks, -- Shuah
On Fri, Sep 25, 2020 at 05:06:02PM -0600, Shuah Khan wrote: > On 9/25/20 3:16 PM, Kees Cook wrote: > > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote: > > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: > > > > > > > > Currently, after generating run_kselftest.sh, there is no way to choose > > > > which test we could run. All the tests are listed together and we have > > > > to run all every time. This patch enhanced the run_kselftest.sh to make > > > > the tests individually selectable. e.g. > > > > > > > > $ ./run_kselftest.sh -t "bpf size timers" > > > > > > My test run break on linux next > > > > > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' > > > ./run_kselftest.sh: line 1331: `-e -s | --summary ) > > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' > > > > Yes, please revert this patch. The resulting script is completely > > trashed: > > > > Thank you both. Now reverted. I've sent an alternative that I think should do nicely. It will work well with LAVA as well.
On Fri, Sep 25, 2020 at 02:16:14PM -0700, Kees Cook wrote: > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote: > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: > > > > > > Currently, after generating run_kselftest.sh, there is no way to choose > > > which test we could run. All the tests are listed together and we have > > > to run all every time. This patch enhanced the run_kselftest.sh to make > > > the tests individually selectable. e.g. > > > > > > $ ./run_kselftest.sh -t "bpf size timers" > > > > My test run break on linux next > > > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' > > ./run_kselftest.sh: line 1331: `-e -s | --summary ) > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' > > Yes, please revert this patch. The resulting script is completely > trashed: > > BASE_DIR=$(realpath $(dirname $0)) > . ./kselftest/runner.sh > TESTS="seccomp" > > run_seccomp() > { > -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg > -e cd seccomp > -en run_many > \ > -ne "seccomp_bpf" > \ > -ne "seccomp_benchmark" > > -e cd $ROOT > } I'm really sorry to make this trouble. And I'm OK to revert the patch. I just a little wondering how do you generate this script. I tested with 'make -C tools/testing/selftests gen_tar FORMAT=.xz' before post the patch and all looks good to me. ``` run_seccomp() { [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg cd seccomp run_many \ "seccomp_bpf" \ "seccomp_benchmark" cd $ROOT } ``` Thanks Hangbin
On Sun, Sep 27, 2020 at 09:53:34AM +0800, Hangbin Liu wrote: > On Fri, Sep 25, 2020 at 02:16:14PM -0700, Kees Cook wrote: > > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote: > > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote: > > > > > > > > Currently, after generating run_kselftest.sh, there is no way to choose > > > > which test we could run. All the tests are listed together and we have > > > > to run all every time. This patch enhanced the run_kselftest.sh to make > > > > the tests individually selectable. e.g. > > > > > > > > $ ./run_kselftest.sh -t "bpf size timers" > > > > > > My test run break on linux next > > > > > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)' > > > ./run_kselftest.sh: line 1331: `-e -s | --summary ) > > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;' > > > > Yes, please revert this patch. The resulting script is completely > > trashed: > > > > BASE_DIR=$(realpath $(dirname $0)) > > . ./kselftest/runner.sh > > TESTS="seccomp" > > > > run_seccomp() > > { > > -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg > > -e cd seccomp > > -en run_many > > \ > > -ne "seccomp_bpf" > > \ > > -ne "seccomp_benchmark" > > > > -e cd $ROOT > > } > > I'm really sorry to make this trouble. And I'm OK to revert the patch. > I just a little wondering how do you generate this script. This issue is with which shell is used. I suspect your /bin/sh is full /bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which lacks "-e" support for the built-in "echo").
On Mon, Sep 28, 2020 at 01:06:15PM -0700, Kees Cook wrote: > > I'm really sorry to make this trouble. And I'm OK to revert the patch. > > I just a little wondering how do you generate this script. > > This issue is with which shell is used. I suspect your /bin/sh is full > /bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which > lacks "-e" support for the built-in "echo"). Ah, got it. Thanks for your explanation. Regards Hangbin
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 469d115a95f1..7b92f9c177f6 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -151,6 +151,14 @@ note some tests will require root privileges:: $ cd kselftest $ ./run_kselftest.sh +Or you can run some specific test cases in the installed Kselftests by:: + + $ ./run_kselftest.sh -t "bpf size timers" + +You can view the available tests to run with:: + + $ ./run_kselftest.sh -l + Packaging selftests =================== diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 15c1c1359c50..4c8159dd2bd7 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -225,13 +225,9 @@ ifdef INSTALL_PATH @# Ask all targets to emit their test scripts echo "#!/bin/sh" > $(ALL_SCRIPT) echo "BASE_DIR=\$$(realpath \$$(dirname \$$0))" >> $(ALL_SCRIPT) - echo "cd \$$BASE_DIR" >> $(ALL_SCRIPT) echo ". ./kselftest/runner.sh" >> $(ALL_SCRIPT) - echo "ROOT=\$$PWD" >> $(ALL_SCRIPT) - echo "if [ \"\$$1\" = \"--summary\" ]; then" >> $(ALL_SCRIPT) - echo " logfile=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) - echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT) - echo "fi" >> $(ALL_SCRIPT) + echo "TESTS=\"$(TARGETS)\"" >> $(ALL_SCRIPT) + echo "" >> $(ALL_SCRIPT); @# While building run_kselftest.sh skip also non-existent TARGET dirs: @# they could be the result of a build failure and should NOT be @@ -239,15 +235,50 @@ ifdef INSTALL_PATH for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ [ ! -d $(INSTALL_PATH)/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \ - echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \ - echo "cd $$TARGET" >> $(ALL_SCRIPT); \ - echo -n "run_many" >> $(ALL_SCRIPT); \ + echo "run_$$TARGET()" | tr -s "/-" "_" >> $(ALL_SCRIPT); \ + echo "{" >> $(ALL_SCRIPT); \ + echo -e "\t[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \ + echo -e "\tcd $$TARGET" >> $(ALL_SCRIPT); \ + echo -en "\trun_many" >> $(ALL_SCRIPT); \ echo -n "Emit Tests for $$TARGET\n"; \ $(MAKE) -s --no-print-directory OUTPUT=$$BUILD_TARGET -C $$TARGET emit_tests >> $(ALL_SCRIPT); \ echo "" >> $(ALL_SCRIPT); \ - echo "cd \$$ROOT" >> $(ALL_SCRIPT); \ + echo -e "\tcd \$$ROOT" >> $(ALL_SCRIPT); \ + echo "}" >> $(ALL_SCRIPT); \ + echo "" >> $(ALL_SCRIPT); \ done; + echo "usage()" >> $(ALL_SCRIPT); + echo "{" >> $(ALL_SCRIPT); + echo -e "\tcat <<EOF" >> $(ALL_SCRIPT); + echo "usage: \$${0##*/} OPTS" >> $(ALL_SCRIPT); + echo -e "\t-s | --summary\t\tOnly print summary info and put detailed log in output.log" >> $(ALL_SCRIPT); + echo -e "\t-t | --tests\t\tTest name you want to run specifically" >> $(ALL_SCRIPT); + echo -e "\t-l | --list\t\tList the available tests" >> $(ALL_SCRIPT); + echo -e "\t-h | --help\t\tShow this usage info" >> $(ALL_SCRIPT); + echo "EOF" >> $(ALL_SCRIPT); + echo "}" >> $(ALL_SCRIPT); + echo "" >> $(ALL_SCRIPT); + + echo "while true; do" >> $(ALL_SCRIPT); + echo -e "\tcase \"\$$1\" in" >> $(ALL_SCRIPT); + echo -e "\t-s | --summary ) logfile=\$$BASE_DIR/output.log; cat /dev/null > \$$logfile; shift ;;" >> $(ALL_SCRIPT); + echo -e "\t-t | --tests ) TESTS=\$$2; shift 2 ;;" >> $(ALL_SCRIPT); + echo -e "\t-l | --list ) echo \$$TESTS; exit 0 ;;" >> $(ALL_SCRIPT); + echo -e "\t-h | --help ) usage; exit 0 ;;" >> $(ALL_SCRIPT); + echo -e "\t\"\" ) break;;" >> $(ALL_SCRIPT); + echo -e "\t* ) usage; exit 1;;" >> $(ALL_SCRIPT); + echo -e "\tesac" >> $(ALL_SCRIPT); + echo "done" >> $(ALL_SCRIPT); + echo "" >> $(ALL_SCRIPT); + + echo "cd \$$BASE_DIR" >> $(ALL_SCRIPT) + echo "ROOT=\$$PWD" >> $(ALL_SCRIPT) + + echo "for folder in \$$TESTS; do" >> $(ALL_SCRIPT); \ + echo -e "\tfolder=\$$(echo \$$folder | tr -s '/-' '_')" >> $(ALL_SCRIPT); \ + echo -e "\trun_\$$folder" >> $(ALL_SCRIPT); \ + echo "done" >> $(ALL_SCRIPT); \ chmod u+x $(ALL_SCRIPT) else $(error Error: set INSTALL_PATH to use install) diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 51124b962d56..3c4c94a5d184 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -108,7 +108,7 @@ emit_tests: for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ echo " \\"; \ - echo -n " \"$$BASENAME_TEST\""; \ + echo -ne "\t\t\"$$BASENAME_TEST\""; \ done; \ # define if isn't already. It is undefined in make O= case.
Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g. $ ./run_kselftest.sh -t "bpf size timers" Before the patch: ================ $ cat run_kselftest.sh \#!/bin/sh BASE_DIR=$(realpath $(dirname $0)) cd $BASE_DIR . ./kselftest/runner.sh ROOT=$PWD if [ "$1" = "--summary" ]; then logfile=$BASE_DIR/output.log cat /dev/null > $logfile fi [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg cd android run_many \ "run.sh" cd $ROOT ...<snip>... [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg cd zram run_many \ "zram.sh" cd $ROOT After the patch: =============== $ cat run_kselftest.sh \#!/bin/sh BASE_DIR=$(realpath $(dirname $0)) . ./kselftest/runner.sh TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram" run_android() { [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg cd android run_many \ "run.sh" cd $ROOT } ...<snip>... run_filesystems_binderfs() { [ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg cd filesystems/binderfs run_many \ "binderfs_test" cd $ROOT } ...<snip>... run_zram() { [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg cd zram run_many \ "zram.sh" cd $ROOT } usage() { cat <<EOF usage: ${0##*/} OPTS -s | --summary Only print summary info and put detailed log in output.log -t | --tests Test name you want to run specifically -h | --help Show this usage info EOF } while true; do case "$1" in -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;; -t | --tests ) TESTS=$2; shift 2 ;; -l | --list ) echo $TESTS; exit 0 ;; -h | --help ) usage; exit 0 ;; "" ) break;; * ) usage; exit 1;; esac done cd $BASE_DIR ROOT=$PWD for folder in $TESTS; do folder=$(echo $folder | tr -s '/-' '_') run_$folder done Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v5: Forgot to update commit description for the new added -l option v4: Add parameter -l to list available tests, suggested by Bird, Tim v3: 1) rebase the patch to latest code 2) move `tr -s "/-" "_"` in for loop at the end so user could use test folder name directly. Before the fix, user need to use ./run_kselftest.sh -t 'networking_forwarding'. Now they can just run ./run_kselftest.sh -t 'networking/forwarding' directly. v2: update document and commit description. --- Documentation/dev-tools/kselftest.rst | 8 +++++ tools/testing/selftests/Makefile | 51 +++++++++++++++++++++------ tools/testing/selftests/lib.mk | 2 +- 3 files changed, 50 insertions(+), 11 deletions(-)