diff mbox series

selftests/vm: enable running select groups of tests

Message ID 20220705185605.3889110-1-jsavitz@redhat.com (mailing list archive)
State New
Headers show
Series selftests/vm: enable running select groups of tests | expand

Commit Message

Joel Savitz July 5, 2022, 6:56 p.m. UTC
Add the ability to run one or more groups of vm tests (specified
by the environment variable TEST_ITEMS). Preserve existing default
behavior of running all tests when TEST_ITEMS is empty or "default".

Documentation of test groups is included in the patch as follows:

    # ./run_vmtests.sh [ -h || --help ]

    usage: ./tools/testing/selftests/vm/run_vmtests.sh [ -h ]
      -h: display this message

    The default behavior is to run all tests.

    Alternatively, specific groups tests can be run by defining
    the TEST_ITEMS shell variable with a space-separated string
    of one or more of the following:
    - mmap
	    tests for mmap(2)
    - gup_test
	    tests for gup using gup_test interface
    - userfaultfd
	    tests for  userfaultfd(2)
    - compaction
	    a test for the patch "Allow compaction of unevictable pages"
    - mlock
	    tests for mlock(2)
    - mremap
	    tests for mremap(2)
    - hugevm
	    tests for very large virtual address space
    - vmalloc
	    vmalloc smoke tests
    - hmm
	    hmm smoke tests
    - madv_populate
	    test memadvise(2) MADV_POPULATE_{READ,WRITE} options
    - memfd_secret
	    test memfd_secret(2)
    - process_mrelease
	    test process_mrelease(2)
    - ksm
	    ksm tests that do not require >=2 NUMA nodes
    - ksm_numa
	    ksm tests that require >=2 NUMA nodes
    example: TEST_ITEMS="hmm mmap ksm" ./run_vmtests.sh

Signed-off-by: Joel Savitz <jsavitz@redhat.com>
---
 tools/testing/selftests/vm/run_vmtests.sh | 224 +++++++++++++++-------
 1 file changed, 150 insertions(+), 74 deletions(-)

Comments

Andrew Morton July 5, 2022, 8:34 p.m. UTC | #1
On Tue,  5 Jul 2022 14:56:05 -0400 Joel Savitz <jsavitz@redhat.com> wrote:

> Add the ability to run one or more groups of vm tests (specified
> by the environment variable TEST_ITEMS). Preserve existing default
> behavior of running all tests when TEST_ITEMS is empty or "default".

What is the reason for this?  What's the use case?

And why via the environment rather than via commandline args?
Nico Pache July 6, 2022, 6:38 p.m. UTC | #2
On Tue, Jul 5, 2022 at 4:35 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Tue,  5 Jul 2022 14:56:05 -0400 Joel Savitz <jsavitz@redhat.com> wrote:
>
> > Add the ability to run one or more groups of vm tests (specified
> > by the environment variable TEST_ITEMS). Preserve existing default
> > behavior of running all tests when TEST_ITEMS is empty or "default".
>
Hi Andrew,

> What is the reason for this?  What's the use case?

The current design of vm selftests is all-or-none. We'd like to be
able to selectively run these tests (We settled for selective groups
of tests rather than individual tests).

The main reason for doing this is our efforts to expand RedHats MM CI
testing. There are two use cases for these changes that relate to our
reasoning:
1) Our current CI has overlapping tests between LTP and vm selftests,
so we'd like a way to prevent running the same test in two places.
2) We'd like the ability to skip a test if it is determined to be
unstable or requires certain hardware requirements.

By adding this functionality we are really expanding what we are able
to do with the stock vm-selftests.

> And why via the environment rather than via commandline args?
Just a design choice I suppose. I'm sure Joel would be willing to
implement it as a cmdline arg if you'd prefer that approach.

Cheers,
-- Nico
Andrew Morton July 6, 2022, 6:50 p.m. UTC | #3
On Wed, 6 Jul 2022 14:38:03 -0400 Nico Pache <npache@redhat.com> wrote:

> On Tue, Jul 5, 2022 at 4:35 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Tue,  5 Jul 2022 14:56:05 -0400 Joel Savitz <jsavitz@redhat.com> wrote:
> >
> > > Add the ability to run one or more groups of vm tests (specified
> > > by the environment variable TEST_ITEMS). Preserve existing default
> > > behavior of running all tests when TEST_ITEMS is empty or "default".
> >
> Hi Andrew,
> 
> > What is the reason for this?  What's the use case?
> 
> The current design of vm selftests is all-or-none. We'd like to be
> able to selectively run these tests (We settled for selective groups
> of tests rather than individual tests).
> 
> The main reason for doing this is our efforts to expand RedHats MM CI
> testing. There are two use cases for these changes that relate to our
> reasoning:
> 1) Our current CI has overlapping tests between LTP and vm selftests,
> so we'd like a way to prevent running the same test in two places.
> 2) We'd like the ability to skip a test if it is determined to be
> unstable or requires certain hardware requirements.
> 
> By adding this functionality we are really expanding what we are able
> to do with the stock vm-selftests.

OK, please let's get this info into the changelog - it's the most
important part.

> > And why via the environment rather than via commandline args?
> Just a design choice I suppose. I'm sure Joel would be willing to
> implement it as a cmdline arg if you'd prefer that approach.

I think that would be best.
Joel Savitz July 6, 2022, 6:53 p.m. UTC | #4
Thanks for the feedback.

Will rework and send v2.

Best,
Joel Savitz

On Wed, Jul 6, 2022 at 2:50 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Wed, 6 Jul 2022 14:38:03 -0400 Nico Pache <npache@redhat.com> wrote:
>
> > On Tue, Jul 5, 2022 at 4:35 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> > >
> > > On Tue,  5 Jul 2022 14:56:05 -0400 Joel Savitz <jsavitz@redhat.com> wrote:
> > >
> > > > Add the ability to run one or more groups of vm tests (specified
> > > > by the environment variable TEST_ITEMS). Preserve existing default
> > > > behavior of running all tests when TEST_ITEMS is empty or "default".
> > >
> > Hi Andrew,
> >
> > > What is the reason for this?  What's the use case?
> >
> > The current design of vm selftests is all-or-none. We'd like to be
> > able to selectively run these tests (We settled for selective groups
> > of tests rather than individual tests).
> >
> > The main reason for doing this is our efforts to expand RedHats MM CI
> > testing. There are two use cases for these changes that relate to our
> > reasoning:
> > 1) Our current CI has overlapping tests between LTP and vm selftests,
> > so we'd like a way to prevent running the same test in two places.
> > 2) We'd like the ability to skip a test if it is determined to be
> > unstable or requires certain hardware requirements.
> >
> > By adding this functionality we are really expanding what we are able
> > to do with the stock vm-selftests.
>
> OK, please let's get this info into the changelog - it's the most
> important part.
>
> > > And why via the environment rather than via commandline args?
> > Just a design choice I suppose. I'm sure Joel would be willing to
> > implement it as a cmdline arg if you'd prefer that approach.
>
> I think that would be best.
>
Andrew Morton Aug. 15, 2022, 1:55 a.m. UTC | #5
On Tue,  5 Jul 2022 14:56:05 -0400 Joel Savitz <jsavitz@redhat.com> wrote:

> Add the ability to run one or more groups of vm tests (specified
> by the environment variable TEST_ITEMS). Preserve existing default
> behavior of running all tests when TEST_ITEMS is empty or "default".

Why are we doing this?   Please describe the use case - something I
can add to the changelog.
Andrew Morton Aug. 15, 2022, 1:56 a.m. UTC | #6
On Sun, 14 Aug 2022 18:55:03 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:

> On Tue,  5 Jul 2022 14:56:05 -0400 Joel Savitz <jsavitz@redhat.com> wrote:
> 
> > Add the ability to run one or more groups of vm tests (specified
> > by the environment variable TEST_ITEMS). Preserve existing default
> > behavior of running all tests when TEST_ITEMS is empty or "default".
> 
> Why are we doing this?   Please describe the use case - something I
> can add to the changelog.

Old patch version, please ignore.
diff mbox series

Patch

diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh
index 41fce8bea929..d31fc66205f8 100755
--- a/tools/testing/selftests/vm/run_vmtests.sh
+++ b/tools/testing/selftests/vm/run_vmtests.sh
@@ -1,22 +1,74 @@ 
 #!/bin/bash
 # SPDX-License-Identifier: GPL-2.0
-#please run as root
+# Please run as root
 
 # Kselftest framework requirement - SKIP code is 4.
 ksft_skip=4
 
 mnt=./huge
 exitcode=0
-
-#get huge pagesize and freepages from /proc/meminfo
-while read -r name size unit; do
-	if [ "$name" = "HugePages_Free:" ]; then
-		freepgs="$size"
-	fi
-	if [ "$name" = "Hugepagesize:" ]; then
-		hpgsize_KB="$size"
+nr_tests_ran=0
+
+if [ ${1:-0} == "-h" ] || [ ${1:-0} == "--help" ]
+then
+	cat <<EOF
+usage: ${BASH_SOURCE[0]:-$0} [ -h ]
+  -h: display this message
+
+The default behavior is to run all tests.
+
+Alternatively, specific groups tests can be run by defining
+the TEST_ITEMS shell variable with a space-separated string
+of one or more of the following:
+- mmap
+	tests for mmap(2)
+- gup_test
+	tests for gup using gup_test interface
+- userfaultfd
+	tests for  userfaultfd(2)
+- compaction
+	a test for the patch "Allow compaction of unevictable pages"
+- mlock
+	tests for mlock(2)
+- mremap
+	tests for mremap(2)
+- hugevm
+	tests for very large virtual address space
+- vmalloc
+	vmalloc smoke tests
+- hmm
+	hmm smoke tests
+- madv_populate
+	test memadvise(2) MADV_POPULATE_{READ,WRITE} options
+- memfd_secret
+	test memfd_secret(2)
+- process_mrelease
+	test process_mrelease(2)
+- ksm
+	ksm tests that do not require >=2 NUMA nodes
+- ksm_numa
+	ksm tests that require >=2 NUMA nodes
+example: TEST_ITEMS="hmm mmap ksm" ./run_vmtests.sh
+EOF
+	exit 0
+fi # $1 == -h || $1 == --help
+
+# default behavior: run all tests
+TEST_ITEMS=${TEST_ITEMS:-default}
+
+echo "Selected test items: ${TEST_ITEMS}"
+
+test_selected() {
+	if [ "$TEST_ITEMS" == "default" ]; then
+		# If no TEST_ITEMS are specified, run all tests
+		return 0
 	fi
-done < /proc/meminfo
+	echo ${TEST_ITEMS} | grep ${1} 2>&1 >/dev/null
+	return ${?}
+}
+
+# Hugepage setup only needed for hugetlb tests
+if test_selected "hugetlb"; then
 
 # Simple hugetlbfs tests have a hardcoded minimum requirement of
 # huge pages totaling 256MB (262144KB) in size.  The userfaultfd
@@ -28,7 +80,17 @@  hpgsize_MB=$((hpgsize_KB / 1024))
 half_ufd_size_MB=$((((nr_cpus * hpgsize_MB + 127) / 128) * 128))
 needmem_KB=$((half_ufd_size_MB * 2 * 1024))
 
-#set proper nr_hugepages
+# get huge pagesize and freepages from /proc/meminfo
+while read -r name size unit; do
+	if [ "$name" = "HugePages_Free:" ]; then
+		freepgs="$size"
+	fi
+	if [ "$name" = "Hugepagesize:" ]; then
+		hpgsize_KB="$size"
+	fi
+done < /proc/meminfo
+
+# set proper nr_hugepages
 if [ -n "$freepgs" ] && [ -n "$hpgsize_KB" ]; then
 	nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
 	needpgs=$((needmem_KB / hpgsize_KB))
@@ -57,126 +119,140 @@  else
 	exit 1
 fi
 
-#filter 64bit architectures
+fi # test_selected "hugetlb"
+
+# filter 64bit architectures
 ARCH64STR="arm64 ia64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sh64 sparc64 x86_64"
 if [ -z "$ARCH" ]; then
 	ARCH=$(uname -m 2>/dev/null | sed -e 's/aarch64.*/arm64/')
 fi
 VADDR64=0
-echo "$ARCH64STR" | grep "$ARCH" && VADDR64=1
+echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
 
 # Usage: run_test [test binary] [arbitrary test arguments...]
 run_test() {
-	local title="running $*"
-	local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
-	printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
-
-	"$@"
-	local ret=$?
-	if [ $ret -eq 0 ]; then
-		echo "[PASS]"
-	elif [ $ret -eq $ksft_skip ]; then
-		echo "[SKIP]"
-		exitcode=$ksft_skip
-	else
-		echo "[FAIL]"
-		exitcode=1
-	fi
+	if test_selected ${CATEGORY}; then
+		local title="running $*"
+		local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
+		printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
+
+		"$@"
+		local ret=$?
+		if [ $ret -eq 0 ]; then
+			echo "[PASS]"
+		elif [ $ret -eq $ksft_skip ]; then
+			echo "[SKIP]"
+			exitcode=$ksft_skip
+		else
+			echo "[FAIL]"
+			exitcode=1
+		fi
+		nr_tests_ran=$((nr_tests_ran + 1))
+	fi # test_selected
 }
 
-mkdir "$mnt"
-mount -t hugetlbfs none "$mnt"
+# setup only needed for hugetlb tests
+if test_selected "hugetlb"; then
+	mkdir "$mnt"
+	mount -t hugetlbfs none "$mnt"
+fi
 
-run_test ./hugepage-mmap
+CATEGORY="hugetlb" run_test ./hugepage-mmap
 
 shmmax=$(cat /proc/sys/kernel/shmmax)
 shmall=$(cat /proc/sys/kernel/shmall)
 echo 268435456 > /proc/sys/kernel/shmmax
 echo 4194304 > /proc/sys/kernel/shmall
-run_test ./hugepage-shm
+CATEGORY="hugetlb" run_test ./hugepage-shm
 echo "$shmmax" > /proc/sys/kernel/shmmax
 echo "$shmall" > /proc/sys/kernel/shmall
 
-run_test ./map_hugetlb
+CATEGORY="hugetlb" run_test ./map_hugetlb
 
-run_test ./hugepage-mremap "$mnt"/huge_mremap
-rm -f "$mnt"/huge_mremap
+CATEGORY="hugetlb" run_test ./hugepage-mremap "$mnt"/huge_mremap
+test_selected "hugetlb" && rm -f "$mnt"/huge_mremap
 
-run_test ./hugepage-vmemmap
+CATEGORY="hugetlb" run_test ./hugepage-vmemmap
 
-run_test ./hugetlb-madvise "$mnt"/madvise-test
-rm -f "$mnt"/madvise-test
+CATEGORY="hugetlb" run_test ./hugetlb-madvise "$mnt"/madvise-test
+test_selected "hugetlb" && rm -f "$mnt"/madvise-test
 
-echo "NOTE: The above hugetlb tests provide minimal coverage.  Use"
-echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
-echo "      hugetlb regression testing."
+if test_selected "hugetlb"; then
+	echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
+	echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
+	echo "      hugetlb regression testing."
+fi
 
-run_test ./map_fixed_noreplace
+CATEGORY="mmap" run_test ./map_fixed_noreplace
 
 # get_user_pages_fast() benchmark
-run_test ./gup_test -u
+CATEGORY="gup_test" run_test ./gup_test -u
 # pin_user_pages_fast() benchmark
-run_test ./gup_test -a
+CATEGORY="gup_test" run_test ./gup_test -a
 # Dump pages 0, 19, and 4096, using pin_user_pages:
-run_test ./gup_test -ct -F 0x1 0 19 0x1000
+CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
 
-run_test ./userfaultfd anon 20 16
+CATEGORY="userfaultfd" run_test ./userfaultfd anon 20 16
 # Test requires source and destination huge pages.  Size of source
 # (half_ufd_size_MB) is passed as argument to test.
-run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
-run_test ./userfaultfd shmem 20 16
-
-#cleanup
-umount "$mnt"
-rm -rf "$mnt"
-echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
+CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
+CATEGORY="userfaultfd" run_test ./userfaultfd shmem 20 16
+
+# cleanup (only needed when running hugetlb tests)
+if test_selected "hugetlb"; then
+	umount "$mnt"
+	rm -rf "$mnt"
+	echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
+fi
 
-run_test ./compaction_test
+CATEGORY="compaction" run_test ./compaction_test
 
-run_test sudo -u nobody ./on-fault-limit
+CATEGORY="mlock" run_test sudo -u nobody ./on-fault-limit
 
-run_test ./map_populate
+CATEGORY="mmap" run_test ./map_populate
 
-run_test ./mlock-random-test
+CATEGORY="mlock" run_test ./mlock-random-test
 
-run_test ./mlock2-tests
+CATEGORY="mlock" run_test ./mlock2-tests
 
-run_test ./mrelease_test
+CATEGORY="process_mrelease" run_test ./mrelease_test
 
-run_test ./mremap_test
+CATEGORY="mremap" run_test ./mremap_test
 
-run_test ./thuge-gen
+CATEGORY="hugetlb" run_test ./thuge-gen
 
 if [ $VADDR64 -ne 0 ]; then
-	run_test ./virtual_address_range
+	CATEGORY="hugevm" run_test ./virtual_address_range
 
 	# virtual address 128TB switch test
-	run_test ./va_128TBswitch
+	CATEGORY="hugevm" run_test ./va_128TBswitch
 fi # VADDR64
 
 # vmalloc stability smoke test
-run_test ./test_vmalloc.sh smoke
+CATEGORY="vmalloc" run_test ./test_vmalloc.sh smoke
 
-run_test ./mremap_dontunmap
+CATEGORY="mremap" run_test ./mremap_dontunmap
 
-run_test ./test_hmm.sh smoke
+CATEGORY="hmm" run_test ./test_hmm.sh smoke
 
 # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
-run_test ./madv_populate
+CATEGORY="madv_populate" run_test ./madv_populate
 
-run_test ./memfd_secret
+CATEGORY="memfd_secret" run_test ./memfd_secret
 
 # KSM MADV_MERGEABLE test with 10 identical pages
-run_test ./ksm_tests -M -p 10
+CATEGORY="ksm" run_test ./ksm_tests -M -p 10
 # KSM unmerge test
-run_test ./ksm_tests -U
+CATEGORY="ksm" run_test ./ksm_tests -U
 # KSM test with 10 zero pages and use_zero_pages = 0
-run_test ./ksm_tests -Z -p 10 -z 0
+CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 0
 # KSM test with 10 zero pages and use_zero_pages = 1
-run_test ./ksm_tests -Z -p 10 -z 1
+CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 1
 # KSM test with 2 NUMA nodes and merge_across_nodes = 1
-run_test ./ksm_tests -N -m 1
+CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 1
 # KSM test with 2 NUMA nodes and merge_across_nodes = 0
-run_test ./ksm_tests -N -m 0
+CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 0
+
+echo "Ran $nr_tests_ran tests"
 
 exit $exitcode