| Message ID | 20230713135440.3651409-10-ryan.roberts@arm.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | selftests/mm fixes for arm64 | expand |
On 13.07.23 15:54, Ryan Roberts wrote: > It is very unclear to me how one is supposed to run all the mm selftests > consistently and get clear results. > > Most of the test programs are launched by both run_vmtests.sh and > run_kselftest.sh: > > hugepage-mmap > hugepage-shm > map_hugetlb > hugepage-mremap > hugepage-vmemmap > hugetlb-madvise > map_fixed_noreplace > gup_test > gup_longterm > uffd-unit-tests > uffd-stress > compaction_test > on-fault-limit > map_populate > mlock-random-test > mlock2-tests > mrelease_test > mremap_test > thuge-gen > virtual_address_range > va_high_addr_switch > mremap_dontunmap > hmm-tests > madv_populate > memfd_secret > ksm_tests > ksm_functional_tests > soft-dirty > cow > Which run_kselftest.sh are you referring to, the one in the parent directory? How to invoke it to run these mm tests? (I never dared invoking something different than run_vmtests.sh ;) ) [...] > > Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> > --- > tools/testing/selftests/mm/Makefile | 79 ++++++++++++----------- > tools/testing/selftests/mm/run_vmtests.sh | 23 +++++++ > tools/testing/selftests/mm/settings | 2 +- > 3 files changed, 64 insertions(+), 40 deletions(-) > > diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile > index 66d7c07dc177..881ed96d96fd 100644 > --- a/tools/testing/selftests/mm/Makefile > +++ b/tools/testing/selftests/mm/Makefile > @@ -35,39 +35,39 @@ MAKEFLAGS += --no-builtin-rules > CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) > LDLIBS = -lrt -lpthread > > -TEST_GEN_PROGS = cow > -TEST_GEN_PROGS += compaction_test > -TEST_GEN_PROGS += gup_longterm > -TEST_GEN_PROGS += gup_test > -TEST_GEN_PROGS += hmm-tests > -TEST_GEN_PROGS += hugetlb-madvise > -TEST_GEN_PROGS += hugepage-mmap > -TEST_GEN_PROGS += hugepage-mremap > -TEST_GEN_PROGS += hugepage-shm > -TEST_GEN_PROGS += hugepage-vmemmap > -TEST_GEN_PROGS += khugepaged > -TEST_GEN_PROGS += madv_populate > -TEST_GEN_PROGS += map_fixed_noreplace > -TEST_GEN_PROGS += map_hugetlb > -TEST_GEN_PROGS += map_populate > -TEST_GEN_PROGS += memfd_secret > -TEST_GEN_PROGS += migration > -TEST_GEN_PROGS += mkdirty > -TEST_GEN_PROGS += mlock-random-test > -TEST_GEN_PROGS += mlock2-tests > -TEST_GEN_PROGS += mrelease_test > -TEST_GEN_PROGS += mremap_dontunmap > -TEST_GEN_PROGS += mremap_test > -TEST_GEN_PROGS += on-fault-limit > -TEST_GEN_PROGS += thuge-gen > -TEST_GEN_PROGS += transhuge-stress > -TEST_GEN_PROGS += uffd-stress > -TEST_GEN_PROGS += uffd-unit-tests > -TEST_GEN_PROGS += soft-dirty > -TEST_GEN_PROGS += split_huge_page_test > -TEST_GEN_PROGS += ksm_tests > -TEST_GEN_PROGS += ksm_functional_tests > -TEST_GEN_PROGS += mdwe_test > +TEST_GEN_FILES = cow > +TEST_GEN_FILES += compaction_test > +TEST_GEN_FILES += gup_longterm > +TEST_GEN_FILES += gup_test > +TEST_GEN_FILES += hmm-tests > +TEST_GEN_FILES += hugetlb-madvise > +TEST_GEN_FILES += hugepage-mmap > +TEST_GEN_FILES += hugepage-mremap > +TEST_GEN_FILES += hugepage-shm > +TEST_GEN_FILES += hugepage-vmemmap > +TEST_GEN_FILES += khugepaged > +TEST_GEN_FILES += madv_populate > +TEST_GEN_FILES += map_fixed_noreplace > +TEST_GEN_FILES += map_hugetlb > +TEST_GEN_FILES += map_populate > +TEST_GEN_FILES += memfd_secret > +TEST_GEN_FILES += migration > +TEST_GEN_FILES += mkdirty > +TEST_GEN_FILES += mlock-random-test > +TEST_GEN_FILES += mlock2-tests > +TEST_GEN_FILES += mrelease_test > +TEST_GEN_FILES += mremap_dontunmap > +TEST_GEN_FILES += mremap_test > +TEST_GEN_FILES += on-fault-limit > +TEST_GEN_FILES += thuge-gen > +TEST_GEN_FILES += transhuge-stress > +TEST_GEN_FILES += uffd-stress > +TEST_GEN_FILES += uffd-unit-tests > +TEST_GEN_FILES += soft-dirty > +TEST_GEN_FILES += split_huge_page_test > +TEST_GEN_FILES += ksm_tests > +TEST_GEN_FILES += ksm_functional_tests > +TEST_GEN_FILES += mdwe_test IIRC, we recently converted all to TEST_GEN_PROGS. See commit aef6fde75d8c6c1cad4a0e017a8d4cbee2143723 Author: Peter Xu <peterx@redhat.com> Date: Wed Apr 12 12:42:18 2023 -0400 selftests/mm: use TEST_GEN_PROGS where proper TEST_GEN_PROGS and TEST_GEN_FILES are used randomly in the mm/Makefile to specify programs that need to build. Logically all these binaries should all fall into TEST_GEN_PROGS. Replace those TEST_GEN_FILES with TEST_GEN_PROGS, so that we can reference all the tests easily later. Why is that change required, and how does it interact with run_kselftest.sh? (Not clear from you patch description.)
On 13/07/2023 15:50, David Hildenbrand wrote: > On 13.07.23 15:54, Ryan Roberts wrote: >> It is very unclear to me how one is supposed to run all the mm selftests >> consistently and get clear results. >> >> Most of the test programs are launched by both run_vmtests.sh and >> run_kselftest.sh: >> >> hugepage-mmap >> hugepage-shm >> map_hugetlb >> hugepage-mremap >> hugepage-vmemmap >> hugetlb-madvise >> map_fixed_noreplace >> gup_test >> gup_longterm >> uffd-unit-tests >> uffd-stress >> compaction_test >> on-fault-limit >> map_populate >> mlock-random-test >> mlock2-tests >> mrelease_test >> mremap_test >> thuge-gen >> virtual_address_range >> va_high_addr_switch >> mremap_dontunmap >> hmm-tests >> madv_populate >> memfd_secret >> ksm_tests >> ksm_functional_tests >> soft-dirty >> cow >> > > Which run_kselftest.sh are you referring to, the one in the parent directory? run_kselftest.sh is the uniform way of executing all the kselftests. mm seems to be trying to be special as far as I can see. Certainly if you run the `install` make target, kselftests will create a list of all the tests (including non-mm tests if you have included them in the TARGETS variable) and copy that test list and run_kselftest.sh to the install path along with all the test binaries. Then the user can invoke any of the collections or specific tests in the collections using that tool. It also wraps everything with tap output, runs tests with a timeout, etc. See Documentation/dev-tools/kselftest.rst > > How to invoke it to run these mm tests? > > (I never dared invoking something different than > run_vmtests.sh ;) ) # single test: $ sudo ./run_kselftest.sh -t mm:<test_name> or # all tests in collection: $ sudo ./run_kselftest.sh -c mm > > [...] > >> >> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> >> --- >> tools/testing/selftests/mm/Makefile | 79 ++++++++++++----------- >> tools/testing/selftests/mm/run_vmtests.sh | 23 +++++++ >> tools/testing/selftests/mm/settings | 2 +- >> 3 files changed, 64 insertions(+), 40 deletions(-) >> >> diff --git a/tools/testing/selftests/mm/Makefile >> b/tools/testing/selftests/mm/Makefile >> index 66d7c07dc177..881ed96d96fd 100644 >> --- a/tools/testing/selftests/mm/Makefile >> +++ b/tools/testing/selftests/mm/Makefile >> @@ -35,39 +35,39 @@ MAKEFLAGS += --no-builtin-rules >> CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) >> LDLIBS = -lrt -lpthread >> -TEST_GEN_PROGS = cow >> -TEST_GEN_PROGS += compaction_test >> -TEST_GEN_PROGS += gup_longterm >> -TEST_GEN_PROGS += gup_test >> -TEST_GEN_PROGS += hmm-tests >> -TEST_GEN_PROGS += hugetlb-madvise >> -TEST_GEN_PROGS += hugepage-mmap >> -TEST_GEN_PROGS += hugepage-mremap >> -TEST_GEN_PROGS += hugepage-shm >> -TEST_GEN_PROGS += hugepage-vmemmap >> -TEST_GEN_PROGS += khugepaged >> -TEST_GEN_PROGS += madv_populate >> -TEST_GEN_PROGS += map_fixed_noreplace >> -TEST_GEN_PROGS += map_hugetlb >> -TEST_GEN_PROGS += map_populate >> -TEST_GEN_PROGS += memfd_secret >> -TEST_GEN_PROGS += migration >> -TEST_GEN_PROGS += mkdirty >> -TEST_GEN_PROGS += mlock-random-test >> -TEST_GEN_PROGS += mlock2-tests >> -TEST_GEN_PROGS += mrelease_test >> -TEST_GEN_PROGS += mremap_dontunmap >> -TEST_GEN_PROGS += mremap_test >> -TEST_GEN_PROGS += on-fault-limit >> -TEST_GEN_PROGS += thuge-gen >> -TEST_GEN_PROGS += transhuge-stress >> -TEST_GEN_PROGS += uffd-stress >> -TEST_GEN_PROGS += uffd-unit-tests >> -TEST_GEN_PROGS += soft-dirty >> -TEST_GEN_PROGS += split_huge_page_test >> -TEST_GEN_PROGS += ksm_tests >> -TEST_GEN_PROGS += ksm_functional_tests >> -TEST_GEN_PROGS += mdwe_test >> +TEST_GEN_FILES = cow >> +TEST_GEN_FILES += compaction_test >> +TEST_GEN_FILES += gup_longterm >> +TEST_GEN_FILES += gup_test >> +TEST_GEN_FILES += hmm-tests >> +TEST_GEN_FILES += hugetlb-madvise >> +TEST_GEN_FILES += hugepage-mmap >> +TEST_GEN_FILES += hugepage-mremap >> +TEST_GEN_FILES += hugepage-shm >> +TEST_GEN_FILES += hugepage-vmemmap >> +TEST_GEN_FILES += khugepaged >> +TEST_GEN_FILES += madv_populate >> +TEST_GEN_FILES += map_fixed_noreplace >> +TEST_GEN_FILES += map_hugetlb >> +TEST_GEN_FILES += map_populate >> +TEST_GEN_FILES += memfd_secret >> +TEST_GEN_FILES += migration >> +TEST_GEN_FILES += mkdirty >> +TEST_GEN_FILES += mlock-random-test >> +TEST_GEN_FILES += mlock2-tests >> +TEST_GEN_FILES += mrelease_test >> +TEST_GEN_FILES += mremap_dontunmap >> +TEST_GEN_FILES += mremap_test >> +TEST_GEN_FILES += on-fault-limit >> +TEST_GEN_FILES += thuge-gen >> +TEST_GEN_FILES += transhuge-stress >> +TEST_GEN_FILES += uffd-stress >> +TEST_GEN_FILES += uffd-unit-tests >> +TEST_GEN_FILES += soft-dirty >> +TEST_GEN_FILES += split_huge_page_test >> +TEST_GEN_FILES += ksm_tests >> +TEST_GEN_FILES += ksm_functional_tests >> +TEST_GEN_FILES += mdwe_test > > IIRC, we recently converted all to TEST_GEN_PROGS. See > > commit aef6fde75d8c6c1cad4a0e017a8d4cbee2143723 > Author: Peter Xu <peterx@redhat.com> > Date: Wed Apr 12 12:42:18 2023 -0400 > > selftests/mm: use TEST_GEN_PROGS where proper > TEST_GEN_PROGS and TEST_GEN_FILES are used randomly in the mm/Makefile to > specify programs that need to build. Logically all these binaries should > all fall into TEST_GEN_PROGS. > Replace those TEST_GEN_FILES with TEST_GEN_PROGS, so that we can reference > all the tests easily later. > > > Why is that change required, and how does it interact with > run_kselftest.sh? (Not clear from you patch description.) TEST_GEN_PROGS will compile and install the tests and will add them to the list of tests that run_kselftest.sh will run. TEST_GEN_FILES will compile and install the tests but will not add them to the test list. Note that run_vmtests.sh is added to TEST_PROGS, which means it ends up in the test list. (the lack of "_GEN" means it won't be compiled, but simply copied). So with this change at the kselftest level, there is a single test in its list; run_vmtests.sh. And all the other tests that were previously in that list are moved into run_vmtests.sh (if they weren't there already). I've only learnt all this today; All based on info in kselftest.rst. >
>> >> Which run_kselftest.sh are you referring to, the one in the parent directory? > > run_kselftest.sh is the uniform way of executing all the kselftests. mm seems to > be trying to be special as far as I can see. Certainly if you run the `install` > make target, kselftests will create a list of all the tests (including non-mm > tests if you have included them in the TARGETS variable) and copy that test list > and run_kselftest.sh to the install path along with all the test binaries. Then > the user can invoke any of the collections or specific tests in the collections > using that tool. It also wraps everything with tap output, runs tests with a > timeout, etc. > > See Documentation/dev-tools/kselftest.rst > Got it, thanks! >> >> How to invoke it to run these mm tests? >> >> (I never dared invoking something different than >> run_vmtests.sh ;) ) > > # single test: > $ sudo ./run_kselftest.sh -t mm:<test_name> > > or > > # all tests in collection: > $ sudo ./run_kselftest.sh -c mm > Ah, that makes sense. So I guess mm is then one "collection". >> >> [...] >> >>> >>> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> >>> --- >>> tools/testing/selftests/mm/Makefile | 79 ++++++++++++----------- >>> tools/testing/selftests/mm/run_vmtests.sh | 23 +++++++ >>> tools/testing/selftests/mm/settings | 2 +- >>> 3 files changed, 64 insertions(+), 40 deletions(-) >>> >>> diff --git a/tools/testing/selftests/mm/Makefile >>> b/tools/testing/selftests/mm/Makefile >>> index 66d7c07dc177..881ed96d96fd 100644 >>> --- a/tools/testing/selftests/mm/Makefile >>> +++ b/tools/testing/selftests/mm/Makefile >>> @@ -35,39 +35,39 @@ MAKEFLAGS += --no-builtin-rules >>> CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) >>> LDLIBS = -lrt -lpthread >>> -TEST_GEN_PROGS = cow >>> -TEST_GEN_PROGS += compaction_test >>> -TEST_GEN_PROGS += gup_longterm >>> -TEST_GEN_PROGS += gup_test >>> -TEST_GEN_PROGS += hmm-tests >>> -TEST_GEN_PROGS += hugetlb-madvise >>> -TEST_GEN_PROGS += hugepage-mmap >>> -TEST_GEN_PROGS += hugepage-mremap >>> -TEST_GEN_PROGS += hugepage-shm >>> -TEST_GEN_PROGS += hugepage-vmemmap >>> -TEST_GEN_PROGS += khugepaged >>> -TEST_GEN_PROGS += madv_populate >>> -TEST_GEN_PROGS += map_fixed_noreplace >>> -TEST_GEN_PROGS += map_hugetlb >>> -TEST_GEN_PROGS += map_populate >>> -TEST_GEN_PROGS += memfd_secret >>> -TEST_GEN_PROGS += migration >>> -TEST_GEN_PROGS += mkdirty >>> -TEST_GEN_PROGS += mlock-random-test >>> -TEST_GEN_PROGS += mlock2-tests >>> -TEST_GEN_PROGS += mrelease_test >>> -TEST_GEN_PROGS += mremap_dontunmap >>> -TEST_GEN_PROGS += mremap_test >>> -TEST_GEN_PROGS += on-fault-limit >>> -TEST_GEN_PROGS += thuge-gen >>> -TEST_GEN_PROGS += transhuge-stress >>> -TEST_GEN_PROGS += uffd-stress >>> -TEST_GEN_PROGS += uffd-unit-tests >>> -TEST_GEN_PROGS += soft-dirty >>> -TEST_GEN_PROGS += split_huge_page_test >>> -TEST_GEN_PROGS += ksm_tests >>> -TEST_GEN_PROGS += ksm_functional_tests >>> -TEST_GEN_PROGS += mdwe_test >>> +TEST_GEN_FILES = cow >>> +TEST_GEN_FILES += compaction_test >>> +TEST_GEN_FILES += gup_longterm >>> +TEST_GEN_FILES += gup_test >>> +TEST_GEN_FILES += hmm-tests >>> +TEST_GEN_FILES += hugetlb-madvise >>> +TEST_GEN_FILES += hugepage-mmap >>> +TEST_GEN_FILES += hugepage-mremap >>> +TEST_GEN_FILES += hugepage-shm >>> +TEST_GEN_FILES += hugepage-vmemmap >>> +TEST_GEN_FILES += khugepaged >>> +TEST_GEN_FILES += madv_populate >>> +TEST_GEN_FILES += map_fixed_noreplace >>> +TEST_GEN_FILES += map_hugetlb >>> +TEST_GEN_FILES += map_populate >>> +TEST_GEN_FILES += memfd_secret >>> +TEST_GEN_FILES += migration >>> +TEST_GEN_FILES += mkdirty >>> +TEST_GEN_FILES += mlock-random-test >>> +TEST_GEN_FILES += mlock2-tests >>> +TEST_GEN_FILES += mrelease_test >>> +TEST_GEN_FILES += mremap_dontunmap >>> +TEST_GEN_FILES += mremap_test >>> +TEST_GEN_FILES += on-fault-limit >>> +TEST_GEN_FILES += thuge-gen >>> +TEST_GEN_FILES += transhuge-stress >>> +TEST_GEN_FILES += uffd-stress >>> +TEST_GEN_FILES += uffd-unit-tests >>> +TEST_GEN_FILES += soft-dirty >>> +TEST_GEN_FILES += split_huge_page_test >>> +TEST_GEN_FILES += ksm_tests >>> +TEST_GEN_FILES += ksm_functional_tests >>> +TEST_GEN_FILES += mdwe_test >> >> IIRC, we recently converted all to TEST_GEN_PROGS. See >> >> commit aef6fde75d8c6c1cad4a0e017a8d4cbee2143723 >> Author: Peter Xu <peterx@redhat.com> >> Date: Wed Apr 12 12:42:18 2023 -0400 >> >> selftests/mm: use TEST_GEN_PROGS where proper >> TEST_GEN_PROGS and TEST_GEN_FILES are used randomly in the mm/Makefile to >> specify programs that need to build. Logically all these binaries should >> all fall into TEST_GEN_PROGS. >> Replace those TEST_GEN_FILES with TEST_GEN_PROGS, so that we can reference >> all the tests easily later. >> >> >> Why is that change required, and how does it interact with >> run_kselftest.sh? (Not clear from you patch description.) > > TEST_GEN_PROGS will compile and install the tests and will add them to the list > of tests that run_kselftest.sh will run. TEST_GEN_FILES will compile and install > the tests but will not add them to the test list. > > Note that run_vmtests.sh is added to TEST_PROGS, which means it ends up in the > test list. (the lack of "_GEN" means it won't be compiled, but simply copied). > > So with this change at the kselftest level, there is a single test in its list; > run_vmtests.sh. And all the other tests that were previously in that list are > moved into run_vmtests.sh (if they weren't there already). That sound good to me. (worth adding to the patch description) Let me CC Peter, so he's aware.
On 13/07/2023 16:25, David Hildenbrand wrote: >>> >>> Which run_kselftest.sh are you referring to, the one in the parent directory? >> >> run_kselftest.sh is the uniform way of executing all the kselftests. mm seems to >> be trying to be special as far as I can see. Certainly if you run the `install` >> make target, kselftests will create a list of all the tests (including non-mm >> tests if you have included them in the TARGETS variable) and copy that test list >> and run_kselftest.sh to the install path along with all the test binaries. Then >> the user can invoke any of the collections or specific tests in the collections >> using that tool. It also wraps everything with tap output, runs tests with a >> timeout, etc. >> >> See Documentation/dev-tools/kselftest.rst >> > > Got it, thanks! > >>> >>> How to invoke it to run these mm tests? >>> >>> (I never dared invoking something different than >>> run_vmtests.sh ;) ) >> >> # single test: >> $ sudo ./run_kselftest.sh -t mm:<test_name> >> >> or >> >> # all tests in collection: >> $ sudo ./run_kselftest.sh -c mm >> > > Ah, that makes sense. So I guess mm is then one "collection". yep, the collections are the directories under tools/testing/selftests with a few special exceptions. > >>> >>> [...] >>> >>>> >>>> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> >>>> --- >>>> tools/testing/selftests/mm/Makefile | 79 ++++++++++++----------- >>>> tools/testing/selftests/mm/run_vmtests.sh | 23 +++++++ >>>> tools/testing/selftests/mm/settings | 2 +- >>>> 3 files changed, 64 insertions(+), 40 deletions(-) >>>> >>>> diff --git a/tools/testing/selftests/mm/Makefile >>>> b/tools/testing/selftests/mm/Makefile >>>> index 66d7c07dc177..881ed96d96fd 100644 >>>> --- a/tools/testing/selftests/mm/Makefile >>>> +++ b/tools/testing/selftests/mm/Makefile >>>> @@ -35,39 +35,39 @@ MAKEFLAGS += --no-builtin-rules >>>> CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) >>>> LDLIBS = -lrt -lpthread >>>> -TEST_GEN_PROGS = cow >>>> -TEST_GEN_PROGS += compaction_test >>>> -TEST_GEN_PROGS += gup_longterm >>>> -TEST_GEN_PROGS += gup_test >>>> -TEST_GEN_PROGS += hmm-tests >>>> -TEST_GEN_PROGS += hugetlb-madvise >>>> -TEST_GEN_PROGS += hugepage-mmap >>>> -TEST_GEN_PROGS += hugepage-mremap >>>> -TEST_GEN_PROGS += hugepage-shm >>>> -TEST_GEN_PROGS += hugepage-vmemmap >>>> -TEST_GEN_PROGS += khugepaged >>>> -TEST_GEN_PROGS += madv_populate >>>> -TEST_GEN_PROGS += map_fixed_noreplace >>>> -TEST_GEN_PROGS += map_hugetlb >>>> -TEST_GEN_PROGS += map_populate >>>> -TEST_GEN_PROGS += memfd_secret >>>> -TEST_GEN_PROGS += migration >>>> -TEST_GEN_PROGS += mkdirty >>>> -TEST_GEN_PROGS += mlock-random-test >>>> -TEST_GEN_PROGS += mlock2-tests >>>> -TEST_GEN_PROGS += mrelease_test >>>> -TEST_GEN_PROGS += mremap_dontunmap >>>> -TEST_GEN_PROGS += mremap_test >>>> -TEST_GEN_PROGS += on-fault-limit >>>> -TEST_GEN_PROGS += thuge-gen >>>> -TEST_GEN_PROGS += transhuge-stress >>>> -TEST_GEN_PROGS += uffd-stress >>>> -TEST_GEN_PROGS += uffd-unit-tests >>>> -TEST_GEN_PROGS += soft-dirty >>>> -TEST_GEN_PROGS += split_huge_page_test >>>> -TEST_GEN_PROGS += ksm_tests >>>> -TEST_GEN_PROGS += ksm_functional_tests >>>> -TEST_GEN_PROGS += mdwe_test >>>> +TEST_GEN_FILES = cow >>>> +TEST_GEN_FILES += compaction_test >>>> +TEST_GEN_FILES += gup_longterm >>>> +TEST_GEN_FILES += gup_test >>>> +TEST_GEN_FILES += hmm-tests >>>> +TEST_GEN_FILES += hugetlb-madvise >>>> +TEST_GEN_FILES += hugepage-mmap >>>> +TEST_GEN_FILES += hugepage-mremap >>>> +TEST_GEN_FILES += hugepage-shm >>>> +TEST_GEN_FILES += hugepage-vmemmap >>>> +TEST_GEN_FILES += khugepaged >>>> +TEST_GEN_FILES += madv_populate >>>> +TEST_GEN_FILES += map_fixed_noreplace >>>> +TEST_GEN_FILES += map_hugetlb >>>> +TEST_GEN_FILES += map_populate >>>> +TEST_GEN_FILES += memfd_secret >>>> +TEST_GEN_FILES += migration >>>> +TEST_GEN_FILES += mkdirty >>>> +TEST_GEN_FILES += mlock-random-test >>>> +TEST_GEN_FILES += mlock2-tests >>>> +TEST_GEN_FILES += mrelease_test >>>> +TEST_GEN_FILES += mremap_dontunmap >>>> +TEST_GEN_FILES += mremap_test >>>> +TEST_GEN_FILES += on-fault-limit >>>> +TEST_GEN_FILES += thuge-gen >>>> +TEST_GEN_FILES += transhuge-stress >>>> +TEST_GEN_FILES += uffd-stress >>>> +TEST_GEN_FILES += uffd-unit-tests >>>> +TEST_GEN_FILES += soft-dirty >>>> +TEST_GEN_FILES += split_huge_page_test >>>> +TEST_GEN_FILES += ksm_tests >>>> +TEST_GEN_FILES += ksm_functional_tests >>>> +TEST_GEN_FILES += mdwe_test >>> >>> IIRC, we recently converted all to TEST_GEN_PROGS. See >>> >>> commit aef6fde75d8c6c1cad4a0e017a8d4cbee2143723 >>> Author: Peter Xu <peterx@redhat.com> >>> Date: Wed Apr 12 12:42:18 2023 -0400 >>> >>> selftests/mm: use TEST_GEN_PROGS where proper >>> TEST_GEN_PROGS and TEST_GEN_FILES are used randomly in the >>> mm/Makefile to >>> specify programs that need to build. Logically all these binaries should >>> all fall into TEST_GEN_PROGS. >>> Replace those TEST_GEN_FILES with TEST_GEN_PROGS, so that we can >>> reference >>> all the tests easily later. >>> >>> >>> Why is that change required, and how does it interact with >>> run_kselftest.sh? (Not clear from you patch description.) >> >> TEST_GEN_PROGS will compile and install the tests and will add them to the list >> of tests that run_kselftest.sh will run. TEST_GEN_FILES will compile and install >> the tests but will not add them to the test list. >> >> Note that run_vmtests.sh is added to TEST_PROGS, which means it ends up in the >> test list. (the lack of "_GEN" means it won't be compiled, but simply copied). >> >> So with this change at the kselftest level, there is a single test in its list; >> run_vmtests.sh. And all the other tests that were previously in that list are >> moved into run_vmtests.sh (if they weren't there already). > > That sound good to me. (worth adding to the patch description) > > Let me CC Peter, so he's aware. Thanks - would be good to hear his opinion!
On Thu, Jul 13, 2023 at 04:04:44PM +0100, Ryan Roberts wrote: > So with this change at the kselftest level, there is a single test in its list; > run_vmtests.sh. And all the other tests that were previously in that list are > moved into run_vmtests.sh (if they weren't there already). The results parsers I'm aware of like the LAVA one will DTRT with nested kselftests since that's required to pull see individual test cases run by a single binary so it's the common case to see at least one level of nesting.
On 13/07/2023 16:30, Mark Brown wrote: > On Thu, Jul 13, 2023 at 04:04:44PM +0100, Ryan Roberts wrote: > >> So with this change at the kselftest level, there is a single test in its list; >> run_vmtests.sh. And all the other tests that were previously in that list are >> moved into run_vmtests.sh (if they weren't there already). > > The results parsers I'm aware of like the LAVA one will DTRT with nested > kselftests since that's required to pull see individual test cases run > by a single binary so it's the common case to see at least one level of > nesting. That's good to hear. But bear in mind that run_vmtests.sh does not use TAP. So you end up with a single top-level test who's result is reported with run_kselftest.sh's TAP output. Then you have a second level (run_vmtests.sh) using custom reporting, then _some_ of the tests invoked use TAP so you sometimes have TAP at level 3. But those tests at level 2 that don't do their own TAP output probably won't be parsed by LAVA? Since you agreed to put this into the CI, I was going to call this part "your problem" ;-)
On Thu, Jul 13, 2023 at 04:36:18PM +0100, Ryan Roberts wrote: > On 13/07/2023 16:30, Mark Brown wrote: > > The results parsers I'm aware of like the LAVA one will DTRT with nested > > kselftests since that's required to pull see individual test cases run > > by a single binary so it's the common case to see at least one level of > > nesting. > That's good to hear. But bear in mind that run_vmtests.sh does not use TAP. So > you end up with a single top-level test who's result is reported with > run_kselftest.sh's TAP output. Then you have a second level (run_vmtests.sh) > using custom reporting, then _some_ of the tests invoked use TAP so you > sometimes have TAP at level 3. But those tests at level 2 that don't do their > own TAP output probably won't be parsed by LAVA? I think that should mostly mean that all the tests that don't individually produce KTAP output get ignored by parsers and those which do produce KTAP output will be seen as nesting one level up from where they are (ie, the individual cases will run directly from vmtest), though there's likely to be confusion about expected run numbers for things that actually pay attention to that. > Since you agreed to put this into the CI, I was going to call this part "your > problem" ;-) It'll run, the results are a different story. :P
On 13/07/2023 16:43, Mark Brown wrote: > On Thu, Jul 13, 2023 at 04:36:18PM +0100, Ryan Roberts wrote: >> On 13/07/2023 16:30, Mark Brown wrote: > >>> The results parsers I'm aware of like the LAVA one will DTRT with nested >>> kselftests since that's required to pull see individual test cases run >>> by a single binary so it's the common case to see at least one level of >>> nesting. > >> That's good to hear. But bear in mind that run_vmtests.sh does not use TAP. So >> you end up with a single top-level test who's result is reported with >> run_kselftest.sh's TAP output. Then you have a second level (run_vmtests.sh) >> using custom reporting, then _some_ of the tests invoked use TAP so you >> sometimes have TAP at level 3. But those tests at level 2 that don't do their >> own TAP output probably won't be parsed by LAVA? > > I think that should mostly mean that all the tests that don't > individually produce KTAP output get ignored by parsers and those which > do produce KTAP output will be seen as nesting one level up from where > they are (ie, the individual cases will run directly from vmtest), > though there's likely to be confusion about expected run numbers for > things that actually pay attention to that. I suspect it wouldn't be technically dififcult to add a --tap option to run_vmtests.sh, which would switch the output format to TAP. If people are amenable. > >> Since you agreed to put this into the CI, I was going to call this part "your >> problem" ;-) > > It'll run, the results are a different story. :P
diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 66d7c07dc177..881ed96d96fd 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -35,39 +35,39 @@ MAKEFLAGS += --no-builtin-rules CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) LDLIBS = -lrt -lpthread -TEST_GEN_PROGS = cow -TEST_GEN_PROGS += compaction_test -TEST_GEN_PROGS += gup_longterm -TEST_GEN_PROGS += gup_test -TEST_GEN_PROGS += hmm-tests -TEST_GEN_PROGS += hugetlb-madvise -TEST_GEN_PROGS += hugepage-mmap -TEST_GEN_PROGS += hugepage-mremap -TEST_GEN_PROGS += hugepage-shm -TEST_GEN_PROGS += hugepage-vmemmap -TEST_GEN_PROGS += khugepaged -TEST_GEN_PROGS += madv_populate -TEST_GEN_PROGS += map_fixed_noreplace -TEST_GEN_PROGS += map_hugetlb -TEST_GEN_PROGS += map_populate -TEST_GEN_PROGS += memfd_secret -TEST_GEN_PROGS += migration -TEST_GEN_PROGS += mkdirty -TEST_GEN_PROGS += mlock-random-test -TEST_GEN_PROGS += mlock2-tests -TEST_GEN_PROGS += mrelease_test -TEST_GEN_PROGS += mremap_dontunmap -TEST_GEN_PROGS += mremap_test -TEST_GEN_PROGS += on-fault-limit -TEST_GEN_PROGS += thuge-gen -TEST_GEN_PROGS += transhuge-stress -TEST_GEN_PROGS += uffd-stress -TEST_GEN_PROGS += uffd-unit-tests -TEST_GEN_PROGS += soft-dirty -TEST_GEN_PROGS += split_huge_page_test -TEST_GEN_PROGS += ksm_tests -TEST_GEN_PROGS += ksm_functional_tests -TEST_GEN_PROGS += mdwe_test +TEST_GEN_FILES = cow +TEST_GEN_FILES += compaction_test +TEST_GEN_FILES += gup_longterm +TEST_GEN_FILES += gup_test +TEST_GEN_FILES += hmm-tests +TEST_GEN_FILES += hugetlb-madvise +TEST_GEN_FILES += hugepage-mmap +TEST_GEN_FILES += hugepage-mremap +TEST_GEN_FILES += hugepage-shm +TEST_GEN_FILES += hugepage-vmemmap +TEST_GEN_FILES += khugepaged +TEST_GEN_FILES += madv_populate +TEST_GEN_FILES += map_fixed_noreplace +TEST_GEN_FILES += map_hugetlb +TEST_GEN_FILES += map_populate +TEST_GEN_FILES += memfd_secret +TEST_GEN_FILES += migration +TEST_GEN_FILES += mkdirty +TEST_GEN_FILES += mlock-random-test +TEST_GEN_FILES += mlock2-tests +TEST_GEN_FILES += mrelease_test +TEST_GEN_FILES += mremap_dontunmap +TEST_GEN_FILES += mremap_test +TEST_GEN_FILES += on-fault-limit +TEST_GEN_FILES += thuge-gen +TEST_GEN_FILES += transhuge-stress +TEST_GEN_FILES += uffd-stress +TEST_GEN_FILES += uffd-unit-tests +TEST_GEN_FILES += soft-dirty +TEST_GEN_FILES += split_huge_page_test +TEST_GEN_FILES += ksm_tests +TEST_GEN_FILES += ksm_functional_tests +TEST_GEN_FILES += mdwe_test ifeq ($(ARCH),x86_64) CAN_BUILD_I386 := $(shell ./../x86/check_cc.sh "$(CC)" ../x86/trivial_32bit_program.c -m32) @@ -83,24 +83,24 @@ CFLAGS += -no-pie endif ifeq ($(CAN_BUILD_I386),1) -TEST_GEN_PROGS += $(BINARIES_32) +TEST_GEN_FILES += $(BINARIES_32) endif ifeq ($(CAN_BUILD_X86_64),1) -TEST_GEN_PROGS += $(BINARIES_64) +TEST_GEN_FILES += $(BINARIES_64) endif else ifneq (,$(findstring $(ARCH),ppc64)) -TEST_GEN_PROGS += protection_keys +TEST_GEN_FILES += protection_keys endif endif ifneq (,$(filter $(ARCH),arm64 ia64 mips64 parisc64 ppc64 riscv64 s390x sparc64 x86_64)) -TEST_GEN_PROGS += va_high_addr_switch -TEST_GEN_PROGS += virtual_address_range -TEST_GEN_PROGS += write_to_hugetlbfs +TEST_GEN_FILES += va_high_addr_switch +TEST_GEN_FILES += virtual_address_range +TEST_GEN_FILES += write_to_hugetlbfs endif TEST_PROGS := run_vmtests.sh @@ -112,6 +112,7 @@ TEST_FILES += va_high_addr_switch.sh include ../lib.mk $(TEST_GEN_PROGS): vm_util.c +$(TEST_GEN_FILES): vm_util.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 3f26f6e15b2a..55fe1d309355 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -55,6 +55,17 @@ separated by spaces: test soft dirty page bit semantics - cow test copy-on-write semantics +- thp + test transparent huge pages +- migration + invoke move_pages(2) to exercise the migration entry code + paths in the kernel +- mkdirty + test handling of code that might set PTE/PMD dirty in + read-only VMAs +- mdwe + test prctl(PR_SET_MDWE, ...) + example: ./run_vmtests.sh -t "hmm mmap ksm" EOF exit 0 @@ -295,6 +306,18 @@ CATEGORY="soft_dirty" run_test ./soft-dirty # COW tests CATEGORY="cow" run_test ./cow +CATEGORY="thp" run_test ./khugepaged + +CATEGORY="thp" run_test ./transhuge-stress -d 20 + +CATEGORY="thp" run_test ./split_huge_page_test + +CATEGORY="migration" run_test ./migration + +CATEGORY="mkdirty" run_test ./mkdirty + +CATEGORY="mdwe" run_test ./mdwe_test + echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" exit $exitcode diff --git a/tools/testing/selftests/mm/settings b/tools/testing/selftests/mm/settings index ba4d85f74cd6..a953c96aa16e 100644 --- a/tools/testing/selftests/mm/settings +++ b/tools/testing/selftests/mm/settings @@ -1 +1 @@ -timeout=90 +timeout=180
It is very unclear to me how one is supposed to run all the mm selftests consistently and get clear results. Most of the test programs are launched by both run_vmtests.sh and run_kselftest.sh: hugepage-mmap hugepage-shm map_hugetlb hugepage-mremap hugepage-vmemmap hugetlb-madvise map_fixed_noreplace gup_test gup_longterm uffd-unit-tests uffd-stress compaction_test on-fault-limit map_populate mlock-random-test mlock2-tests mrelease_test mremap_test thuge-gen virtual_address_range va_high_addr_switch mremap_dontunmap hmm-tests madv_populate memfd_secret ksm_tests ksm_functional_tests soft-dirty cow However, of this set, when launched by run_vmtests.sh, some of the programs are invoked multiple times with different arguments. When invoked by run_kselftest.sh, they are invoked without arguments (and as a consequence, some fail immediately). Some test programs are only launched by run_vmtests.sh: test_vmalloc.sh And some test programs and only launched by run_kselftest.sh: khugepaged migration mkdirty transhuge-stress split_huge_page_test mdwe_test write_to_hugetlbfs Furthermore, run_vmtests.sh is invoked by run_kselftest.sh, so in this case all the test programs invoked by both scripts are run twice! Needless to say, this is a bit of a mess. In the absence of fully understanding the history here, it looks to me like the best solution is to launch ALL test programs from run_vmtests.sh, and ONLY invoke run_vmtests.sh from run_kselftest.sh. This way, we get full control over the parameters, each program is only invoked the intended number of times, and regardless of which script is used, the same tests get run in the same way. The only drawback is that if using run_kselftest.sh, it's top-level tap result reporting reports only a single test and it fails if any of the contained tests fail. I don't see this as a big deal though since we still see all the nested reporting from multiple layers. The other issue with this is that all of run_vmtests.sh must execute within a single kselftest timeout period, so let's increase that to something more suitable. Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> --- tools/testing/selftests/mm/Makefile | 79 ++++++++++++----------- tools/testing/selftests/mm/run_vmtests.sh | 23 +++++++ tools/testing/selftests/mm/settings | 2 +- 3 files changed, 64 insertions(+), 40 deletions(-)