[6/8] tools: make sure that test groups are described in the documentation

Commit Message

Darrick J. Wong Sept. 2, 2021, 11:52 p.m. UTC

From: Darrick J. Wong <[e-mail]>

Create a file to document the purpose of each test group that is
currently defined in fstests, and add a build script to check that every
group mentioned in the tests is also mentioned in the documentation.

Signed-off-by: Darrick J. Wong <[e-mail]>
---
 doc/group-names.txt    |  135 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/buildgrouplist |    1 
 tools/check-groups     |   35 ++++++++++++
 3 files changed, 171 insertions(+)
 create mode 100644 doc/group-names.txt
 create mode 100755 tools/check-groups

Comments

Amir Goldstein Sept. 3, 2021, 3:38 a.m. UTC | #1

> diff --git a/include/buildgrouplist b/include/buildgrouplist
> index d898efa3..489de965 100644
> --- a/include/buildgrouplist
> +++ b/include/buildgrouplist
> @@ -6,3 +6,4 @@
>  group.list:
>         @echo " [GROUP] $$PWD/$@"
>         $(Q)$(TOPDIR)/tools/mkgroupfile $@
> +       $(Q)$(TOPDIR)/tools/check-groups $(TOPDIR)/doc/group-names.txt $@

I would like to argue against checking groups post mkgroupfile
and for checking groups during mkgroupfile

> diff --git a/tools/check-groups b/tools/check-groups
> new file mode 100755
> index 00000000..0d193615
> --- /dev/null
> +++ b/tools/check-groups
> @@ -0,0 +1,35 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Oracle.  All Rights Reserved.
> +#
> +# Make sure that all groups listed in a group.list file are mentioned in the
> +# group description file.
> +
> +if [ -z "$1" ] || [ "$1" = "--help" ]; then
> +       echo "Usage: $0 path_to_group_names [group.list files...]"
> +       exit 1
> +fi
> +
> +groups_doc_file="$1"
> +shift
> +
> +get_group_list() {
> +       for file in "$@"; do
> +               while read testname groups; do
> +                       test -z "${testname}" && continue
> +                       test "${testname:0:1}" = "#" && continue
> +
> +                       echo "${groups}" | tr ' ' '\n'
> +               done < "${file}"
> +       done | sort | uniq
> +}
> +
> +ret=0
> +while read group; do
> +       if ! grep -q "^${group}[[:space:]]" "${groups_doc_file}"; then
> +               echo "${group}: group not mentioned in documentation." 1>&2

This message would have been more informative with the offending
test file.

Now after you crunched all the test files into group.list files and
all the group.list files into a unique group set, this is too late.
But this same check during generate_groupfile() would have
been trivial and would allow reporting the offending test.

While we are on the subject of generate_groupfile(), can you please
explain the rationale behind the method of extracting the test file
groups by executing the test with GENERATE_GROUPS=yes?
As opposed to just getting the list of groups on the stop from the file
using grep?

Thanks,
Amir.

Darrick J. Wong Sept. 4, 2021, 1:29 a.m. UTC | #2

On Fri, Sep 03, 2021 at 06:38:38AM +0300, Amir Goldstein wrote:
> > diff --git a/include/buildgrouplist b/include/buildgrouplist
> > index d898efa3..489de965 100644
> > --- a/include/buildgrouplist
> > +++ b/include/buildgrouplist
> > @@ -6,3 +6,4 @@
> >  group.list:
> >         @echo " [GROUP] $$PWD/$@"
> >         $(Q)$(TOPDIR)/tools/mkgroupfile $@
> > +       $(Q)$(TOPDIR)/tools/check-groups $(TOPDIR)/doc/group-names.txt $@
> 
> I would like to argue against checking groups post mkgroupfile
> and for checking groups during mkgroupfile

Done.

> > diff --git a/tools/check-groups b/tools/check-groups
> > new file mode 100755
> > index 00000000..0d193615
> > --- /dev/null
> > +++ b/tools/check-groups
> > @@ -0,0 +1,35 @@
> > +#!/bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2021 Oracle.  All Rights Reserved.
> > +#
> > +# Make sure that all groups listed in a group.list file are mentioned in the
> > +# group description file.
> > +
> > +if [ -z "$1" ] || [ "$1" = "--help" ]; then
> > +       echo "Usage: $0 path_to_group_names [group.list files...]"
> > +       exit 1
> > +fi
> > +
> > +groups_doc_file="$1"
> > +shift
> > +
> > +get_group_list() {
> > +       for file in "$@"; do
> > +               while read testname groups; do
> > +                       test -z "${testname}" && continue
> > +                       test "${testname:0:1}" = "#" && continue
> > +
> > +                       echo "${groups}" | tr ' ' '\n'
> > +               done < "${file}"
> > +       done | sort | uniq
> > +}
> > +
> > +ret=0
> > +while read group; do
> > +       if ! grep -q "^${group}[[:space:]]" "${groups_doc_file}"; then
> > +               echo "${group}: group not mentioned in documentation." 1>&2
> 
> This message would have been more informative with the offending
> test file.

Hm.  This becomes much easier if I make the _begin_fstest helper do the
checking of the group names.

> Now after you crunched all the test files into group.list files and
> all the group.list files into a unique group set, this is too late.
> But this same check during generate_groupfile() would have
> been trivial and would allow reporting the offending test.
> 
> While we are on the subject of generate_groupfile(), can you please
> explain the rationale behind the method of extracting the test file
> groups by executing the test with GENERATE_GROUPS=yes?
> As opposed to just getting the list of groups on the stop from the file
> using grep?

Well... now that you point that out, it's so that we can put in custom
logic like checking group names. ;)

--D

> 
> Thanks,
> Amir.

Patch

diff --git a/doc/group-names.txt b/doc/group-names.txt
new file mode 100644
index 00000000..e8e3477e
--- /dev/null
+++ b/doc/group-names.txt
@@ -0,0 +1,135 @@ 
+======================= =======================================================
+Group Name:		Description:
+======================= =======================================================
+all			All known tests, automatically generated by ./check at
+			runtime
+auto			Tests that should be run automatically.  These should
+			not require more than ~5 minutes to run.
+quick			Tests that should run in under 30 seconds.
+deprecated		Old tests that should not be run.
+
+acl			Access Control Lists
+admin			xfs_admin functionality
+aio			general libaio async io tests
+atime			file access time
+attr			extended attributes
+attr2			xfs v2 extended aributes
+balance			btrfs tree rebalance
+bigtime			timestamps beyond the year 2038
+blockdev		block device functionality
+broken			broken tests
+cap			Linux capabilities
+casefold		directory name casefolding
+ci			ASCII case-insensitive directory name lookups
+clone			FICLONE/FICLONERANGE ioctls
+clone_stress		stress testing FICLONE/FICLONERANGE
+collapse		fallocate FALLOC_FL_COLLAPSE_RANGE
+compress		file compression
+convert			btrfs ext[34] conversion tool
+copy			xfs_copy functionality
+copy_range		copy_file_range syscall
+copyup			overlayfs copyup
+dangerous		dangerous test that can crash the system
+dangerous_bothrepair	fuzzers to evaluate xfs_scrub + xfs_repair repair
+dangerous_fuzzers	fuzzers that can crash your computer
+dangerous_norepair	fuzzers to evaluate kernel metadata verifiers
+dangerous_online_repair	fuzzers to evaluate xfs_scrub online repair
+dangerous_repair	fuzzers to evaluate xfs_repair offline repair
+dangerous_scrub		fuzzers to evaluate xfs_scrub checking
+data			data loss checkers
+dax			direct access mode for persistent memory files
+db			xfs_db functional tests
+dedupe			FIEDEDUPERANGE ioctl
+defrag			filesystem defragmenters
+dir			directory test functions
+dump			dump and restore utilities
+eio			IO error reporting
+encrypt			encrypted file contents
+enospc			ENOSPC error reporting
+exportfs		file handles
+filestreams		XFS filestreams allocator
+freeze			filesystem freeze tests
+fsck			general fsck tests
+fsmap			FS_IOC_GETFSMAP ioctl
+fsr			XFS free space reorganizer
+fuzzers			filesystem fuzz tests
+growfs			increasing the size of a filesystem
+hardlink		hardlinks
+health			XFS health reporting
+idmapped		idmapped mount functionality
+inobtcount		XFS inode btree count tests
+insert			fallocate FALLOC_FL_INSERT_RANGE
+ioctl			general ioctl tests
+io_uring		general io_uring async io tests
+label			filesystem labelling
+limit			resource limits
+locks			file locking
+log			metadata logging
+logprint		xfs_logprint functional tests
+long_rw			long-soak read write IO path exercisers
+metacopy		overlayfs metadata-only copy-up
+metadata		filesystem metadata update exercisers
+metadump		xfs_metadump/xfs_mdrestore functionality
+mkfs			filesystem formatting tools
+mount			mount option and functionality checks
+nested			nested overlayfs instances
+nfs4_acl		NFSv4 access control lists
+nonsamefs		overlayfs layers on different filesystems
+online_repair		online repair functionality tests
+other			dumping ground, do not add more tests to this group
+pattern			specific IO pattern tests
+perms			access control and permission checking
+pipe			pipe functionality
+pnfs			PNFS
+posix			POSIX behavior conformance
+prealloc		fallocate for preallocating unwritten space
+preallocrw		fallocate, then read and write
+punch			fallocate FALLOC_FL_PUNCH_HOLE
+qgroup			btrfs qgroup feature
+quota			filesystem usage quotas
+raid			btrfs RAID
+realtime		XFS realtime volumes
+recoveryloop		crash recovery loops
+redirect		overlayfs redirect_dir feature
+remote			dump and restore with a remote tape
+remount			remounting filesystems
+rename			rename system call
+repair			xfs_repair functional tests
+replace			btrfs device replace
+replay			dm-logwrites replays
+resize			resize2fs functionality tests
+richacl			rich ACL feature
+rmap			XFS reverse mapping exercisers
+rotate			overlayfs upper layer rotate tests from the unionmount
+			test suite
+rw			read/write IO tests
+samefs			overlayfs when all layers are on the same fs
+scrub			filesystem metadata scrubbers
+seed			btrfs seeded filesystems
+seek			llseek functionality
+send			btrfs send/receive
+shrinkfs		decreasing the size of a filesystem
+shutdown		FS_IOC_SHUTDOWN ioctl
+snapshot		btrfs snapshots
+soak			long running soak tests of any kind
+spaceman		xfs_spaceman functional tests
+splice			splice system call
+stress			fsstress filesystem exerciser
+subvol			btrfs subvolumes
+swap			swap files
+symlink			symbolic links
+tape			dump and restore with a tape
+thin			thin provisioning
+trim			FITRIM ioctl
+udf			UDF functionality tests
+union			tests from the unionmount test suite
+unlink			O_TMPFILE unlinked files
+unshare			fallocate FALLOC_FL_UNSHARE_RANGE
+v2log			XFS v2 log format tests
+verity			fsverity
+volume			btrfs volume management
+whiteout		overlayfs whiteout functionality
+xino			overlayfs xino feature
+zero			fallocate FALLOC_FL_ZERO_RANGE
+zone			zoned (SMR) device support
+======================= =======================================================
diff --git a/include/buildgrouplist b/include/buildgrouplist
index d898efa3..489de965 100644
--- a/include/buildgrouplist
+++ b/include/buildgrouplist
@@ -6,3 +6,4 @@ 
 group.list:
 	@echo " [GROUP] $$PWD/$@"
 	$(Q)$(TOPDIR)/tools/mkgroupfile $@
+	$(Q)$(TOPDIR)/tools/check-groups $(TOPDIR)/doc/group-names.txt $@
diff --git a/tools/check-groups b/tools/check-groups
new file mode 100755
index 00000000..0d193615
--- /dev/null
+++ b/tools/check-groups
@@ -0,0 +1,35 @@ 
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Oracle.  All Rights Reserved.
+#
+# Make sure that all groups listed in a group.list file are mentioned in the
+# group description file.
+
+if [ -z "$1" ] || [ "$1" = "--help" ]; then
+	echo "Usage: $0 path_to_group_names [group.list files...]"
+	exit 1
+fi
+
+groups_doc_file="$1"
+shift
+
+get_group_list() {
+	for file in "$@"; do
+		while read testname groups; do
+			test -z "${testname}" && continue
+			test "${testname:0:1}" = "#" && continue
+
+			echo "${groups}" | tr ' ' '\n'
+		done < "${file}"
+	done | sort | uniq
+}
+
+ret=0
+while read group; do
+	if ! grep -q "^${group}[[:space:]]" "${groups_doc_file}"; then
+		echo "${group}: group not mentioned in documentation." 1>&2
+		ret=1
+	fi
+done < <(get_group_list "$@")
+
+exit "${ret}"