| Message ID | 20220214100119.6795-1-kch@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | loop: cleanup and few improvements | expand |
On 2/14/22 3:01 AM, Chaitanya Kulkarni wrote: > Hi, > > This has few improvment and cleanups such as using sysfs_emit() for the > sysfs dev attributes and removing variables that are only used once and > a cleanup with fixing declaration. > > Below is the test log where 10 loop devices created, each device is > linked to it's own file in ./loopX, formatted with xfs and mounted on > /mnt/loopX. For each device it reads the offset, sizelimit, autoclear, > partscan, and dio attr from sysfs using cat command, then it runs fio > verify job on it. > > In summary write-verify fio job seems to work fine :- > write-and-verify: (groupid=0, jobs=1): err= 0: pid=3495: Mon Feb 14 00:43:19 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=3960: Mon Feb 14 00:45:17 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=4321: Mon Feb 14 00:47:15 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=4369: Mon Feb 14 00:49:20 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=4443: Mon Feb 14 00:51:25 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=4536: Mon Feb 14 00:53:25 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=4770: Mon Feb 14 00:55:24 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=4933: Mon Feb 14 00:57:31 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=5008: Mon Feb 14 00:59:35 2022 > write-and-verify: (groupid=0, jobs=1): err= 0: pid=5048: Mon Feb 14 01:01:58 2022 > > Below is detailed test log. > > Chaitanya Kulkarni (8): > loop: use sysfs_emit() in the sysfs offset show > loop: use sysfs_emit() in the sysfs sizelimit show > loop: use sysfs_emit() in the sysfs autoclear show > loop: use sysfs_emit() in the sysfs partscan show > loop: use sysfs_emit() in the sysfs dio show > loop: remove extra variable in lo_fallocate() > loop: remove extra variable in lo_req_flush > loop: allow user to set the queue depth Please collapse patches 1..5.
Hi, This has few improvment and cleanups such as using sysfs_emit() for the sysfs dev attributes and removing variables that are only used once and a cleanup with fixing declaration. Below is the test log where 10 loop devices created, each device is linked to it's own file in ./loopX, formatted with xfs and mounted on /mnt/loopX. For each device it reads the offset, sizelimit, autoclear, partscan, and dio attr from sysfs using cat command, then it runs fio verify job on it. In summary write-verify fio job seems to work fine :- write-and-verify: (groupid=0, jobs=1): err= 0: pid=3495: Mon Feb 14 00:43:19 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=3960: Mon Feb 14 00:45:17 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=4321: Mon Feb 14 00:47:15 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=4369: Mon Feb 14 00:49:20 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=4443: Mon Feb 14 00:51:25 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=4536: Mon Feb 14 00:53:25 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=4770: Mon Feb 14 00:55:24 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=4933: Mon Feb 14 00:57:31 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=5008: Mon Feb 14 00:59:35 2022 write-and-verify: (groupid=0, jobs=1): err= 0: pid=5048: Mon Feb 14 01:01:58 2022 Below is detailed test log. Chaitanya Kulkarni (8): loop: use sysfs_emit() in the sysfs offset show loop: use sysfs_emit() in the sysfs sizelimit show loop: use sysfs_emit() in the sysfs autoclear show loop: use sysfs_emit() in the sysfs partscan show loop: use sysfs_emit() in the sysfs dio show loop: remove extra variable in lo_fallocate() loop: remove extra variable in lo_req_flush loop: allow user to set the queue depth drivers/block/loop.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) # git am --skip Applying: loop: use sysfs_emit() in the sysfs offset show Applying: loop: use sysfs_emit() in the sysfs sizelimit show Applying: loop: use sysfs_emit() in the sysfs autoclear show Applying: loop: use sysfs_emit() in the sysfs partscan show Applying: loop: use sysfs_emit() in the sysfs dio show Applying: loop: remove extra variable in lo_fallocate() Applying: loop: remove extra variable in lo_req_flush Applying: loop: allow user to set the queue depth root@dev linux-block (for-next) # ./compile_loop.sh 10 + FILE=./loop + LOOP_MNT=/mnt/loop + NN=10 + unload_loop ++ shuf -i 1-10 -n 10 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop5 umount: /mnt/loop5: no mount point specified. + rm -fr ./loop5 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop8 umount: /mnt/loop8: no mount point specified. + rm -fr ./loop8 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop1 umount: /mnt/loop1: no mount point specified. + rm -fr ./loop1 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop4 umount: /mnt/loop4: no mount point specified. + rm -fr ./loop4 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop2 umount: /mnt/loop2: no mount point specified. + rm -fr ./loop2 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop3 umount: /mnt/loop3: no mount point specified. + rm -fr ./loop3 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop9 umount: /mnt/loop9: no mount point specified. + rm -fr ./loop9 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop10 umount: /mnt/loop10: no mount point specified. + rm -fr ./loop10 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop6 umount: /mnt/loop6: no mount point specified. + rm -fr ./loop6 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop7 umount: /mnt/loop7: no mount point specified. + rm -fr ./loop7 + losetup -D + sleep 3 + rmmod loop rmmod: ERROR: Module loop is not currently loaded + modprobe -r loop + lsmod + grep loop ++ shuf -i 1-10 -n 10 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop9 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop3 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop10 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop7 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop5 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop4 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop8 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop2 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop6 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop1 + rm -fr '/mnt/loop*' + compile_loop ++ nproc + make -j 48 M=drivers/block modules CC [M] drivers/block/loop.o MODPOST drivers/block/Module.symvers CC [M] drivers/block/brd.mod.o CC [M] drivers/block/loop.mod.o CC [M] drivers/block/null_blk/null_blk.mod.o CC [M] drivers/block/pktcdvd.mod.o CC [M] drivers/block/rbd.mod.o CC [M] drivers/block/sx8.mod.o CC [M] drivers/block/virtio_blk.mod.o CC [M] drivers/block/xen-blkfront.mod.o CC [M] drivers/block/zram/zram.mod.o LD [M] drivers/block/rbd.ko LD [M] drivers/block/xen-blkfront.ko LD [M] drivers/block/brd.ko LD [M] drivers/block/null_blk/null_blk.ko LD [M] drivers/block/zram/zram.ko LD [M] drivers/block/loop.ko LD [M] drivers/block/pktcdvd.ko LD [M] drivers/block/sx8.ko LD [M] drivers/block/virtio_blk.ko + HOST=drivers/block/ ++ uname -r + HOST_DEST=/lib/modules/5.17.0-rc3blk+/kernel/drivers/block + cp drivers/block//loop.ko /lib/modules/5.17.0-rc3blk+/kernel/drivers/block/ + load_loop + insmod drivers/block/loop.ko max_loop=11 hw_queue_depth=32 ++ shuf -i 1-10 -n 10 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop1 + truncate -s 2048M ./loop1 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop1 ./loop1 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 + sleep 1 + mkfs.xfs -f /dev/loop1 meta-data=/dev/loop1 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop1 /mnt/loop1 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop1/loop/offset : ' cat /sys/block/loop1/loop/offset : + cat /sys/block/loop1/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop1/loop/sizelimit : ' cat /sys/block/loop1/loop/sizelimit : + cat /sys/block/loop1/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop1/loop/autoclear : ' cat /sys/block/loop1/loop/autoclear : + cat /sys/block/loop1/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop1/loop/partscan : ' cat /sys/block/loop1/loop/partscan : + cat /sys/block/loop1/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop1/loop/dio : ' cat /sys/block/loop1/loop/dio : + cat /sys/block/loop1/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop4 + truncate -s 2048M ./loop4 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop4 ./loop4 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 + sleep 1 + mkfs.xfs -f /dev/loop4 meta-data=/dev/loop4 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop4 /mnt/loop4 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop4/loop/offset : ' cat /sys/block/loop4/loop/offset : + cat /sys/block/loop4/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop4/loop/sizelimit : ' cat /sys/block/loop4/loop/sizelimit : + cat /sys/block/loop4/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop4/loop/autoclear : ' cat /sys/block/loop4/loop/autoclear : + cat /sys/block/loop4/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop4/loop/partscan : ' cat /sys/block/loop4/loop/partscan : + cat /sys/block/loop4/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop4/loop/dio : ' cat /sys/block/loop4/loop/dio : + cat /sys/block/loop4/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop9 + truncate -s 2048M ./loop9 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop9 ./loop9 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 + sleep 1 + mkfs.xfs -f /dev/loop9 meta-data=/dev/loop9 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop9 /mnt/loop9 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop9/loop/offset : ' cat /sys/block/loop9/loop/offset : + cat /sys/block/loop9/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop9/loop/sizelimit : ' cat /sys/block/loop9/loop/sizelimit : + cat /sys/block/loop9/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop9/loop/autoclear : ' cat /sys/block/loop9/loop/autoclear : + cat /sys/block/loop9/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop9/loop/partscan : ' cat /sys/block/loop9/loop/partscan : + cat /sys/block/loop9/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop9/loop/dio : ' cat /sys/block/loop9/loop/dio : + cat /sys/block/loop9/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop10 + truncate -s 2048M ./loop10 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop10 ./loop10 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop10 meta-data=/dev/loop10 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop10 /mnt/loop10 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop10/loop/offset : ' cat /sys/block/loop10/loop/offset : + cat /sys/block/loop10/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop10/loop/sizelimit : ' cat /sys/block/loop10/loop/sizelimit : + cat /sys/block/loop10/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop10/loop/autoclear : ' cat /sys/block/loop10/loop/autoclear : + cat /sys/block/loop10/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop10/loop/partscan : ' cat /sys/block/loop10/loop/partscan : + cat /sys/block/loop10/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop10/loop/dio : ' cat /sys/block/loop10/loop/dio : + cat /sys/block/loop10/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop7 + truncate -s 2048M ./loop7 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop7 ./loop7 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop7 0 0 0 0 /mnt/data/linux-block/loop7 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop7 meta-data=/dev/loop7 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop7 /mnt/loop7 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop7/loop/offset : ' cat /sys/block/loop7/loop/offset : + cat /sys/block/loop7/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop7/loop/sizelimit : ' cat /sys/block/loop7/loop/sizelimit : + cat /sys/block/loop7/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop7/loop/autoclear : ' cat /sys/block/loop7/loop/autoclear : + cat /sys/block/loop7/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop7/loop/partscan : ' cat /sys/block/loop7/loop/partscan : + cat /sys/block/loop7/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop7/loop/dio : ' cat /sys/block/loop7/loop/dio : + cat /sys/block/loop7/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop2 + truncate -s 2048M ./loop2 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop2 ./loop2 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop2 0 0 0 0 /mnt/data/linux-block/loop2 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop7 0 0 0 0 /mnt/data/linux-block/loop7 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop2 meta-data=/dev/loop2 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop2 /mnt/loop2 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop2/loop/offset : ' cat /sys/block/loop2/loop/offset : + cat /sys/block/loop2/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop2/loop/sizelimit : ' cat /sys/block/loop2/loop/sizelimit : + cat /sys/block/loop2/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop2/loop/autoclear : ' cat /sys/block/loop2/loop/autoclear : + cat /sys/block/loop2/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop2/loop/partscan : ' cat /sys/block/loop2/loop/partscan : + cat /sys/block/loop2/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop2/loop/dio : ' cat /sys/block/loop2/loop/dio : + cat /sys/block/loop2/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop3 + truncate -s 2048M ./loop3 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop3 ./loop3 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop2 0 0 0 0 /mnt/data/linux-block/loop2 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop7 0 0 0 0 /mnt/data/linux-block/loop7 1 512 /dev/loop3 0 0 0 0 /mnt/data/linux-block/loop3 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop3 meta-data=/dev/loop3 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop3 /mnt/loop3 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop3/loop/offset : ' cat /sys/block/loop3/loop/offset : + cat /sys/block/loop3/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop3/loop/sizelimit : ' cat /sys/block/loop3/loop/sizelimit : + cat /sys/block/loop3/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop3/loop/autoclear : ' cat /sys/block/loop3/loop/autoclear : + cat /sys/block/loop3/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop3/loop/partscan : ' cat /sys/block/loop3/loop/partscan : + cat /sys/block/loop3/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop3/loop/dio : ' cat /sys/block/loop3/loop/dio : + cat /sys/block/loop3/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop5 + truncate -s 2048M ./loop5 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop5 ./loop5 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop2 0 0 0 0 /mnt/data/linux-block/loop2 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop7 0 0 0 0 /mnt/data/linux-block/loop7 1 512 /dev/loop5 0 0 0 0 /mnt/data/linux-block/loop5 1 512 /dev/loop3 0 0 0 0 /mnt/data/linux-block/loop3 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop5 meta-data=/dev/loop5 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop5 /mnt/loop5 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop5/loop/offset : ' cat /sys/block/loop5/loop/offset : + cat /sys/block/loop5/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop5/loop/sizelimit : ' cat /sys/block/loop5/loop/sizelimit : + cat /sys/block/loop5/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop5/loop/autoclear : ' cat /sys/block/loop5/loop/autoclear : + cat /sys/block/loop5/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop5/loop/partscan : ' cat /sys/block/loop5/loop/partscan : + cat /sys/block/loop5/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop5/loop/dio : ' cat /sys/block/loop5/loop/dio : + cat /sys/block/loop5/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop8 + truncate -s 2048M ./loop8 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop8 ./loop8 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop8 0 0 0 0 /mnt/data/linux-block/loop8 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop2 0 0 0 0 /mnt/data/linux-block/loop2 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop7 0 0 0 0 /mnt/data/linux-block/loop7 1 512 /dev/loop5 0 0 0 0 /mnt/data/linux-block/loop5 1 512 /dev/loop3 0 0 0 0 /mnt/data/linux-block/loop3 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop8 meta-data=/dev/loop8 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop8 /mnt/loop8 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop8/loop/offset : ' cat /sys/block/loop8/loop/offset : + cat /sys/block/loop8/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop8/loop/sizelimit : ' cat /sys/block/loop8/loop/sizelimit : + cat /sys/block/loop8/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop8/loop/autoclear : ' cat /sys/block/loop8/loop/autoclear : + cat /sys/block/loop8/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop8/loop/partscan : ' cat /sys/block/loop8/loop/partscan : + cat /sys/block/loop8/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop8/loop/dio : ' cat /sys/block/loop8/loop/dio : + cat /sys/block/loop8/loop/dio 1 + for i in `shuf -i 1-$NN -n $NN` + mkdir -p /mnt/loop6 + truncate -s 2048M ./loop6 + /mnt/data/util-linux/losetup --direct-io=on /dev/loop6 ./loop6 + /mnt/data/util-linux/losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /mnt/data/linux-block/loop1 1 512 /dev/loop8 0 0 0 0 /mnt/data/linux-block/loop8 1 512 /dev/loop6 0 0 0 0 /mnt/data/linux-block/loop6 1 512 /dev/loop4 0 0 0 0 /mnt/data/linux-block/loop4 1 512 /dev/loop2 0 0 0 0 /mnt/data/linux-block/loop2 1 512 /dev/loop9 0 0 0 0 /mnt/data/linux-block/loop9 1 512 /dev/loop7 0 0 0 0 /mnt/data/linux-block/loop7 1 512 /dev/loop5 0 0 0 0 /mnt/data/linux-block/loop5 1 512 /dev/loop3 0 0 0 0 /mnt/data/linux-block/loop3 1 512 /dev/loop10 0 0 0 0 /mnt/data/linux-block/loop10 1 512 + sleep 1 + mkfs.xfs -f /dev/loop6 meta-data=/dev/loop6 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. + mount /dev/loop6 /mnt/loop6 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop6/loop/offset : ' cat /sys/block/loop6/loop/offset : + cat /sys/block/loop6/loop/offset 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop6/loop/sizelimit : ' cat /sys/block/loop6/loop/sizelimit : + cat /sys/block/loop6/loop/sizelimit 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop6/loop/autoclear : ' cat /sys/block/loop6/loop/autoclear : + cat /sys/block/loop6/loop/autoclear 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop6/loop/partscan : ' cat /sys/block/loop6/loop/partscan : + cat /sys/block/loop6/loop/partscan 0 + for attr in offset sizelimit autoclear partscan dio + echo -n 'cat /sys/block/loop6/loop/dio : ' cat /sys/block/loop6/loop/dio : + cat /sys/block/loop6/loop/dio 1 + mount + grep loop /dev/loop1 on /mnt/loop1 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop4 on /mnt/loop4 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop9 on /mnt/loop9 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop10 on /mnt/loop10 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop7 on /mnt/loop7 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop2 on /mnt/loop2 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop3 on /mnt/loop3 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop5 on /mnt/loop5 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop8 on /mnt/loop8 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/loop6 on /mnt/loop6 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) + dmesg -c [ 0.000000] Linux version 5.17.0-rc3blk+ (root@dev) (gcc (GCC) 11.0.1 20210324 (Red Hat 11.0.1-0), GNU ld version 2.35.1-41.fc34) #52 SMP PREEMPT Mon Feb 14 00:35:51 PST 2022 [ 0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.17.0-rc3blk+ root=UUID=e5f9bccb-cc5d-4577-8f74-fddb710fae7f ro rootflags=subvol=root rhgb quiet console=ttyS0,115200 kgdboc=ttyS0,115200 nokaslr [ 88.725708] loop: loading out-of-tree module taints kernel. [ 88.795243] loop: module loaded [ 88.886469] loop1: detected capacity change from 0 to 4194304 [ 90.099519] XFS (loop1): Mounting V5 Filesystem [ 90.137943] XFS (loop1): Ending clean mount [ 90.142241] xfs filesystem being mounted at /mnt/loop1 supports timestamps until 2038 (0x7fffffff) [ 90.248857] loop4: detected capacity change from 0 to 4194304 [ 91.445059] XFS (loop4): Mounting V5 Filesystem [ 91.461389] XFS (loop4): Ending clean mount [ 91.466664] xfs filesystem being mounted at /mnt/loop4 supports timestamps until 2038 (0x7fffffff) [ 91.567786] loop9: detected capacity change from 0 to 4194304 [ 92.762834] XFS (loop9): Mounting V5 Filesystem [ 92.780240] XFS (loop9): Ending clean mount [ 92.784748] xfs filesystem being mounted at /mnt/loop9 supports timestamps until 2038 (0x7fffffff) [ 92.883355] loop10: detected capacity change from 0 to 4194304 [ 94.077276] XFS (loop10): Mounting V5 Filesystem [ 94.095212] XFS (loop10): Ending clean mount [ 94.101121] xfs filesystem being mounted at /mnt/loop10 supports timestamps until 2038 (0x7fffffff) [ 94.200828] loop7: detected capacity change from 0 to 4194304 [ 95.387052] XFS (loop7): Mounting V5 Filesystem [ 95.402344] XFS (loop7): Ending clean mount [ 95.408219] xfs filesystem being mounted at /mnt/loop7 supports timestamps until 2038 (0x7fffffff) [ 95.505530] loop2: detected capacity change from 0 to 4194304 [ 96.696228] XFS (loop2): Mounting V5 Filesystem [ 96.712500] XFS (loop2): Ending clean mount [ 96.717802] xfs filesystem being mounted at /mnt/loop2 supports timestamps until 2038 (0x7fffffff) [ 96.815870] loop3: detected capacity change from 0 to 4194304 [ 98.006189] XFS (loop3): Mounting V5 Filesystem [ 98.022896] XFS (loop3): Ending clean mount [ 98.028990] xfs filesystem being mounted at /mnt/loop3 supports timestamps until 2038 (0x7fffffff) [ 98.128314] loop5: detected capacity change from 0 to 4194304 [ 99.317471] XFS (loop5): Mounting V5 Filesystem [ 99.334259] XFS (loop5): Ending clean mount [ 99.340014] xfs filesystem being mounted at /mnt/loop5 supports timestamps until 2038 (0x7fffffff) [ 99.437944] loop8: detected capacity change from 0 to 4194304 [ 100.640549] XFS (loop8): Mounting V5 Filesystem [ 100.658762] XFS (loop8): Ending clean mount [ 100.663623] xfs filesystem being mounted at /mnt/loop8 supports timestamps until 2038 (0x7fffffff) [ 100.768163] loop6: detected capacity change from 0 to 4194304 [ 101.964471] XFS (loop6): Mounting V5 Filesystem [ 101.985869] XFS (loop6): Ending clean mount [ 101.991076] xfs filesystem being mounted at /mnt/loop6 supports timestamps until 2038 (0x7fffffff) + df -h /mnt/loop1 /mnt/loop10 /mnt/loop2 /mnt/loop3 /mnt/loop4 /mnt/loop5 /mnt/loop6 /mnt/loop7 /mnt/loop8 /mnt/loop9 Filesystem Size Used Avail Use% Mounted on /dev/loop1 2.0G 47M 2.0G 3% /mnt/loop1 /dev/loop10 2.0G 47M 2.0G 3% /mnt/loop10 /dev/loop2 2.0G 47M 2.0G 3% /mnt/loop2 /dev/loop3 2.0G 47M 2.0G 3% /mnt/loop3 /dev/loop4 2.0G 47M 2.0G 3% /mnt/loop4 /dev/loop5 2.0G 47M 2.0G 3% /mnt/loop5 /dev/loop6 2.0G 47M 2.0G 3% /mnt/loop6 /dev/loop7 2.0G 47M 2.0G 3% /mnt/loop7 /dev/loop8 2.0G 47M 2.0G 3% /mnt/loop8 /dev/loop9 2.0G 47M 2.0G 3% /mnt/loop9 ++ shuf -i 1-10 -n 10 + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop5/testfile + fio fio/verify.fio --filename=/mnt/loop5/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.7%][r=18.4MiB/s][r=4722 IOPS][eta 00m:30s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=3495: Mon Feb 14 00:43:19 2022 read: IOPS=4612, BW=18.0MiB/s (18.9MB/s)(647MiB/35913msec) slat (usec): min=7, max=2920, avg=13.50, stdev=23.99 clat (usec): min=269, max=86467, avg=3453.04, stdev=1221.75 lat (usec): min=321, max=86484, avg=3466.65, stdev=1224.18 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2474], 10.00th=[ 2540], 20.00th=[ 2671], | 30.00th=[ 2802], 40.00th=[ 3097], 50.00th=[ 3490], 60.00th=[ 3752], | 70.00th=[ 3851], 80.00th=[ 3982], 90.00th=[ 4228], 95.00th=[ 4424], | 99.00th=[ 6587], 99.50th=[ 7242], 99.90th=[ 9765], 99.95th=[11469], | 99.99th=[46924] write: IOPS=2671, BW=10.4MiB/s (10.9MB/s)(1024MiB/98116msec); 0 zone resets slat (usec): min=9, max=136085, avg=32.59, stdev=613.45 clat (usec): min=255, max=344426, avg=5954.50, stdev=13130.50 lat (usec): min=490, max=344452, avg=5987.24, stdev=13189.93 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 11], | 99.00th=[ 36], 99.50th=[ 93], 99.90th=[ 228], 99.95th=[ 251], | 99.99th=[ 292] bw ( KiB/s): min= 240, max=19040, per=99.67%, avg=10652.53, stdev=4662.38, samples=197 iops : min= 60, max= 4760, avg=2663.01, stdev=1165.58, samples=197 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=63.60%, 10=33.10%, 20=1.91%, 50=0.87% lat (msec) : 100=0.20%, 250=0.25%, 500=0.03% cpu : usr=1.89%, sys=8.56%, ctx=422844, majf=0, minf=3901 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165662,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (18.9MB/s), 18.0MiB/s-18.0MiB/s (18.9MB/s-18.9MB/s), io=647MiB (679MB), run=35913-35913msec WRITE: bw=10.4MiB/s (10.9MB/s), 10.4MiB/s-10.4MiB/s (10.9MB/s-10.9MB/s), io=1024MiB (1074MB), run=98116-98116msec Disk stats (read/write): loop5: ios=165665/274948, merge=0/1146, ticks=570916/1710663, in_queue=2291246, util=99.71% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop3/testfile + fio fio/verify.fio --filename=/mnt/loop3/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.4%][r=19.4MiB/s][r=4959 IOPS][eta 00m:27s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=3960: Mon Feb 14 00:45:17 2022 read: IOPS=4591, BW=17.9MiB/s (18.8MB/s)(647MiB/36059msec) slat (usec): min=7, max=3764, avg=12.77, stdev=37.77 clat (usec): min=328, max=195361, avg=3469.75, stdev=2029.21 lat (usec): min=380, max=195375, avg=3482.63, stdev=2031.19 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2507], 10.00th=[ 2573], 20.00th=[ 2737], | 30.00th=[ 2900], 40.00th=[ 3097], 50.00th=[ 3458], 60.00th=[ 3720], | 70.00th=[ 3851], 80.00th=[ 3949], 90.00th=[ 4228], 95.00th=[ 4424], | 99.00th=[ 6521], 99.50th=[ 7111], 99.90th=[ 9372], 99.95th=[ 9765], | 99.99th=[17433] write: IOPS=3197, BW=12.5MiB/s (13.1MB/s)(1024MiB/81996msec); 0 zone resets slat (usec): min=9, max=36590, avg=26.96, stdev=175.62 clat (usec): min=383, max=140174, avg=4976.36, stdev=4045.11 lat (usec): min=477, max=140202, avg=5003.46, stdev=4051.17 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 2999], 10.00th=[ 3097], 20.00th=[ 3228], | 30.00th=[ 3326], 40.00th=[ 3556], 50.00th=[ 3949], 60.00th=[ 4686], | 70.00th=[ 5211], 80.00th=[ 5604], 90.00th=[ 6325], 95.00th=[ 9241], | 99.00th=[24511], 99.50th=[31065], 99.90th=[47449], 99.95th=[65799], | 99.99th=[96994] bw ( KiB/s): min= 4832, max=18504, per=100.00%, avg=12793.93, stdev=2620.03, samples=164 iops : min= 1208, max= 4626, avg=3198.38, stdev=654.99, samples=164 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=62.80%, 10=34.34%, 20=1.93%, 50=0.81% lat (msec) : 100=0.05%, 250=0.01% cpu : usr=2.02%, sys=9.13%, ctx=422417, majf=0, minf=3898 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165574,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=647MiB (678MB), run=36059-36059msec WRITE: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=1024MiB (1074MB), run=81996-81996msec Disk stats (read/write): loop3: ios=165516/274739, merge=0/1239, ticks=573166/1354400, in_queue=1935490, util=99.86% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop1/testfile + fio fio/verify.fio --filename=/mnt/loop1/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.2%][r=16.4MiB/s][r=4188 IOPS][eta 00m:27s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=4321: Mon Feb 14 00:47:15 2022 read: IOPS=4598, BW=18.0MiB/s (18.8MB/s)(648MiB/36048msec) slat (usec): min=7, max=6302, avg=12.44, stdev=31.83 clat (usec): min=421, max=184955, avg=3464.65, stdev=1920.29 lat (usec): min=450, max=184973, avg=3477.19, stdev=1921.75 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2507], 10.00th=[ 2573], 20.00th=[ 2737], | 30.00th=[ 2868], 40.00th=[ 3163], 50.00th=[ 3523], 60.00th=[ 3752], | 70.00th=[ 3851], 80.00th=[ 3949], 90.00th=[ 4228], 95.00th=[ 4359], | 99.00th=[ 6325], 99.50th=[ 6915], 99.90th=[ 8291], 99.95th=[ 9110], | 99.99th=[15139] write: IOPS=3213, BW=12.6MiB/s (13.2MB/s)(1024MiB/81578msec); 0 zone resets slat (usec): min=9, max=103942, avg=26.21, stdev=275.35 clat (usec): min=462, max=202364, avg=4951.53, stdev=4217.15 lat (usec): min=776, max=202401, avg=4977.88, stdev=4237.99 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 10], | 99.00th=[ 24], 99.50th=[ 30], 99.90th=[ 40], 99.95th=[ 54], | 99.99th=[ 138] bw ( KiB/s): min= 4368, max=19240, per=100.00%, avg=12865.96, stdev=2971.61, samples=163 iops : min= 1092, max= 4810, avg=3216.49, stdev=742.90, samples=163 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=64.55%, 10=32.44%, 20=2.12%, 50=0.83% lat (msec) : 100=0.03%, 250=0.01% cpu : usr=2.03%, sys=9.06%, ctx=423983, majf=0, minf=3903 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165779,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (18.8MB/s), 18.0MiB/s-18.0MiB/s (18.8MB/s-18.8MB/s), io=648MiB (679MB), run=36048-36048msec WRITE: bw=12.6MiB/s (13.2MB/s), 12.6MiB/s-12.6MiB/s (13.2MB/s-13.2MB/s), io=1024MiB (1074MB), run=81578-81578msec Disk stats (read/write): loop1: ios=164590/274753, merge=0/972, ticks=569739/1340209, in_queue=1917586, util=99.90% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop7/testfile + fio fio/verify.fio --filename=/mnt/loop7/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.0%][r=18.5MiB/s][r=4743 IOPS][eta 00m:29s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=4369: Mon Feb 14 00:49:20 2022 read: IOPS=4576, BW=17.9MiB/s (18.7MB/s)(648MiB/36269msec) slat (usec): min=7, max=3455, avg=12.29, stdev=18.44 clat (usec): min=339, max=183878, avg=3481.44, stdev=1929.01 lat (usec): min=392, max=183901, avg=3493.84, stdev=1930.15 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2507], 10.00th=[ 2573], 20.00th=[ 2704], | 30.00th=[ 2868], 40.00th=[ 3163], 50.00th=[ 3523], 60.00th=[ 3752], | 70.00th=[ 3851], 80.00th=[ 3982], 90.00th=[ 4228], 95.00th=[ 4424], | 99.00th=[ 6521], 99.50th=[ 7046], 99.90th=[10159], 99.95th=[11338], | 99.99th=[14222] write: IOPS=2964, BW=11.6MiB/s (12.1MB/s)(1024MiB/88415msec); 0 zone resets slat (usec): min=9, max=22005, avg=26.98, stdev=119.64 clat (usec): min=363, max=331919, avg=5368.06, stdev=8943.04 lat (usec): min=660, max=331946, avg=5395.19, stdev=8945.60 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 11], | 99.00th=[ 27], 99.50th=[ 33], 99.90th=[ 153], 99.95th=[ 232], | 99.99th=[ 313] bw ( KiB/s): min= 200, max=19288, per=99.94%, avg=11853.62, stdev=3752.39, samples=177 iops : min= 50, max= 4822, avg=2963.33, stdev=938.07, samples=177 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=62.68%, 10=33.85%, 20=2.47%, 50=0.82% lat (msec) : 100=0.05%, 250=0.07%, 500=0.02% cpu : usr=1.98%, sys=8.76%, ctx=423946, majf=0, minf=3908 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=166003,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=648MiB (680MB), run=36269-36269msec WRITE: bw=11.6MiB/s (12.1MB/s), 11.6MiB/s-11.6MiB/s (12.1MB/s-12.1MB/s), io=1024MiB (1074MB), run=88415-88415msec Disk stats (read/write): loop7: ios=165505/275157, merge=0/445, ticks=575262/1491091, in_queue=2074451, util=99.80% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop2/testfile + fio fio/verify.fio --filename=/mnt/loop2/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.0%][r=19.8MiB/s][r=5069 IOPS][eta 00m:29s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=4443: Mon Feb 14 00:51:25 2022 read: IOPS=4727, BW=18.5MiB/s (19.4MB/s)(647MiB/35059msec) slat (usec): min=7, max=4887, avg=12.81, stdev=23.98 clat (usec): min=245, max=132700, avg=3369.45, stdev=1488.12 lat (usec): min=288, max=132731, avg=3382.36, stdev=1489.93 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2507], 10.00th=[ 2540], 20.00th=[ 2638], | 30.00th=[ 2769], 40.00th=[ 2966], 50.00th=[ 3294], 60.00th=[ 3621], | 70.00th=[ 3785], 80.00th=[ 3916], 90.00th=[ 4228], 95.00th=[ 4359], | 99.00th=[ 6390], 99.50th=[ 6915], 99.90th=[ 9503], 99.95th=[10552], | 99.99th=[13829] write: IOPS=2931, BW=11.5MiB/s (12.0MB/s)(1024MiB/89414msec); 0 zone resets slat (usec): min=9, max=64714, avg=31.32, stdev=289.47 clat (usec): min=370, max=161699, avg=5424.62, stdev=4208.98 lat (usec): min=524, max=161746, avg=5456.10, stdev=4225.29 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3163], 10.00th=[ 3228], 20.00th=[ 3392], | 30.00th=[ 3621], 40.00th=[ 3851], 50.00th=[ 4178], 60.00th=[ 4621], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[11469], | 99.00th=[25560], 99.50th=[31589], 99.90th=[44303], 99.95th=[54789], | 99.99th=[81265] bw ( KiB/s): min= 1408, max=19264, per=99.91%, avg=11717.60, stdev=3203.42, samples=179 iops : min= 352, max= 4816, avg=2929.39, stdev=800.85, samples=179 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=60.44%, 10=35.67%, 20=2.88%, 50=0.93% lat (msec) : 100=0.04%, 250=0.01% cpu : usr=2.12%, sys=9.57%, ctx=423544, majf=0, minf=3902 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165748,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.5MiB/s (19.4MB/s), 18.5MiB/s-18.5MiB/s (19.4MB/s-19.4MB/s), io=647MiB (679MB), run=35059-35059msec WRITE: bw=11.5MiB/s (12.0MB/s), 11.5MiB/s-11.5MiB/s (12.0MB/s-12.0MB/s), io=1024MiB (1074MB), run=89414-89414msec Disk stats (read/write): loop2: ios=165107/275671, merge=0/849, ticks=555353/1492768, in_queue=2056959, util=99.84% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop9/testfile + fio fio/verify.fio --filename=/mnt/loop9/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.6%][r=17.8MiB/s][r=4544 IOPS][eta 00m:27s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=4536: Mon Feb 14 00:53:25 2022 read: IOPS=4578, BW=17.9MiB/s (18.8MB/s)(648MiB/36203msec) slat (usec): min=7, max=3656, avg=12.64, stdev=30.11 clat (usec): min=433, max=172133, avg=3479.80, stdev=1835.20 lat (usec): min=461, max=172146, avg=3492.54, stdev=1836.82 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2507], 10.00th=[ 2540], 20.00th=[ 2737], | 30.00th=[ 2868], 40.00th=[ 3195], 50.00th=[ 3556], 60.00th=[ 3785], | 70.00th=[ 3851], 80.00th=[ 3982], 90.00th=[ 4228], 95.00th=[ 4359], | 99.00th=[ 6194], 99.50th=[ 6849], 99.90th=[ 9634], 99.95th=[10945], | 99.99th=[27657] write: IOPS=3129, BW=12.2MiB/s (12.8MB/s)(1024MiB/83768msec); 0 zone resets slat (usec): min=9, max=118411, avg=27.16, stdev=269.68 clat (usec): min=399, max=156396, avg=5084.28, stdev=4265.18 lat (usec): min=611, max=156425, avg=5111.58, stdev=4275.66 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 11], | 99.00th=[ 25], 99.50th=[ 31], 99.90th=[ 48], 99.95th=[ 68], | 99.99th=[ 127] bw ( KiB/s): min= 3808, max=17688, per=99.75%, avg=12486.86, stdev=2928.80, samples=168 iops : min= 952, max= 4422, avg=3121.66, stdev=732.17, samples=168 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.06%, 4=62.84%, 10=33.78%, 20=2.44%, 50=0.82% lat (msec) : 100=0.05%, 250=0.01% cpu : usr=1.94%, sys=9.19%, ctx=422880, majf=0, minf=3903 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165766,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=648MiB (679MB), run=36203-36203msec WRITE: bw=12.2MiB/s (12.8MB/s), 12.2MiB/s-12.2MiB/s (12.8MB/s-12.8MB/s), io=1024MiB (1074MB), run=83768-83768msec Disk stats (read/write): loop9: ios=164914/275126, merge=0/801, ticks=573496/1387584, in_queue=1969380, util=99.87% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop8/testfile + fio fio/verify.fio --filename=/mnt/loop8/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) ./kernel_compile.sh Jobs: 1 (f=1): [V(1)][75.7%][r=19.5MiB/s][r=4996 IOPS][eta 00m:36s] Jobs: 1 (f=1): [V(1)][81.5%][r=18.9MiB/s][r=4826 IOPS][eta 00m:27s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=4770: Mon Feb 14 00:55:24 2022 read: IOPS=4440, BW=17.3MiB/s (18.2MB/s)(647MiB/37270msec) slat (usec): min=7, max=2362, avg=13.33, stdev=18.79 clat (usec): min=303, max=197336, avg=3587.52, stdev=2050.10 lat (usec): min=355, max=197349, avg=3600.96, stdev=2051.09 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2540], 10.00th=[ 2638], 20.00th=[ 2802], | 30.00th=[ 2999], 40.00th=[ 3392], 50.00th=[ 3720], 60.00th=[ 3851], | 70.00th=[ 3916], 80.00th=[ 4047], 90.00th=[ 4293], 95.00th=[ 4490], | 99.00th=[ 6521], 99.50th=[ 7046], 99.90th=[ 9110], 99.95th=[10421], | 99.99th=[17433] write: IOPS=3205, BW=12.5MiB/s (13.1MB/s)(1024MiB/81785msec); 0 zone resets slat (usec): min=9, max=90569, avg=27.29, stdev=290.77 clat (usec): min=360, max=145421, avg=4963.20, stdev=4064.59 lat (usec): min=506, max=145449, avg=4990.62, stdev=4085.24 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 2966], 10.00th=[ 3032], 20.00th=[ 3163], | 30.00th=[ 3261], 40.00th=[ 3490], 50.00th=[ 3982], 60.00th=[ 4621], | 70.00th=[ 5145], 80.00th=[ 5604], 90.00th=[ 6521], 95.00th=[ 9372], | 99.00th=[25035], 99.50th=[30540], 99.90th=[45351], 99.95th=[58459], | 99.99th=[88605] bw ( KiB/s): min= 2632, max=20184, per=99.73%, avg=12787.02, stdev=2904.81, samples=164 iops : min= 658, max= 5046, avg=3196.75, stdev=726.20, samples=164 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=60.54%, 10=36.65%, 20=1.84%, 50=0.86% lat (msec) : 100=0.04%, 250=0.01% cpu : usr=1.99%, sys=9.24%, ctx=423283, majf=0, minf=3903 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165505,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.3MiB/s (18.2MB/s), 17.3MiB/s-17.3MiB/s (18.2MB/s-18.2MB/s), io=647MiB (678MB), run=37270-37270msec WRITE: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=1024MiB (1074MB), run=81785-81785msec Disk stats (read/write): loop8: ios=164724/275455, merge=0/496, ticks=589770/1367679, in_queue=1965080, util=99.91% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop10/testfile + fio fio/verify.fio --filename=/mnt/loop10/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.3%][r=17.8MiB/s][r=4556 IOPS][eta 00m:29s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=4933: Mon Feb 14 00:57:31 2022 read: IOPS=4707, BW=18.4MiB/s (19.3MB/s)(647MiB/35192msec) slat (usec): min=7, max=3278, avg=12.42, stdev=17.36 clat (usec): min=188, max=110908, avg=3384.22, stdev=1312.64 lat (usec): min=216, max=110935, avg=3396.74, stdev=1314.44 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2442], 10.00th=[ 2474], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3425], 60.00th=[ 3720], | 70.00th=[ 3851], 80.00th=[ 3949], 90.00th=[ 4228], 95.00th=[ 4359], | 99.00th=[ 6390], 99.50th=[ 6915], 99.90th=[ 9765], 99.95th=[10945], | 99.99th=[13960] write: IOPS=2875, BW=11.2MiB/s (11.8MB/s)(1024MiB/91151msec); 0 zone resets slat (usec): min=9, max=188191, avg=31.96, stdev=677.26 clat (usec): min=327, max=498572, avg=5530.16, stdev=11129.13 lat (usec): min=360, max=498617, avg=5562.26, stdev=11210.29 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 11], | 99.00th=[ 30], 99.50th=[ 48], 99.90th=[ 207], 99.95th=[ 249], | 99.99th=[ 292] bw ( KiB/s): min= 208, max=20048, per=99.67%, avg=11466.34, stdev=4330.86, samples=183 iops : min= 52, max= 5012, avg=2866.48, stdev=1082.72, samples=183 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.04%, 4=63.80%, 10=32.95%, 20=2.05%, 50=0.86% lat (msec) : 100=0.11%, 250=0.15%, 500=0.03% cpu : usr=1.84%, sys=8.64%, ctx=423262, majf=0, minf=3902 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165678,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.4MiB/s (19.3MB/s), 18.4MiB/s-18.4MiB/s (19.3MB/s-19.3MB/s), io=647MiB (679MB), run=35192-35192msec WRITE: bw=11.2MiB/s (11.8MB/s), 11.2MiB/s-11.2MiB/s (11.8MB/s-11.8MB/s), io=1024MiB (1074MB), run=91151-91151msec Disk stats (read/write): loop10: ios=165337/275417, merge=0/558, ticks=558916/1602182, in_queue=2171960, util=99.48% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop6/testfile + fio fio/verify.fio --filename=/mnt/loop6/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.5%][r=17.2MiB/s][r=4407 IOPS][eta 00m:28s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=5008: Mon Feb 14 00:59:35 2022 read: IOPS=4388, BW=17.1MiB/s (18.0MB/s)(648MiB/37775msec) slat (usec): min=7, max=3957, avg=13.73, stdev=18.70 clat (usec): min=214, max=30818, avg=3630.04, stdev=887.40 lat (usec): min=246, max=30833, avg=3643.89, stdev=889.77 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2573], 10.00th=[ 2671], 20.00th=[ 2835], | 30.00th=[ 3032], 40.00th=[ 3326], 50.00th=[ 3654], 60.00th=[ 3851], | 70.00th=[ 3982], 80.00th=[ 4228], 90.00th=[ 4490], 95.00th=[ 4817], | 99.00th=[ 6521], 99.50th=[ 7242], 99.90th=[ 9634], 99.95th=[10814], | 99.99th=[26608] write: IOPS=3058, BW=11.9MiB/s (12.5MB/s)(1024MiB/85702msec); 0 zone resets slat (usec): min=9, max=64165, avg=28.45, stdev=341.57 clat (usec): min=283, max=292636, avg=5201.04, stdev=7778.65 lat (usec): min=419, max=300151, avg=5229.64, stdev=7823.47 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 11], | 99.00th=[ 28], 99.50th=[ 35], 99.90th=[ 121], 99.95th=[ 188], | 99.99th=[ 259] bw ( KiB/s): min= 368, max=19680, per=99.66%, avg=12193.70, stdev=3835.99, samples=172 iops : min= 92, max= 4920, avg=3048.40, stdev=958.99, samples=172 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=61.29%, 10=35.42%, 20=2.16%, 50=0.89% lat (msec) : 100=0.11%, 250=0.07%, 500=0.01% cpu : usr=1.93%, sys=8.80%, ctx=423706, majf=0, minf=3901 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165770,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.1MiB/s (18.0MB/s), 17.1MiB/s-17.1MiB/s (18.0MB/s-18.0MB/s), io=648MiB (679MB), run=37775-37775msec WRITE: bw=11.9MiB/s (12.5MB/s), 11.9MiB/s-11.9MiB/s (12.5MB/s-12.5MB/s), io=1024MiB (1074MB), run=85702-85702msec Disk stats (read/write): loop6: ios=165239/275207, merge=0/399, ticks=599041/1461570, in_queue=2069975, util=99.69% + for i in `shuf -i 1-$NN -n $NN` + fallocate -o 0 -l 524288000 /mnt/loop4/testfile + fio fio/verify.fio --filename=/mnt/loop4/testfile write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.27 Starting 1 process write-and-verify: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [V(1)][81.1%][r=18.5MiB/s][r=4724 IOPS][eta 00m:33s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=5048: Mon Feb 14 01:01:58 2022 read: IOPS=4523, BW=17.7MiB/s (18.5MB/s)(647MiB/36630msec) slat (usec): min=7, max=5014, avg=13.45, stdev=45.87 clat (usec): min=231, max=180178, avg=3521.35, stdev=1920.30 lat (usec): min=263, max=180195, avg=3534.91, stdev=1923.48 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2573], 10.00th=[ 2638], 20.00th=[ 2802], | 30.00th=[ 2933], 40.00th=[ 3130], 50.00th=[ 3425], 60.00th=[ 3687], | 70.00th=[ 3884], 80.00th=[ 4047], 90.00th=[ 4359], 95.00th=[ 4621], | 99.00th=[ 6521], 99.50th=[ 7177], 99.90th=[10945], 99.95th=[12256], | 99.99th=[20055] write: IOPS=2477, BW=9911KiB/s (10.1MB/s)(1024MiB/105799msec); 0 zone resets slat (usec): min=9, max=149056, avg=38.32, stdev=830.44 clat (usec): min=576, max=335748, avg=6417.70, stdev=16705.28 lat (usec): min=677, max=335834, avg=6456.18, stdev=16804.79 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 8], 95.00th=[ 12], | 99.00th=[ 41], 99.50th=[ 132], 99.90th=[ 271], 99.95th=[ 292], | 99.99th=[ 330] bw ( KiB/s): min= 200, max=17667, per=99.85%, avg=9896.22, stdev=4614.79, samples=212 iops : min= 50, max= 4416, avg=2473.96, stdev=1153.68, samples=212 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=60.24%, 10=36.23%, 20=2.09%, 50=0.90% lat (msec) : 100=0.14%, 250=0.29%, 500=0.10% cpu : usr=1.82%, sys=7.77%, ctx=422823, majf=0, minf=3901 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=165702,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.7MiB/s (18.5MB/s), 17.7MiB/s-17.7MiB/s (18.5MB/s-18.5MB/s), io=647MiB (679MB), run=36630-36630msec WRITE: bw=9911KiB/s (10.1MB/s), 9911KiB/s-9911KiB/s (10.1MB/s-10.1MB/s), io=1024MiB (1074MB), run=105799-105799msec Disk stats (read/write): loop4: ios=165422/275347, merge=0/459, ticks=581057/1918398, in_queue=2512842, util=99.54% + df -h /mnt/loop1 /mnt/loop10 /mnt/loop2 /mnt/loop3 /mnt/loop4 /mnt/loop5 /mnt/loop6 /mnt/loop7 /mnt/loop8 /mnt/loop9 Filesystem Size Used Avail Use% Mounted on /dev/loop1 2.0G 1.1G 965M 53% /mnt/loop1 /dev/loop10 2.0G 1.1G 966M 53% /mnt/loop10 /dev/loop2 2.0G 1.1G 966M 53% /mnt/loop2 /dev/loop3 2.0G 1.1G 966M 53% /mnt/loop3 /dev/loop4 2.0G 1.1G 965M 53% /mnt/loop4 /dev/loop5 2.0G 1.1G 966M 53% /mnt/loop5 /dev/loop6 2.0G 1.1G 966M 53% /mnt/loop6 /dev/loop7 2.0G 1.1G 966M 53% /mnt/loop7 /dev/loop8 2.0G 1.1G 965M 53% /mnt/loop8 /dev/loop9 2.0G 1.1G 965M 53% /mnt/loop9 + unload_loop ++ shuf -i 1-10 -n 10 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop3 + rm -fr ./loop3 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop10 + rm -fr ./loop10 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop5 + rm -fr ./loop5 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop8 + rm -fr ./loop8 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop9 + rm -fr ./loop9 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop4 + rm -fr ./loop4 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop7 + rm -fr ./loop7 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop2 + rm -fr ./loop2 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop1 + rm -fr ./loop1 + for i in `shuf -i 1-$NN -n $NN` + umount /mnt/loop6 + rm -fr ./loop6 + losetup -D + sleep 3 + rmmod loop + modprobe -r loop + lsmod + grep loop ++ shuf -i 1-10 -n 10 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop7 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop8 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop1 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop4 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop6 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop10 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop2 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop3 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop5 + for i in `shuf -i 1-$NN -n $NN` + rm -fr ./loop9 + rm -fr /mnt/loop1 /mnt/loop10 /mnt/loop2 /mnt/loop3 /mnt/loop4 /mnt/loop5 /mnt/loop6 /mnt/loop7 /mnt/loop8 /mnt/loop9 + dmesg -c [ 1355.637847] XFS (loop3): Unmounting Filesystem [ 1355.777851] XFS (loop10): Unmounting Filesystem [ 1355.916829] XFS (loop5): Unmounting Filesystem [ 1356.049920] XFS (loop8): Unmounting Filesystem [ 1356.197833] XFS (loop9): Unmounting Filesystem [ 1356.332848] XFS (loop4): Unmounting Filesystem [ 1356.519850] XFS (loop7): Unmounting Filesystem [ 1356.655913] XFS (loop2): Unmounting Filesystem [ 1356.781838] XFS (loop1): Unmounting Filesystem [ 1356.909819] XFS (loop6): Unmounting Filesystem