mbox series

[0/1] null_blk: replace ida_simple_xxx()

Message ID 20220309220222.20931-1-kch@nvidia.com (mailing list archive)
Headers show
Series null_blk: replace ida_simple_xxx() | expand

Message

Chaitanya Kulkarni March 9, 2022, 10:02 p.m. UTC 
Hi,

Like various places in kernel replace deprecated ida_simple_get() and
ida_simple_remove() with ida_alloc() and ida_free().

Below is the testlog for blktests block category and fio verify job with
null_blk memory backed devices.

-ck 

Chaitanya Kulkarni (1):
  null-blk: replace deprecated ida_simple_xxx()

 drivers/block/null_blk/main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


blktests (master) # ./check block
block/001 (stress device hotplugging)                        [passed]
    runtime  107.074s  ...  42.733s
block/002 (remove a device while running blktrace)           [passed]
    runtime  1.852s  ...  0.894s
block/006 (run null-blk in blocking mode)                    [passed]
    read iops  376026    ...  9181926
    runtime    176.328s  ...  8.794s
block/009 (check page-cache coherency after BLKDISCARD)      [passed]
    runtime  0.970s  ...  0.516s
block/010 (run I/O on null_blk with shared and non-shared tags) [passed]
    Individual tags read iops  448657    ...  8516353
    runtime                    315.200s  ...  18.082s
    Shared tags read iops      450129    ...  8680489
block/014 (run null-blk with blk-mq and timeout injection configured) [not run]
    null_blk module does not have parameter timeout
block/015 (run null-blk on different schedulers with requeue injection configured) [not run]
    null_blk module does not have parameter requeue
block/016 (send a signal to a process waiting on a frozen queue) [passed]
    runtime  7.210s  ...  7.069s
block/017 (do I/O and check the inflight counter)            [passed]
    runtime  1.851s  ...  1.681s
block/018 (do I/O and check iostats times)                   [passed]
    runtime  5.321s  ...  5.100s
block/020 (run null-blk on different schedulers with only one hardware tag) [passed]
    runtime  31.852s  ...  30.984s
block/021 (read/write nr_requests on null-blk with different schedulers) [passed]
    runtime  4.031s  ...  3.932s
block/022 (Test hang caused by freeze/unfreeze sequence)     [passed]
    runtime  30.211s  ...  30.113s
block/023 (do I/O on all null_blk queue modes)               [passed]
    runtime  0.629s  ...  0.198s
block/024 (do I/O faster than a jiffy and check iostats times) [passed]
    runtime  3.046s  ...  2.685s
block/025 (do a huge discard with 4k sector size)            [passed]
    runtime  4.397s  ...  3.796s
block/027 (stress device hotplugging with running fio jobs and different schedulers) [passed]
    runtime  21.039s  ...  17.985s
block/028 (do I/O on scsi_debug with DIF/DIX enabled)        [passed]
    runtime  57.230s  ...  8.563s
block/029 (trigger blk_mq_update_nr_hw_queues())             [passed]
    runtime  30.535s  ...  30.287s
block/030 (trigger the blk_mq_realloc_hw_ctxs() error path)  [not run]
    null_blk module does not have parameter init_hctx
block/031 (do IO on null-blk with a host tag set)            [passed]
    runtime  30.529s  ...  30.272s

linux-block (for-next) # git am 0001-null-blk-replace-deprecated-ida_simple_xxx.patch
Applying: null-blk: replace deprecated ida_simple_xxx()
linux-block (for-next) # ./compile_nullb.sh 
+ umount /mnt/nullb0
umount: /mnt/nullb0: not mounted.
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ dmesg -c
+ modprobe -r null_blk
+ lsmod
+ grep null_blk
++ nproc
+ make -j 48 M=drivers/block modules
  CC [M]  drivers/block/null_blk/main.o
  LD [M]  drivers/block/null_blk/null_blk.o
  MODPOST drivers/block/Module.symvers
  CC [M]  drivers/block/brd.mod.o
  CC [M]  drivers/block/drbd/drbd.mod.o
  CC [M]  drivers/block/floppy.mod.o
  CC [M]  drivers/block/loop.mod.o
  CC [M]  drivers/block/mtip32xx/mtip32xx.mod.o
  CC [M]  drivers/block/nbd.mod.o
  CC [M]  drivers/block/null_blk/null_blk.mod.o
  CC [M]  drivers/block/pktcdvd.mod.o
  CC [M]  drivers/block/rbd.mod.o
  CC [M]  drivers/block/sx8.mod.o
  CC [M]  drivers/block/virtio_blk.mod.o
  CC [M]  drivers/block/xen-blkback/xen-blkback.mod.o
  CC [M]  drivers/block/xen-blkfront.mod.o
  CC [M]  drivers/block/zram/zram.mod.o
  LD [M]  drivers/block/drbd/drbd.ko
  LD [M]  drivers/block/brd.ko
  LD [M]  drivers/block/mtip32xx/mtip32xx.ko
  LD [M]  drivers/block/nbd.ko
  LD [M]  drivers/block/virtio_blk.ko
  LD [M]  drivers/block/pktcdvd.ko
  LD [M]  drivers/block/xen-blkfront.ko
  LD [M]  drivers/block/rbd.ko
  LD [M]  drivers/block/xen-blkback/xen-blkback.ko
  LD [M]  drivers/block/zram/zram.ko
  LD [M]  drivers/block/loop.ko
  LD [M]  drivers/block/null_blk/null_blk.ko
  LD [M]  drivers/block/floppy.ko
  LD [M]  drivers/block/sx8.ko
+ HOST=drivers/block/null_blk/
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc7blk+/kernel/drivers/block/null_blk/
+ cp drivers/block/null_blk//null_blk.ko /lib/modules/5.17.0-rc7blk+/kernel/drivers/block/null_blk//
+ ls -lrth /lib/modules/5.17.0-rc7blk+/kernel/drivers/block/null_blk//null_blk.ko
-rw-r--r--. 1 root root 1.1M Mar  9 13:25 /lib/modules/5.17.0-rc7blk+/kernel/drivers/block/null_blk//null_blk.ko
+ sleep 1
+ dmesg -c
linux-block (for-next) # ./nullbtests.sh 4 
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ NN=4
+ modprobe -r null_blk
+ modprobe null_blk nr_devices=0
+ echo loading devices
loading devices
++ seq 0 4
+ for i in `seq 0 $NN`
+ NULLB_DIR=config/nullb/nullb0
+ mkdir config/nullb/nullb0
+ echo 1
+ echo 4096
+ echo 2048
+ echo 2
+ cat config/nullb/nullb0/queue_mode
2
+ echo 1
++ cat config/nullb/nullb0/index
+ IDX=0
+ echo -n ' 0 '
 0 + sleep .50
+ for i in `seq 0 $NN`
+ NULLB_DIR=config/nullb/nullb1
+ mkdir config/nullb/nullb1
+ echo 1
+ echo 4096
+ echo 2048
+ echo 2
+ cat config/nullb/nullb1/queue_mode
2
+ echo 1
++ cat config/nullb/nullb1/index
+ IDX=1
+ echo -n ' 1 '
 1 + sleep .50
+ for i in `seq 0 $NN`
+ NULLB_DIR=config/nullb/nullb2
+ mkdir config/nullb/nullb2
+ echo 1
+ echo 4096
+ echo 2048
+ echo 2
+ cat config/nullb/nullb2/queue_mode
2
+ echo 1
++ cat config/nullb/nullb2/index
+ IDX=2
+ echo -n ' 2 '
 2 + sleep .50
+ for i in `seq 0 $NN`
+ NULLB_DIR=config/nullb/nullb3
+ mkdir config/nullb/nullb3
+ echo 1
+ echo 4096
+ echo 2048
+ echo 2
+ cat config/nullb/nullb3/queue_mode
2
+ echo 1
++ cat config/nullb/nullb3/index
+ IDX=3
+ echo -n ' 3 '
 3 + sleep .50
+ for i in `seq 0 $NN`
+ NULLB_DIR=config/nullb/nullb4
+ mkdir config/nullb/nullb4
+ echo 1
+ echo 4096
+ echo 2048
+ echo 2
+ cat config/nullb/nullb4/queue_mode
2
+ echo 1
++ cat config/nullb/nullb4/index
+ IDX=4
+ echo -n ' 4 '
 4 + sleep .50
+ lsblk
+ grep null
+ sort
nullb0  251:0    0    2G  0 disk 
nullb1  251:1    0    2G  0 disk 
nullb2  251:2    0    2G  0 disk 
nullb3  251:3    0    2G  0 disk 
nullb4  251:4    0    2G  0 disk 
+ sleep 1
+ dmesg -c
[   62.827976] null_blk: loading out-of-tree module taints kernel.
[   62.828048] null_blk: module verification failed: signature and/or required key missing - tainting kernel
[   62.833367] null_blk: module loaded
+ lsblk
+ grep null
nullb0  251:0    0    2G  0 disk 
nullb1  251:1    0    2G  0 disk 
nullb2  251:2    0    2G  0 disk 
nullb3  251:3    0    2G  0 disk 
nullb4  251:4    0    2G  0 disk 
++ seq 0 4
+ for i in `seq 0 $NN`
+ fio fio/verify.fio --filename=/dev/nullb0
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process

write-and-verify: (groupid=0, jobs=1): err= 0: pid=2862: Wed Mar  9 13:25:49 2022
  read: IOPS=445k, BW=1737MiB/s (1822MB/s)(158MiB/91msec)
    slat (nsec): min=1062, max=12223, avg=1349.08, stdev=315.29
    clat (nsec): min=932, max=46237, avg=33631.24, stdev=1793.19
     lat (nsec): min=2265, max=47981, avg=35022.15, stdev=1838.86
    clat percentiles (nsec):
     |  1.00th=[32128],  5.00th=[32384], 10.00th=[32384], 20.00th=[32640],
     | 30.00th=[33024], 40.00th=[33024], 50.00th=[33024], 60.00th=[33536],
     | 70.00th=[33536], 80.00th=[34048], 90.00th=[35072], 95.00th=[38144],
     | 99.00th=[40704], 99.50th=[41728], 99.90th=[44800], 99.95th=[45824],
     | 99.99th=[45824]
  write: IOPS=268k, BW=1046MiB/s (1097MB/s)(250MiB/239msec); 0 zone resets
    slat (nsec): min=2214, max=42160, avg=3124.73, stdev=1051.23
    clat (nsec): min=771, max=116021, avg=56206.26, stdev=14239.21
     lat (usec): min=3, max=131, avg=59.39, stdev=15.00
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   47], 10.00th=[   48], 20.00th=[   49],
     | 30.00th=[   49], 40.00th=[   50], 50.00th=[   50], 60.00th=[   51],
     | 70.00th=[   52], 80.00th=[   58], 90.00th=[   88], 95.00th=[   91],
     | 99.00th=[  100], 99.50th=[  103], 99.90th=[  109], 99.95th=[  110],
     | 99.99th=[  115]
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=68.57%, 100=30.79%
  lat (usec)   : 250=0.63%
  cpu          : usr=48.02%, sys=51.67%, ctx=0, majf=0, minf=966
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40470,64000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1737MiB/s (1822MB/s), 1737MiB/s-1737MiB/s (1822MB/s-1822MB/s), io=158MiB (166MB), run=91-91msec
  WRITE: bw=1046MiB/s (1097MB/s), 1046MiB/s-1046MiB/s (1097MB/s-1097MB/s), io=250MiB (262MB), run=239-239msec

Disk stats (read/write):
  nullb0: ios=0/35688, merge=0/0, ticks=0/30, in_queue=30, util=58.85%
+ for i in `seq 0 $NN`
+ fio fio/verify.fio --filename=/dev/nullb1
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process

write-and-verify: (groupid=0, jobs=1): err= 0: pid=2865: Wed Mar  9 13:25:50 2022
  read: IOPS=455k, BW=1776MiB/s (1862MB/s)(158MiB/89msec)
    slat (nsec): min=1052, max=14828, avg=1334.48, stdev=311.47
    clat (nsec): min=932, max=46809, avg=33151.56, stdev=1804.86
     lat (nsec): min=2225, max=48121, avg=34528.03, stdev=1852.32
    clat percentiles (nsec):
     |  1.00th=[31616],  5.00th=[31616], 10.00th=[31872], 20.00th=[32128],
     | 30.00th=[32384], 40.00th=[32384], 50.00th=[32640], 60.00th=[33024],
     | 70.00th=[33024], 80.00th=[33536], 90.00th=[35072], 95.00th=[37632],
     | 99.00th=[40192], 99.50th=[40704], 99.90th=[43776], 99.95th=[44800],
     | 99.99th=[46848]
  write: IOPS=275k, BW=1073MiB/s (1125MB/s)(250MiB/233msec); 0 zone resets
    slat (nsec): min=2114, max=43754, avg=3045.27, stdev=1031.67
    clat (nsec): min=721, max=126910, avg=54771.05, stdev=14107.80
     lat (usec): min=3, max=132, avg=57.87, stdev=14.86
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   46], 10.00th=[   47], 20.00th=[   48],
     | 30.00th=[   48], 40.00th=[   49], 50.00th=[   49], 60.00th=[   50],
     | 70.00th=[   51], 80.00th=[   56], 90.00th=[   87], 95.00th=[   91],
     | 99.00th=[   99], 99.50th=[  102], 99.90th=[  110], 99.95th=[  112],
     | 99.99th=[  124]
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=77.56%, 100=21.97%
  lat (usec)   : 250=0.46%
  cpu          : usr=48.60%, sys=51.40%, ctx=0, majf=0, minf=966
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40465,64000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1776MiB/s (1862MB/s), 1776MiB/s-1776MiB/s (1862MB/s-1862MB/s), io=158MiB (166MB), run=89-89msec
  WRITE: bw=1073MiB/s (1125MB/s), 1073MiB/s-1073MiB/s (1125MB/s-1125MB/s), io=250MiB (262MB), run=233-233msec

Disk stats (read/write):
  nullb1: ios=0/38401, merge=0/0, ticks=0/31, in_queue=31, util=60.08%
+ for i in `seq 0 $NN`
+ fio fio/verify.fio --filename=/dev/nullb2
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process

write-and-verify: (groupid=0, jobs=1): err= 0: pid=2868: Wed Mar  9 13:25:50 2022
  read: IOPS=426k, BW=1663MiB/s (1744MB/s)(158MiB/95msec)
    slat (nsec): min=1082, max=11932, avg=1424.57, stdev=381.94
    clat (nsec): min=952, max=73750, avg=35268.03, stdev=2448.22
     lat (nsec): min=2254, max=75514, avg=36739.98, stdev=2516.34
    clat percentiles (nsec):
     |  1.00th=[33024],  5.00th=[33536], 10.00th=[33536], 20.00th=[34048],
     | 30.00th=[34048], 40.00th=[34560], 50.00th=[34560], 60.00th=[35072],
     | 70.00th=[35072], 80.00th=[35584], 90.00th=[37120], 95.00th=[40192],
     | 99.00th=[45312], 99.50th=[50432], 99.90th=[57600], 99.95th=[61184],
     | 99.99th=[71168]
  write: IOPS=259k, BW=1012MiB/s (1061MB/s)(250MiB/247msec); 0 zone resets
    slat (nsec): min=2214, max=32622, avg=3230.35, stdev=1091.64
    clat (nsec): min=621, max=126089, avg=57938.70, stdev=14139.18
     lat (usec): min=3, max=131, avg=61.23, stdev=14.89
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   49], 10.00th=[   50], 20.00th=[   50],
     | 30.00th=[   51], 40.00th=[   52], 50.00th=[   52], 60.00th=[   53],
     | 70.00th=[   55], 80.00th=[   60], 90.00th=[   89], 95.00th=[   92],
     | 99.00th=[  103], 99.50th=[  106], 99.90th=[  116], 99.95th=[  121],
     | 99.99th=[  126]
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=50.54%, 100=48.53%
  lat (usec)   : 250=0.92%
  cpu          : usr=45.59%, sys=54.12%, ctx=1, majf=0, minf=966
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40444,64000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1663MiB/s (1744MB/s), 1663MiB/s-1663MiB/s (1744MB/s-1744MB/s), io=158MiB (166MB), run=95-95msec
  WRITE: bw=1012MiB/s (1061MB/s), 1012MiB/s-1012MiB/s (1061MB/s-1061MB/s), io=250MiB (262MB), run=247-247msec

Disk stats (read/write):
  nullb2: ios=0/35202, merge=0/0, ticks=0/31, in_queue=31, util=59.59%
+ for i in `seq 0 $NN`
+ fio fio/verify.fio --filename=/dev/nullb3
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process

write-and-verify: (groupid=0, jobs=1): err= 0: pid=2871: Wed Mar  9 13:25:51 2022
  read: IOPS=454k, BW=1774MiB/s (1860MB/s)(158MiB/89msec)
    slat (nsec): min=1052, max=18965, avg=1330.48, stdev=327.51
    clat (nsec): min=932, max=50977, avg=33145.60, stdev=1853.80
     lat (nsec): min=2466, max=52239, avg=34517.16, stdev=1902.39
    clat percentiles (nsec):
     |  1.00th=[31360],  5.00th=[31616], 10.00th=[31872], 20.00th=[32128],
     | 30.00th=[32384], 40.00th=[32384], 50.00th=[32640], 60.00th=[33024],
     | 70.00th=[33024], 80.00th=[33536], 90.00th=[35072], 95.00th=[37632],
     | 99.00th=[40192], 99.50th=[41216], 99.90th=[45312], 99.95th=[46848],
     | 99.99th=[50944]
  write: IOPS=271k, BW=1059MiB/s (1111MB/s)(250MiB/236msec); 0 zone resets
    slat (nsec): min=2154, max=25269, avg=3073.42, stdev=1039.66
    clat (nsec): min=672, max=125849, avg=55367.59, stdev=14444.12
     lat (usec): min=3, max=130, avg=58.50, stdev=15.21
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   46], 10.00th=[   47], 20.00th=[   48],
     | 30.00th=[   48], 40.00th=[   49], 50.00th=[   50], 60.00th=[   50],
     | 70.00th=[   52], 80.00th=[   57], 90.00th=[   87], 95.00th=[   91],
     | 99.00th=[  100], 99.50th=[  102], 99.90th=[  109], 99.95th=[  112],
     | 99.99th=[  122]
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=76.05%, 100=23.34%
  lat (usec)   : 250=0.60%
  cpu          : usr=47.84%, sys=51.85%, ctx=1, majf=0, minf=965
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40416,64000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1774MiB/s (1860MB/s), 1774MiB/s-1774MiB/s (1860MB/s-1860MB/s), io=158MiB (166MB), run=89-89msec
  WRITE: bw=1059MiB/s (1111MB/s), 1059MiB/s-1059MiB/s (1111MB/s-1111MB/s), io=250MiB (262MB), run=236-236msec

Disk stats (read/write):
  nullb3: ios=0/35934, merge=0/0, ticks=0/29, in_queue=29, util=59.09%
+ for i in `seq 0 $NN`
+ fio fio/verify.fio --filename=/dev/nullb4
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process

write-and-verify: (groupid=0, jobs=1): err= 0: pid=2876: Wed Mar  9 13:25:51 2022
  read: IOPS=444k, BW=1733MiB/s (1817MB/s)(158MiB/91msec)
    slat (nsec): min=1062, max=10130, avg=1342.86, stdev=297.90
    clat (nsec): min=952, max=51177, avg=33670.92, stdev=1803.05
     lat (nsec): min=2214, max=52590, avg=35055.68, stdev=1845.21
    clat percentiles (nsec):
     |  1.00th=[31872],  5.00th=[32128], 10.00th=[32384], 20.00th=[32640],
     | 30.00th=[33024], 40.00th=[33024], 50.00th=[33024], 60.00th=[33536],
     | 70.00th=[33536], 80.00th=[34048], 90.00th=[35072], 95.00th=[38144],
     | 99.00th=[40704], 99.50th=[41728], 99.90th=[44288], 99.95th=[46336],
     | 99.99th=[47872]
  write: IOPS=268k, BW=1046MiB/s (1097MB/s)(250MiB/239msec); 0 zone resets
    slat (nsec): min=2174, max=36739, avg=3118.08, stdev=1032.84
    clat (nsec): min=681, max=114448, avg=56228.67, stdev=14207.31
     lat (usec): min=3, max=129, avg=59.40, stdev=14.96
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   48], 10.00th=[   48], 20.00th=[   49],
     | 30.00th=[   49], 40.00th=[   50], 50.00th=[   50], 60.00th=[   51],
     | 70.00th=[   52], 80.00th=[   57], 90.00th=[   88], 95.00th=[   91],
     | 99.00th=[  101], 99.50th=[  103], 99.90th=[  109], 99.95th=[  111],
     | 99.99th=[  113]
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=67.98%, 100=31.30%
  lat (usec)   : 250=0.70%
  cpu          : usr=46.81%, sys=52.89%, ctx=0, majf=0, minf=963
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40366,64000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1733MiB/s (1817MB/s), 1733MiB/s-1733MiB/s (1817MB/s-1817MB/s), io=158MiB (165MB), run=91-91msec
  WRITE: bw=1046MiB/s (1097MB/s), 1046MiB/s-1046MiB/s (1097MB/s-1097MB/s), io=250MiB (262MB), run=239-239msec

Disk stats (read/write):
  nullb4: ios=0/36626, merge=0/0, ticks=0/31, in_queue=31, util=59.51%