| Message ID | 1516218304.2820.51.camel@wdc.com (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Delegated to: | Mike Snitzer |
| Headers | show |
On Wed, Jan 17 2018 at 2:45pm -0500, Bart Van Assche <Bart.VanAssche@wdc.com> wrote: > On Wed, 2018-01-17 at 14:36 -0500, Mike Snitzer wrote: > > That tree doesn't have the revert you think it does (because it doesn't have _any_ of the dm-4.16 changes). > > > > So there is some other reason for your hang. > > Hello Mike, > > Please have another look, namely at the following commit: > https://github.com/bvanassche/linux/commit/cd0b6b9f8779a0b692b1ef1d1f5ba6cb2fd981c5 > > That contents of that commit is as follows: > > Subject: [PATCH] Revert "dm rq: Avoid that request processing stalls sporadically" > > This reverts commit 6077c2d706097c00d8f2fed57d3f3c45cd228ee8. > --- > drivers/md/dm-rq.c | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c > index 85d80b8ccc33..a9e0f6ec62b6 100644 > --- a/drivers/md/dm-rq.c > +++ b/drivers/md/dm-rq.c > @@ -773,7 +773,6 @@ static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx, > /* Undo dm_start_request() before requeuing */ > rq_end_stats(md, rq); > rq_completed(md, rq_data_dir(rq), false); > - blk_mq_delay_run_hw_queue(hctx, 100/*ms*/); > return BLK_STS_RESOURCE; > } > Fine, the chrome browser on my phone displayed "master" rather than your other branch. BUT my broader point stands: you aren't testing the dm-4.16 changes. By just reverting that commit you're creating a self-fulfilling prophecy (that you'll see hangs without it). Fact is you should pull all of dm-4.16 in, see: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.16 But these dm-4.16 changes are particularly important: 050af08ffb1b dm mpath: return DM_MAPIO_REQUEUE on blk-mq rq allocation failure 459b54019cfe dm mpath: return DM_MAPIO_DELAY_REQUEUE if QUEUE_IO or PG_INIT_REQUIRED ec3eaf9a6731 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE 4dd6edd23e7e dm mpath: delay the retry of a request if the target responded as busy This last one is the commit that _should_ serve as a proper replacement for the change you manually reverted in your branch. Please re-test after pulling in dm-4.16 and let us know how things fair. Thanks! Mike -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Wed, 2018-01-17 at 15:14 -0500, Mike Snitzer wrote: > BUT my broader point stands: you aren't testing the dm-4.16 changes. By > just reverting that commit you're creating a self-fulfilling prophecy > (that you'll see hangs without it). > > Fact is you should pull all of dm-4.16 in, see: > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.16 > > But these dm-4.16 changes are particularly important: > 050af08ffb1b dm mpath: return DM_MAPIO_REQUEUE on blk-mq rq allocation failure > 459b54019cfe dm mpath: return DM_MAPIO_DELAY_REQUEUE if QUEUE_IO or PG_INIT_REQUIRED > ec3eaf9a6731 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE > 4dd6edd23e7e dm mpath: delay the retry of a request if the target responded as busy > > This last one is the commit that _should_ serve as a proper replacement > for the change you manually reverted in your branch. > > Please re-test after pulling in dm-4.16 and let us know how things fair. Hello Mike, If I replace the patch I referred to in my previous e-mail with your dm-4.16 branch then I see the following: * Without I/O scheduler: dm path removal at the end of the test fails. This succeeded reliably in the past so I think this is a regression: # srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq [ ... ] Unmounting /root/mnt1 from /dev/mapper/mpathb SRP LUN /sys/class/scsi_device/4:0:0:0 / sdc: removing /dev/dm-1: done SRP LUN /sys/class/scsi_device/4:0:0:1 / sde: removing /dev/dm-2: done SRP LUN /sys/class/scsi_device/4:0:0:2 / sdd: removing /dev/dm-0: dm=$(dev_to_mpath "/dev/dm-0"): failed [ ... ] * With the Kyber I/O scheduler: I/O hangs. # srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq -e kyber [ ... ] Using /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000 -> ../../dm-0 (hangs) I have attached a queue state dump for the last test case to this e-mail. Bart. ./dm-2/hctx0/cpu3/completed:2 0 ./dm-2/hctx0/cpu3/merged:0 ./dm-2/hctx0/cpu3/dispatched:2 0 ./dm-2/hctx0/cpu2/completed:29 0 ./dm-2/hctx0/cpu2/merged:0 ./dm-2/hctx0/cpu2/dispatched:29 0 ./dm-2/hctx0/cpu1/completed:0 0 ./dm-2/hctx0/cpu1/merged:0 ./dm-2/hctx0/cpu1/dispatched:0 0 ./dm-2/hctx0/cpu0/completed:0 0 ./dm-2/hctx0/cpu0/merged:0 ./dm-2/hctx0/cpu0/dispatched:0 0 ./dm-2/hctx0/active:0 ./dm-2/hctx0/run:33 ./dm-2/hctx0/queued:31 ./dm-2/hctx0/dispatched: 0 6 ./dm-2/hctx0/dispatched: 1 31 ./dm-2/hctx0/dispatched: 2 0 ./dm-2/hctx0/dispatched: 4 0 ./dm-2/hctx0/dispatched: 8 0 ./dm-2/hctx0/dispatched: 16 0 ./dm-2/hctx0/dispatched: 32+ 0 ./dm-2/hctx0/io_poll:considered=0 ./dm-2/hctx0/io_poll:invoked=0 ./dm-2/hctx0/io_poll:success=0 ./dm-2/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/sched_tags:nr_tags=256 ./dm-2/hctx0/sched_tags:nr_reserved_tags=0 ./dm-2/hctx0/sched_tags:active_queues=0 ./dm-2/hctx0/sched_tags: ./dm-2/hctx0/sched_tags:bitmap_tags: ./dm-2/hctx0/sched_tags:depth=256 ./dm-2/hctx0/sched_tags:busy=0 ./dm-2/hctx0/sched_tags:bits_per_word=64 ./dm-2/hctx0/sched_tags:map_nr=4 ./dm-2/hctx0/sched_tags:alloc_hint={33, 216, 163, 187} ./dm-2/hctx0/sched_tags:wake_batch=8 ./dm-2/hctx0/sched_tags:wake_index=0 ./dm-2/hctx0/sched_tags:ws={ ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/sched_tags:} ./dm-2/hctx0/sched_tags:round_robin=0 ./dm-2/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-2/hctx0/tags:nr_tags=2048 ./dm-2/hctx0/tags:nr_reserved_tags=0 ./dm-2/hctx0/tags:active_queues=0 ./dm-2/hctx0/tags: ./dm-2/hctx0/tags:bitmap_tags: ./dm-2/hctx0/tags:depth=2048 ./dm-2/hctx0/tags:busy=0 ./dm-2/hctx0/tags:bits_per_word=64 ./dm-2/hctx0/tags:map_nr=32 ./dm-2/hctx0/tags:alloc_hint={760, 1496, 132, 171} ./dm-2/hctx0/tags:wake_batch=8 ./dm-2/hctx0/tags:wake_index=0 ./dm-2/hctx0/tags:ws={ ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-2/hctx0/tags:} ./dm-2/hctx0/tags:round_robin=0 ./dm-2/hctx0/ctx_map:00000000: 00 ./dm-2/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE ./dm-2/hctx0/state: ./dm-2/sched/starved:0 ./dm-2/sched/batching:1 ./dm-2/write_hints:hint0: 0 ./dm-2/write_hints:hint1: 0 ./dm-2/write_hints:hint2: 0 ./dm-2/write_hints:hint3: 0 ./dm-2/write_hints:hint4: 0 ./dm-2/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED ./dm-2/poll_stat:read (512 Bytes): samples=0 ./dm-2/poll_stat:write (512 Bytes): samples=0 ./dm-2/poll_stat:read (1024 Bytes): samples=0 ./dm-2/poll_stat:write (1024 Bytes): samples=0 ./dm-2/poll_stat:read (2048 Bytes): samples=0 ./dm-2/poll_stat:write (2048 Bytes): samples=0 ./dm-2/poll_stat:read (4096 Bytes): samples=0 ./dm-2/poll_stat:write (4096 Bytes): samples=0 ./dm-2/poll_stat:read (8192 Bytes): samples=0 ./dm-2/poll_stat:write (8192 Bytes): samples=0 ./dm-2/poll_stat:read (16384 Bytes): samples=0 ./dm-2/poll_stat:write (16384 Bytes): samples=0 ./dm-2/poll_stat:read (32768 Bytes): samples=0 ./dm-2/poll_stat:write (32768 Bytes): samples=0 ./dm-2/poll_stat:read (65536 Bytes): samples=0 ./dm-2/poll_stat:write (65536 Bytes): samples=0 ./sde/hctx3/cpu3/completed:41 10 ./sde/hctx3/cpu3/merged:0 ./sde/hctx3/cpu3/dispatched:41 10 ./sde/hctx3/active:0 ./sde/hctx3/run:19 ./sde/hctx3/queued:51 ./sde/hctx3/dispatched: 0 5 ./sde/hctx3/dispatched: 1 14 ./sde/hctx3/dispatched: 2 0 ./sde/hctx3/dispatched: 4 0 ./sde/hctx3/dispatched: 8 0 ./sde/hctx3/dispatched: 16 0 ./sde/hctx3/dispatched: 32+ 0 ./sde/hctx3/io_poll:considered=0 ./sde/hctx3/io_poll:invoked=0 ./sde/hctx3/io_poll:success=0 ./sde/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000 ./sde/hctx3/tags:nr_tags=62 ./sde/hctx3/tags:nr_reserved_tags=0 ./sde/hctx3/tags:active_queues=0 ./sde/hctx3/tags: ./sde/hctx3/tags:bitmap_tags: ./sde/hctx3/tags:depth=62 ./sde/hctx3/tags:busy=0 ./sde/hctx3/tags:bits_per_word=8 ./sde/hctx3/tags:map_nr=8 ./sde/hctx3/tags:alloc_hint={28, 33, 59, 34} ./sde/hctx3/tags:wake_batch=7 ./sde/hctx3/tags:wake_index=0 ./sde/hctx3/tags:ws={ ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx3/tags:} ./sde/hctx3/tags:round_robin=0 ./sde/hctx3/ctx_map:00000000: 00 ./sde/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sde/hctx3/state: ./sde/hctx2/cpu2/completed:97 46 ./sde/hctx2/cpu2/merged:0 ./sde/hctx2/cpu2/dispatched:97 46 ./sde/hctx2/active:0 ./sde/hctx2/run:80 ./sde/hctx2/queued:143 ./sde/hctx2/dispatched: 0 11 ./sde/hctx2/dispatched: 1 70 ./sde/hctx2/dispatched: 2 0 ./sde/hctx2/dispatched: 4 0 ./sde/hctx2/dispatched: 8 0 ./sde/hctx2/dispatched: 16 0 ./sde/hctx2/dispatched: 32+ 0 ./sde/hctx2/io_poll:considered=0 ./sde/hctx2/io_poll:invoked=0 ./sde/hctx2/io_poll:success=0 ./sde/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000 ./sde/hctx2/tags:nr_tags=62 ./sde/hctx2/tags:nr_reserved_tags=0 ./sde/hctx2/tags:active_queues=0 ./sde/hctx2/tags: ./sde/hctx2/tags:bitmap_tags: ./sde/hctx2/tags:depth=62 ./sde/hctx2/tags:busy=0 ./sde/hctx2/tags:bits_per_word=8 ./sde/hctx2/tags:map_nr=8 ./sde/hctx2/tags:alloc_hint={2, 0, 24, 0} ./sde/hctx2/tags:wake_batch=7 ./sde/hctx2/tags:wake_index=0 ./sde/hctx2/tags:ws={ ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx2/tags:} ./sde/hctx2/tags:round_robin=0 ./sde/hctx2/ctx_map:00000000: 00 ./sde/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sde/hctx2/state: ./sde/hctx1/cpu1/completed:75 17 ./sde/hctx1/cpu1/merged:0 ./sde/hctx1/cpu1/dispatched:75 17 ./sde/hctx1/active:0 ./sde/hctx1/run:45 ./sde/hctx1/queued:92 ./sde/hctx1/dispatched: 0 11 ./sde/hctx1/dispatched: 1 36 ./sde/hctx1/dispatched: 2 0 ./sde/hctx1/dispatched: 4 0 ./sde/hctx1/dispatched: 8 0 ./sde/hctx1/dispatched: 16 0 ./sde/hctx1/dispatched: 32+ 0 ./sde/hctx1/io_poll:considered=0 ./sde/hctx1/io_poll:invoked=0 ./sde/hctx1/io_poll:success=0 ./sde/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000 ./sde/hctx1/tags:nr_tags=62 ./sde/hctx1/tags:nr_reserved_tags=0 ./sde/hctx1/tags:active_queues=0 ./sde/hctx1/tags: ./sde/hctx1/tags:bitmap_tags: ./sde/hctx1/tags:depth=62 ./sde/hctx1/tags:busy=0 ./sde/hctx1/tags:bits_per_word=8 ./sde/hctx1/tags:map_nr=8 ./sde/hctx1/tags:alloc_hint={39, 24, 20, 46} ./sde/hctx1/tags:wake_batch=7 ./sde/hctx1/tags:wake_index=0 ./sde/hctx1/tags:ws={ ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx1/tags:} ./sde/hctx1/tags:round_robin=0 ./sde/hctx1/ctx_map:00000000: 00 ./sde/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sde/hctx1/state: ./sde/hctx0/cpu0/completed:11 20 ./sde/hctx0/cpu0/merged:0 ./sde/hctx0/cpu0/dispatched:11 20 ./sde/hctx0/active:0 ./sde/hctx0/run:29 ./sde/hctx0/queued:31 ./sde/hctx0/dispatched: 0 1 ./sde/hctx0/dispatched: 1 28 ./sde/hctx0/dispatched: 2 0 ./sde/hctx0/dispatched: 4 0 ./sde/hctx0/dispatched: 8 0 ./sde/hctx0/dispatched: 16 0 ./sde/hctx0/dispatched: 32+ 0 ./sde/hctx0/io_poll:considered=0 ./sde/hctx0/io_poll:invoked=0 ./sde/hctx0/io_poll:success=0 ./sde/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 ./sde/hctx0/tags:nr_tags=62 ./sde/hctx0/tags:nr_reserved_tags=0 ./sde/hctx0/tags:active_queues=0 ./sde/hctx0/tags: ./sde/hctx0/tags:bitmap_tags: ./sde/hctx0/tags:depth=62 ./sde/hctx0/tags:busy=0 ./sde/hctx0/tags:bits_per_word=8 ./sde/hctx0/tags:map_nr=8 ./sde/hctx0/tags:alloc_hint={7, 56, 59, 26} ./sde/hctx0/tags:wake_batch=7 ./sde/hctx0/tags:wake_index=0 ./sde/hctx0/tags:ws={ ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sde/hctx0/tags:} ./sde/hctx0/tags:round_robin=0 ./sde/hctx0/ctx_map:00000000: 00 ./sde/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sde/hctx0/state: ./sde/write_hints:hint0: 0 ./sde/write_hints:hint1: 0 ./sde/write_hints:hint2: 0 ./sde/write_hints:hint3: 0 ./sde/write_hints:hint4: 0 ./sde/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH ./sde/poll_stat:read (512 Bytes): samples=0 ./sde/poll_stat:write (512 Bytes): samples=0 ./sde/poll_stat:read (1024 Bytes): samples=0 ./sde/poll_stat:write (1024 Bytes): samples=0 ./sde/poll_stat:read (2048 Bytes): samples=0 ./sde/poll_stat:write (2048 Bytes): samples=0 ./sde/poll_stat:read (4096 Bytes): samples=0 ./sde/poll_stat:write (4096 Bytes): samples=0 ./sde/poll_stat:read (8192 Bytes): samples=0 ./sde/poll_stat:write (8192 Bytes): samples=0 ./sde/poll_stat:read (16384 Bytes): samples=0 ./sde/poll_stat:write (16384 Bytes): samples=0 ./sde/poll_stat:read (32768 Bytes): samples=0 ./sde/poll_stat:write (32768 Bytes): samples=0 ./sde/poll_stat:read (65536 Bytes): samples=0 ./sde/poll_stat:write (65536 Bytes): samples=0 ./sdf/hctx3/cpu3/completed:7 40 ./sdf/hctx3/cpu3/merged:0 ./sdf/hctx3/cpu3/dispatched:7 40 ./sdf/hctx3/active:0 ./sdf/hctx3/run:48 ./sdf/hctx3/queued:47 ./sdf/hctx3/dispatched: 0 5 ./sdf/hctx3/dispatched: 1 43 ./sdf/hctx3/dispatched: 2 0 ./sdf/hctx3/dispatched: 4 0 ./sdf/hctx3/dispatched: 8 0 ./sdf/hctx3/dispatched: 16 0 ./sdf/hctx3/dispatched: 32+ 0 ./sdf/hctx3/io_poll:considered=0 ./sdf/hctx3/io_poll:invoked=0 ./sdf/hctx3/io_poll:success=0 ./sdf/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdf/hctx3/tags:nr_tags=62 ./sdf/hctx3/tags:nr_reserved_tags=0 ./sdf/hctx3/tags:active_queues=0 ./sdf/hctx3/tags: ./sdf/hctx3/tags:bitmap_tags: ./sdf/hctx3/tags:depth=62 ./sdf/hctx3/tags:busy=0 ./sdf/hctx3/tags:bits_per_word=8 ./sdf/hctx3/tags:map_nr=8 ./sdf/hctx3/tags:alloc_hint={28, 33, 59, 34} ./sdf/hctx3/tags:wake_batch=7 ./sdf/hctx3/tags:wake_index=0 ./sdf/hctx3/tags:ws={ ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx3/tags:} ./sdf/hctx3/tags:round_robin=0 ./sdf/hctx3/ctx_map:00000000: 00 ./sdf/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdf/hctx3/state: ./sdf/hctx2/cpu2/completed:35 7 ./sdf/hctx2/cpu2/merged:0 ./sdf/hctx2/cpu2/dispatched:35 7 ./sdf/hctx2/active:0 ./sdf/hctx2/run:7 ./sdf/hctx2/queued:42 ./sdf/hctx2/dispatched: 0 0 ./sdf/hctx2/dispatched: 1 7 ./sdf/hctx2/dispatched: 2 0 ./sdf/hctx2/dispatched: 4 0 ./sdf/hctx2/dispatched: 8 0 ./sdf/hctx2/dispatched: 16 0 ./sdf/hctx2/dispatched: 32+ 0 ./sdf/hctx2/io_poll:considered=0 ./sdf/hctx2/io_poll:invoked=0 ./sdf/hctx2/io_poll:success=0 ./sdf/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdf/hctx2/tags:nr_tags=62 ./sdf/hctx2/tags:nr_reserved_tags=0 ./sdf/hctx2/tags:active_queues=0 ./sdf/hctx2/tags: ./sdf/hctx2/tags:bitmap_tags: ./sdf/hctx2/tags:depth=62 ./sdf/hctx2/tags:busy=0 ./sdf/hctx2/tags:bits_per_word=8 ./sdf/hctx2/tags:map_nr=8 ./sdf/hctx2/tags:alloc_hint={2, 0, 24, 0} ./sdf/hctx2/tags:wake_batch=7 ./sdf/hctx2/tags:wake_index=0 ./sdf/hctx2/tags:ws={ ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx2/tags:} ./sdf/hctx2/tags:round_robin=0 ./sdf/hctx2/ctx_map:00000000: 00 ./sdf/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdf/hctx2/state: ./sdf/hctx1/cpu1/completed:52 11 ./sdf/hctx1/cpu1/merged:0 ./sdf/hctx1/cpu1/dispatched:52 11 ./sdf/hctx1/active:0 ./sdf/hctx1/run:63 ./sdf/hctx1/queued:63 ./sdf/hctx1/dispatched: 0 6 ./sdf/hctx1/dispatched: 1 56 ./sdf/hctx1/dispatched: 2 0 ./sdf/hctx1/dispatched: 4 0 ./sdf/hctx1/dispatched: 8 0 ./sdf/hctx1/dispatched: 16 0 ./sdf/hctx1/dispatched: 32+ 0 ./sdf/hctx1/io_poll:considered=0 ./sdf/hctx1/io_poll:invoked=0 ./sdf/hctx1/io_poll:success=0 ./sdf/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdf/hctx1/tags:nr_tags=62 ./sdf/hctx1/tags:nr_reserved_tags=0 ./sdf/hctx1/tags:active_queues=0 ./sdf/hctx1/tags: ./sdf/hctx1/tags:bitmap_tags: ./sdf/hctx1/tags:depth=62 ./sdf/hctx1/tags:busy=0 ./sdf/hctx1/tags:bits_per_word=8 ./sdf/hctx1/tags:map_nr=8 ./sdf/hctx1/tags:alloc_hint={39, 24, 20, 46} ./sdf/hctx1/tags:wake_batch=7 ./sdf/hctx1/tags:wake_index=0 ./sdf/hctx1/tags:ws={ ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx1/tags:} ./sdf/hctx1/tags:round_robin=0 ./sdf/hctx1/ctx_map:00000000: 00 ./sdf/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdf/hctx1/state: ./sdf/hctx0/cpu0/completed:1 23 ./sdf/hctx0/cpu0/merged:0 ./sdf/hctx0/cpu0/dispatched:1 23 ./sdf/hctx0/active:0 ./sdf/hctx0/run:28 ./sdf/hctx0/queued:24 ./sdf/hctx0/dispatched: 0 5 ./sdf/hctx0/dispatched: 1 23 ./sdf/hctx0/dispatched: 2 0 ./sdf/hctx0/dispatched: 4 0 ./sdf/hctx0/dispatched: 8 0 ./sdf/hctx0/dispatched: 16 0 ./sdf/hctx0/dispatched: 32+ 0 ./sdf/hctx0/io_poll:considered=0 ./sdf/hctx0/io_poll:invoked=0 ./sdf/hctx0/io_poll:success=0 ./sdf/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdf/hctx0/tags:nr_tags=62 ./sdf/hctx0/tags:nr_reserved_tags=0 ./sdf/hctx0/tags:active_queues=0 ./sdf/hctx0/tags: ./sdf/hctx0/tags:bitmap_tags: ./sdf/hctx0/tags:depth=62 ./sdf/hctx0/tags:busy=0 ./sdf/hctx0/tags:bits_per_word=8 ./sdf/hctx0/tags:map_nr=8 ./sdf/hctx0/tags:alloc_hint={7, 56, 59, 26} ./sdf/hctx0/tags:wake_batch=7 ./sdf/hctx0/tags:wake_index=0 ./sdf/hctx0/tags:ws={ ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdf/hctx0/tags:} ./sdf/hctx0/tags:round_robin=0 ./sdf/hctx0/ctx_map:00000000: 00 ./sdf/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdf/hctx0/state: ./sdf/write_hints:hint0: 0 ./sdf/write_hints:hint1: 0 ./sdf/write_hints:hint2: 0 ./sdf/write_hints:hint3: 0 ./sdf/write_hints:hint4: 0 ./sdf/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH ./sdf/poll_stat:read (512 Bytes): samples=0 ./sdf/poll_stat:write (512 Bytes): samples=0 ./sdf/poll_stat:read (1024 Bytes): samples=0 ./sdf/poll_stat:write (1024 Bytes): samples=0 ./sdf/poll_stat:read (2048 Bytes): samples=0 ./sdf/poll_stat:write (2048 Bytes): samples=0 ./sdf/poll_stat:read (4096 Bytes): samples=0 ./sdf/poll_stat:write (4096 Bytes): samples=0 ./sdf/poll_stat:read (8192 Bytes): samples=0 ./sdf/poll_stat:write (8192 Bytes): samples=0 ./sdf/poll_stat:read (16384 Bytes): samples=0 ./sdf/poll_stat:write (16384 Bytes): samples=0 ./sdf/poll_stat:read (32768 Bytes): samples=0 ./sdf/poll_stat:write (32768 Bytes): samples=0 ./sdf/poll_stat:read (65536 Bytes): samples=0 ./sdf/poll_stat:write (65536 Bytes): samples=0 ./sdc/hctx3/cpu3/completed:1 5 ./sdc/hctx3/cpu3/merged:0 ./sdc/hctx3/cpu3/dispatched:1 5 ./sdc/hctx3/active:0 ./sdc/hctx3/run:11 ./sdc/hctx3/queued:6 ./sdc/hctx3/dispatched: 0 5 ./sdc/hctx3/dispatched: 1 6 ./sdc/hctx3/dispatched: 2 0 ./sdc/hctx3/dispatched: 4 0 ./sdc/hctx3/dispatched: 8 0 ./sdc/hctx3/dispatched: 16 0 ./sdc/hctx3/dispatched: 32+ 0 ./sdc/hctx3/io_poll:considered=0 ./sdc/hctx3/io_poll:invoked=0 ./sdc/hctx3/io_poll:success=0 ./sdc/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdc/hctx3/tags:nr_tags=62 ./sdc/hctx3/tags:nr_reserved_tags=0 ./sdc/hctx3/tags:active_queues=0 ./sdc/hctx3/tags: ./sdc/hctx3/tags:bitmap_tags: ./sdc/hctx3/tags:depth=62 ./sdc/hctx3/tags:busy=0 ./sdc/hctx3/tags:bits_per_word=8 ./sdc/hctx3/tags:map_nr=8 ./sdc/hctx3/tags:alloc_hint={28, 33, 59, 34} ./sdc/hctx3/tags:wake_batch=7 ./sdc/hctx3/tags:wake_index=0 ./sdc/hctx3/tags:ws={ ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx3/tags:} ./sdc/hctx3/tags:round_robin=0 ./sdc/hctx3/ctx_map:00000000: 00 ./sdc/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdc/hctx3/state: ./sdc/hctx2/cpu2/completed:1 54 ./sdc/hctx2/cpu2/merged:0 ./sdc/hctx2/cpu2/dispatched:1 54 ./sdc/hctx2/active:0 ./sdc/hctx2/run:45 ./sdc/hctx2/queued:55 ./sdc/hctx2/dispatched: 0 6 ./sdc/hctx2/dispatched: 1 39 ./sdc/hctx2/dispatched: 2 0 ./sdc/hctx2/dispatched: 4 0 ./sdc/hctx2/dispatched: 8 0 ./sdc/hctx2/dispatched: 16 0 ./sdc/hctx2/dispatched: 32+ 0 ./sdc/hctx2/io_poll:considered=0 ./sdc/hctx2/io_poll:invoked=0 ./sdc/hctx2/io_poll:success=0 ./sdc/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdc/hctx2/tags:nr_tags=62 ./sdc/hctx2/tags:nr_reserved_tags=0 ./sdc/hctx2/tags:active_queues=0 ./sdc/hctx2/tags: ./sdc/hctx2/tags:bitmap_tags: ./sdc/hctx2/tags:depth=62 ./sdc/hctx2/tags:busy=0 ./sdc/hctx2/tags:bits_per_word=8 ./sdc/hctx2/tags:map_nr=8 ./sdc/hctx2/tags:alloc_hint={2, 0, 24, 0} ./sdc/hctx2/tags:wake_batch=7 ./sdc/hctx2/tags:wake_index=0 ./sdc/hctx2/tags:ws={ ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx2/tags:} ./sdc/hctx2/tags:round_robin=0 ./sdc/hctx2/ctx_map:00000000: 00 ./sdc/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdc/hctx2/state: ./sdc/hctx1/cpu1/completed:28 29 ./sdc/hctx1/cpu1/merged:0 ./sdc/hctx1/cpu1/dispatched:28 29 ./sdc/hctx1/active:0 ./sdc/hctx1/run:65 ./sdc/hctx1/queued:57 ./sdc/hctx1/dispatched: 0 13 ./sdc/hctx1/dispatched: 1 55 ./sdc/hctx1/dispatched: 2 0 ./sdc/hctx1/dispatched: 4 0 ./sdc/hctx1/dispatched: 8 0 ./sdc/hctx1/dispatched: 16 0 ./sdc/hctx1/dispatched: 32+ 0 ./sdc/hctx1/io_poll:considered=0 ./sdc/hctx1/io_poll:invoked=0 ./sdc/hctx1/io_poll:success=0 ./sdc/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdc/hctx1/tags:nr_tags=62 ./sdc/hctx1/tags:nr_reserved_tags=0 ./sdc/hctx1/tags:active_queues=0 ./sdc/hctx1/tags: ./sdc/hctx1/tags:bitmap_tags: ./sdc/hctx1/tags:depth=62 ./sdc/hctx1/tags:busy=0 ./sdc/hctx1/tags:bits_per_word=8 ./sdc/hctx1/tags:map_nr=8 ./sdc/hctx1/tags:alloc_hint={39, 24, 20, 46} ./sdc/hctx1/tags:wake_batch=7 ./sdc/hctx1/tags:wake_index=0 ./sdc/hctx1/tags:ws={ ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx1/tags:} ./sdc/hctx1/tags:round_robin=0 ./sdc/hctx1/ctx_map:00000000: 00 ./sdc/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdc/hctx1/state: ./sdc/hctx0/cpu0/completed:30 20 ./sdc/hctx0/cpu0/merged:0 ./sdc/hctx0/cpu0/dispatched:30 20 ./sdc/hctx0/active:0 ./sdc/hctx0/run:44 ./sdc/hctx0/queued:50 ./sdc/hctx0/dispatched: 0 1 ./sdc/hctx0/dispatched: 1 43 ./sdc/hctx0/dispatched: 2 0 ./sdc/hctx0/dispatched: 4 0 ./sdc/hctx0/dispatched: 8 0 ./sdc/hctx0/dispatched: 16 0 ./sdc/hctx0/dispatched: 32+ 0 ./sdc/hctx0/io_poll:considered=0 ./sdc/hctx0/io_poll:invoked=0 ./sdc/hctx0/io_poll:success=0 ./sdc/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 ./sdc/hctx0/tags:nr_tags=62 ./sdc/hctx0/tags:nr_reserved_tags=0 ./sdc/hctx0/tags:active_queues=0 ./sdc/hctx0/tags: ./sdc/hctx0/tags:bitmap_tags: ./sdc/hctx0/tags:depth=62 ./sdc/hctx0/tags:busy=0 ./sdc/hctx0/tags:bits_per_word=8 ./sdc/hctx0/tags:map_nr=8 ./sdc/hctx0/tags:alloc_hint={7, 56, 59, 26} ./sdc/hctx0/tags:wake_batch=7 ./sdc/hctx0/tags:wake_index=0 ./sdc/hctx0/tags:ws={ ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags: {.wait_cnt=7, .wait=inactive}, ./sdc/hctx0/tags:} ./sdc/hctx0/tags:round_robin=0 ./sdc/hctx0/ctx_map:00000000: 00 ./sdc/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|TAG_SHARED|SG_MERGE ./sdc/hctx0/state: ./sdc/write_hints:hint0: 0 ./sdc/write_hints:hint1: 0 ./sdc/write_hints:hint2: 0 ./sdc/write_hints:hint3: 0 ./sdc/write_hints:hint4: 0 ./sdc/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|REGISTERED|SCSI_PASSTHROUGH ./sdc/poll_stat:read (512 Bytes): samples=0 ./sdc/poll_stat:write (512 Bytes): samples=0 ./sdc/poll_stat:read (1024 Bytes): samples=0 ./sdc/poll_stat:write (1024 Bytes): samples=0 ./sdc/poll_stat:read (2048 Bytes): samples=0 ./sdc/poll_stat:write (2048 Bytes): samples=0 ./sdc/poll_stat:read (4096 Bytes): samples=0 ./sdc/poll_stat:write (4096 Bytes): samples=0 ./sdc/poll_stat:read (8192 Bytes): samples=0 ./sdc/poll_stat:write (8192 Bytes): samples=0 ./sdc/poll_stat:read (16384 Bytes): samples=0 ./sdc/poll_stat:write (16384 Bytes): samples=0 ./sdc/poll_stat:read (32768 Bytes): samples=0 ./sdc/poll_stat:write (32768 Bytes): samples=0 ./sdc/poll_stat:read (65536 Bytes): samples=0 ./sdc/poll_stat:write (65536 Bytes): samples=0 ./dm-1/hctx0/cpu3/completed:32 0 ./dm-1/hctx0/cpu3/merged:0 ./dm-1/hctx0/cpu3/dispatched:32 0 ./dm-1/hctx0/cpu2/completed:56 0 ./dm-1/hctx0/cpu2/merged:0 ./dm-1/hctx0/cpu2/dispatched:56 0 ./dm-1/hctx0/cpu1/completed:39 0 ./dm-1/hctx0/cpu1/merged:0 ./dm-1/hctx0/cpu1/dispatched:39 0 ./dm-1/hctx0/cpu0/completed:2 0 ./dm-1/hctx0/cpu0/merged:0 ./dm-1/hctx0/cpu0/dispatched:2 0 ./dm-1/hctx0/active:0 ./dm-1/hctx0/run:140 ./dm-1/hctx0/queued:129 ./dm-1/hctx0/dispatched: 0 33 ./dm-1/hctx0/dispatched: 1 130 ./dm-1/hctx0/dispatched: 2 0 ./dm-1/hctx0/dispatched: 4 0 ./dm-1/hctx0/dispatched: 8 0 ./dm-1/hctx0/dispatched: 16 0 ./dm-1/hctx0/dispatched: 32+ 0 ./dm-1/hctx0/io_poll:considered=0 ./dm-1/hctx0/io_poll:invoked=0 ./dm-1/hctx0/io_poll:success=0 ./dm-1/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/sched_tags:nr_tags=256 ./dm-1/hctx0/sched_tags:nr_reserved_tags=0 ./dm-1/hctx0/sched_tags:active_queues=0 ./dm-1/hctx0/sched_tags: ./dm-1/hctx0/sched_tags:bitmap_tags: ./dm-1/hctx0/sched_tags:depth=256 ./dm-1/hctx0/sched_tags:busy=0 ./dm-1/hctx0/sched_tags:bits_per_word=64 ./dm-1/hctx0/sched_tags:map_nr=4 ./dm-1/hctx0/sched_tags:alloc_hint={165, 112, 86, 40} ./dm-1/hctx0/sched_tags:wake_batch=8 ./dm-1/hctx0/sched_tags:wake_index=0 ./dm-1/hctx0/sched_tags:ws={ ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/sched_tags:} ./dm-1/hctx0/sched_tags:round_robin=0 ./dm-1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-1/hctx0/tags:nr_tags=2048 ./dm-1/hctx0/tags:nr_reserved_tags=0 ./dm-1/hctx0/tags:active_queues=0 ./dm-1/hctx0/tags: ./dm-1/hctx0/tags:bitmap_tags: ./dm-1/hctx0/tags:depth=2048 ./dm-1/hctx0/tags:busy=0 ./dm-1/hctx0/tags:bits_per_word=64 ./dm-1/hctx0/tags:map_nr=32 ./dm-1/hctx0/tags:alloc_hint={1468, 1663, 1467, 1182} ./dm-1/hctx0/tags:wake_batch=8 ./dm-1/hctx0/tags:wake_index=0 ./dm-1/hctx0/tags:ws={ ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-1/hctx0/tags:} ./dm-1/hctx0/tags:round_robin=0 ./dm-1/hctx0/ctx_map:00000000: 00 ./dm-1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE ./dm-1/hctx0/state: ./dm-1/sched/starved:0 ./dm-1/sched/batching:1 ./dm-1/write_hints:hint0: 0 ./dm-1/write_hints:hint1: 0 ./dm-1/write_hints:hint2: 0 ./dm-1/write_hints:hint3: 0 ./dm-1/write_hints:hint4: 0 ./dm-1/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED ./dm-1/poll_stat:read (512 Bytes): samples=0 ./dm-1/poll_stat:write (512 Bytes): samples=0 ./dm-1/poll_stat:read (1024 Bytes): samples=0 ./dm-1/poll_stat:write (1024 Bytes): samples=0 ./dm-1/poll_stat:read (2048 Bytes): samples=0 ./dm-1/poll_stat:write (2048 Bytes): samples=0 ./dm-1/poll_stat:read (4096 Bytes): samples=0 ./dm-1/poll_stat:write (4096 Bytes): samples=0 ./dm-1/poll_stat:read (8192 Bytes): samples=0 ./dm-1/poll_stat:write (8192 Bytes): samples=0 ./dm-1/poll_stat:read (16384 Bytes): samples=0 ./dm-1/poll_stat:write (16384 Bytes): samples=0 ./dm-1/poll_stat:read (32768 Bytes): samples=0 ./dm-1/poll_stat:write (32768 Bytes): samples=0 ./dm-1/poll_stat:read (65536 Bytes): samples=0 ./dm-1/poll_stat:write (65536 Bytes): samples=0 ./dm-0/sched/async_depth:48 ./dm-0/sched/other_tokens:depth=64 ./dm-0/sched/other_tokens:busy=0 ./dm-0/sched/other_tokens:bits_per_word=64 ./dm-0/sched/other_tokens:map_nr=1 ./dm-0/sched/other_tokens:alloc_hint={5, 1695, 36, 42} ./dm-0/sched/other_tokens:wake_batch=8 ./dm-0/sched/other_tokens:wake_index=0 ./dm-0/sched/other_tokens:ws={ ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/other_tokens:} ./dm-0/sched/other_tokens:round_robin=0 ./dm-0/sched/sync_write_tokens:depth=24 ./dm-0/sched/sync_write_tokens:busy=24 ./dm-0/sched/sync_write_tokens:bits_per_word=64 ./dm-0/sched/sync_write_tokens:map_nr=1 ./dm-0/sched/sync_write_tokens:alloc_hint={0, 0, 0, 11} ./dm-0/sched/sync_write_tokens:wake_batch=3 ./dm-0/sched/sync_write_tokens:wake_index=1 ./dm-0/sched/sync_write_tokens:ws={ ./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=1, .wait=active}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=1, .wait=inactive}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=1, .wait=inactive}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive}, ./dm-0/sched/sync_write_tokens: {.wait_cnt=3, .wait=inactive}, ./dm-0/sched/sync_write_tokens:} ./dm-0/sched/sync_write_tokens:round_robin=0 ./dm-0/sched/read_tokens:depth=256 ./dm-0/sched/read_tokens:busy=256 ./dm-0/sched/read_tokens:bits_per_word=64 ./dm-0/sched/read_tokens:map_nr=4 ./dm-0/sched/read_tokens:alloc_hint={0, 44, 0, 43} ./dm-0/sched/read_tokens:wake_batch=8 ./dm-0/sched/read_tokens:wake_index=0 ./dm-0/sched/read_tokens:ws={ ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=active}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens: {.wait_cnt=8, .wait=inactive}, ./dm-0/sched/read_tokens:} ./dm-0/sched/read_tokens:round_robin=0 ./dm-0/hctx0/sched/batching:0 ./dm-0/hctx0/sched/cur_domain:OTHER ./dm-0/hctx0/sched/other_waiting:0 ./dm-0/hctx0/sched/sync_write_waiting:1 ./dm-0/hctx0/sched/sync_write_rqs:000000001f573771 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=79} ./dm-0/hctx0/sched/sync_write_rqs:00000000e3b936ff {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=104} ./dm-0/hctx0/sched/sync_write_rqs:000000003cf95da1 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=105} ./dm-0/hctx0/sched/sync_write_rqs:00000000a533203e {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=102} ./dm-0/hctx0/sched/sync_write_rqs:0000000077ff59b5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=99} ./dm-0/hctx0/sched/sync_write_rqs:00000000d891eb85 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=59} ./dm-0/hctx0/sched/sync_write_rqs:00000000f8c029e7 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=123} ./dm-0/hctx0/sched/sync_write_rqs:0000000059a91beb {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=80} ./dm-0/hctx0/sched/sync_write_rqs:000000003f13f5bd {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=69} ./dm-0/hctx0/sched/sync_write_rqs:00000000aa3459ed {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=ELVPRIV|IO_STAT, complete=0, .tag=-1, .internal_tag=72} ./dm-0/hctx0/sched/sync_write_rqs:000000004f65153b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=107} ./dm-0/hctx0/sched/sync_write_rqs:00000000156ef8c1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=40} ./dm-0/hctx0/sched/sync_write_rqs:000000006874301c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=103} ./dm-0/hctx0/sched/sync_write_rqs:000000006fd73113 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=70} ./dm-0/hctx0/sched/sync_write_rqs:00000000ce8cc3aa {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=241} ./dm-0/hctx0/sched/sync_write_rqs:000000009bfc32cd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=57} ./dm-0/hctx0/sched/sync_write_rqs:000000008ae86f6d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=62} ./dm-0/hctx0/sched/sync_write_rqs:000000002bea67f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=2} ./dm-0/hctx0/sched/sync_write_rqs:000000004acf53b2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=5} ./dm-0/hctx0/sched/sync_write_rqs:00000000418f9bcb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=7} ./dm-0/hctx0/sched/sync_write_rqs:00000000487ad874 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=12} ./dm-0/hctx0/sched/sync_write_rqs:00000000d7cbf463 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=15} ./dm-0/hctx0/sched/sync_write_rqs:00000000f6bd8e3a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=20} ./dm-0/hctx0/sched/sync_write_rqs:0000000054db3535 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=106} ./dm-0/hctx0/sched/sync_write_rqs:00000000eb35cbdb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=110} ./dm-0/hctx0/sched/sync_write_rqs:000000007bdd531c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=113} ./dm-0/hctx0/sched/sync_write_rqs:00000000f4b1f373 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=133} ./dm-0/hctx0/sched/sync_write_rqs:000000009d0a7052 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=67} ./dm-0/hctx0/sched/sync_write_rqs:00000000cb5775b4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=64} ./dm-0/hctx0/sched/sync_write_rqs:00000000c49a47e2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=136} ./dm-0/hctx0/sched/sync_write_rqs:00000000ce2123f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=98} ./dm-0/hctx0/sched/sync_write_rqs:00000000693d5140 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=87} ./dm-0/hctx0/sched/sync_write_rqs:000000004bf481dd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=138} ./dm-0/hctx0/sched/sync_write_rqs:0000000019b5d899 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=86} ./dm-0/hctx0/sched/sync_write_rqs:000000006815432d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=42} ./dm-0/hctx0/sched/sync_write_rqs:00000000751a985b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=46} ./dm-0/hctx0/sched/sync_write_rqs:00000000d14f2300 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=47} ./dm-0/hctx0/sched/sync_write_rqs:00000000c701ede2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=25} ./dm-0/hctx0/sched/sync_write_rqs:0000000078fbfaee {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=31} ./dm-0/hctx0/sched/sync_write_rqs:00000000c1654839 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=190} ./dm-0/hctx0/sched/sync_write_rqs:0000000045c6f16b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=131} ./dm-0/hctx0/sched/sync_write_rqs:00000000a6d12f31 {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=78} ./dm-0/hctx0/sched/sync_write_rqs:00000000cdcc74c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=100} ./dm-0/hctx0/sched/sync_write_rqs:00000000cd5796df {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=101} ./dm-0/hctx0/sched/sync_write_rqs:00000000bf164c9e {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=75} ./dm-0/hctx0/sched/sync_write_rqs:00000000df2e49ec {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=76} ./dm-0/hctx0/sched/sync_write_rqs:00000000ebeb68f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=172} ./dm-0/hctx0/sched/sync_write_rqs:000000000526aa72 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=174} ./dm-0/hctx0/sched/sync_write_rqs:000000005abced20 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=177} ./dm-0/hctx0/sched/sync_write_rqs:000000009dc6fe76 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=179} ./dm-0/hctx0/sched/sync_write_rqs:000000001d5deaf7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=185} ./dm-0/hctx0/sched/sync_write_rqs:000000005539190f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=191} ./dm-0/hctx0/sched/sync_write_rqs:000000003b7659de {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=202} ./dm-0/hctx0/sched/sync_write_rqs:0000000013e8b2dd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=143} ./dm-0/hctx0/sched/sync_write_rqs:00000000407e2164 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=149} ./dm-0/hctx0/sched/sync_write_rqs:000000003e25a32f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=154} ./dm-0/hctx0/sched/sync_write_rqs:00000000ea72147a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=227} ./dm-0/hctx0/sched/sync_write_rqs:00000000d79b8d48 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=229} ./dm-0/hctx0/sched/sync_write_rqs:00000000a1889d65 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=212} ./dm-0/hctx0/sched/read_waiting:1 ./dm-0/hctx0/sched/read_rqs:00000000f1d18e6a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=24} ./dm-0/hctx0/sched/read_rqs:00000000a199844e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=26} ./dm-0/hctx0/sched/read_rqs:000000007efcae62 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=28} ./dm-0/hctx0/sched/read_rqs:00000000af80be3b {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=29} ./dm-0/hctx0/sched/read_rqs:00000000aa25e9cc {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=34} ./dm-0/hctx0/sched/read_rqs:00000000449b58fa {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=35} ./dm-0/hctx0/sched/read_rqs:00000000083e07f2 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=38} ./dm-0/hctx0/sched/read_rqs:000000007815ced8 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=39} ./dm-0/hctx0/sched/read_rqs:00000000a2b71248 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=44} ./dm-0/hctx0/sched/read_rqs:00000000f5d2e05e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=45} ./dm-0/hctx0/sched/read_rqs:000000005f198b9e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=48} ./dm-0/hctx0/sched/read_rqs:000000008ee34144 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=49} ./dm-0/hctx0/sched/read_rqs:0000000076e8cbf9 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=71} ./dm-0/hctx0/sched/read_rqs:0000000069f53dae {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=73} ./dm-0/hctx0/sched/read_rqs:0000000029fa385c {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=222} ./dm-0/hctx0/sched/read_rqs:000000008a5edba7 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=247} ./dm-0/hctx0/sched/read_rqs:000000009ac21dec {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=200} ./dm-0/hctx0/sched/read_rqs:00000000e01ff63e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=74} ./dm-0/hctx0/sched/read_rqs:000000006c3f8b64 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=1} ./dm-0/hctx0/sched/read_rqs:0000000006f095fa {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=6} ./dm-0/hctx0/sched/read_rqs:000000006a0ec63c {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=10} ./dm-0/hctx0/sched/read_rqs:000000000a56cd0f {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=16} ./dm-0/hctx0/sched/read_rqs:00000000d2c3813a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=27} ./dm-0/hctx0/sched/read_rqs:000000003598c30a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=30} ./dm-0/hctx0/sched/read_rqs:000000001d9cbd54 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=209} ./dm-0/hctx0/sched/read_rqs:00000000bd9be489 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=228} ./dm-0/hctx0/sched/read_rqs:00000000e7f8caa3 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=236} ./dm-0/hctx0/sched/read_rqs:000000004c771941 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=254} ./dm-0/hctx0/sched/read_rqs:0000000006a08c24 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=219} ./dm-0/hctx0/sched/read_rqs:000000001e816561 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=220} ./dm-0/hctx0/sched/read_rqs:00000000370d6c62 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=225} ./dm-0/hctx0/sched/read_rqs:00000000f213e06d {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=237} ./dm-0/hctx0/sched/read_rqs:00000000df4bd4d4 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=238} ./dm-0/hctx0/sched/read_rqs:00000000181a5782 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=19} ./dm-0/hctx0/sched/read_rqs:00000000a50d648b {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=244} ./dm-0/hctx0/sched/read_rqs:0000000036dc390a {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=246} ./dm-0/hctx0/sched/read_rqs:00000000e0cb7ef4 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=36} ./dm-0/hctx0/sched/read_rqs:00000000c09c951c {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=248} ./dm-0/hctx0/sched/read_rqs:000000006944c854 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=207} ./dm-0/hctx0/sched/read_rqs:000000008ce5e0ef {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=201} ./dm-0/hctx0/sched/read_rqs:00000000152a8d0d {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=203} ./dm-0/hctx0/sched/read_rqs:00000000db3bebb5 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=204} ./dm-0/hctx0/sched/read_rqs:00000000dd2875cf {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=206} ./dm-0/hctx0/sched/read_rqs:00000000a6c0229e {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=215} ./dm-0/hctx0/sched/read_rqs:00000000803073e9 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=88} ./dm-0/hctx0/sched/read_rqs:00000000801ebbaa {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=52} ./dm-0/hctx0/sched/read_rqs:0000000031200a0d {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=4} ./dm-0/hctx0/sched/read_rqs:0000000095b25961 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=9} ./dm-0/hctx0/sched/read_rqs:00000000b3358e33 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=17} ./dm-0/hctx0/sched/read_rqs:0000000089c4cdb8 {.op=READ, .cmd_flags=, .rq_flags=SORTED|ELVPRIV|IO_STAT|STATS, complete=0, .tag=-1, .internal_tag=18} ./dm-0/hctx0/cpu3/completed:2658 0 ./dm-0/hctx0/cpu3/merged:460 ./dm-0/hctx0/cpu3/dispatched:2689 0 ./dm-0/hctx0/cpu2/completed:2341 8 ./dm-0/hctx0/cpu2/merged:371 ./dm-0/hctx0/cpu2/dispatched:2358 8 ./dm-0/hctx0/cpu1/completed:2015 0 ./dm-0/hctx0/cpu1/merged:392 ./dm-0/hctx0/cpu1/dispatched:2052 0 ./dm-0/hctx0/cpu0/completed:1124 0 ./dm-0/hctx0/cpu0/merged:193 ./dm-0/hctx0/cpu0/dispatched:1148 0 ./dm-0/hctx0/active:0 ./dm-0/hctx0/run:15802 ./dm-0/hctx0/queued:8255 ./dm-0/hctx0/dispatched: 0 15202 ./dm-0/hctx0/dispatched: 1 8354 ./dm-0/hctx0/dispatched: 2 0 ./dm-0/hctx0/dispatched: 4 0 ./dm-0/hctx0/dispatched: 8 0 ./dm-0/hctx0/dispatched: 16 0 ./dm-0/hctx0/dispatched: 32+ 1 ./dm-0/hctx0/io_poll:considered=0 ./dm-0/hctx0/io_poll:invoked=0 ./dm-0/hctx0/io_poll:success=0 ./dm-0/hctx0/sched_tags_bitmap:00000000: f696 1fff dcf5 134a e9df c101 fc4f 0208 ./dm-0/hctx0/sched_tags_bitmap:00000010: 2885 2004 0050 0ac2 00df 9258 3a70 d241 ./dm-0/hctx0/sched_tags:nr_tags=256 ./dm-0/hctx0/sched_tags:nr_reserved_tags=0 ./dm-0/hctx0/sched_tags:active_queues=0 ./dm-0/hctx0/sched_tags: ./dm-0/hctx0/sched_tags:bitmap_tags: ./dm-0/hctx0/sched_tags:depth=256 ./dm-0/hctx0/sched_tags:busy=109 ./dm-0/hctx0/sched_tags:bits_per_word=64 ./dm-0/hctx0/sched_tags:map_nr=4 ./dm-0/hctx0/sched_tags:alloc_hint={125, 75, 84, 53} ./dm-0/hctx0/sched_tags:wake_batch=8 ./dm-0/hctx0/sched_tags:wake_index=1 ./dm-0/hctx0/sched_tags:ws={ ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/sched_tags:} ./dm-0/hctx0/sched_tags:round_robin=0 ./dm-0/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags_bitmap:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ./dm-0/hctx0/tags:nr_tags=2048 ./dm-0/hctx0/tags:nr_reserved_tags=0 ./dm-0/hctx0/tags:active_queues=0 ./dm-0/hctx0/tags: ./dm-0/hctx0/tags:bitmap_tags: ./dm-0/hctx0/tags:depth=2048 ./dm-0/hctx0/tags:busy=0 ./dm-0/hctx0/tags:bits_per_word=64 ./dm-0/hctx0/tags:map_nr=32 ./dm-0/hctx0/tags:alloc_hint={1173, 1146, 1156, 1172} ./dm-0/hctx0/tags:wake_batch=8 ./dm-0/hctx0/tags:wake_index=0 ./dm-0/hctx0/tags:ws={ ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./dm-0/hctx0/tags:} ./dm-0/hctx0/tags:round_robin=0 ./dm-0/hctx0/ctx_map:00000000: 00 ./dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE ./dm-0/hctx0/state: ./dm-0/write_hints:hint0: 0 ./dm-0/write_hints:hint1: 0 ./dm-0/write_hints:hint2: 0 ./dm-0/write_hints:hint3: 0 ./dm-0/write_hints:hint4: 0 ./dm-0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|STATS|REGISTERED ./dm-0/poll_stat:read (512 Bytes): samples=0 ./dm-0/poll_stat:write (512 Bytes): samples=0 ./dm-0/poll_stat:read (1024 Bytes): samples=0 ./dm-0/poll_stat:write (1024 Bytes): samples=0 ./dm-0/poll_stat:read (2048 Bytes): samples=0 ./dm-0/poll_stat:write (2048 Bytes): samples=0 ./dm-0/poll_stat:read (4096 Bytes): samples=0 ./dm-0/poll_stat:write (4096 Bytes): samples=0 ./dm-0/poll_stat:read (8192 Bytes): samples=0 ./dm-0/poll_stat:write (8192 Bytes): samples=0 ./dm-0/poll_stat:read (16384 Bytes): samples=0 ./dm-0/poll_stat:write (16384 Bytes): samples=0 ./dm-0/poll_stat:read (32768 Bytes): samples=0 ./dm-0/poll_stat:write (32768 Bytes): samples=0 ./dm-0/poll_stat:read (65536 Bytes): samples=0 ./dm-0/poll_stat:write (65536 Bytes): samples=0 ./sdb/hctx0/cpu3/completed:118 11 ./sdb/hctx0/cpu3/merged:0 ./sdb/hctx0/cpu3/dispatched:118 11 ./sdb/hctx0/cpu2/completed:150 47 ./sdb/hctx0/cpu2/merged:0 ./sdb/hctx0/cpu2/dispatched:150 47 ./sdb/hctx0/cpu1/completed:119 9 ./sdb/hctx0/cpu1/merged:0 ./sdb/hctx0/cpu1/dispatched:119 9 ./sdb/hctx0/cpu0/completed:46 17 ./sdb/hctx0/cpu0/merged:0 ./sdb/hctx0/cpu0/dispatched:46 17 ./sdb/hctx0/active:0 ./sdb/hctx0/run:511 ./sdb/hctx0/queued:517 ./sdb/hctx0/dispatched: 0 0 ./sdb/hctx0/dispatched: 1 517 ./sdb/hctx0/dispatched: 2 0 ./sdb/hctx0/dispatched: 4 0 ./sdb/hctx0/dispatched: 8 0 ./sdb/hctx0/dispatched: 16 0 ./sdb/hctx0/dispatched: 32+ 0 ./sdb/hctx0/io_poll:considered=0 ./sdb/hctx0/io_poll:invoked=0 ./sdb/hctx0/io_poll:success=0 ./sdb/hctx0/sched_tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./sdb/hctx0/sched_tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ./sdb/hctx0/sched_tags:nr_tags=256 ./sdb/hctx0/sched_tags:nr_reserved_tags=0 ./sdb/hctx0/sched_tags:active_queues=0 ./sdb/hctx0/sched_tags: ./sdb/hctx0/sched_tags:bitmap_tags: ./sdb/hctx0/sched_tags:depth=256 ./sdb/hctx0/sched_tags:busy=0 ./sdb/hctx0/sched_tags:bits_per_word=64 ./sdb/hctx0/sched_tags:map_nr=4 ./sdb/hctx0/sched_tags:alloc_hint={47, 230, 128, 219} ./sdb/hctx0/sched_tags:wake_batch=8 ./sdb/hctx0/sched_tags:wake_index=0 ./sdb/hctx0/sched_tags:ws={ ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/sched_tags:} ./sdb/hctx0/sched_tags:round_robin=0 ./sdb/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./sdb/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 ./sdb/hctx0/tags:nr_tags=192 ./sdb/hctx0/tags:nr_reserved_tags=0 ./sdb/hctx0/tags:active_queues=0 ./sdb/hctx0/tags: ./sdb/hctx0/tags:bitmap_tags: ./sdb/hctx0/tags:depth=192 ./sdb/hctx0/tags:busy=0 ./sdb/hctx0/tags:bits_per_word=32 ./sdb/hctx0/tags:map_nr=6 ./sdb/hctx0/tags:alloc_hint={112, 51, 6, 134} ./sdb/hctx0/tags:wake_batch=8 ./sdb/hctx0/tags:wake_index=0 ./sdb/hctx0/tags:ws={ ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sdb/hctx0/tags:} ./sdb/hctx0/tags:round_robin=0 ./sdb/hctx0/ctx_map:00000000: 00 ./sdb/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE ./sdb/hctx0/state: ./sdb/sched/starved:0 ./sdb/sched/batching:1 ./sdb/write_hints:hint0: 0 ./sdb/write_hints:hint1: 0 ./sdb/write_hints:hint2: 0 ./sdb/write_hints:hint3: 0 ./sdb/write_hints:hint4: 0 ./sdb/state:BIDI|SAME_COMP|NONROT|IO_STAT|INIT_DONE|POLL|WC|FUA|REGISTERED|SCSI_PASSTHROUGH ./sdb/poll_stat:read (512 Bytes): samples=0 ./sdb/poll_stat:write (512 Bytes): samples=0 ./sdb/poll_stat:read (1024 Bytes): samples=0 ./sdb/poll_stat:write (1024 Bytes): samples=0 ./sdb/poll_stat:read (2048 Bytes): samples=0 ./sdb/poll_stat:write (2048 Bytes): samples=0 ./sdb/poll_stat:read (4096 Bytes): samples=0 ./sdb/poll_stat:write (4096 Bytes): samples=0 ./sdb/poll_stat:read (8192 Bytes): samples=0 ./sdb/poll_stat:write (8192 Bytes): samples=0 ./sdb/poll_stat:read (16384 Bytes): samples=0 ./sdb/poll_stat:write (16384 Bytes): samples=0 ./sdb/poll_stat:read (32768 Bytes): samples=0 ./sdb/poll_stat:write (32768 Bytes): samples=0 ./sdb/poll_stat:read (65536 Bytes): samples=0 ./sdb/poll_stat:write (65536 Bytes): samples=0 ./sda/hctx0/cpu3/completed:6 9 ./sda/hctx0/cpu3/merged:0 ./sda/hctx0/cpu3/dispatched:6 9 ./sda/hctx0/cpu2/completed:41 46 ./sda/hctx0/cpu2/merged:0 ./sda/hctx0/cpu2/dispatched:41 46 ./sda/hctx0/cpu1/completed:40 11 ./sda/hctx0/cpu1/merged:0 ./sda/hctx0/cpu1/dispatched:40 11 ./sda/hctx0/cpu0/completed:3 32 ./sda/hctx0/cpu0/merged:0 ./sda/hctx0/cpu0/dispatched:3 32 ./sda/hctx0/active:0 ./sda/hctx0/run:161 ./sda/hctx0/queued:188 ./sda/hctx0/dispatched: 0 0 ./sda/hctx0/dispatched: 1 188 ./sda/hctx0/dispatched: 2 0 ./sda/hctx0/dispatched: 4 0 ./sda/hctx0/dispatched: 8 0 ./sda/hctx0/dispatched: 16 0 ./sda/hctx0/dispatched: 32+ 0 ./sda/hctx0/io_poll:considered=0 ./sda/hctx0/io_poll:invoked=0 ./sda/hctx0/io_poll:success=0 ./sda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./sda/hctx0/tags:nr_tags=128 ./sda/hctx0/tags:nr_reserved_tags=0 ./sda/hctx0/tags:active_queues=0 ./sda/hctx0/tags: ./sda/hctx0/tags:bitmap_tags: ./sda/hctx0/tags:depth=128 ./sda/hctx0/tags:busy=0 ./sda/hctx0/tags:bits_per_word=32 ./sda/hctx0/tags:map_nr=4 ./sda/hctx0/tags:alloc_hint={112, 68, 4, 98} ./sda/hctx0/tags:wake_batch=8 ./sda/hctx0/tags:wake_index=0 ./sda/hctx0/tags:ws={ ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./sda/hctx0/tags:} ./sda/hctx0/tags:round_robin=0 ./sda/hctx0/ctx_map:00000000: 00 ./sda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE ./sda/hctx0/state: ./sda/write_hints:hint0: 0 ./sda/write_hints:hint1: 0 ./sda/write_hints:hint2: 0 ./sda/write_hints:hint3: 0 ./sda/write_hints:hint4: 0 ./sda/state:SAME_COMP|IO_STAT|DISCARD|INIT_DONE|POLL|WC|REGISTERED|SCSI_PASSTHROUGH ./sda/poll_stat:read (512 Bytes): samples=0 ./sda/poll_stat:write (512 Bytes): samples=0 ./sda/poll_stat:read (1024 Bytes): samples=0 ./sda/poll_stat:write (1024 Bytes): samples=0 ./sda/poll_stat:read (2048 Bytes): samples=0 ./sda/poll_stat:write (2048 Bytes): samples=0 ./sda/poll_stat:read (4096 Bytes): samples=0 ./sda/poll_stat:write (4096 Bytes): samples=0 ./sda/poll_stat:read (8192 Bytes): samples=0 ./sda/poll_stat:write (8192 Bytes): samples=0 ./sda/poll_stat:read (16384 Bytes): samples=0 ./sda/poll_stat:write (16384 Bytes): samples=0 ./sda/poll_stat:read (32768 Bytes): samples=0 ./sda/poll_stat:write (32768 Bytes): samples=0 ./sda/poll_stat:read (65536 Bytes): samples=0 ./sda/poll_stat:write (65536 Bytes): samples=0 ./vda/hctx0/cpu3/completed:1348 162 ./vda/hctx0/cpu3/merged:4 ./vda/hctx0/cpu3/dispatched:1348 162 ./vda/hctx0/cpu2/completed:1088 46 ./vda/hctx0/cpu2/merged:7 ./vda/hctx0/cpu2/dispatched:1088 46 ./vda/hctx0/cpu1/completed:2092 20 ./vda/hctx0/cpu1/merged:0 ./vda/hctx0/cpu1/dispatched:2092 20 ./vda/hctx0/cpu0/completed:2114 79 ./vda/hctx0/cpu0/merged:16 ./vda/hctx0/cpu0/dispatched:2114 79 ./vda/hctx0/active:0 ./vda/hctx0/run:6902 ./vda/hctx0/queued:6949 ./vda/hctx0/dispatched: 0 0 ./vda/hctx0/dispatched: 1 6810 ./vda/hctx0/dispatched: 2 42 ./vda/hctx0/dispatched: 4 19 ./vda/hctx0/dispatched: 8 16 ./vda/hctx0/dispatched: 16 8 ./vda/hctx0/dispatched: 32+ 0 ./vda/hctx0/io_poll:considered=0 ./vda/hctx0/io_poll:invoked=0 ./vda/hctx0/io_poll:success=0 ./vda/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ./vda/hctx0/tags:nr_tags=128 ./vda/hctx0/tags:nr_reserved_tags=0 ./vda/hctx0/tags:active_queues=0 ./vda/hctx0/tags: ./vda/hctx0/tags:bitmap_tags: ./vda/hctx0/tags:depth=128 ./vda/hctx0/tags:busy=0 ./vda/hctx0/tags:bits_per_word=32 ./vda/hctx0/tags:map_nr=4 ./vda/hctx0/tags:alloc_hint={97, 74, 111, 53} ./vda/hctx0/tags:wake_batch=8 ./vda/hctx0/tags:wake_index=0 ./vda/hctx0/tags:ws={ ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags: {.wait_cnt=8, .wait=inactive}, ./vda/hctx0/tags:} ./vda/hctx0/tags:round_robin=0 ./vda/hctx0/ctx_map:00000000: 00 ./vda/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE ./vda/hctx0/state: ./vda/write_hints:hint0: 0 ./vda/write_hints:hint1: 0 ./vda/write_hints:hint2: 0 ./vda/write_hints:hint3: 0 ./vda/write_hints:hint4: 0 ./vda/state:SAME_COMP|IO_STAT|INIT_DONE|NO_SG_MERGE|POLL|WC|REGISTERED ./vda/poll_stat:read (512 Bytes): samples=0 ./vda/poll_stat:write (512 Bytes): samples=0 ./vda/poll_stat:read (1024 Bytes): samples=0 ./vda/poll_stat:write (1024 Bytes): samples=0 ./vda/poll_stat:read (2048 Bytes): samples=0 ./vda/poll_stat:write (2048 Bytes): samples=0 ./vda/poll_stat:read (4096 Bytes): samples=0 ./vda/poll_stat:write (4096 Bytes): samples=0 ./vda/poll_stat:read (8192 Bytes): samples=0 ./vda/poll_stat:write (8192 Bytes): samples=0 ./vda/poll_stat:read (16384 Bytes): samples=0 ./vda/poll_stat:write (16384 Bytes): samples=0 ./vda/poll_stat:read (32768 Bytes): samples=0 ./vda/poll_stat:write (32768 Bytes): samples=0 ./vda/poll_stat:read (65536 Bytes): samples=0 ./vda/poll_stat:write (65536 Bytes): samples=0 ./sr0/hctx0/cpu3/completed:7 10 ./sr0/hctx0/cpu3/merged:0 ./sr0/hctx0/cpu3/dispatched:7 10 ./sr0/hctx0/cpu2/completed:39 299 ./sr0/hctx0/cpu2/merged:0 ./sr0/hctx0/cpu2/dispatched:39 299 ./sr0/hctx0/cpu1/completed:0 37 ./sr0/hctx0/cpu1/merged:0 ./sr0/hctx0/cpu1/dispatched:0 37 ./sr0/hctx0/cpu0/completed:0 26 ./sr0/hctx0/cpu0/merged:0 ./sr0/hctx0/cpu0/dispatched:0 26 ./sr0/hctx0/active:0 ./sr0/hctx0/run:418 ./sr0/hctx0/queued:418 ./sr0/hctx0/dispatched: 0 0 ./sr0/hctx0/dispatched: 1 418 ./sr0/hctx0/dispatched: 2 0 ./sr0/hctx0/dispatched: 4 0 ./sr0/hctx0/dispatched: 8 0 ./sr0/hctx0/dispatched: 16 0 ./sr0/hctx0/dispatched: 32+ 0 ./sr0/hctx0/io_poll:considered=0 ./sr0/hctx0/io_poll:invoked=0 ./sr0/hctx0/io_poll:success=0 ./sr0/hctx0/tags_bitmap:00000000: 00 ./sr0/hctx0/tags:nr_tags=1 ./sr0/hctx0/tags:nr_reserved_tags=0 ./sr0/hctx0/tags:active_queues=0 ./sr0/hctx0/tags: ./sr0/hctx0/tags:bitmap_tags: ./sr0/hctx0/tags:depth=1 ./sr0/hctx0/tags:busy=0 ./sr0/hctx0/tags:bits_per_word=64 ./sr0/hctx0/tags:map_nr=1 ./sr0/hctx0/tags:alloc_hint={0, 0, 0, 0} ./sr0/hctx0/tags:wake_batch=1 ./sr0/hctx0/tags:wake_index=7 ./sr0/hctx0/tags:ws={ ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags: {.wait_cnt=1, .wait=inactive}, ./sr0/hctx0/tags:} ./sr0/hctx0/tags:round_robin=1 ./sr0/hctx0/ctx_map:00000000: 00 ./sr0/hctx0/flags:alloc_policy=RR SHOULD_MERGE|SG_MERGE ./sr0/hctx0/state: ./sr0/write_hints:hint0: 0 ./sr0/write_hints:hint1: 0 ./sr0/write_hints:hint2: 0 ./sr0/write_hints:hint3: 0 ./sr0/write_hints:hint4: 0 ./sr0/state:SAME_COMP|IO_STAT|INIT_DONE|POLL|FLUSH_NQ|REGISTERED|SCSI_PASSTHROUGH ./sr0/poll_stat:read (512 Bytes): samples=0 ./sr0/poll_stat:write (512 Bytes): samples=0 ./sr0/poll_stat:read (1024 Bytes): samples=0 ./sr0/poll_stat:write (1024 Bytes): samples=0 ./sr0/poll_stat:read (2048 Bytes): samples=0 ./sr0/poll_stat:write (2048 Bytes): samples=0 ./sr0/poll_stat:read (4096 Bytes): samples=0 ./sr0/poll_stat:write (4096 Bytes): samples=0 ./sr0/poll_stat:read (8192 Bytes): samples=0 ./sr0/poll_stat:write (8192 Bytes): samples=0 ./sr0/poll_stat:read (16384 Bytes): samples=0 ./sr0/poll_stat:write (16384 Bytes): samples=0 ./sr0/poll_stat:read (32768 Bytes): samples=0 ./sr0/poll_stat:write (32768 Bytes): samples=0 ./sr0/poll_stat:read (65536 Bytes): samples=0 ./sr0/poll_stat:write (65536 Bytes): samples=0 -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Wed, Jan 17 2018 at 4:27pm -0500, Bart Van Assche <Bart.VanAssche@wdc.com> wrote: > On Wed, 2018-01-17 at 15:14 -0500, Mike Snitzer wrote: > > BUT my broader point stands: you aren't testing the dm-4.16 changes. By > > just reverting that commit you're creating a self-fulfilling prophecy > > (that you'll see hangs without it). > > > > Fact is you should pull all of dm-4.16 in, see: > > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.16 > > > > But these dm-4.16 changes are particularly important: > > 050af08ffb1b dm mpath: return DM_MAPIO_REQUEUE on blk-mq rq allocation failure > > 459b54019cfe dm mpath: return DM_MAPIO_DELAY_REQUEUE if QUEUE_IO or PG_INIT_REQUIRED > > ec3eaf9a6731 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE > > 4dd6edd23e7e dm mpath: delay the retry of a request if the target responded as busy > > > > This last one is the commit that _should_ serve as a proper replacement > > for the change you manually reverted in your branch. > > > > Please re-test after pulling in dm-4.16 and let us know how things fair. > > Hello Mike, > > If I replace the patch I referred to in my previous e-mail with your dm-4.16 > branch then I see the following: > * Without I/O scheduler: dm path removal at the end of the test fails. This > succeeded reliably in the past so I think this is a regression: > # srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq > [ ... ] > Unmounting /root/mnt1 from /dev/mapper/mpathb > SRP LUN /sys/class/scsi_device/4:0:0:0 / sdc: removing /dev/dm-1: done > SRP LUN /sys/class/scsi_device/4:0:0:1 / sde: removing /dev/dm-2: done > SRP LUN /sys/class/scsi_device/4:0:0:2 / sdd: removing /dev/dm-0: dm=$(dev_to_mpath "/dev/dm-0"): failed > [ ... ] So no IO hangs? Just removal of a dm device fails at the end? Anything in the kernel log that might give a hint as to why? I'll need to appreciate what the test is doing. Like why is a single SRP scsi device being used to create a dm device? What type of DM device? > * With the Kyber I/O scheduler: I/O hangs. > # srp-test/run_tests -c -d -r 10 -q 1 -t 02-mq -e kyber > [ ... ] > Using /dev/disk/by-id/dm-uuid-mpath-3600140572616d6469736b31000000000 -> ../../dm-0 > (hangs) Again this says little to me. But hopefully I'll find time to dig in further and in parallel Laurence will be able to reproduce on his testbed. How critical is it to have the latest SCSI changes that are queued for 4.16? Thanks, Mike -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Wed, 2018-01-17 at 16:37 -0500, Mike Snitzer wrote: > How critical is it to have the latest SCSI changes that are queued for > 4.16? Hello Mike, The IB tests I ran do not depend on any recent SCSI initiator (drivers/scsi) or SRP driver changes. However, if you want to run the SRP drivers on top of the rdma_rxe driver then you will need the latest version of the SRP initiator and target drivers, including some patches that are not yet upstream (but should go upstream soon). The test I ran may depend on the SCSI target (drivers/target) patches in my tree. The SCSI target patches in my tree significantly simplify the session shutdown code. The patch for simplification of session shutdown may be a bug fix. However, the original code is so hairy that I have not yet tried to figure out whether or not my changes improve the behavior of the session shutdown code. Bart. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 85d80b8ccc33..a9e0f6ec62b6 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -773,7 +773,6 @@ static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx, /* Undo dm_start_request() before requeuing */ rq_end_stats(md, rq); rq_completed(md, rq_data_dir(rq), false); - blk_mq_delay_run_hw_queue(hctx, 100/*ms*/); return BLK_STS_RESOURCE; }