Message ID | 1553068921-6605-1-git-send-email-zhengbin13@huawei.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | blk-mq: fix a hung issue when set device state to blocked and restore running | expand |
On Wed, Mar 20, 2019 at 04:02:01PM +0800, zhengbin wrote: > When I use dd test a SCSI device which use blk-mq in the following steps: > 1.echo "blocked" >/sys/block/sda/device/state > 2.dd if=/dev/sda of=/mnt/t.log bs=1M count=10 > 3.echo "running" >/sys/block/sda/device/state > dd should finish this work after step 3, unfortunately, still hung. > > After step2, the key code process is like this: > blk_mq_dispatch_rq_list-->scsi_queue_rq-->prep_to_mq > -->if ret is BLK_STS_RESOURCE, delay run hw queue > > prep_to_mq will return BLK_STS_RESOURCE, and scsi_queue_rq will transter > it to BLK_STS_DEV_RESOURCE. In this situtation, we should delay run hw BLK_STS_DEV_RESOURCE means that the driver will rerun hw queue, so maybe you need to investigate why it is returned from scsi driver first. BTW, I'd suggest you read the big comment on BLK_STS_DEV_RESOURCE first. Thanks, Ming
Hi Bin On 3/20/19 4:02 PM, zhengbin wrote: > When I use dd test a SCSI device which use blk-mq in the following steps: > 1.echo "blocked" >/sys/block/sda/device/state > 2.dd if=/dev/sda of=/mnt/t.log bs=1M count=10 > 3.echo "running" >/sys/block/sda/device/state > dd should finish this work after step 3, unfortunately, still hung. If this test case really matters for you, we should try to run the hw queues after set state to 'running'. Thanks Jianchao > > After step2, the key code process is like this: > blk_mq_dispatch_rq_list-->scsi_queue_rq-->prep_to_mq > -->if ret is BLK_STS_RESOURCE, delay run hw queue > > prep_to_mq will return BLK_STS_RESOURCE, and scsi_queue_rq will transter > it to BLK_STS_DEV_RESOURCE. In this situtation, we should delay run hw > queue. This patch fixes that. > > Fixes: 86ff7c2a80cd ("blk-mq: introduce BLK_STS_DEV_RESOURCE") > Signed-off-by: zhengbin <zhengbin13@huawei.com> > --- > block/blk-mq.c | 10 ++++++---- > 1 file changed, 6 insertions(+), 4 deletions(-) > > diff --git a/block/blk-mq.c b/block/blk-mq.c > index a9c1816..556d606 100644 > --- a/block/blk-mq.c > +++ b/block/blk-mq.c > @@ -1309,15 +1309,17 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list, > * returning BLK_STS_RESOURCE. Two exceptions are scsi-mq > * and dm-rq. > * > - * If driver returns BLK_STS_RESOURCE and SCHED_RESTART > - * bit is set, run queue after a delay to avoid IO stalls > - * that could otherwise occur if the queue is idle. > + * If driver returns BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE > + * and SCHED_RESTART bit is set, run queue after a delay to > + * avoid IO stalls that could otherwise occur if the queue > + * is idle. > */ > needs_restart = blk_mq_sched_needs_restart(hctx); > if (!needs_restart || > (no_tag && list_empty_careful(&hctx->dispatch_wait.entry))) > blk_mq_run_hw_queue(hctx, true); > - else if (needs_restart && (ret == BLK_STS_RESOURCE)) > + else if (needs_restart && ((ret == BLK_STS_RESOURCE) || > + (ret == BLK_STS_DEV_RESOURCE))) > blk_mq_delay_run_hw_queue(hctx, BLK_MQ_RESOURCE_DELAY); > > blk_mq_update_dispatch_busy(hctx, true); > -- > 2.7.4 > >
Thanks for your quick reply, I will study BLK_STS_DEV_RESOURCE in detail > BLK_STS_DEV_RESOURCE means that the driver will rerun hw queue, so > maybe you need to investigate why it is returned from scsi driver first. because we set the device state to blocked, scsi_queue_rq-->prep_to_mq(return BLK_STS_RESOURCE) -->out_put_budget transfer BLK_STS_RESOURCE to BLK_STS_DEV_RESOURCE In this situtation, the request does not send to the driver. If the device use sq, when we we set the device state to blocked and test dd, it will continue to call blk_delay_queue. If this test case really matters for you, we should try to run the hw queues after set state to 'running'. --->Maybe we should call blk_mq_run_hw_queue in scsi_device_set_state? On 2019/3/20 16:11, Ming Lei wrote: > On Wed, Mar 20, 2019 at 04:02:01PM +0800, zhengbin wrote: >> When I use dd test a SCSI device which use blk-mq in the following steps: >> 1.echo "blocked" >/sys/block/sda/device/state >> 2.dd if=/dev/sda of=/mnt/t.log bs=1M count=10 >> 3.echo "running" >/sys/block/sda/device/state >> dd should finish this work after step 3, unfortunately, still hung. >> >> After step2, the key code process is like this: >> blk_mq_dispatch_rq_list-->scsi_queue_rq-->prep_to_mq >> -->if ret is BLK_STS_RESOURCE, delay run hw queue >> >> prep_to_mq will return BLK_STS_RESOURCE, and scsi_queue_rq will transter >> it to BLK_STS_DEV_RESOURCE. In this situtation, we should delay run hw > > BLK_STS_DEV_RESOURCE means that the driver will rerun hw queue, so > maybe you need to investigate why it is returned from scsi driver first. > > BTW, I'd suggest you read the big comment on BLK_STS_DEV_RESOURCE first. > > Thanks, > Ming > > . >
On Wed, Mar 20, 2019 at 04:52:40PM +0800, zhengbin (A) wrote: > Thanks for your quick reply, I will study BLK_STS_DEV_RESOURCE in detail > > > BLK_STS_DEV_RESOURCE means that the driver will rerun hw queue, so > > maybe you need to investigate why it is returned from scsi driver first. > because we set the device state to blocked, > scsi_queue_rq-->prep_to_mq(return BLK_STS_RESOURCE) > -->out_put_budget transfer BLK_STS_RESOURCE to BLK_STS_DEV_RESOURCE > In this situtation, the request does not send to the driver. Then the queue will be run when the scsi_device becomes un-blocked, see scsi_internal_device_unblock_nowait(). Thanks, Ming
diff --git a/block/blk-mq.c b/block/blk-mq.c index a9c1816..556d606 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1309,15 +1309,17 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list, * returning BLK_STS_RESOURCE. Two exceptions are scsi-mq * and dm-rq. * - * If driver returns BLK_STS_RESOURCE and SCHED_RESTART - * bit is set, run queue after a delay to avoid IO stalls - * that could otherwise occur if the queue is idle. + * If driver returns BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE + * and SCHED_RESTART bit is set, run queue after a delay to + * avoid IO stalls that could otherwise occur if the queue + * is idle. */ needs_restart = blk_mq_sched_needs_restart(hctx); if (!needs_restart || (no_tag && list_empty_careful(&hctx->dispatch_wait.entry))) blk_mq_run_hw_queue(hctx, true); - else if (needs_restart && (ret == BLK_STS_RESOURCE)) + else if (needs_restart && ((ret == BLK_STS_RESOURCE) || + (ret == BLK_STS_DEV_RESOURCE))) blk_mq_delay_run_hw_queue(hctx, BLK_MQ_RESOURCE_DELAY); blk_mq_update_dispatch_busy(hctx, true);
When I use dd test a SCSI device which use blk-mq in the following steps: 1.echo "blocked" >/sys/block/sda/device/state 2.dd if=/dev/sda of=/mnt/t.log bs=1M count=10 3.echo "running" >/sys/block/sda/device/state dd should finish this work after step 3, unfortunately, still hung. After step2, the key code process is like this: blk_mq_dispatch_rq_list-->scsi_queue_rq-->prep_to_mq -->if ret is BLK_STS_RESOURCE, delay run hw queue prep_to_mq will return BLK_STS_RESOURCE, and scsi_queue_rq will transter it to BLK_STS_DEV_RESOURCE. In this situtation, we should delay run hw queue. This patch fixes that. Fixes: 86ff7c2a80cd ("blk-mq: introduce BLK_STS_DEV_RESOURCE") Signed-off-by: zhengbin <zhengbin13@huawei.com> --- block/blk-mq.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) -- 2.7.4