[v1,2/3] blk-mq: comment on races related with timeout handler
diff mbox

Message ID 20170317095711.5819-3-tom.leiming@gmail.com
State New
Headers show

Commit Message

Ming Lei March 17, 2017, 9:57 a.m. UTC
This patch adds comment on two races related with
timeout handler:

- requeue from queue busy vs. timeout
- rq free & reallocation vs. timeout

Both the races themselves and current solution aren't
explicit enough, so add comments on them.

Cc: Bart Van Assche <bart.vanassche@sandisk.com>
Signed-off-by: Ming Lei <tom.leiming@gmail.com>
---
 block/blk-mq.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

Comments

Bart Van Assche March 17, 2017, 5:39 p.m. UTC | #1
On Fri, 2017-03-17 at 17:57 +0800, Ming Lei wrote:
> +/*
> + * When we reach here because queue is busy, REQ_ATOM_COMPLETE
> + * flag isn't set yet, so there may be race with timeout hanlder,
> + * but given rq->deadline is just set in .queue_rq() under
> + * this sitation, the race won't be possible in reality because
> + * rq->timeout should be set as big enough to cover the window
> + * between blk_mq_start_request() called from .queue_rq() and
> + * clearing REQ_ATOM_STARTED here.
> + */
>  static void __blk_mq_requeue_request(struct request *rq)
>  {
>  	struct request_queue *q = rq->q;
> @@ -700,6 +709,19 @@ static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx,
>  	if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
>  		return;
>  
> +	/*
> +	 * The rq being checked may have been freed and reallocated
> +	 * out already here, we avoid this race by checking rq->deadline
> +	 * and REQ_ATOM_COMPLETE flag together:
> +	 *
> +	 * - if rq->deadline is observed as new value because of
> +	 *   reusing, the rq won't be timed out because of timing.
> +	 * - if rq->deadline is observed as previous value,
> +	 *   REQ_ATOM_COMPLETE flag won't be cleared in reuse path
> +	 *   because we put a barrier between setting rq->deadline
> +	 *   and clearing the flag in blk_mq_start_request(), so
> +	 *   this rq won't be timed out too.
> +	 */
>  	if (time_after_eq(jiffies, rq->deadline)) {
>  		if (!blk_mark_rq_complete(rq))
>  			blk_mq_rq_timed_out(rq, reserved);

Since this explanation applies to the same race addressed by patch 1/3,
please consider squashing this patch into patch 1/3.

Thanks,

Bart.
Ming Lei March 17, 2017, 6:23 p.m. UTC | #2
On Sat, Mar 18, 2017 at 1:39 AM, Bart Van Assche
<Bart.VanAssche@sandisk.com> wrote:
> On Fri, 2017-03-17 at 17:57 +0800, Ming Lei wrote:
>> +/*
>> + * When we reach here because queue is busy, REQ_ATOM_COMPLETE
>> + * flag isn't set yet, so there may be race with timeout hanlder,
>> + * but given rq->deadline is just set in .queue_rq() under
>> + * this sitation, the race won't be possible in reality because
>> + * rq->timeout should be set as big enough to cover the window
>> + * between blk_mq_start_request() called from .queue_rq() and
>> + * clearing REQ_ATOM_STARTED here.
>> + */
>>  static void __blk_mq_requeue_request(struct request *rq)
>>  {
>>       struct request_queue *q = rq->q;
>> @@ -700,6 +709,19 @@ static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx,
>>       if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
>>               return;
>>
>> +     /*
>> +      * The rq being checked may have been freed and reallocated
>> +      * out already here, we avoid this race by checking rq->deadline
>> +      * and REQ_ATOM_COMPLETE flag together:
>> +      *
>> +      * - if rq->deadline is observed as new value because of
>> +      *   reusing, the rq won't be timed out because of timing.
>> +      * - if rq->deadline is observed as previous value,
>> +      *   REQ_ATOM_COMPLETE flag won't be cleared in reuse path
>> +      *   because we put a barrier between setting rq->deadline
>> +      *   and clearing the flag in blk_mq_start_request(), so
>> +      *   this rq won't be timed out too.
>> +      */
>>       if (time_after_eq(jiffies, rq->deadline)) {
>>               if (!blk_mark_rq_complete(rq))
>>                       blk_mq_rq_timed_out(rq, reserved);
>
> Since this explanation applies to the same race addressed by patch 1/3,

First, this explains how we deal with the race of reuse vs. timeout, and 1/3
fixes another race or rq corruption. Did you see anywhere I mentioned STARTED
flag in above comment?

In case of 1/3, the rq to be dispatched can be destroyed simply by the
blk_mq_end_request() from timeout. Or even it can survive, the same rq
can be allocated into another I/O path, and this situation is different with
reuse vs. timeout. And I can't see any help from the comment for explaining
1/3's issue, can you? Maybe I need to mention rq corruption in 1/3 explicitly.

Secondly introducing this comment to 1/3 just causes unnecessary
backporting burden, as we have to make it into -stable.

> please consider squashing this patch into patch 1/3.

So please do not consider that.

Thanks,
Ming Lei
Hannes Reinecke March 18, 2017, 11:25 a.m. UTC | #3
On 03/17/2017 10:57 AM, Ming Lei wrote:
> This patch adds comment on two races related with
> timeout handler:
> 
> - requeue from queue busy vs. timeout
> - rq free & reallocation vs. timeout
> 
> Both the races themselves and current solution aren't
> explicit enough, so add comments on them.
> 
> Cc: Bart Van Assche <bart.vanassche@sandisk.com>
> Signed-off-by: Ming Lei <tom.leiming@gmail.com>
> ---
>  block/blk-mq.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 

Reviewed-by: Hannes Reinecke <hare@suse.com>

Cheers,

Hannes

Patch
diff mbox

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 08a49c69738b..7068779d3bac 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -527,6 +527,15 @@  void blk_mq_start_request(struct request *rq)
 }
 EXPORT_SYMBOL(blk_mq_start_request);
 
+/*
+ * When we reach here because queue is busy, REQ_ATOM_COMPLETE
+ * flag isn't set yet, so there may be race with timeout hanlder,
+ * but given rq->deadline is just set in .queue_rq() under
+ * this sitation, the race won't be possible in reality because
+ * rq->timeout should be set as big enough to cover the window
+ * between blk_mq_start_request() called from .queue_rq() and
+ * clearing REQ_ATOM_STARTED here.
+ */
 static void __blk_mq_requeue_request(struct request *rq)
 {
 	struct request_queue *q = rq->q;
@@ -700,6 +709,19 @@  static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx,
 	if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
 		return;
 
+	/*
+	 * The rq being checked may have been freed and reallocated
+	 * out already here, we avoid this race by checking rq->deadline
+	 * and REQ_ATOM_COMPLETE flag together:
+	 *
+	 * - if rq->deadline is observed as new value because of
+	 *   reusing, the rq won't be timed out because of timing.
+	 * - if rq->deadline is observed as previous value,
+	 *   REQ_ATOM_COMPLETE flag won't be cleared in reuse path
+	 *   because we put a barrier between setting rq->deadline
+	 *   and clearing the flag in blk_mq_start_request(), so
+	 *   this rq won't be timed out too.
+	 */
 	if (time_after_eq(jiffies, rq->deadline)) {
 		if (!blk_mark_rq_complete(rq))
 			blk_mq_rq_timed_out(rq, reserved);