diff mbox series

[1/2] xenbus: req->body should be updated before req->state

Message ID 20200303015859.18813-1-dongli.zhang@oracle.com (mailing list archive)
State Superseded
Headers show
Series [1/2] xenbus: req->body should be updated before req->state | expand

Commit Message

Dongli Zhang March 3, 2020, 1:58 a.m. UTC
The req->body should be updated before req->state is updated and the
order should be guaranteed by a barrier.

Otherwise, read_reply() might return req->body = NULL.

Below is sample callstack when the issue is reproduced on purpose by
reordering the updates of req->body and req->state and adding delay in
code between updates of req->state and req->body.

[   22.356105] general protection fault: 0000 [#1] SMP PTI
[   22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6
[   22.366727] Hardware name: Xen HVM domU, BIOS ...
[   22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
... ...
[   22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246
[   22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000
[   22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d
[   22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd
[   22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000
[   22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000
[   22.423726] FS:  0000000000000000(0000) GS:ffff9d953bc80000(0000) knlGS:0000000000000000
[   22.429898] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0
[   22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   22.450342] Call Trace:
[   22.452509]  simple_strtoull+0x27/0x70
[   22.455572]  xenbus_transaction_start+0x31/0x50
[   22.459104]  netback_changed+0x76c/0xcc1 [xen_netfront]
[   22.463279]  ? find_watch+0x40/0x40
[   22.466156]  xenwatch_thread+0xb4/0x150
[   22.469309]  ? wait_woken+0x80/0x80
[   22.472198]  kthread+0x10e/0x130
[   22.474925]  ? kthread_park+0x80/0x80
[   22.477946]  ret_from_fork+0x35/0x40
[   22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront xen_blkfront
[   22.486783] ---[ end trace a9222030a747c3f7 ]---
[   22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60

The "while" is changed to "do while" so that wait_event() is used as a
barrier.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 drivers/xen/xenbus/xenbus_comms.c | 2 ++
 drivers/xen/xenbus/xenbus_xs.c    | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

Comments

Julien Grall March 3, 2020, 9:40 a.m. UTC | #1
Hi,

On 03/03/2020 01:58, Dongli Zhang wrote:
> The req->body should be updated before req->state is updated and the
> order should be guaranteed by a barrier.
> 
> Otherwise, read_reply() might return req->body = NULL.
> 
> Below is sample callstack when the issue is reproduced on purpose by
> reordering the updates of req->body and req->state and adding delay in
> code between updates of req->state and req->body.
> 
> [   22.356105] general protection fault: 0000 [#1] SMP PTI
> [   22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6
> [   22.366727] Hardware name: Xen HVM domU, BIOS ...
> [   22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
> ... ...
> [   22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246
> [   22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000
> [   22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d
> [   22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd
> [   22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000
> [   22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000
> [   22.423726] FS:  0000000000000000(0000) GS:ffff9d953bc80000(0000) knlGS:0000000000000000
> [   22.429898] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0
> [   22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [   22.450342] Call Trace:
> [   22.452509]  simple_strtoull+0x27/0x70
> [   22.455572]  xenbus_transaction_start+0x31/0x50
> [   22.459104]  netback_changed+0x76c/0xcc1 [xen_netfront]
> [   22.463279]  ? find_watch+0x40/0x40
> [   22.466156]  xenwatch_thread+0xb4/0x150
> [   22.469309]  ? wait_woken+0x80/0x80
> [   22.472198]  kthread+0x10e/0x130
> [   22.474925]  ? kthread_park+0x80/0x80
> [   22.477946]  ret_from_fork+0x35/0x40
> [   22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront xen_blkfront
> [   22.486783] ---[ end trace a9222030a747c3f7 ]---
> [   22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
> 
> The "while" is changed to "do while" so that wait_event() is used as a
> barrier.

The correct barrier for read_reply() should be virt_rmb(). While on x86, 
this is equivalent to barrier(), on Arm this will be a dmb(ish) to 
prevent the processor re-ordering memory access.

Therefore the barrier in test_reply() (called by wait_event()) is not 
going to be sufficient for Arm.

> 
> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
> ---
>   drivers/xen/xenbus/xenbus_comms.c | 2 ++
>   drivers/xen/xenbus/xenbus_xs.c    | 4 ++--
>   2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
> index d239fc3c5e3d..852ed161fc2a 100644
> --- a/drivers/xen/xenbus/xenbus_comms.c
> +++ b/drivers/xen/xenbus/xenbus_comms.c
> @@ -313,6 +313,8 @@ static int process_msg(void)
>   			req->msg.type = state.msg.type;
>   			req->msg.len = state.msg.len;
>   			req->body = state.body;
> +			/* write body, then update state */
> +			virt_wmb();
>   			req->state = xb_req_state_got_reply;
>   			req->cb(req);
>   		} else
> diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c
> index ddc18da61834..f5b0a6a72ad3 100644
> --- a/drivers/xen/xenbus/xenbus_xs.c
> +++ b/drivers/xen/xenbus/xenbus_xs.c
> @@ -202,7 +202,7 @@ static bool test_reply(struct xb_req_data *req)
>   
>   static void *read_reply(struct xb_req_data *req)
>   {
> -	while (req->state != xb_req_state_got_reply) {
> +	do {
>   		wait_event(req->wq, test_reply(req));
>   
>   		if (!xenbus_ok())
> @@ -216,7 +216,7 @@ static void *read_reply(struct xb_req_data *req)
>   		if (req->err)
>   			return ERR_PTR(req->err);
>   
> -	}
> +	} while (req->state != xb_req_state_got_reply);
>   
>   	return req->body;
>   }
> 

Cheers,
Dongli Zhang March 3, 2020, 5:36 p.m. UTC | #2
On 3/3/20 1:40 AM, Julien Grall wrote:
> Hi,
> 
> On 03/03/2020 01:58, Dongli Zhang wrote:
>> The req->body should be updated before req->state is updated and the
>> order should be guaranteed by a barrier.
>>
>> Otherwise, read_reply() might return req->body = NULL.
>>
>> Below is sample callstack when the issue is reproduced on purpose by
>> reordering the updates of req->body and req->state and adding delay in
>> code between updates of req->state and req->body.
>>
>> [   22.356105] general protection fault: 0000 [#1] SMP PTI
>> [   22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6
>> [   22.366727] Hardware name: Xen HVM domU, BIOS ...
>> [   22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
>> ... ...
>> [   22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246
>> [   22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000
>> [   22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d
>> [   22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd
>> [   22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000
>> [   22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000
>> [   22.423726] FS:  0000000000000000(0000) GS:ffff9d953bc80000(0000)
>> knlGS:0000000000000000
>> [   22.429898] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [   22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0
>> [   22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [   22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [   22.450342] Call Trace:
>> [   22.452509]  simple_strtoull+0x27/0x70
>> [   22.455572]  xenbus_transaction_start+0x31/0x50
>> [   22.459104]  netback_changed+0x76c/0xcc1 [xen_netfront]
>> [   22.463279]  ? find_watch+0x40/0x40
>> [   22.466156]  xenwatch_thread+0xb4/0x150
>> [   22.469309]  ? wait_woken+0x80/0x80
>> [   22.472198]  kthread+0x10e/0x130
>> [   22.474925]  ? kthread_park+0x80/0x80
>> [   22.477946]  ret_from_fork+0x35/0x40
>> [   22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront
>> xen_blkfront
>> [   22.486783] ---[ end trace a9222030a747c3f7 ]---
>> [   22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
>>
>> The "while" is changed to "do while" so that wait_event() is used as a
>> barrier.
> 
> The correct barrier for read_reply() should be virt_rmb(). While on x86, this is
> equivalent to barrier(), on Arm this will be a dmb(ish) to prevent the processor
> re-ordering memory access.
> 
> Therefore the barrier in test_reply() (called by wait_event()) is not going to
> be sufficient for Arm.

Sorry that I just erroneously thought wait_event() would be used as read barrier.

I would change barrier() to virt_rmb() for read_reply() in v2.

Thank you very much!

Dongli Zhang
Julien Grall March 3, 2020, 6:32 p.m. UTC | #3
Hi,

On 03/03/2020 17:36, dongli.zhang@oracle.com wrote:
> 
> 
> On 3/3/20 1:40 AM, Julien Grall wrote:
>> Hi,
>>
>> On 03/03/2020 01:58, Dongli Zhang wrote:
>>> The req->body should be updated before req->state is updated and the
>>> order should be guaranteed by a barrier.
>>>
>>> Otherwise, read_reply() might return req->body = NULL.
>>>
>>> Below is sample callstack when the issue is reproduced on purpose by
>>> reordering the updates of req->body and req->state and adding delay in
>>> code between updates of req->state and req->body.
>>>
>>> [   22.356105] general protection fault: 0000 [#1] SMP PTI
>>> [   22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6
>>> [   22.366727] Hardware name: Xen HVM domU, BIOS ...
>>> [   22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
>>> ... ...
>>> [   22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246
>>> [   22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000
>>> [   22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d
>>> [   22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd
>>> [   22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000
>>> [   22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000
>>> [   22.423726] FS:  0000000000000000(0000) GS:ffff9d953bc80000(0000)
>>> knlGS:0000000000000000
>>> [   22.429898] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [   22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0
>>> [   22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [   22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>> [   22.450342] Call Trace:
>>> [   22.452509]  simple_strtoull+0x27/0x70
>>> [   22.455572]  xenbus_transaction_start+0x31/0x50
>>> [   22.459104]  netback_changed+0x76c/0xcc1 [xen_netfront]
>>> [   22.463279]  ? find_watch+0x40/0x40
>>> [   22.466156]  xenwatch_thread+0xb4/0x150
>>> [   22.469309]  ? wait_woken+0x80/0x80
>>> [   22.472198]  kthread+0x10e/0x130
>>> [   22.474925]  ? kthread_park+0x80/0x80
>>> [   22.477946]  ret_from_fork+0x35/0x40
>>> [   22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront
>>> xen_blkfront
>>> [   22.486783] ---[ end trace a9222030a747c3f7 ]---
>>> [   22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
>>>
>>> The "while" is changed to "do while" so that wait_event() is used as a
>>> barrier.
>>
>> The correct barrier for read_reply() should be virt_rmb(). While on x86, this is
>> equivalent to barrier(), on Arm this will be a dmb(ish) to prevent the processor
>> re-ordering memory access.
>>
>> Therefore the barrier in test_reply() (called by wait_event()) is not going to
>> be sufficient for Arm.
> 
> Sorry that I just erroneously thought wait_event() would be used as read barrier.

I was also kind of expecting wait_event() to contain a memory barrier. 
But it does not at least if condition is valid before waiting.

Cheers,
diff mbox series

Patch

diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
index d239fc3c5e3d..852ed161fc2a 100644
--- a/drivers/xen/xenbus/xenbus_comms.c
+++ b/drivers/xen/xenbus/xenbus_comms.c
@@ -313,6 +313,8 @@  static int process_msg(void)
 			req->msg.type = state.msg.type;
 			req->msg.len = state.msg.len;
 			req->body = state.body;
+			/* write body, then update state */
+			virt_wmb();
 			req->state = xb_req_state_got_reply;
 			req->cb(req);
 		} else
diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c
index ddc18da61834..f5b0a6a72ad3 100644
--- a/drivers/xen/xenbus/xenbus_xs.c
+++ b/drivers/xen/xenbus/xenbus_xs.c
@@ -202,7 +202,7 @@  static bool test_reply(struct xb_req_data *req)
 
 static void *read_reply(struct xb_req_data *req)
 {
-	while (req->state != xb_req_state_got_reply) {
+	do {
 		wait_event(req->wq, test_reply(req));
 
 		if (!xenbus_ok())
@@ -216,7 +216,7 @@  static void *read_reply(struct xb_req_data *req)
 		if (req->err)
 			return ERR_PTR(req->err);
 
-	}
+	} while (req->state != xb_req_state_got_reply);
 
 	return req->body;
 }