Stub domain crash on Xen v4.6.1

Commit Message

Wei Liu April 12, 2016, 8:36 a.m. UTC

On Tue, Apr 05, 2016 at 05:17:00PM +0200, Fanny Dwargee wrote:
> Hi,
> 
> after adding the 'device_model_stubdomain_override = 1' to an otherwise
> fine configuration the domain crashes on start.
> 
> Xen is v4.6.1 compiled from source on Debian Jessie 64bits this way:
> 
>    - ./configure --enable-stubdom --enable-githttp
>    - make dist-xen
>    - make dist-tools
>    - make dist-stubdom
>    - make install-xen
>    - make install-tools
>    - make install-stubdom
> 
> As pointed out before the same configuration file without the '
> device_model_stubdomain_override' works flawlessly.
> 
> This is the 'xl list' command output while the domain is starting:
> 
> Name                    ID   Mem VCPUs      State   Time(s)
> Domain-0                 0  1022     2     r-----     318.3
> win7-sp1-x64-2          20  2048     1     r-----       5.5
> win7-sp1-x64-2-dm       21    44     1     r-----       6.0
> 
> 
> As you can see both domains are started (the stub and the original domain)
> 
> Find attached the domain configuration file, 'xl info' output, 'xl create'
> output and the /var/log/xen/console/hypervisor.log file, notice the grant
> table error on console-hypervisor.log
> 
> I'd very grateful for any help finding the cause of this problem.
> 
> Best regards,
> 
> Fanny


(d21) read error -1 on /local/domain/21/device/vbd/768 at offset 0, num bytes 512
(XEN) grant_table.c:525:d0v0 Bad flags (0) or dom (0). (expected dom 0)
(d21) read error -1 on /local/domain/21/device/vbd/768 at offset 0, num bytes 512
(XEN) grant_table.c:525:d0v0 Bad flags (0) or dom (0). (expected dom 0)

This reminds me of a bug that would cause error in disk:

http://lists.xen.org/archives/html/minios-devel/2016-04/msg00000.html

Are you able to apply the patch in that thread and test?

For your convenience I've attached the patch. It needs to be applied to
xen.git/extras/mini-os.

---8<---
From c519e3dfcdbc1edeac994dfa3918c175aae44983 Mon Sep 17 00:00:00 2001
From: Samuel Thibault <[e-mail]>
Date: Fri, 1 Apr 2016 20:17:01 +0200
Subject: [PATCH] Mini-OS: netfront: fix off-by-one error introduced in
 7c8f3483

7c8f3483 introduced a break within a loop in netfront.c such that
cons and nr_consumed were no longer always being incremented. The
offset at cons will be processed multiple times with the break in
place.

This commit reverts to using the "some" variable in the loop condition,
but avoids ifdefs for the non-libc case. It also renames it to dobreak
to make its usage clearer.

Signed-off-by: Samuel Thibault <[e-mail]>
Tested-by: Sarah Newman <[e-mail]>
---
 netfront.c | 20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)

Comments

Fanny Dwargee April 12, 2016, 9:44 a.m. UTC | #1

Wei,

It works now!

Many, many thanks for your invaluable time.

By the way, Do you know when the patch will be included in the stable-4.6
branch? Maybe will be on time for the 4.6.2 version?

IMHO is an important patch because of the added security shielding it
provides (allowing the use of stub domains on some cases).

Best regards,

Fanny

2016-04-12 10:36 GMT+02:00 Wei Liu <[e-mail]>:

> On Tue, Apr 05, 2016 at 05:17:00PM +0200, Fanny Dwargee wrote:
>
> (d21) read error -1 on /local/domain/21/device/vbd/768 at offset 0, num
> bytes 512
> (XEN) grant_table.c:525:d0v0 Bad flags (0) or dom (0). (expected dom 0)
> (d21) read error -1 on /local/domain/21/device/vbd/768 at offset 0, num
> bytes 512
> (XEN) grant_table.c:525:d0v0 Bad flags (0) or dom (0). (expected dom 0)
>
> This reminds me of a bug that would cause error in disk:
>
> http://lists.xen.org/archives/html/minios-devel/2016-04/msg00000.html
>
> Are you able to apply the patch in that thread and test?
>
> For your convenience I've attached the patch. It needs to be applied to
> xen.git/extras/mini-os.
>
> ---8<---
> From c519e3dfcdbc1edeac994dfa3918c175aae44983 Mon Sep 17 00:00:00 2001
> From: Samuel Thibault <[e-mail]>
> Date: Fri, 1 Apr 2016 20:17:01 +0200
> Subject: [PATCH] Mini-OS: netfront: fix off-by-one error introduced in
>  7c8f3483
>
> 7c8f3483 introduced a break within a loop in netfront.c such that
> cons and nr_consumed were no longer always being incremented. The
> offset at cons will be processed multiple times with the break in
> place.
>
> This commit reverts to using the "some" variable in the loop condition,
> but avoids ifdefs for the non-libc case. It also renames it to dobreak
> to make its usage clearer.
>
> Signed-off-by: Samuel Thibault <[e-mail]>
> Tested-by: Sarah Newman <[e-mail]>
> ---
>  netfront.c | 20 ++++++--------------
>  1 file changed, 6 insertions(+), 14 deletions(-)
>
> diff --git a/netfront.c b/netfront.c
> index 0eca5b5..b8fac62 100644
> --- a/netfront.c
> +++ b/netfront.c
> @@ -97,19 +97,15 @@ void network_rx(struct netfront_dev *dev)
>  {
>      RING_IDX rp,cons,req_prod;
>      int nr_consumed, more, i, notify;
> -#ifdef HAVE_LIBC
> -    int some;
> -#endif
> +    int dobreak;
>
>      nr_consumed = 0;
>  moretodo:
>      rp = dev->rx.sring->rsp_prod;
>      rmb(); /* Ensure we see queued responses up to 'rp'. */
>
> -#ifdef HAVE_LIBC
> -    some = 0;
> -#endif
> -    for (cons = dev->rx.rsp_cons; cons != rp; nr_consumed++, cons++)
> +    dobreak = 0;
> +    for (cons = dev->rx.rsp_cons; cons != rp && !dobreak; nr_consumed++,
> cons++)
>      {
>          struct net_buffer* buf;
>          unsigned char* page;
> @@ -134,8 +130,8 @@ moretodo:
>                     len = dev->len;
>                 memcpy(dev->data, page+rx->offset, len);
>                 dev->rlen = len;
> -               some = 1;
> -                break;
> +               /* No need to receive the rest for now */
> +               dobreak = 1;
>             } else
>  #endif
>                 dev->netif_rx(page+rx->offset,rx->status);
> @@ -144,11 +140,7 @@ moretodo:
>      dev->rx.rsp_cons=cons;
>
>      RING_FINAL_CHECK_FOR_RESPONSES(&dev->rx,more);
> -#ifdef HAVE_LIBC
> -    if(more && !some) goto moretodo;
> -#else
> -    if(more) goto moretodo;
> -#endif
> +    if(more && !dobreak) goto moretodo;
>
>      req_prod = dev->rx.req_prod_pvt;
>
> --
> 2.1.4
>
>

Wei Liu April 12, 2016, 9:49 a.m. UTC | #2

On Tue, Apr 12, 2016 at 11:44:16AM +0200, Fanny Dwargee wrote:
> Wei,
> 
> It works now!
> 

Good to know.

> Many, many thanks for your invaluable time.
> 

You're welcome.

> By the way, Do you know when the patch will be included in the stable-4.6
> branch? Maybe will be on time for the 4.6.2 version?
> 
> IMHO is an important patch because of the added security shielding it
> provides (allowing the use of stub domains on some cases).
> 

Yes, it's on my radar. I hope to get it committed to xen-unstable soon.
And then it will need to wait a bit to land in 4.6.2.

Thanks for testing.

Wei.

Patch

diff --git a/netfront.c b/netfront.c
index 0eca5b5..b8fac62 100644
--- a/netfront.c
+++ b/netfront.c
@@ -97,19 +97,15 @@  void network_rx(struct netfront_dev *dev)
 {
     RING_IDX rp,cons,req_prod;
     int nr_consumed, more, i, notify;
-#ifdef HAVE_LIBC
-    int some;
-#endif
+    int dobreak;
 
     nr_consumed = 0;
 moretodo:
     rp = dev->rx.sring->rsp_prod;
     rmb(); /* Ensure we see queued responses up to 'rp'. */
 
-#ifdef HAVE_LIBC
-    some = 0;
-#endif
-    for (cons = dev->rx.rsp_cons; cons != rp; nr_consumed++, cons++)
+    dobreak = 0;
+    for (cons = dev->rx.rsp_cons; cons != rp && !dobreak; nr_consumed++, cons++)
     {
         struct net_buffer* buf;
         unsigned char* page;
@@ -134,8 +130,8 @@  moretodo:
 		    len = dev->len;
 		memcpy(dev->data, page+rx->offset, len);
 		dev->rlen = len;
-		some = 1;
-                break;
+		/* No need to receive the rest for now */
+		dobreak = 1;
 	    } else
 #endif
 		dev->netif_rx(page+rx->offset,rx->status);
@@ -144,11 +140,7 @@  moretodo:
     dev->rx.rsp_cons=cons;
 
     RING_FINAL_CHECK_FOR_RESPONSES(&dev->rx,more);
-#ifdef HAVE_LIBC
-    if(more && !some) goto moretodo;
-#else
-    if(more) goto moretodo;
-#endif
+    if(more && !dobreak) goto moretodo;
 
     req_prod = dev->rx.req_prod_pvt;