[0/6] virtio: refactor host notifiers

Commit Message

Paolo Bonzini March 24, 2016, 5:06 p.m. UTC

On 24/03/2016 17:15, Cornelia Huck wrote:
> Here's the next version of my refactoring of the virtio host notifiers.
> This one actually survives a bit of testing for me (reboot loop).
> 
> As this patchset fixes a latent bug exposed by the recent dataplane
> changes (we have a deassigned ioeventfd for a short period of time
> during dataplane start, which leads to the virtqueue handler being
> called in both the vcpu thread and the iothread simultaneously), I'd
> like to see this in 2.6.

Tested-by: Paolo Bonzini <[e-mail]>

Resisted 6 minutes versus 10 seconds.  At about 2.5 seconds per reboot, 
that means the failure happened at the fourth reboot before, and 
resisted about 150 reboots with your patches.

My testcase was to add "systemd.unit=reboot.target" to a Fedora 21's 
kernel command line and run the following

./+build/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -m 512 \
  -smp 4 -serial mon:stdio -display none -object iothread,id=io \
  -drive if=none,id=hd,file=/vm/virt_test/images/jeos-21-64.qcow2 \
  -device virtio-blk-pci,drive=hd,iothread=io \
  -drive if=none,file=null-co://,id=n1 \
  -drive if=none,file=null-co://,id=n2 \
  -drive if=none,file=null-co://,id=n3 \
  -drive if=none,file=null-co://,id=n4 \
  -drive if=none,file=null-co://,id=n5 \
  -drive if=none,file=null-co://,id=n6 \
  -drive if=none,file=null-co://,id=n7 \
  -drive if=none,file=null-co://,id=n8 \
  -device virtio-blk-pci,iothread=io,drive=n1 \
  -device virtio-blk-pci,iothread=io,drive=n2 \
  -device virtio-blk-pci,iothread=io,drive=n3 \
  -device virtio-blk-pci,iothread=io,drive=n4 \
  -device virtio-blk-pci,iothread=io,drive=n5 \
  -device virtio-blk-pci,iothread=io,drive=n6 \
  -device virtio-blk-pci,iothread=io,drive=n7 \
  -device virtio-blk-pci,iothread=io,drive=n8

with the assertion patch applied:


Thanks for your help!

Paolo

> Changes from RFC:
> - Fixed some silly errors (checking for !disabled instead of disabled,
>   virtio_ccw_stop_ioeventfd() calling virtio_bus_start_ioeventfd()).
> - Completely reworked set_host_notifier(): We only want to set/unset
>   the actual handler function and don't want to do anything to the
>   ioeventfd backing, so reduce the function to actually doing only
>   that.
> - With the change above, we can lose the 'assign' parameter in
>   virtio_bus_stop_ioeventfd() again.
> - Added more comments that hopefully make it clearer what is going on.
> 
> I'd appreciate it if people could give it some testing; I'll be back
> to look at the fallout after Easter.
> 
> Cornelia Huck (6):
>   virtio-bus: common ioeventfd infrastructure
>   virtio-bus: have callers tolerate new host notifier api
>   virtio-ccw: convert to ioeventfd callbacks
>   virtio-pci: convert to ioeventfd callbacks
>   virtio-mmio: convert to ioeventfd callbacks
>   virtio-bus: remove old set_host_notifier callback
> 
>  hw/block/dataplane/virtio-blk.c |   6 +-
>  hw/s390x/virtio-ccw.c           | 133 ++++++++++++++--------------------------
>  hw/scsi/virtio-scsi-dataplane.c |   9 ++-
>  hw/virtio/vhost.c               |  13 ++--
>  hw/virtio/virtio-bus.c          | 132 +++++++++++++++++++++++++++++++++++++++
>  hw/virtio/virtio-mmio.c         | 128 +++++++++++++-------------------------
>  hw/virtio/virtio-pci.c          | 124 +++++++++++++------------------------
>  include/hw/virtio/virtio-bus.h  |  31 +++++++++-
>  8 files changed, 303 insertions(+), 273 deletions(-)
>

Comments

Cornelia Huck March 29, 2016, 8:18 a.m. UTC | #1

On Thu, 24 Mar 2016 18:06:21 +0100
Paolo Bonzini <[e-mail]> wrote:

> On 24/03/2016 17:15, Cornelia Huck wrote:
> > Here's the next version of my refactoring of the virtio host notifiers.
> > This one actually survives a bit of testing for me (reboot loop).
> > 
> > As this patchset fixes a latent bug exposed by the recent dataplane
> > changes (we have a deassigned ioeventfd for a short period of time
> > during dataplane start, which leads to the virtqueue handler being
> > called in both the vcpu thread and the iothread simultaneously), I'd
> > like to see this in 2.6.
> 
> Tested-by: Paolo Bonzini <[e-mail]>
> 
> Resisted 6 minutes versus 10 seconds.  At about 2.5 seconds per reboot, 
> that means the failure happened at the fourth reboot before, and 
> resisted about 150 reboots with your patches.

Thanks for testing!

Is the failure still the same? I thought I had understood the problem
by now, and I'm wondering which hole we're still missing.

Paolo Bonzini March 29, 2016, 9:15 a.m. UTC | #2

On 29/03/2016 10:18, Cornelia Huck wrote:
>> > 
>> > Tested-by: Paolo Bonzini <[e-mail]>
>> > 
>> > Resisted 6 minutes versus 10 seconds.  At about 2.5 seconds per reboot, 
>> > that means the failure happened at the fourth reboot before, and 
>> > resisted about 150 reboots with your patches.
> 
> Thanks for testing!
> 
> Is the failure still the same? I thought I had understood the problem
> by now, and I'm wondering which hole we're still missing.

No, I just had to leave after the 6 minutes so I turned it off. :)

Paolo

Patch

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index cb710f1..d0b8248 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -591,6 +591,7 @@ 
         return;
     }
 
+    assert(atomic_fetch_inc(&s->reentrancy_test) == 0);
     blk_io_plug(s->blk);
 
     while ((req = virtio_blk_get_request(s))) {
@@ -602,6 +603,7 @@ 
     }
 
     blk_io_unplug(s->blk);
+    atomic_dec(&s->reentrancy_test);
 }
 
 static void virtio_blk_dma_restart_bh(void *opaque)
diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
index ae84d92..5cb66cd 100644
--- a/include/hw/virtio/virtio-blk.h
+++ b/include/hw/virtio/virtio-blk.h
@@ -54,6 +54,7 @@  typedef struct VirtIOBlock {
     bool original_wce;
     VMChangeStateEntry *change;
     bool dataplane_started;
+    int reentrancy_test;
     struct VirtIOBlockDataPlane *dataplane;
 } VirtIOBlock;