From patchwork Wed Apr 7 14:34:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12188567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D40EAC433B4 for ; Wed, 7 Apr 2021 14:41:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7940C61362 for ; Wed, 7 Apr 2021 14:41:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7940C61362 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kaod.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:45724 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lU9NR-0007l6-Iw for qemu-devel@archiver.kernel.org; Wed, 07 Apr 2021 10:41:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42872) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9Gy-0008FT-SY for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:12 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:60315) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9Gx-0006MY-4V for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:12 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-296-QgOBYmJVMcm-khQXL7kLKw-1; Wed, 07 Apr 2021 10:35:08 -0400 X-MC-Unique: QgOBYmJVMcm-khQXL7kLKw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 63FB581426D; Wed, 7 Apr 2021 14:35:07 +0000 (UTC) Received: from bahia.redhat.com (ovpn-114-100.ams2.redhat.com [10.36.114.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id D31DA5D9CA; Wed, 7 Apr 2021 14:35:04 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [for-6.1 1/4] virtio-blk: Fix rollback path in virtio_blk_data_plane_start() Date: Wed, 7 Apr 2021 16:34:58 +0200 Message-Id: <20210407143501.244343-2-groug@kaod.org> In-Reply-To: <20210407143501.244343-1-groug@kaod.org> References: <20210407143501.244343-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=205.139.111.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , qemu-block@nongnu.org, "Michael S. Tsirkin" , Greg Kurz , Max Reitz , Stefan Hajnoczi , Paolo Bonzini , David Gibson Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When dataplane multiqueue support was added in QEMU 2.7, the path that would rollback guest notifiers assignment in case of error simply got dropped. Later on, when Error was added to blk_set_aio_context() in QEMU 4.1, another error path was introduced, but it ommits to rollback both host and guest notifiers. It seems cleaner to fix the rollback path in one go. The patch is simple enough that it can be adjusted if backported to a pre-4.1 QEMU. Fixes: 51b04ac5c6a6 ("virtio-blk: dataplane multiqueue support") Cc: stefanha@redhat.com Fixes: 97896a4887a0 ("block: Add Error to blk_set_aio_context()") Cc: kwolf@redhat.com Signed-off-by: Greg Kurz Reviewed-by: Stefan Hajnoczi --- hw/block/dataplane/virtio-blk.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c index e9050c8987e7..d7b5c95d26d9 100644 --- a/hw/block/dataplane/virtio-blk.c +++ b/hw/block/dataplane/virtio-blk.c @@ -207,7 +207,7 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev) virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); } - goto fail_guest_notifiers; + goto fail_host_notifiers; } } @@ -221,7 +221,7 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev) aio_context_release(old_context); if (r < 0) { error_report_err(local_err); - goto fail_guest_notifiers; + goto fail_aio_context; } /* Process queued requests before the ones in vring */ @@ -245,6 +245,13 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev) aio_context_release(s->ctx); return 0; + fail_aio_context: + for (i = 0; i < nvqs; i++) { + virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); + virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); + } + fail_host_notifiers: + k->set_guest_notifiers(qbus->parent, nvqs, false); fail_guest_notifiers: /* * If we failed to set up the guest notifiers queued requests will be From patchwork Wed Apr 7 14:34:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12188549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4921C433B4 for ; Wed, 7 Apr 2021 14:36:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7000561279 for ; Wed, 7 Apr 2021 14:36:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7000561279 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kaod.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59762 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lU9Ig-0001mu-JD for qemu-devel@archiver.kernel.org; Wed, 07 Apr 2021 10:36:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42902) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9H0-0008JG-RW for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:14 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:60230) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9Gz-0006NS-7H for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:14 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-64-qTzJfcA9OsaiRgafA9kwKQ-1; Wed, 07 Apr 2021 10:35:10 -0400 X-MC-Unique: qTzJfcA9OsaiRgafA9kwKQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B2ABB801814; Wed, 7 Apr 2021 14:35:09 +0000 (UTC) Received: from bahia.redhat.com (ovpn-114-100.ams2.redhat.com [10.36.114.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1F4E5D9CA; Wed, 7 Apr 2021 14:35:07 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [for-6.1 2/4] virtio-blk: Configure all host notifiers in a single MR transaction Date: Wed, 7 Apr 2021 16:34:59 +0200 Message-Id: <20210407143501.244343-3-groug@kaod.org> In-Reply-To: <20210407143501.244343-1-groug@kaod.org> References: <20210407143501.244343-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=groug@kaod.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=205.139.111.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , qemu-block@nongnu.org, "Michael S. Tsirkin" , Greg Kurz , Max Reitz , Stefan Hajnoczi , Paolo Bonzini , David Gibson Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This allows the virtio-blk-pci device to batch the setup of all its host notifiers. This significantly improves boot time of VMs with a high number of vCPUs, e.g. from 3m26.186s down to 0m58.023s for a pseries machine with 384 vCPUs. Note that memory_region_transaction_commit() must be called before virtio_bus_cleanup_host_notifier() because the latter might close ioeventfds that the transaction still assumes to be around when it commits. Signed-off-by: Greg Kurz --- hw/block/dataplane/virtio-blk.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c index d7b5c95d26d9..cd81893d1d01 100644 --- a/hw/block/dataplane/virtio-blk.c +++ b/hw/block/dataplane/virtio-blk.c @@ -198,19 +198,30 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev) goto fail_guest_notifiers; } + memory_region_transaction_begin(); + /* Set up virtqueue notify */ for (i = 0; i < nvqs; i++) { r = virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, true); if (r != 0) { + int j = i; + fprintf(stderr, "virtio-blk failed to set host notifier (%d)\n", r); while (i--) { virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); + } + + memory_region_transaction_commit(); + + while (j--) { virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); } goto fail_host_notifiers; } } + memory_region_transaction_commit(); + s->starting = false; vblk->dataplane_started = true; trace_virtio_blk_data_plane_start(s); @@ -246,8 +257,15 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev) return 0; fail_aio_context: + memory_region_transaction_begin(); + for (i = 0; i < nvqs; i++) { virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); + } + + memory_region_transaction_commit(); + + for (i = 0; i < nvqs; i++) { virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); } fail_host_notifiers: @@ -312,8 +330,15 @@ void virtio_blk_data_plane_stop(VirtIODevice *vdev) aio_context_release(s->ctx); + memory_region_transaction_begin(); + for (i = 0; i < nvqs; i++) { virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); + } + + memory_region_transaction_commit(); + + for (i = 0; i < nvqs; i++) { virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); } From patchwork Wed Apr 7 14:35:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12188599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C52E9C433B4 for ; Wed, 7 Apr 2021 14:43:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3D1176121E for ; Wed, 7 Apr 2021 14:43:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D1176121E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kaod.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50218 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lU9Oz-0001Al-59 for qemu-devel@archiver.kernel.org; Wed, 07 Apr 2021 10:43:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42930) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9H3-0008NN-6F for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:17 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:50698) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9H1-0006QY-Gs for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:16 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-398--6mxb72hNdOeyyGRhDgAaA-1; Wed, 07 Apr 2021 10:35:13 -0400 X-MC-Unique: -6mxb72hNdOeyyGRhDgAaA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 12B59100A911; Wed, 7 Apr 2021 14:35:12 +0000 (UTC) Received: from bahia.redhat.com (ovpn-114-100.ams2.redhat.com [10.36.114.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 08BDE5D9CA; Wed, 7 Apr 2021 14:35:09 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [for-6.1 3/4] virtio-scsi: Set host notifiers and callbacks separately Date: Wed, 7 Apr 2021 16:35:00 +0200 Message-Id: <20210407143501.244343-4-groug@kaod.org> In-Reply-To: <20210407143501.244343-1-groug@kaod.org> References: <20210407143501.244343-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=groug@kaod.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=205.139.111.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , qemu-block@nongnu.org, "Michael S. Tsirkin" , Greg Kurz , Max Reitz , Stefan Hajnoczi , Paolo Bonzini , David Gibson Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Host notifiers are guaranteed to be idle until the callbacks are hooked up with virtio_queue_aio_set_host_notifier_handler(). They thus don't need to be set or unset with the AioContext lock held. Do this outside the critical section, like virtio-blk already does : basically downgrading virtio_scsi_vring_init() to only setup the host notifier and set the callback in the caller. This will allow to batch addition/deletion of ioeventds in a single memory transaction, which is expected to greatly improve initialization time. Signed-off-by: Greg Kurz Reviewed-by: Stefan Hajnoczi --- hw/scsi/virtio-scsi-dataplane.c | 40 ++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/hw/scsi/virtio-scsi-dataplane.c b/hw/scsi/virtio-scsi-dataplane.c index 4ad879340645..b2cb3d9dcc64 100644 --- a/hw/scsi/virtio-scsi-dataplane.c +++ b/hw/scsi/virtio-scsi-dataplane.c @@ -94,8 +94,7 @@ static bool virtio_scsi_data_plane_handle_event(VirtIODevice *vdev, return progress; } -static int virtio_scsi_vring_init(VirtIOSCSI *s, VirtQueue *vq, int n, - VirtIOHandleAIOOutput fn) +static int virtio_scsi_set_host_notifier(VirtIOSCSI *s, VirtQueue *vq, int n) { BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(s))); int rc; @@ -109,7 +108,6 @@ static int virtio_scsi_vring_init(VirtIOSCSI *s, VirtQueue *vq, int n, return rc; } - virtio_queue_aio_set_host_notifier_handler(vq, s->ctx, fn); return 0; } @@ -154,38 +152,44 @@ int virtio_scsi_dataplane_start(VirtIODevice *vdev) goto fail_guest_notifiers; } - aio_context_acquire(s->ctx); - rc = virtio_scsi_vring_init(s, vs->ctrl_vq, 0, - virtio_scsi_data_plane_handle_ctrl); - if (rc) { - goto fail_vrings; + rc = virtio_scsi_set_host_notifier(s, vs->ctrl_vq, 0); + if (rc != 0) { + goto fail_host_notifiers; } vq_init_count++; - rc = virtio_scsi_vring_init(s, vs->event_vq, 1, - virtio_scsi_data_plane_handle_event); - if (rc) { - goto fail_vrings; + rc = virtio_scsi_set_host_notifier(s, vs->event_vq, 1); + if (rc != 0) { + goto fail_host_notifiers; } vq_init_count++; + for (i = 0; i < vs->conf.num_queues; i++) { - rc = virtio_scsi_vring_init(s, vs->cmd_vqs[i], i + 2, - virtio_scsi_data_plane_handle_cmd); + rc = virtio_scsi_set_host_notifier(s, vs->cmd_vqs[i], i + 2); if (rc) { - goto fail_vrings; + goto fail_host_notifiers; } vq_init_count++; } + aio_context_acquire(s->ctx); + virtio_queue_aio_set_host_notifier_handler(vs->ctrl_vq, s->ctx, + virtio_scsi_data_plane_handle_ctrl); + virtio_queue_aio_set_host_notifier_handler(vs->event_vq, s->ctx, + virtio_scsi_data_plane_handle_event); + + for (i = 0; i < vs->conf.num_queues; i++) { + virtio_queue_aio_set_host_notifier_handler(vs->cmd_vqs[i], s->ctx, + virtio_scsi_data_plane_handle_cmd); + } + s->dataplane_starting = false; s->dataplane_started = true; aio_context_release(s->ctx); return 0; -fail_vrings: - aio_wait_bh_oneshot(s->ctx, virtio_scsi_dataplane_stop_bh, s); - aio_context_release(s->ctx); +fail_host_notifiers: for (i = 0; i < vq_init_count; i++) { virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); From patchwork Wed Apr 7 14:35:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12188565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE07EC433ED for ; Wed, 7 Apr 2021 14:39:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 415CA600D4 for ; Wed, 7 Apr 2021 14:39:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 415CA600D4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kaod.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38426 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lU9LJ-0004cs-Bw for qemu-devel@archiver.kernel.org; Wed, 07 Apr 2021 10:39:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42996) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9H7-0008WM-Sp for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:21 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:40134) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU9H5-0006Uu-S0 for qemu-devel@nongnu.org; Wed, 07 Apr 2021 10:35:21 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-293-7zb2v2gHOm2YabSQRoHijg-1; Wed, 07 Apr 2021 10:35:15 -0400 X-MC-Unique: 7zb2v2gHOm2YabSQRoHijg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 58C6C84B9A3; Wed, 7 Apr 2021 14:35:14 +0000 (UTC) Received: from bahia.redhat.com (ovpn-114-100.ams2.redhat.com [10.36.114.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C1135D9CA; Wed, 7 Apr 2021 14:35:12 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [for-6.1 4/4] virtio-scsi: Configure all host notifiers in a single MR transaction Date: Wed, 7 Apr 2021 16:35:01 +0200 Message-Id: <20210407143501.244343-5-groug@kaod.org> In-Reply-To: <20210407143501.244343-1-groug@kaod.org> References: <20210407143501.244343-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=207.211.30.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , qemu-block@nongnu.org, "Michael S. Tsirkin" , Greg Kurz , Max Reitz , Stefan Hajnoczi , Paolo Bonzini , David Gibson Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This allows the virtio-scsi-pci device to batch the setup of all its host notifiers. This significantly improves boot time of VMs with a high number of vCPUs, e.g. from 6m5.563s down to 1m2.884s for a pseries machine with 384 vCPUs. Note that memory_region_transaction_commit() must be called before virtio_bus_cleanup_host_notifier() because the latter might close ioeventfds that the transaction still assumes to be around when it commits. Signed-off-by: Greg Kurz Reviewed-by: Stefan Hajnoczi --- hw/scsi/virtio-scsi-dataplane.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/hw/scsi/virtio-scsi-dataplane.c b/hw/scsi/virtio-scsi-dataplane.c index b2cb3d9dcc64..28e003250a11 100644 --- a/hw/scsi/virtio-scsi-dataplane.c +++ b/hw/scsi/virtio-scsi-dataplane.c @@ -152,6 +152,8 @@ int virtio_scsi_dataplane_start(VirtIODevice *vdev) goto fail_guest_notifiers; } + memory_region_transaction_begin(); + rc = virtio_scsi_set_host_notifier(s, vs->ctrl_vq, 0); if (rc != 0) { goto fail_host_notifiers; @@ -173,6 +175,8 @@ int virtio_scsi_dataplane_start(VirtIODevice *vdev) vq_init_count++; } + memory_region_transaction_commit(); + aio_context_acquire(s->ctx); virtio_queue_aio_set_host_notifier_handler(vs->ctrl_vq, s->ctx, virtio_scsi_data_plane_handle_ctrl); @@ -192,6 +196,11 @@ int virtio_scsi_dataplane_start(VirtIODevice *vdev) fail_host_notifiers: for (i = 0; i < vq_init_count; i++) { virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); + } + + memory_region_transaction_commit(); + + for (i = 0; i < vq_init_count; i++) { virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); } k->set_guest_notifiers(qbus->parent, vs->conf.num_queues + 2, false); @@ -229,8 +238,15 @@ void virtio_scsi_dataplane_stop(VirtIODevice *vdev) blk_drain_all(); /* ensure there are no in-flight requests */ + memory_region_transaction_begin(); + for (i = 0; i < vs->conf.num_queues + 2; i++) { virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false); + } + + memory_region_transaction_commit(); + + for (i = 0; i < vs->conf.num_queues + 2; i++) { virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i); }