From patchwork Fri Mar 20 10:30:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 11448843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 685D96CA for ; Fri, 20 Mar 2020 10:33:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3B47A2051A for ; Fri, 20 Mar 2020 10:33:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Evrv/9Vq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3B47A2051A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFExq-0006p3-DD for patchwork-qemu-devel@patchwork.kernel.org; Fri, 20 Mar 2020 06:33:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37146) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEvj-00049a-Nq for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jFEvi-00016v-F5 for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:07 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:60233) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jFEvi-00016r-BQ for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584700266; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ij6wvb5J1RcSkz7+GPTNk0YJefNIdneAbM1aUHavAsw=; b=Evrv/9VqFQhTOZ2GZs1TNq+r4xk4hQmp1xqAmAES19yKyvTKCaJhzFo2xMxUAdEjreADaE VNbUSjp6m6b2DrqNsJ7I+ym4YCkeEterz4SjyTPtG8xFm9ilQKwP0nlcOcs1pl1/mGnBUm nQfprAsw71BXZTnLx/dsfFg90+LErZw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-346-TFvvHJAlOTme1vvA12aczQ-1; Fri, 20 Mar 2020 06:31:04 -0400 X-MC-Unique: TFvvHJAlOTme1vvA12aczQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5B31A800D4E; Fri, 20 Mar 2020 10:31:03 +0000 (UTC) Received: from localhost (ovpn-114-252.ams2.redhat.com [10.36.114.252]) by smtp.corp.redhat.com (Postfix) with ESMTP id 01E2019C58; Fri, 20 Mar 2020 10:30:54 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Subject: [PATCH RESEND v3 1/4] virtio-scsi: introduce a constant for fixed virtqueues Date: Fri, 20 Mar 2020 10:30:38 +0000 Message-Id: <20200320103041.129527-2-stefanha@redhat.com> In-Reply-To: <20200320103041.129527-1-stefanha@redhat.com> References: <20200320103041.129527-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , Eduardo Habkost , slp@redhat.com, "Michael S. Tsirkin" , cohuck@redhat.com, Max Reitz , Stefan Hajnoczi , Paolo Bonzini , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" The event and control virtqueues are always present, regardless of the multi-queue configuration. Define a constant so that virtqueue number calculations are easier to read. Signed-off-by: Stefan Hajnoczi Reviewed-by: Cornelia Huck --- hw/scsi/vhost-user-scsi.c | 2 +- hw/scsi/virtio-scsi.c | 7 ++++--- include/hw/virtio/virtio-scsi.h | 3 +++ 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/hw/scsi/vhost-user-scsi.c b/hw/scsi/vhost-user-scsi.c index a01bf63a08..e9752baa89 100644 --- a/hw/scsi/vhost-user-scsi.c +++ b/hw/scsi/vhost-user-scsi.c @@ -115,7 +115,7 @@ static void vhost_user_scsi_realize(DeviceState *dev, Error **errp) goto free_virtio; } - vsc->dev.nvqs = 2 + vs->conf.num_queues; + vsc->dev.nvqs = VIRTIO_SCSI_VQ_NUM_FIXED + vs->conf.num_queues; vsc->dev.vqs = g_new0(struct vhost_virtqueue, vsc->dev.nvqs); vsc->dev.vq_index = 0; vsc->dev.backend_features = 0; diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c index 472bbd233b..427ad83c50 100644 --- a/hw/scsi/virtio-scsi.c +++ b/hw/scsi/virtio-scsi.c @@ -191,7 +191,7 @@ static void virtio_scsi_save_request(QEMUFile *f, SCSIRequest *sreq) VirtIOSCSIReq *req = sreq->hba_private; VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(req->dev); VirtIODevice *vdev = VIRTIO_DEVICE(req->dev); - uint32_t n = virtio_get_queue_index(req->vq) - 2; + uint32_t n = virtio_get_queue_index(req->vq) - VIRTIO_SCSI_VQ_NUM_FIXED; assert(n < vs->conf.num_queues); qemu_put_be32s(f, &n); @@ -892,10 +892,11 @@ void virtio_scsi_common_realize(DeviceState *dev, sizeof(VirtIOSCSIConfig)); if (s->conf.num_queues == 0 || - s->conf.num_queues > VIRTIO_QUEUE_MAX - 2) { + s->conf.num_queues > VIRTIO_QUEUE_MAX - VIRTIO_SCSI_VQ_NUM_FIXED) { error_setg(errp, "Invalid number of queues (= %" PRIu32 "), " "must be a positive integer less than %d.", - s->conf.num_queues, VIRTIO_QUEUE_MAX - 2); + s->conf.num_queues, + VIRTIO_QUEUE_MAX - VIRTIO_SCSI_VQ_NUM_FIXED); virtio_cleanup(vdev); return; } diff --git a/include/hw/virtio/virtio-scsi.h b/include/hw/virtio/virtio-scsi.h index 24e768909d..9f293bcb80 100644 --- a/include/hw/virtio/virtio-scsi.h +++ b/include/hw/virtio/virtio-scsi.h @@ -36,6 +36,9 @@ #define VIRTIO_SCSI_MAX_TARGET 255 #define VIRTIO_SCSI_MAX_LUN 16383 +/* Number of virtqueues that are always present */ +#define VIRTIO_SCSI_VQ_NUM_FIXED 2 + typedef struct virtio_scsi_cmd_req VirtIOSCSICmdReq; typedef struct virtio_scsi_cmd_resp VirtIOSCSICmdResp; typedef struct virtio_scsi_ctrl_tmf_req VirtIOSCSICtrlTMFReq; From patchwork Fri Mar 20 10:30:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 11448841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA2D314B4 for ; Fri, 20 Mar 2020 10:32:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AFE0A2051A for ; Fri, 20 Mar 2020 10:32:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZwwLNynA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AFE0A2051A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEwm-0005XI-Sd for patchwork-qemu-devel@patchwork.kernel.org; Fri, 20 Mar 2020 06:32:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37237) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEvo-0004Fd-WE for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jFEvn-00018u-77 for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:12 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:20573) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jFEvn-00018f-3o for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584700270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YJhMB++vOK7g4a4RMh+UMtMgWzKOV3Ns5UnGJbhKJM4=; b=ZwwLNynA1BNsr2Pc5gmX81Qr0279WjdnzTP5xGsWpNpgX3f7NVMvNlQdwWvFmqpsd7h+1Y 5PdJ7g7dnru2I1r6tuHZDUbIG+NboqbmC+ISnGSGbfTP7wCFhsW/RJpnURdlZYHeHaawjp NzKyPXoJEPS7DNU7BpaMraQnmIF8UNU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-193-Ag49-QQbPDCTohpzfJZIyw-1; Fri, 20 Mar 2020 06:31:07 -0400 X-MC-Unique: Ag49-QQbPDCTohpzfJZIyw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 07288801A01; Fri, 20 Mar 2020 10:31:06 +0000 (UTC) Received: from localhost (ovpn-114-252.ams2.redhat.com [10.36.114.252]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2B83819756; Fri, 20 Mar 2020 10:31:04 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Subject: [PATCH RESEND v3 2/4] virtio-scsi: default num_queues to -smp N Date: Fri, 20 Mar 2020 10:30:39 +0000 Message-Id: <20200320103041.129527-3-stefanha@redhat.com> In-Reply-To: <20200320103041.129527-1-stefanha@redhat.com> References: <20200320103041.129527-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 216.205.24.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , Eduardo Habkost , slp@redhat.com, "Michael S. Tsirkin" , cohuck@redhat.com, Max Reitz , Stefan Hajnoczi , Paolo Bonzini , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Automatically size the number of virtio-scsi-pci, vhost-scsi-pci, and vhost-user-scsi-pci request virtqueues to match the number of vCPUs. Other transports continue to default to 1 request virtqueue. A 1:1 virtqueue:vCPU mapping ensures that completion interrupts are handled on the same vCPU that submitted the request. No IPI is necessary to complete an I/O request and performance is improved. Signed-off-by: Stefan Hajnoczi --- hw/core/machine.c | 3 +++ hw/scsi/vhost-scsi.c | 3 ++- hw/scsi/vhost-user-scsi.c | 3 ++- hw/scsi/virtio-scsi.c | 6 +++++- hw/virtio/vhost-scsi-pci.c | 10 ++++++++-- hw/virtio/vhost-user-scsi-pci.c | 10 ++++++++-- hw/virtio/virtio-scsi-pci.c | 10 ++++++++-- include/hw/virtio/virtio-scsi.h | 2 ++ 8 files changed, 38 insertions(+), 9 deletions(-) diff --git a/hw/core/machine.c b/hw/core/machine.c index 9e8c06036f..4bbcec8fbd 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -33,8 +33,11 @@ GlobalProperty hw_compat_4_2[] = { { "virtio-scsi-device", "virtqueue_size", "128"}, { "virtio-blk-device", "x-enable-wce-if-config-wce", "off" }, { "virtio-blk-device", "seg-max-adjust", "off"}, + { "virtio-scsi-device", "num_queues", "1"}, { "virtio-scsi-device", "seg_max_adjust", "off"}, { "vhost-blk-device", "seg_max_adjust", "off"}, + { "vhost-scsi", "num_queues", "1"}, + { "vhost-user-scsi", "num_queues", "1"}, { "usb-host", "suppress-remote-wake", "off" }, { "usb-redir", "suppress-remote-wake", "off" }, { "qxl", "revision", "4" }, diff --git a/hw/scsi/vhost-scsi.c b/hw/scsi/vhost-scsi.c index f052377b7e..8cb7e3825f 100644 --- a/hw/scsi/vhost-scsi.c +++ b/hw/scsi/vhost-scsi.c @@ -272,7 +272,8 @@ static Property vhost_scsi_properties[] = { DEFINE_PROP_STRING("vhostfd", VirtIOSCSICommon, conf.vhostfd), DEFINE_PROP_STRING("wwpn", VirtIOSCSICommon, conf.wwpn), DEFINE_PROP_UINT32("boot_tpgt", VirtIOSCSICommon, conf.boot_tpgt, 0), - DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1), + DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, + VIRTIO_SCSI_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size, 128), DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSICommon, conf.seg_max_adjust, diff --git a/hw/scsi/vhost-user-scsi.c b/hw/scsi/vhost-user-scsi.c index e9752baa89..f0a7e76280 100644 --- a/hw/scsi/vhost-user-scsi.c +++ b/hw/scsi/vhost-user-scsi.c @@ -163,7 +163,8 @@ static void vhost_user_scsi_unrealize(DeviceState *dev, Error **errp) static Property vhost_user_scsi_properties[] = { DEFINE_PROP_CHR("chardev", VirtIOSCSICommon, conf.chardev), DEFINE_PROP_UINT32("boot_tpgt", VirtIOSCSICommon, conf.boot_tpgt, 0), - DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1), + DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, + VIRTIO_SCSI_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size, 128), DEFINE_PROP_UINT32("max_sectors", VirtIOSCSICommon, conf.max_sectors, diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c index 427ad83c50..3bf97836d9 100644 --- a/hw/scsi/virtio-scsi.c +++ b/hw/scsi/virtio-scsi.c @@ -891,6 +891,9 @@ void virtio_scsi_common_realize(DeviceState *dev, virtio_init(vdev, "virtio-scsi", VIRTIO_ID_SCSI, sizeof(VirtIOSCSIConfig)); + if (s->conf.num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + s->conf.num_queues = 1; + } if (s->conf.num_queues == 0 || s->conf.num_queues > VIRTIO_QUEUE_MAX - VIRTIO_SCSI_VQ_NUM_FIXED) { error_setg(errp, "Invalid number of queues (= %" PRIu32 "), " @@ -964,7 +967,8 @@ static void virtio_scsi_device_unrealize(DeviceState *dev, Error **errp) } static Property virtio_scsi_properties[] = { - DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues, 1), + DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues, + VIRTIO_SCSI_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI, parent_obj.conf.virtqueue_size, 256), DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI, diff --git a/hw/virtio/vhost-scsi-pci.c b/hw/virtio/vhost-scsi-pci.c index 5dce640eaf..a0b7cdc1ac 100644 --- a/hw/virtio/vhost-scsi-pci.c +++ b/hw/virtio/vhost-scsi-pci.c @@ -17,6 +17,7 @@ #include "qemu/osdep.h" #include "standard-headers/linux/virtio_pci.h" +#include "hw/boards.h" #include "hw/qdev-properties.h" #include "hw/virtio/vhost-scsi.h" #include "qapi/error.h" @@ -47,10 +48,15 @@ static void vhost_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VHostSCSIPCI *dev = VHOST_SCSI_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.parent_obj.conf; + + /* 1:1 vq to vcpu mapping is ideal because it avoids IPIs */ + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + conf->num_queues = current_machine->smp.cpus; + } if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = vs->conf.num_queues + 3; + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; } qdev_set_parent_bus(vdev, BUS(&vpci_dev->bus)); diff --git a/hw/virtio/vhost-user-scsi-pci.c b/hw/virtio/vhost-user-scsi-pci.c index 32febb2daa..2787123c8d 100644 --- a/hw/virtio/vhost-user-scsi-pci.c +++ b/hw/virtio/vhost-user-scsi-pci.c @@ -18,6 +18,7 @@ #include "qemu/osdep.h" #include "standard-headers/linux/virtio_pci.h" +#include "hw/boards.h" #include "hw/virtio/vhost-user-scsi.h" #include "hw/virtio/virtio.h" #include "hw/virtio/virtio-scsi.h" @@ -53,10 +54,15 @@ static void vhost_user_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VHostUserSCSIPCI *dev = VHOST_USER_SCSI_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.parent_obj.conf; + + /* 1:1 vq to vcpu mapping is ideal because it avoids IPIs */ + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + conf->num_queues = current_machine->smp.cpus; + } if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = vs->conf.num_queues + 3; + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; } qdev_set_parent_bus(vdev, BUS(&vpci_dev->bus)); diff --git a/hw/virtio/virtio-scsi-pci.c b/hw/virtio/virtio-scsi-pci.c index e82e7e5680..45618ed2d1 100644 --- a/hw/virtio/virtio-scsi-pci.c +++ b/hw/virtio/virtio-scsi-pci.c @@ -15,6 +15,7 @@ #include "qemu/osdep.h" +#include "hw/boards.h" #include "hw/qdev-properties.h" #include "hw/virtio/virtio-scsi.h" #include "qemu/module.h" @@ -46,12 +47,17 @@ static void virtio_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VirtIOSCSIPCI *dev = VIRTIO_SCSI_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); DeviceState *proxy = DEVICE(vpci_dev); + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.conf; char *bus_name; + /* 1:1 vq to vcpu mapping is ideal because it avoids IPIs */ + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + conf->num_queues = current_machine->smp.cpus; + } + if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = vs->conf.num_queues + 3; + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; } /* diff --git a/include/hw/virtio/virtio-scsi.h b/include/hw/virtio/virtio-scsi.h index 9f293bcb80..c0b8e4dd7e 100644 --- a/include/hw/virtio/virtio-scsi.h +++ b/include/hw/virtio/virtio-scsi.h @@ -39,6 +39,8 @@ /* Number of virtqueues that are always present */ #define VIRTIO_SCSI_VQ_NUM_FIXED 2 +#define VIRTIO_SCSI_AUTO_NUM_QUEUES UINT32_MAX + typedef struct virtio_scsi_cmd_req VirtIOSCSICmdReq; typedef struct virtio_scsi_cmd_resp VirtIOSCSICmdResp; typedef struct virtio_scsi_ctrl_tmf_req VirtIOSCSICtrlTMFReq; From patchwork Fri Mar 20 10:30:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 11448847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC1EC6CA for ; Fri, 20 Mar 2020 10:34:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8291E2051A for ; Fri, 20 Mar 2020 10:34:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fEhvV65A" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8291E2051A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50634 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEyr-00085R-N2 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 20 Mar 2020 06:34:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37269) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEvw-0004Nm-LN for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jFEvv-0001Ax-CQ for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:20 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:59770) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jFEvv-0001Ap-8U for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584700278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t32fXUBu3z7m4utGiErxnaXstVu42nkJQtglpvtuQh0=; b=fEhvV65AlgwHJsiE0hhcLnamDEZIjoprbgjIE6aAnEm6m+O6BLnDPxyeu/icoy72fyWsks WWHh/o+HY5jlGfdN9X5Lkeuyfk5wit1PfwqyyQ0GTeriwB0j9/ksAIb5osnSkrhk1PNNju QlM0OTCBEpBrGIrOfHCNJ5qpKDMD5rI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-296-K8s1Hv7XPg6BswRa9Pec1w-1; Fri, 20 Mar 2020 06:31:17 -0400 X-MC-Unique: K8s1Hv7XPg6BswRa9Pec1w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 74319DB20; Fri, 20 Mar 2020 10:31:16 +0000 (UTC) Received: from localhost (ovpn-114-252.ams2.redhat.com [10.36.114.252]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3CB0D60BFB; Fri, 20 Mar 2020 10:31:08 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Subject: [PATCH RESEND v3 3/4] virtio-blk: default num_queues to -smp N Date: Fri, 20 Mar 2020 10:30:40 +0000 Message-Id: <20200320103041.129527-4-stefanha@redhat.com> In-Reply-To: <20200320103041.129527-1-stefanha@redhat.com> References: <20200320103041.129527-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , Eduardo Habkost , slp@redhat.com, "Michael S. Tsirkin" , cohuck@redhat.com, Max Reitz , Stefan Hajnoczi , Paolo Bonzini , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Automatically size the number of virtio-blk-pci request virtqueues to match the number of vCPUs. Other transports continue to default to 1 request virtqueue. A 1:1 virtqueue:vCPU mapping ensures that completion interrupts are handled on the same vCPU that submitted the request. No IPI is necessary to complete an I/O request and performance is improved. Performance improves from 78k to 104k IOPS on a 32 vCPU guest with 101 virtio-blk-pci devices (ioengine=libaio, iodepth=1, bs=4k, rw=randread with NVMe storage). Signed-off-by: Stefan Hajnoczi Reviewed-by: Cornelia Huck --- hw/block/virtio-blk.c | 6 +++++- hw/core/machine.c | 1 + hw/virtio/virtio-blk-pci.c | 9 ++++++++- include/hw/virtio/virtio-blk.h | 2 ++ 4 files changed, 16 insertions(+), 2 deletions(-) diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c index 142863a3b2..ed5df71779 100644 --- a/hw/block/virtio-blk.c +++ b/hw/block/virtio-blk.c @@ -1135,6 +1135,9 @@ static void virtio_blk_device_realize(DeviceState *dev, Error **errp) error_setg(errp, "Device needs media, but drive is empty"); return; } + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = 1; + } if (!conf->num_queues) { error_setg(errp, "num-queues property must be larger than 0"); return; @@ -1271,7 +1274,8 @@ static Property virtio_blk_properties[] = { #endif DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0, true), - DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1), + DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, + VIRTIO_BLK_AUTO_NUM_QUEUES), DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256), DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true), DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD, diff --git a/hw/core/machine.c b/hw/core/machine.c index 4bbcec8fbd..c993b8f489 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -31,6 +31,7 @@ GlobalProperty hw_compat_4_2[] = { { "virtio-blk-device", "queue-size", "128"}, { "virtio-scsi-device", "virtqueue_size", "128"}, + { "virtio-blk-device", "num-queues", "1"}, { "virtio-blk-device", "x-enable-wce-if-config-wce", "off" }, { "virtio-blk-device", "seg-max-adjust", "off"}, { "virtio-scsi-device", "num_queues", "1"}, diff --git a/hw/virtio/virtio-blk-pci.c b/hw/virtio/virtio-blk-pci.c index efb2c22a1d..bf628de675 100644 --- a/hw/virtio/virtio-blk-pci.c +++ b/hw/virtio/virtio-blk-pci.c @@ -17,6 +17,7 @@ #include "qemu/osdep.h" +#include "hw/boards.h" #include "hw/qdev-properties.h" #include "hw/virtio/virtio-blk.h" #include "virtio-pci.h" @@ -50,9 +51,15 @@ static void virtio_blk_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VirtIOBlkPCI *dev = VIRTIO_BLK_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); + VirtIOBlkConf *conf = &dev->vdev.conf; + + /* 1:1 vq to vcpu mapping is ideal because it avoids IPIs */ + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = current_machine->smp.cpus; + } if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = dev->vdev.conf.num_queues + 1; + vpci_dev->nvectors = conf->num_queues + 1; } qdev_set_parent_bus(vdev, BUS(&vpci_dev->bus)); diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h index 1e62f869b2..4e5e903f4a 100644 --- a/include/hw/virtio/virtio-blk.h +++ b/include/hw/virtio/virtio-blk.h @@ -30,6 +30,8 @@ struct virtio_blk_inhdr unsigned char status; }; +#define VIRTIO_BLK_AUTO_NUM_QUEUES UINT16_MAX + struct VirtIOBlkConf { BlockConf conf; From patchwork Fri Mar 20 10:30:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 11448849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 512D414B4 for ; Fri, 20 Mar 2020 10:35:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 267782072D for ; Fri, 20 Mar 2020 10:35:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GYZULzy3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 267782072D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50650 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEzW-0000rp-7K for patchwork-qemu-devel@patchwork.kernel.org; Fri, 20 Mar 2020 06:35:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37319) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFEw7-0004VB-JG for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jFEw6-0001Ck-Ew for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:31 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:58367) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jFEw6-0001Cg-Bj for qemu-devel@nongnu.org; Fri, 20 Mar 2020 06:31:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584700290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MFhchIj2XThDo52KiqEGTbwZeUi7LE4DUlc0yBM6SkI=; b=GYZULzy3OFJtsruOpQTxFJkc/YdaUtE+jGDPmHaAGO3Kg0QfSRnW9h+5B5lqI6LYRDZyu4 vkSJGurQH6MTS9TqHaLnk1A0CDUqQn28vN6kcsOFAsHxkU+yC7GMl1sQT+kGq+rtmsyHx6 lkDkgckvOZT14hr7r0ZC2zFnwZAs6z8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-168-5qRIbVOnMYGf6B9SVSr0nQ-1; Fri, 20 Mar 2020 06:31:28 -0400 X-MC-Unique: 5qRIbVOnMYGf6B9SVSr0nQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 99400800D4E; Fri, 20 Mar 2020 10:31:27 +0000 (UTC) Received: from localhost (ovpn-114-252.ams2.redhat.com [10.36.114.252]) by smtp.corp.redhat.com (Postfix) with ESMTP id D42CF19C58; Fri, 20 Mar 2020 10:31:17 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Subject: [PATCH RESEND v3 4/4] vhost-user-blk: default num_queues to -smp N Date: Fri, 20 Mar 2020 10:30:41 +0000 Message-Id: <20200320103041.129527-5-stefanha@redhat.com> In-Reply-To: <20200320103041.129527-1-stefanha@redhat.com> References: <20200320103041.129527-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , Eduardo Habkost , slp@redhat.com, "Michael S. Tsirkin" , cohuck@redhat.com, Max Reitz , Stefan Hajnoczi , Paolo Bonzini , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Automatically size the number of request virtqueues to match the number of vCPUs. This ensures that completion interrupts are handled on the same vCPU that submitted the request. No IPI is necessary to complete an I/O request and performance is improved. Signed-off-by: Stefan Hajnoczi Reviewed-by: Cornelia Huck --- hw/block/vhost-user-blk.c | 6 +++++- hw/core/machine.c | 1 + hw/virtio/vhost-user-blk-pci.c | 6 ++++++ include/hw/virtio/vhost-user-blk.h | 2 ++ 4 files changed, 14 insertions(+), 1 deletion(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 12925a47ec..5c275af935 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -403,6 +403,9 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) return; } + if (s->num_queues == VHOST_USER_BLK_AUTO_NUM_QUEUES) { + s->num_queues = 1; + } if (!s->num_queues || s->num_queues > VIRTIO_QUEUE_MAX) { error_setg(errp, "vhost-user-blk: invalid number of IO queues"); return; @@ -511,7 +514,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { static Property vhost_user_blk_properties[] = { DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), - DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, 1), + DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, + VHOST_USER_BLK_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("queue-size", VHostUserBlk, queue_size, 128), DEFINE_PROP_BIT("config-wce", VHostUserBlk, config_wce, 0, true), DEFINE_PROP_END_OF_LIST(), diff --git a/hw/core/machine.c b/hw/core/machine.c index c993b8f489..13d00abdc4 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -38,6 +38,7 @@ GlobalProperty hw_compat_4_2[] = { { "virtio-scsi-device", "seg_max_adjust", "off"}, { "vhost-blk-device", "seg_max_adjust", "off"}, { "vhost-scsi", "num_queues", "1"}, + { "vhost-user-blk", "num-queues", "1"}, { "vhost-user-scsi", "num_queues", "1"}, { "usb-host", "suppress-remote-wake", "off" }, { "usb-redir", "suppress-remote-wake", "off" }, diff --git a/hw/virtio/vhost-user-blk-pci.c b/hw/virtio/vhost-user-blk-pci.c index 8d3d766427..846fec83ac 100644 --- a/hw/virtio/vhost-user-blk-pci.c +++ b/hw/virtio/vhost-user-blk-pci.c @@ -19,6 +19,7 @@ #include "qemu/osdep.h" #include "standard-headers/linux/virtio_pci.h" +#include "hw/boards.h" #include "hw/virtio/virtio.h" #include "hw/virtio/vhost-user-blk.h" #include "hw/pci/pci.h" @@ -54,6 +55,11 @@ static void vhost_user_blk_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) VHostUserBlkPCI *dev = VHOST_USER_BLK_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); + /* 1:1 vq to vcpu mapping is ideal because it avoids IPIs */ + if (dev->vdev.num_queues == VHOST_USER_BLK_AUTO_NUM_QUEUES) { + dev->vdev.num_queues = current_machine->smp.cpus; + } + if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { vpci_dev->nvectors = dev->vdev.num_queues + 1; } diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-user-blk.h index 05ea0ad183..c28027c7c8 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -25,6 +25,8 @@ #define VHOST_USER_BLK(obj) \ OBJECT_CHECK(VHostUserBlk, (obj), TYPE_VHOST_USER_BLK) +#define VHOST_USER_BLK_AUTO_NUM_QUEUES UINT16_MAX + typedef struct VHostUserBlk { VirtIODevice parent_obj; CharBackend chardev;