| Message ID | 20200706135650.438362-4-stefanha@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | virtio-pci: enable blk and scsi multi-queue by default | expand |
On Mon, 6 Jul 2020 14:56:48 +0100 Stefan Hajnoczi <stefanha@redhat.com> wrote: Maybe mention 'pci' in the subject as well? > Automatically size the number of virtio-scsi-pci, vhost-scsi-pci, and > vhost-user-scsi-pci request virtqueues to match the number of vCPUs. > Other transports continue to default to 1 request virtqueue. > > A 1:1 virtqueue:vCPU mapping ensures that completion interrupts are > handled on the same vCPU that submitted the request. No IPI is > necessary to complete an I/O request and performance is improved. The > maximum number of MSI-X vectors and virtqueues limit are respected. > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > --- > include/hw/virtio/virtio-scsi.h | 2 ++ > hw/core/machine.c | 3 +++ > hw/scsi/vhost-scsi.c | 3 ++- > hw/scsi/vhost-user-scsi.c | 3 ++- > hw/scsi/virtio-scsi.c | 6 +++++- > hw/virtio/vhost-scsi-pci.c | 10 +++++++--- > hw/virtio/vhost-user-scsi-pci.c | 10 +++++++--- > hw/virtio/virtio-scsi-pci.c | 10 +++++++--- > 8 files changed, 35 insertions(+), 12 deletions(-) (...) > diff --git a/hw/virtio/virtio-scsi-pci.c b/hw/virtio/virtio-scsi-pci.c > index 3ff9eb7ef6..fa4b3bfb50 100644 > --- a/hw/virtio/virtio-scsi-pci.c > +++ b/hw/virtio/virtio-scsi-pci.c > @@ -46,13 +46,17 @@ static void virtio_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) > { > VirtIOSCSIPCI *dev = VIRTIO_SCSI_PCI(vpci_dev); > DeviceState *vdev = DEVICE(&dev->vdev); > - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); > DeviceState *proxy = DEVICE(vpci_dev); > + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.conf; > char *bus_name; > > + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { > + conf->num_queues = > + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); > + } > + > if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { > - vpci_dev->nvectors = vs->conf.num_queues + > - VIRTIO_SCSI_VQ_NUM_FIXED + 1; > + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; > } > > /* One corner case where the setup may end up being a bit odd is a situation where nvectors was specified, but num_queues was not, and the device suddenly ends up with more queues than vectors. But I don't see a reason why you would want to specify nvectors but not num_queues in a real word scenario, so I think we can ignore that corner case. Reviewed-by: Cornelia Huck <cohuck@redhat.com>
On Tue, Jul 07, 2020 at 05:44:53PM +0200, Cornelia Huck wrote: > On Mon, 6 Jul 2020 14:56:48 +0100 > Stefan Hajnoczi <stefanha@redhat.com> wrote: > > Maybe mention 'pci' in the subject as well? I think this patch does too many things. I'll split up the generic and PCI parts so that the commit message is more accurate. > > Automatically size the number of virtio-scsi-pci, vhost-scsi-pci, and > > vhost-user-scsi-pci request virtqueues to match the number of vCPUs. > > Other transports continue to default to 1 request virtqueue. > > > > A 1:1 virtqueue:vCPU mapping ensures that completion interrupts are > > handled on the same vCPU that submitted the request. No IPI is > > necessary to complete an I/O request and performance is improved. The > > maximum number of MSI-X vectors and virtqueues limit are respected. > > > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > > --- > > include/hw/virtio/virtio-scsi.h | 2 ++ > > hw/core/machine.c | 3 +++ > > hw/scsi/vhost-scsi.c | 3 ++- > > hw/scsi/vhost-user-scsi.c | 3 ++- > > hw/scsi/virtio-scsi.c | 6 +++++- > > hw/virtio/vhost-scsi-pci.c | 10 +++++++--- > > hw/virtio/vhost-user-scsi-pci.c | 10 +++++++--- > > hw/virtio/virtio-scsi-pci.c | 10 +++++++--- > > 8 files changed, 35 insertions(+), 12 deletions(-) > > (...) > > > diff --git a/hw/virtio/virtio-scsi-pci.c b/hw/virtio/virtio-scsi-pci.c > > index 3ff9eb7ef6..fa4b3bfb50 100644 > > --- a/hw/virtio/virtio-scsi-pci.c > > +++ b/hw/virtio/virtio-scsi-pci.c > > @@ -46,13 +46,17 @@ static void virtio_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) > > { > > VirtIOSCSIPCI *dev = VIRTIO_SCSI_PCI(vpci_dev); > > DeviceState *vdev = DEVICE(&dev->vdev); > > - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); > > DeviceState *proxy = DEVICE(vpci_dev); > > + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.conf; > > char *bus_name; > > > > + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { > > + conf->num_queues = > > + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); > > + } > > + > > if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { > > - vpci_dev->nvectors = vs->conf.num_queues + > > - VIRTIO_SCSI_VQ_NUM_FIXED + 1; > > + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; > > } > > > > /* > > One corner case where the setup may end up being a bit odd is a > situation where nvectors was specified, but num_queues was not, and the > device suddenly ends up with more queues than vectors. But I don't see > a reason why you would want to specify nvectors but not num_queues in a > real word scenario, so I think we can ignore that corner case. I agree, I've ignored that case. Other options include printing a warning or even an error when num_queues disagrees with nvectors. > Reviewed-by: Cornelia Huck <cohuck@redhat.com>
On Tue, Jul 07, 2020 at 05:44:53PM +0200, Cornelia Huck wrote: > On Mon, 6 Jul 2020 14:56:48 +0100 > Stefan Hajnoczi <stefanha@redhat.com> wrote: > > Maybe mention 'pci' in the subject as well? Actually splitting up the patch is hard due to the nvectors dependency on num_queues. I will leave it as a single patch and rewrite the commit message. Stefan
On Wed, 8 Jul 2020 14:05:26 +0100 Stefan Hajnoczi <stefanha@redhat.com> wrote: > On Tue, Jul 07, 2020 at 05:44:53PM +0200, Cornelia Huck wrote: > > On Mon, 6 Jul 2020 14:56:48 +0100 > > Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > diff --git a/hw/virtio/virtio-scsi-pci.c b/hw/virtio/virtio-scsi-pci.c > > > index 3ff9eb7ef6..fa4b3bfb50 100644 > > > --- a/hw/virtio/virtio-scsi-pci.c > > > +++ b/hw/virtio/virtio-scsi-pci.c > > > @@ -46,13 +46,17 @@ static void virtio_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) > > > { > > > VirtIOSCSIPCI *dev = VIRTIO_SCSI_PCI(vpci_dev); > > > DeviceState *vdev = DEVICE(&dev->vdev); > > > - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); > > > DeviceState *proxy = DEVICE(vpci_dev); > > > + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.conf; > > > char *bus_name; > > > > > > + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { > > > + conf->num_queues = > > > + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); > > > + } > > > + > > > if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { > > > - vpci_dev->nvectors = vs->conf.num_queues + > > > - VIRTIO_SCSI_VQ_NUM_FIXED + 1; > > > + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; > > > } > > > > > > /* > > > > One corner case where the setup may end up being a bit odd is a > > situation where nvectors was specified, but num_queues was not, and the > > device suddenly ends up with more queues than vectors. But I don't see > > a reason why you would want to specify nvectors but not num_queues in a > > real word scenario, so I think we can ignore that corner case. > > I agree, I've ignored that case. Other options include printing a > warning or even an error when num_queues disagrees with nvectors. I think an error would be too harsh, but a warning sounds useful.
On Wed, Jul 08, 2020 at 06:50:12PM +0200, Cornelia Huck wrote: > On Wed, 8 Jul 2020 14:05:26 +0100 > Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > On Tue, Jul 07, 2020 at 05:44:53PM +0200, Cornelia Huck wrote: > > > On Mon, 6 Jul 2020 14:56:48 +0100 > > > Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > > > diff --git a/hw/virtio/virtio-scsi-pci.c b/hw/virtio/virtio-scsi-pci.c > > > > index 3ff9eb7ef6..fa4b3bfb50 100644 > > > > --- a/hw/virtio/virtio-scsi-pci.c > > > > +++ b/hw/virtio/virtio-scsi-pci.c > > > > @@ -46,13 +46,17 @@ static void virtio_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) > > > > { > > > > VirtIOSCSIPCI *dev = VIRTIO_SCSI_PCI(vpci_dev); > > > > DeviceState *vdev = DEVICE(&dev->vdev); > > > > - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); > > > > DeviceState *proxy = DEVICE(vpci_dev); > > > > + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.conf; > > > > char *bus_name; > > > > > > > > + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { > > > > + conf->num_queues = > > > > + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); > > > > + } > > > > + > > > > if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { > > > > - vpci_dev->nvectors = vs->conf.num_queues + > > > > - VIRTIO_SCSI_VQ_NUM_FIXED + 1; > > > > + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; > > > > } > > > > > > > > /* > > > > > > One corner case where the setup may end up being a bit odd is a > > > situation where nvectors was specified, but num_queues was not, and the > > > device suddenly ends up with more queues than vectors. But I don't see > > > a reason why you would want to specify nvectors but not num_queues in a > > > real word scenario, so I think we can ignore that corner case. > > > > I agree, I've ignored that case. Other options include printing a > > warning or even an error when num_queues disagrees with nvectors. > > I think an error would be too harsh, but a warning sounds useful. I'll send this as a separate patch because at least virtio-serial is also affected. Stefan
diff --git a/include/hw/virtio/virtio-scsi.h b/include/hw/virtio/virtio-scsi.h index 9f293bcb80..c0b8e4dd7e 100644 --- a/include/hw/virtio/virtio-scsi.h +++ b/include/hw/virtio/virtio-scsi.h @@ -39,6 +39,8 @@ /* Number of virtqueues that are always present */ #define VIRTIO_SCSI_VQ_NUM_FIXED 2 +#define VIRTIO_SCSI_AUTO_NUM_QUEUES UINT32_MAX + typedef struct virtio_scsi_cmd_req VirtIOSCSICmdReq; typedef struct virtio_scsi_cmd_resp VirtIOSCSICmdResp; typedef struct virtio_scsi_ctrl_tmf_req VirtIOSCSICtrlTMFReq; diff --git a/hw/core/machine.c b/hw/core/machine.c index 211b4e077a..3df534405b 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -29,7 +29,10 @@ #include "migration/vmstate.h" GlobalProperty hw_compat_5_0[] = { + { "vhost-scsi", "num_queues", "1"}, + { "vhost-user-scsi", "num_queues", "1"}, { "virtio-balloon-device", "page-poison", "false" }, + { "virtio-scsi-device", "num_queues", "1"}, { "vmport", "x-read-set-eax", "off" }, { "vmport", "x-signal-unsupported-cmd", "off" }, { "vmport", "x-report-vmx-type", "off" }, diff --git a/hw/scsi/vhost-scsi.c b/hw/scsi/vhost-scsi.c index c1b012aea4..5e3bc319ab 100644 --- a/hw/scsi/vhost-scsi.c +++ b/hw/scsi/vhost-scsi.c @@ -272,7 +272,8 @@ static Property vhost_scsi_properties[] = { DEFINE_PROP_STRING("vhostfd", VirtIOSCSICommon, conf.vhostfd), DEFINE_PROP_STRING("wwpn", VirtIOSCSICommon, conf.wwpn), DEFINE_PROP_UINT32("boot_tpgt", VirtIOSCSICommon, conf.boot_tpgt, 0), - DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1), + DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, + VIRTIO_SCSI_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size, 128), DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSICommon, conf.seg_max_adjust, diff --git a/hw/scsi/vhost-user-scsi.c b/hw/scsi/vhost-user-scsi.c index a8b821466f..7c0631656c 100644 --- a/hw/scsi/vhost-user-scsi.c +++ b/hw/scsi/vhost-user-scsi.c @@ -162,7 +162,8 @@ static void vhost_user_scsi_unrealize(DeviceState *dev) static Property vhost_user_scsi_properties[] = { DEFINE_PROP_CHR("chardev", VirtIOSCSICommon, conf.chardev), DEFINE_PROP_UINT32("boot_tpgt", VirtIOSCSICommon, conf.boot_tpgt, 0), - DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1), + DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, + VIRTIO_SCSI_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size, 128), DEFINE_PROP_UINT32("max_sectors", VirtIOSCSICommon, conf.max_sectors, diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c index eecdd05af5..3a71ea7097 100644 --- a/hw/scsi/virtio-scsi.c +++ b/hw/scsi/virtio-scsi.c @@ -891,6 +891,9 @@ void virtio_scsi_common_realize(DeviceState *dev, virtio_init(vdev, "virtio-scsi", VIRTIO_ID_SCSI, sizeof(VirtIOSCSIConfig)); + if (s->conf.num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + s->conf.num_queues = 1; + } if (s->conf.num_queues == 0 || s->conf.num_queues > VIRTIO_QUEUE_MAX - VIRTIO_SCSI_VQ_NUM_FIXED) { error_setg(errp, "Invalid number of queues (= %" PRIu32 "), " @@ -964,7 +967,8 @@ static void virtio_scsi_device_unrealize(DeviceState *dev) } static Property virtio_scsi_properties[] = { - DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues, 1), + DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues, + VIRTIO_SCSI_AUTO_NUM_QUEUES), DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI, parent_obj.conf.virtqueue_size, 256), DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI, diff --git a/hw/virtio/vhost-scsi-pci.c b/hw/virtio/vhost-scsi-pci.c index 06e814d30e..a6bb0dc60d 100644 --- a/hw/virtio/vhost-scsi-pci.c +++ b/hw/virtio/vhost-scsi-pci.c @@ -47,11 +47,15 @@ static void vhost_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VHostSCSIPCI *dev = VHOST_SCSI_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.parent_obj.conf; + + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + conf->num_queues = + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); + } if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = vs->conf.num_queues + - VIRTIO_SCSI_VQ_NUM_FIXED + 1; + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; } qdev_realize(vdev, BUS(&vpci_dev->bus), errp); diff --git a/hw/virtio/vhost-user-scsi-pci.c b/hw/virtio/vhost-user-scsi-pci.c index ab6dfb71a9..25e97ca54e 100644 --- a/hw/virtio/vhost-user-scsi-pci.c +++ b/hw/virtio/vhost-user-scsi-pci.c @@ -53,11 +53,15 @@ static void vhost_user_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VHostUserSCSIPCI *dev = VHOST_USER_SCSI_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.parent_obj.conf; + + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + conf->num_queues = + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); + } if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = vs->conf.num_queues + - VIRTIO_SCSI_VQ_NUM_FIXED + 1; + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; } qdev_realize(vdev, BUS(&vpci_dev->bus), errp); diff --git a/hw/virtio/virtio-scsi-pci.c b/hw/virtio/virtio-scsi-pci.c index 3ff9eb7ef6..fa4b3bfb50 100644 --- a/hw/virtio/virtio-scsi-pci.c +++ b/hw/virtio/virtio-scsi-pci.c @@ -46,13 +46,17 @@ static void virtio_scsi_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VirtIOSCSIPCI *dev = VIRTIO_SCSI_PCI(vpci_dev); DeviceState *vdev = DEVICE(&dev->vdev); - VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(vdev); DeviceState *proxy = DEVICE(vpci_dev); + VirtIOSCSIConf *conf = &dev->vdev.parent_obj.conf; char *bus_name; + if (conf->num_queues == VIRTIO_SCSI_AUTO_NUM_QUEUES) { + conf->num_queues = + virtio_pci_optimal_num_queues(VIRTIO_SCSI_VQ_NUM_FIXED); + } + if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) { - vpci_dev->nvectors = vs->conf.num_queues + - VIRTIO_SCSI_VQ_NUM_FIXED + 1; + vpci_dev->nvectors = conf->num_queues + VIRTIO_SCSI_VQ_NUM_FIXED + 1; } /*
Automatically size the number of virtio-scsi-pci, vhost-scsi-pci, and vhost-user-scsi-pci request virtqueues to match the number of vCPUs. Other transports continue to default to 1 request virtqueue. A 1:1 virtqueue:vCPU mapping ensures that completion interrupts are handled on the same vCPU that submitted the request. No IPI is necessary to complete an I/O request and performance is improved. The maximum number of MSI-X vectors and virtqueues limit are respected. Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/hw/virtio/virtio-scsi.h | 2 ++ hw/core/machine.c | 3 +++ hw/scsi/vhost-scsi.c | 3 ++- hw/scsi/vhost-user-scsi.c | 3 ++- hw/scsi/virtio-scsi.c | 6 +++++- hw/virtio/vhost-scsi-pci.c | 10 +++++++--- hw/virtio/vhost-user-scsi-pci.c | 10 +++++++--- hw/virtio/virtio-scsi-pci.c | 10 +++++++--- 8 files changed, 35 insertions(+), 12 deletions(-)