From patchwork Mon Jul 6 13:56:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 11645921 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 334B413B4 for ; Mon, 6 Jul 2020 13:59:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 099BD2070C for ; Mon, 6 Jul 2020 13:59:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aTYJSo26" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 099BD2070C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:40274 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsReV-0006Un-BC for patchwork-qemu-devel@patchwork.kernel.org; Mon, 06 Jul 2020 09:59:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46342) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jsRcT-0001bc-TE for qemu-devel@nongnu.org; Mon, 06 Jul 2020 09:57:17 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:46389 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jsRcN-0006Sa-Sy for qemu-devel@nongnu.org; Mon, 06 Jul 2020 09:57:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594043831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aSP7/djMhy25kGi63d6m5Qu/kXKQoqJaN2oMWV9dQSg=; b=aTYJSo26MV2RrGR6Vr1EAC9MWuXyymzSCxHiXQQbrGqG6E7toFCQPyBX54YL0kl9J8X7GM 4xDRiOmBHi9tvEP5WL3gBk+7erPZCemwE1jNd/6E8x932H6RW96EBGMSvzm+/ypDDeyOSv mexevkPvW3088wpVX3ObJUmaLx7NDzQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-183-goE85UbSMSSoS1l_CdAbsg-1; Mon, 06 Jul 2020 09:57:07 -0400 X-MC-Unique: goE85UbSMSSoS1l_CdAbsg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1B4CF107ACCA; Mon, 6 Jul 2020 13:57:06 +0000 (UTC) Received: from localhost (ovpn-114-217.ams2.redhat.com [10.36.114.217]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9AF295C1B2; Mon, 6 Jul 2020 13:56:59 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Subject: [PATCH v5 1/5] virtio-pci: add virtio_pci_optimal_num_queues() helper Date: Mon, 6 Jul 2020 14:56:46 +0100 Message-Id: <20200706135650.438362-2-stefanha@redhat.com> In-Reply-To: <20200706135650.438362-1-stefanha@redhat.com> References: <20200706135650.438362-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=stefanha@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/06 01:39:15 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -13 X-Spam_score: -1.4 X-Spam_bar: - X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MIME_BASE64_TEXT=1.741, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , Eduardo Habkost , qemu-block@nongnu.org, "Michael S. Tsirkin" , cohuck@redhat.com, Max Reitz , Stefan Hajnoczi , Pankaj Gupta , Paolo Bonzini , Raphael Norwitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Multi-queue devices achieve the best performance when each vCPU has a dedicated queue. This ensures that virtqueue used notifications are handled on the same vCPU that submitted virtqueue buffers. When another vCPU handles the the notification an IPI will be necessary to wake the submission vCPU and this incurs a performance overhead. Provide a helper function that virtio-pci devices will use in later patches to automatically select the optimal number of queues. The function handles guests with large numbers of CPUs by limiting the number of queues to fit within the following constraints: 1. The maximum number of MSI-X vectors. 2. The maximum number of virtqueues. Signed-off-by: Stefan Hajnoczi Reviewed-by: Cornelia Huck --- hw/virtio/virtio-pci.h | 9 +++++++++ hw/virtio/virtio-pci.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/hw/virtio/virtio-pci.h b/hw/virtio/virtio-pci.h index e2eaaa9182..91096f0291 100644 --- a/hw/virtio/virtio-pci.h +++ b/hw/virtio/virtio-pci.h @@ -243,4 +243,13 @@ typedef struct VirtioPCIDeviceTypeInfo { /* Register virtio-pci type(s). @t must be static. */ void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t); +/** + * virtio_pci_optimal_num_queues: + * @fixed_queues: number of queues that are always present + * + * Returns: The optimal number of queues for a multi-queue device, excluding + * @fixed_queues. + */ +unsigned virtio_pci_optimal_num_queues(unsigned fixed_queues); + #endif diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 7bc8c1c056..c48570c03f 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -19,6 +19,7 @@ #include "exec/memop.h" #include "standard-headers/linux/virtio_pci.h" +#include "hw/boards.h" #include "hw/virtio/virtio.h" #include "migration/qemu-file-types.h" #include "hw/pci/pci.h" @@ -2028,6 +2029,37 @@ void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t) g_free(base_name); } +unsigned virtio_pci_optimal_num_queues(unsigned fixed_queues) +{ + /* + * 1:1 vq to vCPU mapping is ideal because the same vCPU that submitted + * virtqueue buffers can handle their completion. When a different vCPU + * handles completion it may need to IPI the vCPU that submitted the + * request and this adds overhead. + * + * Virtqueues consume guest RAM and MSI-X vectors. This is wasteful in + * guests with very many vCPUs and a device that is only used by a few + * vCPUs. Unfortunately optimizing that case requires manual pinning inside + * the guest, so those users might as well manually set the number of + * queues. There is no upper limit that can be applied automatically and + * doing so arbitrarily would result in a sudden performance drop once the + * threshold number of vCPUs is exceeded. + */ + unsigned num_queues = current_machine->smp.cpus; + + /* + * The maximum number of MSI-X vectors is PCI_MSIX_FLAGS_QSIZE + 1, but the + * config change interrupt and the fixed virtqueues must be taken into + * account too. + */ + num_queues = MIN(num_queues, PCI_MSIX_FLAGS_QSIZE - fixed_queues); + + /* + * There is a limit to how many virtqueues a device can have. + */ + return MIN(num_queues, VIRTIO_QUEUE_MAX - fixed_queues); +} + /* virtio-pci-bus */ static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,