From patchwork Fri Aug 30 04:35:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13784250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1BF4CA0EDB for ; Fri, 30 Aug 2024 04:41:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Reply-To:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To: From:Subject:Message-ID:References:Mime-Version:In-Reply-To:Date: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=M2s+onp5tR3Lhbit2H5d49uth+heyyaX98MlcrFx54Y=; b=OeargROMg1/rfrcylBMgL2fUB9 51qzWQ/A6Yki7dpJ0DIJ/T8BiPZ9uDg3ObV8ypw/svhHrqnEkKoqQGherHUFXC2tYfex64DKBmEIj PMZ73Iq+T2kmYzShoz+coq2AwGRAlK5DV77SzgtxxHAWs3hJpYv7aatTF886zRoN5JV4hqDGuR2LC aK54BepUwebr4tXedHth3cVszPowSylPkhTNh05uGznfi1ddhYOV1gmaxSteXv3TpHG5k8E/7mQIf orKRTVtYiZEvhWuECQtQ4uoqsmHnisFVfjLmo1Xp58xy6rY6t46Nl6OwC0d78lIaxPFRCtoJZweY7 jajt0Psw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sjtSE-00000004hN3-2yNW; Fri, 30 Aug 2024 04:41:46 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sjtMy-00000004fqt-0VIo for linux-arm-kernel@lists.infradead.org; Fri, 30 Aug 2024 04:36:22 +0000 Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-2052918b4f4so2975215ad.1 for ; Thu, 29 Aug 2024 21:36:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1724992579; x=1725597379; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=M2s+onp5tR3Lhbit2H5d49uth+heyyaX98MlcrFx54Y=; b=iC2kvufb9/3aWUAssOwsi80vcITi+G1pUTlgtGJI9wF8zQEzk7f+dhVfLoDrN9bnku VBxw9NEQGm0e9gpmMLxr+m2hiHmG1vFac4M91GxtOWiIkY3n9uVzbU/nBb0P35ylCJUZ /BOK9k8us4COUw+EBhL6VZqkV8a8yCTLrQH5OmHGPqTS4JXsgzEqHs7ajzq5BkeDTNQb JhVSgglbzgovv7tD/JQCROMlVN/3i37L9ggPqh79aayBLhHHGtw1Bp432Ih5sg9qTyvn vERpKf/ImEmzXFgXcafthuahoZsqxxZHdfSbO4I8+avfe4+5MFN3LX8dt7nwTmw8gaA4 i9lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724992579; x=1725597379; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=M2s+onp5tR3Lhbit2H5d49uth+heyyaX98MlcrFx54Y=; b=eAI/BEKZF6O+nh0lDNs2DprU/V1j7EPtDVDU1GKanCt8pjKVZXJnnPfOty9CmapOMa 6sh2UnoBlIMLrDh6zBcxKbUy60fRMOoQuYcBc8oDuPievdd7LX5dturQGejx5j8ghjq0 asGuUYviIfVIGyVswHkeKCsKs/CBWfZ9YEDn28d7z9rlP5rUh5E7zmhLblu4ZyguaJ2P 6dgBtFMY6J5zCHLwcC16rPQ0YiUiaVXKp/Ycjn3slEREOHrIiqsGNPoNVvhVQFAhGBTd Lh4OMXYGsUPRon+0oDbWoiGaGCOWWM/Mq8n5D8CjxnysJaqTBj83/uxIiWh3pR4c6crx 6vLw== X-Forwarded-Encrypted: i=1; AJvYcCVPciojZFKjJfjL5/NziHdV4dMfc/8g+K3/HzuUQ5kHjfr003G1xDNAg/tLUBONA2TWk+Y1PXbOi6VlXHIv6cU3@lists.infradead.org X-Gm-Message-State: AOJu0YxLGMpJgDWjFd7ZQDAgFKiClA3pyXClT5+W2fA7YY003pdcdxfa nRxO5GzYiqoPVrFlmNziC1ygy1To6huXXOhauQlDAPvFvWwphQgWYwys9h1OrxBQLme1w8mN/00 dGQ== X-Google-Smtp-Source: AGHT+IHsUVl4EvyuChNsDlGcFLHLP8TQ15E11S1rfoAfjJIcPKgkB6hh3D01h9rTMbJijEqYHxNp/J3BFZg= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:c411:b0:205:32b5:b541 with SMTP id d9443c01a7336-20532b5b641mr369845ad.3.1724992578679; Thu, 29 Aug 2024 21:36:18 -0700 (PDT) Date: Thu, 29 Aug 2024 21:35:57 -0700 In-Reply-To: <20240830043600.127750-1-seanjc@google.com> Mime-Version: 1.0 References: <20240830043600.127750-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240830043600.127750-8-seanjc@google.com> Subject: [PATCH v4 07/10] KVM: Add a module param to allow enabling virtualization when KVM is loaded From: Sean Christopherson To: Paolo Bonzini , Marc Zyngier , Oliver Upton , Tianrui Zhao , Bibo Mao , Huacai Chen , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Sean Christopherson Cc: kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Chao Gao , Kai Huang , Farrah Chen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240829_213620_192475_CB2F2AED X-CRM114-Status: GOOD ( 18.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Sean Christopherson Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add an on-by-default module param, enable_virt_at_load, to let userspace force virtualization to be enabled in hardware when KVM is initialized, i.e. just before /dev/kvm is exposed to userspace. Enabling virtualization during KVM initialization allows userspace to avoid the additional latency when creating/destroying the first/last VM (or more specifically, on the 0=>1 and 1=>0 edges of creation/destruction). Now that KVM uses the cpuhp framework to do per-CPU enabling, the latency could be non-trivial as the cpuhup bringup/teardown is serialized across CPUs, e.g. the latency could be problematic for use case that need to spin up VMs quickly. Prior to commit 10474ae8945c ("KVM: Activate Virtualization On Demand"), KVM _unconditionally_ enabled virtualization during load, i.e. there's no fundamental reason KVM needs to dynamically toggle virtualization. These days, the only known argument for not enabling virtualization is to allow KVM to be autoloaded without blocking other out-of-tree hypervisors, and such use cases can simply change the module param, e.g. via command line. Note, the aforementioned commit also mentioned that enabling SVM (AMD's virtualization extensions) can result in "using invalid TLB entries". It's not clear whether the changelog was referring to a KVM bug, a CPU bug, or something else entirely. Regardless, leaving virtualization off by default is not a robust "fix", as any protection provided is lost the instant userspace creates the first VM. Reviewed-by: Chao Gao Acked-by: Kai Huang Reviewed-by: Kai Huang Tested-by: Farrah Chen Signed-off-by: Sean Christopherson --- .../admin-guide/kernel-parameters.txt | 17 +++++++++ virt/kvm/kvm_main.c | 35 +++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 09126bb8cc9f..1b52b1b7bbc4 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2648,6 +2648,23 @@ Default is Y (on). + kvm.enable_virt_at_load=[KVM,ARM64,LOONGARCH,MIPS,RISCV,X86] + If enabled, KVM will enable virtualization in hardware + when KVM is loaded, and disable virtualization when KVM + is unloaded (if KVM is built as a module). + + If disabled, KVM will dynamically enable and disable + virtualization on-demand when creating and destroying + VMs, i.e. on the 0=>1 and 1=>0 transitions of the + number of VMs. + + Enabling virtualization at module lode avoids potential + latency for creation of the 0=>1 VM, as KVM serializes + virtualization enabling across all online CPUs. The + "cost" of enabling virtualization when KVM is loaded, + is that doing so may interfere with using out-of-tree + hypervisors that want to "own" virtualization hardware. + kvm.enable_vmware_backdoor=[KVM] Support VMware backdoor PV interface. Default is false (don't support). diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index b000f221abfb..55779fbb37ec 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5572,6 +5572,9 @@ static struct miscdevice kvm_dev = { }; #ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING +static bool enable_virt_at_load = true; +module_param(enable_virt_at_load, bool, 0444); + __visible bool kvm_rebooting; EXPORT_SYMBOL_GPL(kvm_rebooting); @@ -5722,15 +5725,39 @@ static void kvm_disable_virtualization(void) unregister_syscore_ops(&kvm_syscore_ops); cpuhp_remove_state(CPUHP_AP_KVM_ONLINE); } + +static int kvm_init_virtualization(void) +{ + if (enable_virt_at_load) + return kvm_enable_virtualization(); + + return 0; +} + +static void kvm_uninit_virtualization(void) +{ + if (enable_virt_at_load) + kvm_disable_virtualization(); +} #else /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */ static int kvm_enable_virtualization(void) { return 0; } +static int kvm_init_virtualization(void) +{ + return 0; +} + static void kvm_disable_virtualization(void) { +} + +static void kvm_uninit_virtualization(void) +{ + } #endif /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */ @@ -6475,6 +6502,10 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) kvm_gmem_init(module); + r = kvm_init_virtualization(); + if (r) + goto err_virt; + /* * Registration _must_ be the very last thing done, as this exposes * /dev/kvm to userspace, i.e. all infrastructure must be setup! @@ -6488,6 +6519,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) return 0; err_register: + kvm_uninit_virtualization(); +err_virt: kvm_vfio_ops_exit(); err_vfio: kvm_async_pf_deinit(); @@ -6513,6 +6546,8 @@ void kvm_exit(void) */ misc_deregister(&kvm_dev); + kvm_uninit_virtualization(); + debugfs_remove_recursive(kvm_debugfs_dir); for_each_possible_cpu(cpu) free_cpumask_var(per_cpu(cpu_kick_mask, cpu));