From patchwork Wed Feb 1 19:45:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13124909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9562C636D4 for ; Wed, 1 Feb 2023 19:44:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232161AbjBATob (ORCPT ); Wed, 1 Feb 2023 14:44:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232088AbjBATo2 (ORCPT ); Wed, 1 Feb 2023 14:44:28 -0500 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 508687BBFD for ; Wed, 1 Feb 2023 11:44:27 -0800 (PST) Received: by mail-ej1-x632.google.com with SMTP id mf7so35916527ejc.6 for ; Wed, 01 Feb 2023 11:44:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HICwk+ZaGfDJCbBt6gOrUiuKx+JyONa1rsd399znELE=; b=s+TkGtd0CJKn0fgzwiRNSQ0hciitR9Qd6pS4R55tq+1KVcodwIh7hzdgMhVzvVJQl/ Vogax6v8Gs19BrA4b+x/qdMusyTdBCEx4Y7dfR68DJXA7Iha1agsWxF5Bn6Tryp4+0o3 iA53H/yFXVCL79uMOWiCoCkgi/QGS9xr1UlpMZsSMFL4zjLwN6bRgumgIBMmy+uZ9w+8 ifMxfGaPOdhU2cgK1wRoX5Swx0HuIiG6bM/tkO/ugP/iLROL4hMcMxJvk8YxgtJq6CbA GsDlvTkmOpcp5+nRgvXWHg7dHGEhDVIvHPIUJ2wOvtewT6J+x5xjElC4R23hET6sLn+J yQ8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HICwk+ZaGfDJCbBt6gOrUiuKx+JyONa1rsd399znELE=; b=awnKoUGNb6F+D2trjy9lIauU10m4A/g8c+k/obhN41MXcEKqgLWVKAoMH+hrrFteKk fDtnxVliGhhSaYfTfHHzErRXZHqDftguA0DPuOsIOtQQgw/I4ft6c24Ps91XE6acOyW6 t0cwzelXy0o41Pcjzj8X3lgS66kvoQTI5kiRbYKo0Ne/tZJtT7gOFCK0c+bLw442Aon/ NRjVf+Jl/5OLz0fxj9GZr6Pcvc7hKLk/PqMm0GdsmGqJRXfTxJzeN+CpjYQUIm/+DOUF d9nH/E9kM+QeNwVACiFs78QQfCRZFcV3H8scM7MksRBEK2sgSGVYvG3+DbphPtgmwwct 7Wdg== X-Gm-Message-State: AO0yUKVvN9SvWklLI9LGvFgNPO0/g8Db2DP3RUOv01WzfL9IANiYzWcW eSV/qDMBypd9lII7sr5F/aJwhNssVN1WQhr9Lks= X-Google-Smtp-Source: AK7set/j6gNNmp0yFuQKk+e8rNCH+mRK3HddNhFIoeOWrjreVn5bODqIdpKpIdRW7/Wo+taUerV1pA== X-Received: by 2002:a17:907:990b:b0:87b:dac1:bbe6 with SMTP id ka11-20020a170907990b00b0087bdac1bbe6mr4042757ejc.36.1675280665653; Wed, 01 Feb 2023 11:44:25 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:25 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 1/6] KVM: x86/mmu: Avoid indirect call for get_cr3 Date: Wed, 1 Feb 2023 20:45:59 +0100 Message-Id: <20230201194604.11135-2-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Paolo Bonzini Most of the time, calls to get_guest_pgd result in calling kvm_read_cr3 (the exception is only nested TDP). Hardcode the default instead of using the get_cr3 function, avoiding a retpoline if they are enabled. Signed-off-by: Paolo Bonzini Signed-off-by: Mathias Krause --- arch/x86/kvm/mmu/mmu.c | 31 ++++++++++++++++++++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index aeb240b339f5..505768631614 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -241,6 +241,20 @@ static struct kvm_mmu_role_regs vcpu_to_role_regs(struct kvm_vcpu *vcpu) return regs; } +static unsigned long get_guest_cr3(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr3(vcpu); +} + +static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu) +{ + if (IS_ENABLED(CONFIG_RETPOLINE) && mmu->get_guest_pgd == get_guest_cr3) + return kvm_read_cr3(vcpu); + + return mmu->get_guest_pgd(vcpu); +} + static inline bool kvm_available_flush_tlb_with_range(void) { return kvm_x86_ops.tlb_remote_flush_with_range; @@ -3722,7 +3736,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) int quadrant, i, r; hpa_t root; - root_pgd = mmu->get_guest_pgd(vcpu); + root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; if (mmu_check_root(vcpu, root_gfn)) @@ -4172,7 +4186,7 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, arch.token = alloc_apf_token(vcpu); arch.gfn = gfn; arch.direct_map = vcpu->arch.mmu->root_role.direct; - arch.cr3 = vcpu->arch.mmu->get_guest_pgd(vcpu); + arch.cr3 = kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu); return kvm_setup_async_pf(vcpu, cr2_or_gpa, kvm_vcpu_gfn_to_hva(vcpu, gfn), &arch); @@ -4191,7 +4205,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) return; if (!vcpu->arch.mmu->root_role.direct && - work->arch.cr3 != vcpu->arch.mmu->get_guest_pgd(vcpu)) + work->arch.cr3 != kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) return; kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true); @@ -4592,11 +4606,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); -static unsigned long get_cr3(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr3(vcpu); -} - static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) { @@ -5147,7 +5156,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu, context->page_fault = kvm_tdp_page_fault; context->sync_page = nonpaging_sync_page; context->invlpg = NULL; - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; @@ -5297,7 +5306,7 @@ static void init_kvm_softmmu(struct kvm_vcpu *vcpu, kvm_init_shadow_mmu(vcpu, cpu_role); - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; } @@ -5311,7 +5320,7 @@ static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu, return; g_context->cpu_role.as_u64 = new_mode.as_u64; - g_context->get_guest_pgd = get_cr3; + g_context->get_guest_pgd = get_guest_cr3; g_context->get_pdptr = kvm_pdptr_read; g_context->inject_page_fault = kvm_inject_page_fault; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index e5662dbd519c..78448fb84bd6 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -324,7 +324,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: walker->level = mmu->cpu_role.base.level; - pte = mmu->get_guest_pgd(vcpu); + pte = kvm_mmu_get_guest_pgd(vcpu, mmu); have_ad = PT_HAVE_ACCESSED_DIRTY(mmu); #if PTTYPE == 64 From patchwork Wed Feb 1 19:46:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13124908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3194AC63797 for ; Wed, 1 Feb 2023 19:44:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232181AbjBAToc (ORCPT ); Wed, 1 Feb 2023 14:44:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232113AbjBATo3 (ORCPT ); Wed, 1 Feb 2023 14:44:29 -0500 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 067B87CC8D for ; Wed, 1 Feb 2023 11:44:28 -0800 (PST) Received: by mail-ej1-x634.google.com with SMTP id m2so54177094ejb.8 for ; Wed, 01 Feb 2023 11:44:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SwDZjz3R/xUrA9TRfuAPMHW2TTSVBtiLMqPJ61VaYAk=; b=RaxIe/1DMFnMimcNs7o86F/NH2QJAimOB2DuKkM8Rlp3M6jd8ZwN2g3mowK0Iz+Pqg bYC5ACF10ECbb0drnKJUP3eHL/RHsNmek0qc7K+fOTMcmg7/SFSpv2rJKMJEx3GdsV07 rT9ycN16bqmTAY+53MXGTPGDbAMHlLjodfIG0FT7QxHQpmyYi2igCaIjsIehkX2f4CCl 4qjCG11Xd6eatyCxEBzgvTJT5UBgwiW3lcZf46NgbwCU5LLkDkuZAUX0KTkWTcmt4Otn zwrh/hdBGwSjBdsza6opiEjbdZvqe+jcqP9ZK9WfeWU1nlFVG97Ldwg+a19E3iJeccod 1Fmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SwDZjz3R/xUrA9TRfuAPMHW2TTSVBtiLMqPJ61VaYAk=; b=de1QU5gcWcDm2zP6iwBwRt2/k9n2WkZ6INPMoPp+RvKm/nzmtmlIxb4+xwxv5+Ufhi 9uRut9upMyEiAeZRULb3w9vrYl0xWY9a1w6oeiePmBevUZgnTNhQ7TRoOIhRAvYzwTzy yZoOtW1Fzyoinb1m9RPbIyZCmhtFJuNX0RKBUw2rm7eVRlpWWjkHHfqVnT20vBairxjn oyYt4VchBYIwNE2Oe5fA+jHmO6t2zVFeAIZcjg6ceJeMZS5GAXxKgMVlvMSsASjRhOWW o6O6KrXyr5U/NEAoHPS4YofWbtJcdOXgkcXgSfT2qcZz5UNl/v03MnMOF8H6LUxPugkU 7fpg== X-Gm-Message-State: AO0yUKVTbQhvjiGDa7IU6FXn5iHafvFrEMpRNa46hHLzqmZKYq7mRmGt E7Vj7jHqVI3Aok0FpQLBWkfMEe3PBn/qgDVCfCo= X-Google-Smtp-Source: AK7set9WlNbz3bkf6/JRn2TnBfZf0ctA61mbKzUuRSsK5tJf/PCFFaf38agpEEofskCEfNIoGDqsOw== X-Received: by 2002:a17:906:3a91:b0:88b:a30:25f0 with SMTP id y17-20020a1709063a9100b0088b0a3025f0mr3397279ejd.32.1675280666358; Wed, 01 Feb 2023 11:44:26 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:26 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 2/6] KVM: VMX: Avoid retpoline call for control register caused exits Date: Wed, 1 Feb 2023 20:46:00 +0100 Message-Id: <20230201194604.11135-3-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Complement commit 4289d2728664 ("KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers") and avoid a retpoline call for control register accesses as well. This speeds up guests that make heavy use of it, like grsecurity kernels toggling CR0.WP to implement kernel W^X. Signed-off-by: Mathias Krause --- Meanwhile I got my hands on a AMD system and while doing a similar change for SVM gives a small measurable win (1.1% faster for grsecurity guests), it would provide nothing for other guests, as the change I was testing was specifically targeting CR0 caused exits. A more general approach would instead cover CR3 and, maybe, CR4 as well. However, that would require a lot more exit code compares, likely vanishing the gains in the general case. So this tweak is VMX only. arch/x86/kvm/vmx/vmx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c788aa382611..c8198c8a9b55 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6538,6 +6538,8 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) return handle_external_interrupt(vcpu); else if (exit_reason.basic == EXIT_REASON_HLT) return kvm_emulate_halt(vcpu); + else if (exit_reason.basic == EXIT_REASON_CR_ACCESS) + return handle_cr(vcpu); else if (exit_reason.basic == EXIT_REASON_EPT_MISCONFIG) return handle_ept_misconfig(vcpu); #endif From patchwork Wed Feb 1 19:46:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13124910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AF1CC05027 for ; Wed, 1 Feb 2023 19:44:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232203AbjBAToe (ORCPT ); Wed, 1 Feb 2023 14:44:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232129AbjBATo3 (ORCPT ); Wed, 1 Feb 2023 14:44:29 -0500 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C56A67D2BD for ; Wed, 1 Feb 2023 11:44:28 -0800 (PST) Received: by mail-ej1-x62d.google.com with SMTP id m2so54177195ejb.8 for ; Wed, 01 Feb 2023 11:44:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2E5YMmSOIqZe/jnf+n11QdKNSwd3QkONdPmR1/5wHHU=; b=dN6uhAsKaM6VDdtUOahHFzOhYO0AzcwTuctDdbSBi1eK7+SJZo9eHgYPeHkXfTqYXM jFGUyi6ConRUKTp/znJ+WnsVXoR8yHvTEM9JJt/hz5C4/Hi16zD1lVck944gFlUofPyA X4LxeoB7DxASdjidq8sq/Aw9rQKVqp8uRZIspm/O2CSouFTY0go1AI3HK6vy4ZNjnfeh C3BKsRtassll4W9+PjLg6hklei4tWVPFFoChPdCsl8zxVIycdYQAQE6iZ5qxqgKsBpnS AoBvG6UClRQ3ZoCpCwcOwIxtk2ViC8B6QPuFmDUxCtBE4h9vlap6TJAXndDECJckgs75 ynKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2E5YMmSOIqZe/jnf+n11QdKNSwd3QkONdPmR1/5wHHU=; b=VZljZiT7yP3RVAbuvLhZyS8OW5l36H3IxrWwlCneX3MiDyXgaLi+wIOTBJv+C0xP1z xrZ4Qgxw1FidR/HqRHN73LT3ZLr+OLXrqSSe6sC6UYQnYj5zG/cCY10Qs0f9FEjSck5+ ZtMczRjT/HYCCP4uCeXV1rUILdvjRUcJr0rSescG0fxY6HUM3pKgJv/iB+sseoC7cCWV nQBj6klmld9e6ClobnbQYrC+JSRU/Zwt3EGPjKBvnpey2JpQGDEceffmi+x/LFqqJgwM I7crnKnwsv3WH96OJDpq0xWN9c4T5rOrFa6zJWHa493NhdE83Bxcybr98V1AGe5XU/Z6 nBHQ== X-Gm-Message-State: AO0yUKWyde2BLyr9JxkJ5dLPC87L14ThzE7u3qy4SE64F6/s/Nz3e9Mg f4/E8c1MVx7xGPawnZDFxuHtUrkh7X6077zKZio= X-Google-Smtp-Source: AK7set+NXo9TG0IVeGQsdO6K2ylckwuZ/VPruLUn8EfGOGIBcpGHMzEq3mWOaJq14cXbn3rYN1VB0g== X-Received: by 2002:a17:907:2d28:b0:88e:682e:3a9e with SMTP id gs40-20020a1709072d2800b0088e682e3a9emr2765126ejc.61.1675280667157; Wed, 01 Feb 2023 11:44:27 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:26 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 3/6] KVM: x86: Do not unload MMU roots when only toggling CR0.WP Date: Wed, 1 Feb 2023 20:46:01 +0100 Message-Id: <20230201194604.11135-4-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org There is no need to unload the MMU roots for a direct MMU role when only CR0.WP has changed -- the paging structures are still valid, only the permission bitmap needs to be updated. One heavy user of toggling CR0.WP is grsecurity's KERNEXEC feature to implement kernel W^X. The optimization brings a huge performance gain for this case as the following micro-benchmark running 'ssdd 10 50000' from rt-tests[1] on a grsecurity L1 VM shows (runtime in seconds, lower is better): legacy TDP shadow kvm.git/queue 11.55s 13.91s 75.2s kvm.git/queue+patch 7.32s 7.31s 74.6s For legacy MMU this is ~36% faster, for TTP MMU even ~47% faster. Also TDP and legacy MMU now both have around the same runtime which vanishes the need to disable TDP MMU for grsecurity. Shadow MMU sees no measurable difference and is still slow, as expected. [1] https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git Co-developed-by: Sean Christopherson Signed-off-by: Mathias Krause --- v2: handle the CR0.WP case directly in kvm_post_set_cr0() and only for the direct MMU role -- Sean I re-ran the benchmark and it's even faster than with my patch, as the critical path is now the first one handled and is now inline. Thanks a lot for the suggestion, Sean! arch/x86/kvm/x86.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 508074e47bc0..f09bfc0a3cc1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -902,6 +902,15 @@ EXPORT_SYMBOL_GPL(load_pdptrs); void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) { + /* + * Toggling just CR0.WP doesn't invalidate page tables per se, only the + * permission bits. + */ + if (vcpu->arch.mmu->root_role.direct && (cr0 ^ old_cr0) == X86_CR0_WP) { + kvm_init_mmu(vcpu); + return; + } + if ((cr0 ^ old_cr0) & X86_CR0_PG) { kvm_clear_async_pf_completion_queue(vcpu); kvm_async_pf_hash_reset(vcpu); From patchwork Wed Feb 1 19:46:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13124911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EE80C636D4 for ; Wed, 1 Feb 2023 19:44:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231663AbjBATog (ORCPT ); Wed, 1 Feb 2023 14:44:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232154AbjBAToa (ORCPT ); Wed, 1 Feb 2023 14:44:30 -0500 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9393A7BBFD for ; Wed, 1 Feb 2023 11:44:29 -0800 (PST) Received: by mail-ej1-x631.google.com with SMTP id hx15so35096128ejc.11 for ; Wed, 01 Feb 2023 11:44:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KunpYARU3Z8opuHOQvkXAppY0penEFLU7lOORGdMuTM=; b=AI1HafiS7yvl+3Vfl6WZGwTbF4dqFNC7qE0AlRRxIcILQypAR3O1OWlF8mQVoGQ/aN mPFTcy+IcLwRC9EpggiAmx2I7s2RFGMAWK3OWFi8Qo36/dHwJnxDx4Z7q3dTJP35uKan EjiialoxgW7rIayXBv3gpS2QYrrVFaOnZZPeXZpVs3xdSEPsXpz91JLi40vUF26p5HK6 RImt/CnMpqcbH88FaixMQAmxzE6C57B+3hSyFkaCI3IvkQyGK2lqHBhFCWzhkNcK5FMJ sl68khinMx9LmNefwLC8hP5uKyf64IkbT9FRHCvBQB9EEDkQBfaB66dsVn88wIQwazFW GApw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KunpYARU3Z8opuHOQvkXAppY0penEFLU7lOORGdMuTM=; b=6tAR0zkDX1cShaE45fUYIZgMRWbgnS2HqUjgrMB6WXwKOoJksCmLOi41r8b/k2Yk3K EnRcHaDTU/FQVYtB3bALaWCVCuQcrPjEoa1F0gVKkivJhcx1PxXfk33cjFM8EqCwtjdY V1bRrEINuxbWPk4t5mY7w5hkRaUqsJ6oOzv0EcazmaHV6mCXEcf2kcPD3l7hydYtnU2p fD9JD0fh6PLoKT+Ag2VawX+OJkvhBN2EE5jy+LK28U1iS+C6isLyvMAtyCtVu1qLPzIO Vi1zu9nFADVEGp2W+mYDWQJezbNIB/NkmQ9aKtmzB5gGkyFB+UwByjAac1WvUEqTNbSl ZHmQ== X-Gm-Message-State: AO0yUKWuy0YaidB80KXSp5zfWWNaUFsBFg7cp7gs4ow2hV1AIUuBFX0+ mSL8yyOp8E6xcIzi+uU92FANSzfGwDPFZIny+gE= X-Google-Smtp-Source: AK7set/JFfkn/b5GsffBgc4e8Fy+BrGURO7rdpefg0PJWlsJ1HiIwkSDMzFQxUbFqL/TkWiGPU1ktw== X-Received: by 2002:a17:906:dd4:b0:877:667b:f1e2 with SMTP id p20-20020a1709060dd400b00877667bf1e2mr3330879eji.11.1675280667922; Wed, 01 Feb 2023 11:44:27 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:27 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 4/6] KVM: x86: Make use of kvm_read_cr*_bits() when testing bits Date: Wed, 1 Feb 2023 20:46:02 +0100 Message-Id: <20230201194604.11135-5-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Make use of the kvm_read_cr{0,4}_bits() helper functions when we only want to know the state of certain bits instead of the whole register. This not only makes the intend cleaner, it also avoids a VMREAD in case the tested bits aren't guest owned. Signed-off-by: Mathias Krause --- arch/x86/kvm/pmu.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index d939d3b84e6f..d9922277df67 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -439,9 +439,9 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) if (!pmc) return 1; - if (!(kvm_read_cr4(vcpu) & X86_CR4_PCE) && + if (!(kvm_read_cr4_bits(vcpu, X86_CR4_PCE)) && (static_call(kvm_x86_get_cpl)(vcpu) != 0) && - (kvm_read_cr0(vcpu) & X86_CR0_PE)) + (kvm_read_cr0_bits(vcpu, X86_CR0_PE))) return 1; *data = pmc_read_counter(pmc) & mask; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c8198c8a9b55..d3b49e0b6c32 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5487,7 +5487,7 @@ static int handle_cr(struct kvm_vcpu *vcpu) break; case 3: /* lmsw */ val = (exit_qualification >> LMSW_SOURCE_DATA_SHIFT) & 0x0f; - trace_kvm_cr_write(0, (kvm_read_cr0(vcpu) & ~0xful) | val); + trace_kvm_cr_write(0, (kvm_read_cr0_bits(vcpu, ~0xful) | val)); kvm_lmsw(vcpu, val); return kvm_skip_emulated_instruction(vcpu); @@ -7547,7 +7547,7 @@ static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) if (!kvm_arch_has_noncoherent_dma(vcpu->kvm)) return (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; - if (kvm_read_cr0(vcpu) & X86_CR0_CD) { + if (kvm_read_cr0_bits(vcpu, X86_CR0_CD)) { if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) cache = MTRR_TYPE_WRBACK; else From patchwork Wed Feb 1 19:46:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13124912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FCF2C636D7 for ; Wed, 1 Feb 2023 19:44:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232240AbjBAToh (ORCPT ); Wed, 1 Feb 2023 14:44:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232110AbjBATob (ORCPT ); Wed, 1 Feb 2023 14:44:31 -0500 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C2357D986 for ; Wed, 1 Feb 2023 11:44:30 -0800 (PST) Received: by mail-ej1-x631.google.com with SMTP id me3so54640618ejb.7 for ; Wed, 01 Feb 2023 11:44:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4WIfgFu7sWBxHubiDa704oZ6rZ4jmUPQhefB2XMq/DY=; b=HWSMoT32VGWmMlCTrhxHivAlxKAAZNSvyoygy4yjRPBu0W4L15o7HIYq85B7iah+I/ Z8PNyxMMnBzbh7df65L7txwlSvM4GY0KOAyEkXIRwH8BLecSy6HwHBBKPhcgTgPAMoXz o8jExm4r5vV/LZ4W32jN8QUXeTT5lPMYKcMsECiA+wYAsgkZbKBlgdHyMu5TTxk7DODO dPsPPeEqIoqS9WC9kReJaacLryNx8eM9ZTa6zlD5QZvG1kHwckHUNEolVKTlKaSBPWzQ d7qoDWqz/ZNeISAZOGJnH1tCuM7GVVaOIFg7s2fTi1gNcWU4FkXs9x4IPupNRMhT4NDM b8Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4WIfgFu7sWBxHubiDa704oZ6rZ4jmUPQhefB2XMq/DY=; b=HvqChR7NgjQvhUjG3wsARhYykAM6hWvSLicyVuJG/oYI+K6UjJuWcmB0gMJO0bU8F+ VBnNJILq+Z480iSl/G3xiQqzP7M1NTCqxvwnK73oBqnV6M+VKwCFvWT++aeAX+N5GAay IT9wAVDtGviQjR12/M2Wl9JglS8JA0UWMDLfu5u/kKNeWq/OLJlb6LnpCGq2t8zsUtzn KlBOilBh30ladynPti2A0hA0w2rC1nFKZudRv+EvatSSb4ctd7Y3Lbu8HigHOLwz/Ur4 /lD4fL015ke8fQKGRfrfKsl0af0+xKAzbDN8ua88JkskaLRjGIlxLlti6LLM3im627Jm 7Gag== X-Gm-Message-State: AO0yUKVmwlfQAxVo+cfaYEJWSdqFJiCpwrbfb/bL4nG9xZVVTffj/XGp zMvpirnJeFV/YT6b4EHyEyYHH26S/W5zFJusHak= X-Google-Smtp-Source: AK7set/ywQ2ewoJPFz8eXYB+A/CbDDxQByUaY0mydA0gVNfFG5Fk7tFrMcHQkPuLvpyohPEn+SX5yw== X-Received: by 2002:a17:906:8281:b0:862:11f6:a082 with SMTP id h1-20020a170906828100b0086211f6a082mr3412157ejx.17.1675280668753; Wed, 01 Feb 2023 11:44:28 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:28 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 5/6] KVM: x86/mmu: Fix comment typo Date: Wed, 1 Feb 2023 20:46:03 +0100 Message-Id: <20230201194604.11135-6-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Fix a small comment typo in make_spte(). Signed-off-by: Mathias Krause --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index fce6f047399f..95441ffdccc8 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -164,7 +164,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, /* * For simplicity, enforce the NX huge page mitigation even if not * strictly necessary. KVM could ignore the mitigation if paging is - * disabled in the guest, as the guest doesn't have an page tables to + * disabled in the guest, as the guest doesn't have any page tables to * abuse. But to safely ignore the mitigation, KVM would have to * ensure a new MMU is loaded (or all shadow pages zapped) when CR0.PG * is toggled on, and that's a net negative for performance when TDP is From patchwork Wed Feb 1 19:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13124913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99603C6379F for ; Wed, 1 Feb 2023 19:44:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231996AbjBAToi (ORCPT ); Wed, 1 Feb 2023 14:44:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232166AbjBATob (ORCPT ); Wed, 1 Feb 2023 14:44:31 -0500 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42F277D6E3 for ; Wed, 1 Feb 2023 11:44:30 -0800 (PST) Received: by mail-ej1-x62f.google.com with SMTP id gr7so30044933ejb.5 for ; Wed, 01 Feb 2023 11:44:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qsy2Uw5CdDxyHnHzJNf7tebm5waQcm3qgkQypIRD0TQ=; b=u5nOIOiIHGHsUUIO2RFWXjN11TEIX9e+ptL1+CFtVB/c9QeQUBW4CCMKhj+d+jZC9W o/pFuvwXHpvKU8vlW/cQMaNDWw9ok4uB7LnEfQZYqdJ4RbicM3mJwYF33qErrH0McdbG dol1+omltPpOKQY+gJs/XG9hjc4gUAUwj6uo+Uf29HCeDeyK9S7kBCmVzfj+WAdTaiQq GBlkXnew/QD5OUivpAPWqzpIL6dpx4aP3jDDVIlYTRhATKAcyEr1YPyss0/IcWNpOZ3C B7UvaRb7awdj/0U0aX0rRbASJ8Glxqm26OFAY87zPlrP9k5ra2Xo6PpZYUfrgVtaR6CQ zcrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qsy2Uw5CdDxyHnHzJNf7tebm5waQcm3qgkQypIRD0TQ=; b=Q2P8xjaNvbQQxIrNw8WQJRp/259IyHqh98esbf0V7ojGk0OhkjIHKwdS7fR3OrG1W6 U3ajg7XReLNW+4/rXQsqDG1T3S3DilO9dsUYnXfpWjPuVcutT+ioSyg0iz7//sa5b4RL zO4f3kYJnTy/W4PQvgbBI0MetxQ1d6pHlWCkEWUYNzHTcHW4gv5/biIwl7LwnDsVc9Uv WD9+9OS6QghqASl59r+RlIuxU7UqVv8ETlpHrR+ucjFfO3jTRUHcr1Z6xpMSrUqy5O1L jm0riyc5ijuWntgkHwc2LXSqh/re6JVosu1kaM5H6eaRqL0ECjypXuqznpwpk0rhws2O GuEg== X-Gm-Message-State: AO0yUKUEziHLIAbJyXgDsrQaHk9GQ265KmiEnLG/39KIh8NDIhFPq75o 5inU2CuanuVxj2ksCXId7YZqhKxSPjhIdu6zJ1k= X-Google-Smtp-Source: AK7set+H1klZL37ScJpHXCuuMJhTfaEvIr9rEf8uMIyTSOnXzcZN68vTuJtwKY8/D3K4EEBkvU7O9A== X-Received: by 2002:a17:906:4c8a:b0:887:c44a:b8e9 with SMTP id q10-20020a1709064c8a00b00887c44ab8e9mr3727788eju.76.1675280669450; Wed, 01 Feb 2023 11:44:29 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:29 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 6/6] KVM: VMX: Make CR0.WP a guest owned bit Date: Wed, 1 Feb 2023 20:46:04 +0100 Message-Id: <20230201194604.11135-7-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Guests like grsecurity that make heavy use of CR0.WP to implement kernel level W^X will suffer from the implied VMEXITs. For a direct MMU role there is no need to intercept a guest change of CR0.WP, so simply make it a guest owned bit if we can do so. This implies that a read of a guest's CR0.WP bit might need a VMREAD. However, the only potentially affected user seems to be kvm_init_mmu() which is a heavy operation to begin with. But also most callers already cache the full value of CR0 anyway, so no additional VMREAD is needed. The only exception is nested_vmx_load_cr3(). Add a new module parameter 'lazycr0' to allow users to revert back to the old behaviour by loading kvm-intel.ko with 'lazycr0=0'. This change is VMX-specific, as SVM has no such fine grained control register intercept control. Suggested-by: Sean Christopherson Signed-off-by: Mathias Krause --- Initially I wanted to implement the scheme Sean sketched[1]: having a threshold where we would switch from eager to lazy CR0.WP tracking after toggling the bit often enough, make the bit guest owned afterwards and VMREAD CR0 when needed. However, when starting to look for users that would be affected, I only found kvm_init_mmu() (via kvm_init_mmu() -> vcpu_to_role_regs() -> kvm_read_cr0_bits(KVM_MMU_CR0_ROLE_BITS)). It has only these three interesting callers: 1/ kvm_mmu_reset_context(), which isn't all that interesting, as it's a heavy weight operation anyway and many of the control flows leading to it already cache the value of CR0, so no additional VMREAD is needed, 2/ nested_vmx_load_cr3() and 3/ kvm_post_set_cr0(), only when CR0.WP was toggled and the MMU is in direct mode (optimization introduced by patch 3). The last case's most interesting caller is likely kvm_set_cr0(), which already caches the written CR0 value, thereby vanishes the need for another VMREAD in vcpu_to_role_regs(). That's why I went with the much simpler approach and always allow CR0.WP to be guest owned if EPT is enabled as well. There's nothing we can do for SVM, though :/ [1] https://lore.kernel.org/kvm/Y8cTMnyBzNdO5dY3@google.com/ --- arch/x86/kvm/kvm_cache_regs.h | 3 ++- arch/x86/kvm/vmx/capabilities.h | 1 + arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 9 ++++++++- arch/x86/kvm/vmx/vmx.h | 8 ++++++++ 5 files changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index c09174f73a34..495ae0204933 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -4,7 +4,8 @@ #include -#define KVM_POSSIBLE_CR0_GUEST_BITS X86_CR0_TS +#define KVM_LAZY_CR0_GUEST_BITS X86_CR0_WP +#define KVM_POSSIBLE_CR0_GUEST_BITS (X86_CR0_TS | KVM_LAZY_CR0_GUEST_BITS) #define KVM_POSSIBLE_CR4_GUEST_BITS \ (X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR \ | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE) diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h index 45162c1bcd8f..41d48a3a651e 100644 --- a/arch/x86/kvm/vmx/capabilities.h +++ b/arch/x86/kvm/vmx/capabilities.h @@ -12,6 +12,7 @@ extern bool __read_mostly enable_vpid; extern bool __read_mostly flexpriority_enabled; extern bool __read_mostly enable_ept; +extern bool __read_mostly enable_lazy_cr0; extern bool __read_mostly enable_unrestricted_guest; extern bool __read_mostly enable_ept_ad_bits; extern bool __read_mostly enable_pml; diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 557b9c468734..2a0010ca7277 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4478,7 +4478,7 @@ static void load_vmcs12_host_state(struct kvm_vcpu *vcpu, * CR0_GUEST_HOST_MASK is already set in the original vmcs01 * (KVM doesn't change it); */ - vcpu->arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vcpu->arch.cr0_guest_owned_bits = vmx_guest_owned_cr0_bits(); vmx_set_cr0(vcpu, vmcs12->host_cr0); /* Same as above - no reason to call set_cr4_guest_host_mask(). */ @@ -4629,7 +4629,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) */ vmx_set_efer(vcpu, nested_vmx_get_vmcs01_guest_efer(vmx)); - vcpu->arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vcpu->arch.cr0_guest_owned_bits = vmx_guest_owned_cr0_bits(); vmx_set_cr0(vcpu, vmcs_readl(CR0_READ_SHADOW)); vcpu->arch.cr4_guest_owned_bits = ~vmcs_readl(CR4_GUEST_HOST_MASK); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d3b49e0b6c32..1969360d2744 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -91,6 +91,9 @@ module_param_named(flexpriority, flexpriority_enabled, bool, S_IRUGO); bool __read_mostly enable_ept = 1; module_param_named(ept, enable_ept, bool, S_IRUGO); +bool __read_mostly enable_lazy_cr0 = 1; +module_param_named(lazycr0, enable_lazy_cr0, bool, S_IRUGO); + bool __read_mostly enable_unrestricted_guest = 1; module_param_named(unrestricted_guest, enable_unrestricted_guest, bool, S_IRUGO); @@ -4765,7 +4768,7 @@ static void init_vmcs(struct vcpu_vmx *vmx) /* 22.2.1, 20.8.1 */ vm_entry_controls_set(vmx, vmx_vmentry_ctrl()); - vmx->vcpu.arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vmx->vcpu.arch.cr0_guest_owned_bits = vmx_guest_owned_cr0_bits(); vmcs_writel(CR0_GUEST_HOST_MASK, ~vmx->vcpu.arch.cr0_guest_owned_bits); set_cr4_guest_host_mask(vmx); @@ -8370,6 +8373,10 @@ static __init int hardware_setup(void) return -EOPNOTSUPP; } + /* Need EPT for lazy CR0.WP synchronization. */ + if (!enable_ept) + enable_lazy_cr0 = 0; + if (!cpu_has_vmx_ept_ad_bits() || !enable_ept) enable_ept_ad_bits = 0; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index a3da84f4ea45..e899c2291a3f 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -640,6 +640,14 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) (1 << VCPU_EXREG_EXIT_INFO_1) | \ (1 << VCPU_EXREG_EXIT_INFO_2)) +static inline unsigned long vmx_guest_owned_cr0_bits(void) +{ + unsigned long bits = KVM_POSSIBLE_CR0_GUEST_BITS; + if (!enable_lazy_cr0) + bits &= ~KVM_LAZY_CR0_GUEST_BITS; + return bits; +} + static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) { return container_of(kvm, struct kvm_vmx, kvm);