From patchwork Tue Mar 2 18:45:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEFA9C4332D for ; Wed, 3 Mar 2021 07:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7B99364EE3 for ; Wed, 3 Mar 2021 07:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379677AbhCCHRt (ORCPT ); Wed, 3 Mar 2021 02:17:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381050AbhCBS4f (ORCPT ); Tue, 2 Mar 2021 13:56:35 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E558DC0617AB for ; Tue, 2 Mar 2021 10:45:49 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id t6so13533329qkt.14 for ; Tue, 02 Mar 2021 10:45:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=YZHRewQoZy371sjGbfTjC6cuAY8+Z64Uwc7ykPHV30o=; b=Tbqt6GSTs/cBKpRsz6Z+ZO3EOqFtoyaE5wLJwyCohe9UBaJ+04rH/ZyrEb5X7Y10Cy C/8FCMfuhyOCoDsAmNty55rljW6fkw/HhEe9oBIwWC6d61ZKYz2/mB3PQFngKX6ygnaU 7tBhzaHJd/QPLDw1Zyr32w05Uya3/je3mowIXG6jfktVoWtGv8jLAH6M+zEEAYQaslAk dJaJbBJ5akhLEGlSMIsQKQbLfCTSrdzehEggpS2NeN+y1cWihXNRMgsCg+HMX3ES2W2t GU84i46tSdbyjtPSUEuCnhz2z7GAhD7kuD6KQx0yNsz/PRrY5D4Qs5WuKhGobCCGcmS7 /DNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=YZHRewQoZy371sjGbfTjC6cuAY8+Z64Uwc7ykPHV30o=; b=ojbhd0shrUKrgzvFYmD7rAHX4v+PY4qRNO2JZtW3t1mRhDCfBY+CdbvN8RYqDR6EeO y3nCWnUHxzeG/7tpYxFUcWWWKsYXvdQiRKkdmG+SOIJOrIxgRSjmiYkS31wGrqCzSkX9 tVi9FJb+ESnE9HS2v5sNdiqj/xSpRbsBQnEpFBXtabGxT8G0pgLaAb1sdsfQCYdttR/n 7qarJrb1GjPGzD7Jrfr+g3ZRRqzfZRSu2R3Z3M9dFrPWAjf3LODiw7vZetGUuc3mFT1Q vX56cUFSgbYI7+KxghZfH/OYuMOIE6teyOq/t9rqW8rBTz0QIawMOufkzw7nkNiBsbgH 8z0A== X-Gm-Message-State: AOAM531RpQh2ZSgRW5/EMZsF0ftuvb8xDqNjfJnnjjdaFdwLmad/5tAg t0fLwWAnrraJs6JGJt+SecJ6NgBFgBA= X-Google-Smtp-Source: ABdhPJyttoZq05JERdsWEl6MPTgri49/ddXi5SOOkf3NBUVWaM1TTiXfV4sKhSpQtqTr1SNA2Y76kLMylNg= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:c1cc:: with SMTP id v12mr20391617qvh.47.1614710749045; Tue, 02 Mar 2021 10:45:49 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:26 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-2-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 01/15] KVM: nSVM: Set the shadow root level to the TDP level for nested NPT From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Override the shadow root level in the MMU context when configuring NPT for shadowing nested NPT. The level is always tied to the TDP level of the host, not whatever level the guest happens to be using. Fixes: 096586fda522 ("KVM: nSVM: Correctly set the shadow NPT root level in its MMU role") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c462062d36aa..0987cc1d53eb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4618,12 +4618,17 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, u32 cr0, u32 cr4, u32 efer, struct kvm_mmu *context = &vcpu->arch.guest_mmu; union kvm_mmu_role new_role = kvm_calc_shadow_npt_root_page_role(vcpu); - context->shadow_root_level = new_role.base.level; - __kvm_mmu_new_pgd(vcpu, nested_cr3, new_role.base, false, false); - if (new_role.as_u64 != context->mmu_role.as_u64) + if (new_role.as_u64 != context->mmu_role.as_u64) { shadow_mmu_init_context(vcpu, context, cr0, cr4, efer, new_role); + + /* + * Override the level set by the common init helper, nested TDP + * always uses the host's TDP configuration. + */ + context->shadow_root_level = new_role.base.level; + } } EXPORT_SYMBOL_GPL(kvm_init_shadow_npt_mmu); From patchwork Tue Mar 2 18:45:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 716D7C43142 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 607AF64F02 for ; Wed, 3 Mar 2021 07:23:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1380390AbhCCHSW (ORCPT ); Wed, 3 Mar 2021 02:18:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581624AbhCBTAq (ORCPT ); Tue, 2 Mar 2021 14:00:46 -0500 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45B69C061788 for ; Tue, 2 Mar 2021 10:45:52 -0800 (PST) Received: by mail-qv1-xf4a.google.com with SMTP id s16so11878668qvw.3 for ; Tue, 02 Mar 2021 10:45:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=wHB5rTht2wbUwkXtbWaUHLq8QDHf2xz7pZGkxQoWdEU=; b=QVLW01bcTBWnkqf2e6KLYMQTgLvLIp5fsHf51XhYWbunjl+jSctBGZ/3ZheeFQhk/d vhvYoPOahtlj2T5a1NXp7DivbiwyfHpAzbH41UKn7Z9+yRWYAsqlsfo6Nqnqhuv1ifYJ 9Nnldr2Zn0/KzvOKX+hrA6OLQa3KX1zPsQfBPp0wvDD+T+FiRUrzRcTWYhiQpL8QBZN/ bZK3ECs205YmFI88fDj/OWowFdZlvkD1jmJD3XYkASeE8R+DZD4DwCWVmX6bAMDqYNJp 0ICX1yjazabfXs2dqOe2q/Hd2THTDUrQ3eNXZ6Vf0qwkRi0rzlzJV7U7TS9iuuBhpaz/ fqOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=wHB5rTht2wbUwkXtbWaUHLq8QDHf2xz7pZGkxQoWdEU=; b=Pkjo76G0PnvKTfTWqkD3+K4KV0SjFE6xmAMQtVpmP/bX7hSgDJhyg81PeHBSSdF958 ljU7LLQRy85xYBwErhPnfyEVdCyW6YTVcKjHOAVyhNmsHHJsA70OJoTE/c3icrSnEc3p dxSBZY0m9JCsA/CqnudqvkcJ/WAPKHI1TIcnHiw1cv+TvJmzQasiJAMr81PZPJYgIYRa jDg71qTWRk/i2SIMbg5/tNFy6BRGhqmf6nm9GDXT5E8RlbTKAKHyGC9LI3B/8aAdmMNX UtewYzOGZGd2M9PQeWbt/AIJbrTts6wjcplUPO8CSvk8+YVN4FeeMBqgbdP/6/PnUJOD cNkA== X-Gm-Message-State: AOAM5308c7eKbVOalxr3srsNRq+mbOJExAPbrb0mI4TRYlDUV6GahygJ UrrCuACWuaZPByFW5CPLljjfazuBheQ= X-Google-Smtp-Source: ABdhPJxddJibr9G6EZKclLhSikto3yLEmV7Ap8MHT9rMpmd+rJT07TmFbLzbRXqnYqJ3Nhm6sPuAlFSzpEY= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:c78c:: with SMTP id k12mr4967337qvj.47.1614710751429; Tue, 02 Mar 2021 10:45:51 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:27 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-3-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 02/15] KVM: x86/mmu: Alloc page for PDPTEs when shadowing 32-bit NPT with 64-bit From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Allocate the so called pae_root page on-demand, along with the lm_root page, when shadowing 32-bit NPT with 64-bit NPT, i.e. when running a 32-bit L1. KVM currently only allocates the page when NPT is disabled, or when L0 is 32-bit (using PAE paging). Note, there is an existing memory leak involving the MMU roots, as KVM fails to free the PAE roots on failure. This will be addressed in a future commit. Fixes: ee6268ba3a68 ("KVM: x86: Skip pae_root shadow allocation if tdp enabled") Fixes: b6b80c78af83 ("KVM: x86/mmu: Allocate PAE root array when using SVM's 32-bit NPT") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 44 ++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 0987cc1d53eb..2ed3fac1244e 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3187,14 +3187,14 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, if (mmu->shadow_root_level >= PT64_ROOT_4LEVEL && (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) { mmu_free_root_page(kvm, &mmu->root_hpa, &invalid_list); - } else { + } else if (mmu->pae_root) { for (i = 0; i < 4; ++i) if (mmu->pae_root[i] != 0) mmu_free_root_page(kvm, &mmu->pae_root[i], &invalid_list); - mmu->root_hpa = INVALID_PAGE; } + mmu->root_hpa = INVALID_PAGE; mmu->root_pgd = 0; } @@ -3306,9 +3306,23 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * the shadow page table may be a PAE or a long mode page table. */ pm_mask = PT_PRESENT_MASK; - if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) + if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) { pm_mask |= PT_ACCESSED_MASK | PT_WRITABLE_MASK | PT_USER_MASK; + /* + * Allocate the page for the PDPTEs when shadowing 32-bit NPT + * with 64-bit only when needed. Unlike 32-bit NPT, it doesn't + * need to be in low mem. See also lm_root below. + */ + if (!vcpu->arch.mmu->pae_root) { + WARN_ON_ONCE(!tdp_enabled); + + vcpu->arch.mmu->pae_root = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); + if (!vcpu->arch.mmu->pae_root) + return -ENOMEM; + } + } + for (i = 0; i < 4; ++i) { MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i])); if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) { @@ -3331,21 +3345,19 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); /* - * If we shadow a 32 bit page table with a long mode page - * table we enter this path. + * When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP + * tables are allocated and initialized at MMU creation as there is no + * equivalent level in the guest's NPT to shadow. Allocate the tables + * on demand, as running a 32-bit L1 VMM is very rare. The PDP is + * handled above (to share logic with PAE), deal with the PML4 here. */ if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) { if (vcpu->arch.mmu->lm_root == NULL) { - /* - * The additional page necessary for this is only - * allocated on demand. - */ - u64 *lm_root; lm_root = (void*)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (lm_root == NULL) - return 1; + if (!lm_root) + return -ENOMEM; lm_root[0] = __pa(vcpu->arch.mmu->pae_root) | pm_mask; @@ -5248,9 +5260,11 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) * while the PDP table is a per-vCPU construct that's allocated at MMU * creation. When emulating 32-bit mode, cr3 is only 32 bits even on * x86_64. Therefore we need to allocate the PDP table in the first - * 4GB of memory, which happens to fit the DMA32 zone. Except for - * SVM's 32-bit NPT support, TDP paging doesn't use PAE paging and can - * skip allocating the PDP table. + * 4GB of memory, which happens to fit the DMA32 zone. TDP paging + * generally doesn't use PAE paging and can skip allocating the PDP + * table. The main exception, handled here, is SVM's 32-bit NPT. The + * other exception is for shadowing L1's 32-bit or PAE NPT on 64-bit + * KVM; that horror is handled on-demand by mmu_alloc_shadow_roots(). */ if (tdp_enabled && kvm_mmu_get_tdp_level(vcpu) > PT32E_ROOT_LEVEL) return 0; From patchwork Tue Mar 2 18:45:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0871CC282E0 for ; Wed, 3 Mar 2021 07:23:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C858C64EDF for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381339AbhCCHSn (ORCPT ); Wed, 3 Mar 2021 02:18:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581642AbhCBTBX (ORCPT ); Tue, 2 Mar 2021 14:01:23 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5930FC0611C3 for ; Tue, 2 Mar 2021 10:46:05 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id lk3so2525990pjb.9 for ; Tue, 02 Mar 2021 10:46:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=o1VKZ6+LkEpFaDICD3QOjbJ8GdnaSlQdzWdXunTpsjE=; b=TtChc1k0rg4FV+bsfKDIAX1dpUzBoncazX8loWSo0PhZXBnzWTuyMTMJaVmepKzUAa RRE/xBkcYaoezCgX6Jj4z+UHGz4VTuRK4mQGfVp3SPNBG2O0WNXE2SpCxrU7cRYoL9Zu ZUv1hbr/rYZV61H8137hCE/WStVtAdVWrNKtyvnDLSgZYwz5VqT8X0JlAbdQhFQpSGTG e57MM0CNGFjTsuG/6oZPDcUuU7yKeAYJxO7XkcfgTAIWnELSeTq+T7QNsKQLaH5PbVKJ tqtje1+XSMVxZe4bNCXdRrn5s6eiL54g25w21h3PTGbr24L8+38PL7ujvb+DsZE5sJ6F hocA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=o1VKZ6+LkEpFaDICD3QOjbJ8GdnaSlQdzWdXunTpsjE=; b=bHwRWnix2XbJt92roetkkLpYq8L+7u1L+h0eKhdQGqA9qJvfHljrOdeQIDjaTGB8EL P2vSfPYjTtVrmG6l5KlRNlYmOv6hfBnNmu2RYjwlRVRwmaa1TD/ayaoFNpXYYYfe2Bxc 5VKqJVKRoCym/DBLXDSmSHxfMuUnEBWQixPWOlDyMZnCQJ+nA5Gsu9Bz0ab+rmId2b1R Ks8VkI3AVVKczSNnhp92owr3rW12ulr0UWFOPV/D/b+89GdokqZbZUAMq1cv431AKKsi SIAgD00jpDIqQ7buYjQTiB+o+/9EheI225V30dGS+CHbOU6dI3g+k94jKy242WO/de4z Fniw== X-Gm-Message-State: AOAM532B0VPbOmpDp/6FQftxTTn/0C9SaYk0b2HkCHByFjZ7zaz7yLdG PvM68CedWjx9JBg8+7UHoDvb//vpMkg= X-Google-Smtp-Source: ABdhPJx+UnXRWkcfqxPFPEu0Ip6uAE9yR6VcRfIEH3w0TM1YNbrpPpQpU/4GnGH7WZ0/NuKdc0GV0yqEetU= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:bdaf:: with SMTP id n47mr4819595qvg.48.1614710753763; Tue, 02 Mar 2021 10:45:53 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:28 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-4-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 03/15] KVM: x86/mmu: Ensure MMU pages are available when allocating roots From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hold the mmu_lock for write for the entire duration of allocating and initializing an MMU's roots. This ensures there are MMU pages available and thus prevents root allocations from failing. That in turn fixes a bug where KVM would fail to free valid PAE roots if a one of the later roots failed to allocate. Note, KVM still leaks the PAE roots if the lm_root allocation fails. This will be addressed in a future commit. Cc: Ben Gardon Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 41 ++++++++++++-------------------------- arch/x86/kvm/mmu/tdp_mmu.c | 23 +++++---------------- 2 files changed, 18 insertions(+), 46 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 2ed3fac1244e..1f129001a30c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2398,6 +2398,9 @@ static int make_mmu_pages_available(struct kvm_vcpu *vcpu) { unsigned long avail = kvm_mmu_available_pages(vcpu->kvm); + /* Ensure all four PAE roots can be allocated in a single pass. */ + BUILD_BUG_ON(KVM_MIN_FREE_MMU_PAGES < 4); + if (likely(avail >= KVM_MIN_FREE_MMU_PAGES)) return 0; @@ -3220,16 +3223,9 @@ static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gva, { struct kvm_mmu_page *sp; - write_lock(&vcpu->kvm->mmu_lock); - - if (make_mmu_pages_available(vcpu)) { - write_unlock(&vcpu->kvm->mmu_lock); - return INVALID_PAGE; - } sp = kvm_mmu_get_page(vcpu, gfn, gva, level, direct, ACC_ALL); ++sp->root_count; - write_unlock(&vcpu->kvm->mmu_lock); return __pa(sp->spt); } @@ -3241,16 +3237,10 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) if (is_tdp_mmu_enabled(vcpu->kvm)) { root = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu); - - if (!VALID_PAGE(root)) - return -ENOSPC; vcpu->arch.mmu->root_hpa = root; } else if (shadow_root_level >= PT64_ROOT_4LEVEL) { root = mmu_alloc_root(vcpu, 0, 0, shadow_root_level, true); - - if (!VALID_PAGE(root)) - return -ENOSPC; vcpu->arch.mmu->root_hpa = root; } else if (shadow_root_level == PT32E_ROOT_LEVEL) { for (i = 0; i < 4; ++i) { @@ -3258,8 +3248,6 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root = mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), i << 30, PT32_ROOT_LEVEL, true); - if (!VALID_PAGE(root)) - return -ENOSPC; vcpu->arch.mmu->pae_root[i] = root | PT_PRESENT_MASK; } vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); @@ -3294,8 +3282,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root = mmu_alloc_root(vcpu, root_gfn, 0, vcpu->arch.mmu->shadow_root_level, false); - if (!VALID_PAGE(root)) - return -ENOSPC; vcpu->arch.mmu->root_hpa = root; goto set_root_pgd; } @@ -3325,6 +3311,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) for (i = 0; i < 4; ++i) { MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i])); + if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) { pdptr = vcpu->arch.mmu->get_pdptr(vcpu, i); if (!(pdptr & PT_PRESENT_MASK)) { @@ -3338,8 +3325,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root = mmu_alloc_root(vcpu, root_gfn, i << 30, PT32_ROOT_LEVEL, false); - if (!VALID_PAGE(root)) - return -ENOSPC; vcpu->arch.mmu->pae_root[i] = root | pm_mask; } vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); @@ -3373,14 +3358,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) return 0; } -static int mmu_alloc_roots(struct kvm_vcpu *vcpu) -{ - if (vcpu->arch.mmu->direct_map) - return mmu_alloc_direct_roots(vcpu); - else - return mmu_alloc_shadow_roots(vcpu); -} - void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) { int i; @@ -4822,7 +4799,15 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) r = mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->direct_map); if (r) goto out; - r = mmu_alloc_roots(vcpu); + write_lock(&vcpu->kvm->mmu_lock); + if (make_mmu_pages_available(vcpu)) + r = -ENOSPC; + else if (vcpu->arch.mmu->direct_map) + r = mmu_alloc_direct_roots(vcpu); + else + r = mmu_alloc_shadow_roots(vcpu); + write_unlock(&vcpu->kvm->mmu_lock); + kvm_mmu_sync_roots(vcpu); if (r) goto out; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 70226e0875fe..50ef757c5586 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -137,22 +137,21 @@ static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, return sp; } -static struct kvm_mmu_page *get_tdp_mmu_vcpu_root(struct kvm_vcpu *vcpu) +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) { union kvm_mmu_page_role role; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_page *root; + lockdep_assert_held_write(&kvm->mmu_lock); + role = page_role_for_level(vcpu, vcpu->arch.mmu->shadow_root_level); - write_lock(&kvm->mmu_lock); - /* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root(kvm, root) { if (root->role.word == role.word) { kvm_mmu_get_root(kvm, root); - write_unlock(&kvm->mmu_lock); - return root; + goto out; } } @@ -161,19 +160,7 @@ static struct kvm_mmu_page *get_tdp_mmu_vcpu_root(struct kvm_vcpu *vcpu) list_add(&root->link, &kvm->arch.tdp_mmu_roots); - write_unlock(&kvm->mmu_lock); - - return root; -} - -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) -{ - struct kvm_mmu_page *root; - - root = get_tdp_mmu_vcpu_root(vcpu); - if (!root) - return INVALID_PAGE; - +out: return __pa(root->spt); } From patchwork Tue Mar 2 18:45:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BF9FC433DB for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A30E764EE8 for ; Wed, 3 Mar 2021 07:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379964AbhCCHSI (ORCPT ); Wed, 3 Mar 2021 02:18:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581587AbhCBS7P (ORCPT ); Tue, 2 Mar 2021 13:59:15 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A517C061222 for ; Tue, 2 Mar 2021 10:45:57 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id h126so17687219qkd.4 for ; Tue, 02 Mar 2021 10:45:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=J9+GlmlwlBYdlrT0JTLp/uAtDmX9L8zEya3ghwKZEGA=; b=tvhcBBrrgkXavFJY5bBbzP2Swe8VNCGt80pZSYzvqr/EHh9fliGx+4IlE5/Sqg1Trc RnOQGU9NdBkDWFhCVBAzRPTkNuL6KSpC09x9eQ76nXYJUGk+JIX/Cuc3wCQ6wtWCvfDD kW0i+57Xyo6UxAblgdxIxlY1Kf8qb67lVRvfdqlUcE873nmJcVr5b55CC6khTdGjDbZS hNPFkE9EMDBO+v0R8WflmgywQakD3cdXOvq9gn4JOd3hmvvhOI+ZFt8DnHA+g4qs0Pu2 NyGB2JarvZhjg2q5/61DLyUCelQdybWVkc866tUIZE2elNp6xPW1bN+/fB7eMlxy/Ph5 NyPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=J9+GlmlwlBYdlrT0JTLp/uAtDmX9L8zEya3ghwKZEGA=; b=Un/FE/W94JkArrsbgVEcOymHZ/ymu1DvtsUZEmEni3CHNaYd8Oh/wXFg8gsfeU6K7n TF+QbyV94hJk3AwYmPhW+eXjLmKlVg+mziXzXuhxMe4iyyZxlsddU+apoVzeALMsktJJ huXrkPsXLTPAG3x9vlHN1soYv+CeO1G1fghT0dyi9DuA744yUprcjB22aetzWYbWWBF0 2c8AZ+eJm0LNM5ozyc1stz+EC9M0nzBPA60AJzbusSr3UTwYQl9onMsQV/fr/FaSh59S pvZBjsspJQQQ3mC/g7evTccyZSwlVPSKgym97qjGQJ7JDjqfYOGP69VKRzrpgX7jQFLP SXCA== X-Gm-Message-State: AOAM531Ji78bIUDSMvVgPUYeGqPzzT/nCMve4JCXK9ixw/TvIjg8+sLo lFsrUS2/7lD6b/P9DraqjnlLuec7i3E= X-Google-Smtp-Source: ABdhPJxx3HS/5Rnb3Lb6mt6u5vJ+ymSDXuni5FaZ477WaCV+KwPK5g/CwhAbVJjcWEa/FwlcoAHADyZkDbQ= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:b509:: with SMTP id d9mr4749962qve.29.1614710756204; Tue, 02 Mar 2021 10:45:56 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:29 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-5-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 04/15] KVM: x86/mmu: Allocate the lm_root before allocating PAE roots From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Allocate lm_root before the PAE roots so that the PAE roots aren't leaked if the memory allocation for the lm_root happens to fail. Note, KVM can _still_ leak PAE roots if mmu_check_root() fails on a guest's PDPTR. That too will be fixed in a future commit. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 65 +++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 33 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1f129001a30c..e5c3701112f8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3292,21 +3292,39 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * the shadow page table may be a PAE or a long mode page table. */ pm_mask = PT_PRESENT_MASK; - if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) { + if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) pm_mask |= PT_ACCESSED_MASK | PT_WRITABLE_MASK | PT_USER_MASK; - /* - * Allocate the page for the PDPTEs when shadowing 32-bit NPT - * with 64-bit only when needed. Unlike 32-bit NPT, it doesn't - * need to be in low mem. See also lm_root below. - */ - if (!vcpu->arch.mmu->pae_root) { - WARN_ON_ONCE(!tdp_enabled); + /* + * When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP + * tables are allocated and initialized at root creation as there is no + * equivalent level in the guest's NPT to shadow. Allocate the tables + * on demand, as running a 32-bit L1 VMM is very rare. Unlike 32-bit + * NPT, the PDP table doesn't need to be in low mem. Preallocate the + * pages so that the PAE roots aren't leaked on failure. + */ + if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL && + (!vcpu->arch.mmu->pae_root || !vcpu->arch.mmu->lm_root)) { + u64 *lm_root, *pae_root; - vcpu->arch.mmu->pae_root = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (!vcpu->arch.mmu->pae_root) - return -ENOMEM; + if (WARN_ON_ONCE(!tdp_enabled || vcpu->arch.mmu->pae_root || + vcpu->arch.mmu->lm_root)) + return -EIO; + + pae_root = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); + if (!pae_root) + return -ENOMEM; + + lm_root = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); + if (!lm_root) { + free_page((unsigned long)pae_root); + return -ENOMEM; } + + vcpu->arch.mmu->pae_root = pae_root; + vcpu->arch.mmu->lm_root = lm_root; + + lm_root[0] = __pa(vcpu->arch.mmu->pae_root) | pm_mask; } for (i = 0; i < 4; ++i) { @@ -3327,30 +3345,11 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) PT32_ROOT_LEVEL, false); vcpu->arch.mmu->pae_root[i] = root | pm_mask; } - vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); - - /* - * When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP - * tables are allocated and initialized at MMU creation as there is no - * equivalent level in the guest's NPT to shadow. Allocate the tables - * on demand, as running a 32-bit L1 VMM is very rare. The PDP is - * handled above (to share logic with PAE), deal with the PML4 here. - */ - if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) { - if (vcpu->arch.mmu->lm_root == NULL) { - u64 *lm_root; - - lm_root = (void*)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (!lm_root) - return -ENOMEM; - - lm_root[0] = __pa(vcpu->arch.mmu->pae_root) | pm_mask; - - vcpu->arch.mmu->lm_root = lm_root; - } + if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->lm_root); - } + else + vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); set_root_pgd: vcpu->arch.mmu->root_pgd = root_pgd; From patchwork Tue Mar 2 18:45:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EE84C4360C for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 67D3564EDF for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381151AbhCCHSi (ORCPT ); Wed, 3 Mar 2021 02:18:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581639AbhCBTBV (ORCPT ); Tue, 2 Mar 2021 14:01:21 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 613D6C061223 for ; Tue, 2 Mar 2021 10:45:59 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id i69so7924829qke.19 for ; Tue, 02 Mar 2021 10:45:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=xSeBS7Kkuswt1KbMfW4UczMoQ+fxzPvhhz5Kktd9Pv0=; b=YUxL4pYACB8SQ3QzMqoxEsPYOaUy16pHdcAVDxMDQfIzuizatcQ8GzpfhI8eXsDb63 NADj9FSNeN27FxGFguuHLUI+HmvBOp3Su47bqhl1LK45RdU9SWj4T3KxFOMp9h/DkyNX l3NHkgOaHDcoZMImZFOBRObMOmkcT0WoJLROuTqwrS7qOuqyzUfjYQ7pQNR7k6FRAdD3 ZMvMwpkTx3tskoHMwaFpdfvL8zaWpXXy52K857z5n4sxkC2qVgbNnZ+jH1pWlRJ/Opxn SduuuqU1Umn5RvBjPDLAFOAXu7Dx8gkxVZGOpEse15hfhk6bD5vd0yv7EgWyOpu+Bjsb +vDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=xSeBS7Kkuswt1KbMfW4UczMoQ+fxzPvhhz5Kktd9Pv0=; b=ZbDtNJRuPNQem62LR3rtPd2zFu1sEV6+9xmppBz+bi7+ezNeMk4sBA9SbnJ28M8MDk c9OxsbH7fFedxosugBV7WswebnZU7z2r3KX1Lz79ObtGWAjaauLZN4p09zClYrZnTvKA 2FpfbT9qGp6YBqqMOnxR1MqeKB/AXzTUF3mDq2fNWs7GqkNJus9VCsWPFmKFpBEvxgRL nFGOJhcfBGcsFgkMCh6SbXTqBK6I+BW1nVDZSsgeZG7ZfuQCK/0uLqD2/uZcbVxMWeAf BG3Ont6RrIaXwQjmeOHTBOfh5Jwskhw47sAsAIcGtIjl91em5lPQQmd93486QfgduaGj 4oug== X-Gm-Message-State: AOAM532sfwChPY7xtnhYVoFmEW8JsmDxTf9wXCdz0MOftQwB73jLDg5r zK0phrQmpWktsod+xKhY2BeO6xTn7Qk= X-Google-Smtp-Source: ABdhPJw8xbuyNh4DfpQ3wgjXW+zW+ZzUIgplHu54WbdcY1+OcDQ8ex7kSdNUsQKBPIVmB3+nw0Fmh/TIPQg= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:9148:: with SMTP id q66mr21367972qvq.0.1614710758535; Tue, 02 Mar 2021 10:45:58 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:30 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-6-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 05/15] KVM: x86/mmu: Check PDPTRs before allocating PAE roots From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Check the validity of the PDPTRs before allocating any of the PAE roots, otherwise a bad PDPTR will cause KVM to leak any previously allocated roots. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index e5c3701112f8..aa20e8d32197 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3262,7 +3262,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { - u64 pdptr, pm_mask; + u64 pdptrs[4], pm_mask; gfn_t root_gfn, root_pgd; hpa_t root; int i; @@ -3273,6 +3273,17 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) if (mmu_check_root(vcpu, root_gfn)) return 1; + if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) { + for (i = 0; i < 4; ++i) { + pdptrs[i] = vcpu->arch.mmu->get_pdptr(vcpu, i); + if (!(pdptrs[i] & PT_PRESENT_MASK)) + continue; + + if (mmu_check_root(vcpu, pdptrs[i] >> PAGE_SHIFT)) + return 1; + } + } + /* * Do we shadow a long mode page table? If so we need to * write-protect the guests page table root. @@ -3331,14 +3342,11 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i])); if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) { - pdptr = vcpu->arch.mmu->get_pdptr(vcpu, i); - if (!(pdptr & PT_PRESENT_MASK)) { + if (!(pdptrs[i] & PT_PRESENT_MASK)) { vcpu->arch.mmu->pae_root[i] = 0; continue; } - root_gfn = pdptr >> PAGE_SHIFT; - if (mmu_check_root(vcpu, root_gfn)) - return 1; + root_gfn = pdptrs[i] >> PAGE_SHIFT; } root = mmu_alloc_root(vcpu, root_gfn, i << 30, From patchwork Tue Mar 2 18:45:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E38E7C43331 for ; Wed, 3 Mar 2021 07:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA73764EDF for ; Wed, 3 Mar 2021 07:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379923AbhCCHR4 (ORCPT ); Wed, 3 Mar 2021 02:17:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381093AbhCBS4f (ORCPT ); Tue, 2 Mar 2021 13:56:35 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC7C1C061226 for ; Tue, 2 Mar 2021 10:46:01 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v6so23498261ybk.9 for ; Tue, 02 Mar 2021 10:46:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=EnZFLski9RQiJKTWgrYgjdNat8xIrKSM9EmpDdRBoag=; b=scxjXT29yqaE1bBIE5x4Zgc6F4LOv8EyyWUiNAAv+qDm/hGXjM6o9kg5KhmWRVLUFM zhdR8/FEoWylqMQhxxHRXOmgM4H+qgRCyEhB8wDHUUXuiNZCbdlP9I1NtEUqdVkaTeam gugwntXjBdNu0JeTdAQsJcohYbSAh+LKkDEEfxFRTv1e3Ptcv+o+csDw8rXsx+ROPOPq PsNJl9LbRTh46hey3Z5YBQMwM9I1yUcTUhXsGy/vZXyLm/z6foWh2VFoUYuK3nA681DM DTp0zxmhLdpOzkoivjLs9wnh+z7T/CGXbwor9IaPLSpVlgJPeOWYp7nz0Gn2oUSa9Y78 3gLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=EnZFLski9RQiJKTWgrYgjdNat8xIrKSM9EmpDdRBoag=; b=hjdCuqW8v+uxB5/tqgkrq4urEztRvZfrUb9bPb/dzbDkBBWuDp3J2KaDNz+7UhOTfz JF0YFEbVrucVyR9Q2PNkSVZr8gsetY1rUl+4ji5FkcsuAb1HFvJ5NE371FpFaZHNgLYL a/2PiL5baE5eBu/QT/HQ12+KBBPl0M+lCfiVg3qbaQNqHV3ssaWr9+0M0nNLdevmdD6C yZnnGC5uUwwJvG4BtkX98RilmswUd4rPAlv7q1ckYrdPcp1vCbYgjAfIDMyc2n/tT38o Q32LVUMOs1I8F0BkGKX+fwVKPGFhEBavzXkjXfiS543OjzXFMx0tcai1PD2Wvum15jjC Hjvg== X-Gm-Message-State: AOAM530lY0U9hFshYojQwhJwcvYnhdHR8AHLUNMEXSYtFPsPUH8uJdLq yJP1fQ7SVtLtRTJkHXcCOCNsm6fwXi4= X-Google-Smtp-Source: ABdhPJwv3m9DetMSaQqbcfD5ZJfQ7ppifSrZOl1LVpPtrJ2Cq+kVO1NCxwUULwBeK8zws6nAxNFtsyNNbBo= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:e08b:: with SMTP id x133mr32155992ybg.138.1614710761217; Tue, 02 Mar 2021 10:46:01 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:31 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-7-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 06/15] KVM: x86/mmu: Fix and unconditionally enable WARNs to detect PAE leaks From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Exempt NULL PAE roots from the check to detect leaks, since kvm_mmu_free_roots() doesn't set them back to INVALID_PAGE. Stop hiding the WARNs to detect PAE root leaks behind MMU_WARN_ON, the hidden WARNs obviously didn't do their job given the hilarious number of bugs that could lead to PAE roots being leaked, not to mention the above false positive. Opportunistically delete a warning on root_hpa being valid, there's nothing special about 4/5-level shadow pages that warrants a WARN. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index aa20e8d32197..3ef7fb2a9878 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3244,7 +3244,8 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_hpa = root; } else if (shadow_root_level == PT32E_ROOT_LEVEL) { for (i = 0; i < 4; ++i) { - MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i])); + WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i] && + VALID_PAGE(vcpu->arch.mmu->pae_root[i])); root = mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), i << 30, PT32_ROOT_LEVEL, true); @@ -3289,8 +3290,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * write-protect the guests page table root. */ if (vcpu->arch.mmu->root_level >= PT64_ROOT_4LEVEL) { - MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->root_hpa)); - root = mmu_alloc_root(vcpu, root_gfn, 0, vcpu->arch.mmu->shadow_root_level, false); vcpu->arch.mmu->root_hpa = root; @@ -3339,7 +3338,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) } for (i = 0; i < 4; ++i) { - MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i])); + WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i] && + VALID_PAGE(vcpu->arch.mmu->pae_root[i])); if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) { if (!(pdptrs[i] & PT_PRESENT_MASK)) { From patchwork Tue Mar 2 18:45:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E9F8C432C3 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3542D64EF3 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381127AbhCCHSf (ORCPT ); Wed, 3 Mar 2021 02:18:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581632AbhCBTBL (ORCPT ); Tue, 2 Mar 2021 14:01:11 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49C39C0611C1 for ; Tue, 2 Mar 2021 10:46:04 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id l10so23737797ybt.6 for ; Tue, 02 Mar 2021 10:46:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=kSiLnMDkk0ckb4albKKYSCbZaXElHQozT1Z/Ps7GfuI=; b=OHQSzC1bWC97Wo53/ZEgH7WOW5Ud0ZEnUvSaVvwdq0Ra5F8k+TKijqaCAAblPm/5Dj oS9Z5TX7fNhYfkDqOWOquzZMzgUZ0T+Al3eaKVABvAHfEVJQUcb015O9QRMbVLhNcCw0 ThjWF23s8VRE2VMwpEsEZxFwRs0OJwpQ6n6IZpB4w/v9lJukl0TqknRXeh/2uGIOQeNe hkX6ChPh1Iq1RNBAMpC/+dx0BD5GseF6sHk/ffR1CAz4dXqRAmkjyQdy78zTtft+S+cc tKq4/73jmYQ9JJhdJ1oGQbX5eAof9aOBtgXzkB1Np4O+3uF8TGNWJ5cvdDzIThnAjv+Z juMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=kSiLnMDkk0ckb4albKKYSCbZaXElHQozT1Z/Ps7GfuI=; b=ZThz7WVGsbkipZRUS6+B4mYr9GYzCopSIdxPBoddC5seg566kfVx/D0UFPmsYhSaVc +zzMcfl0RvbuNB7ani6jhd1PvPEnW/Fj1Tgl72VAa0DUK9iBZwV7azKt5upB15+2/Upn Gq7TKyVKNx7/u89XOtsPSS1GABztM+MS66EABCIX8iI0b0UHhHvvArJzahF2YLdJunGF EI00346zrZz+UQyHe8GTI+0a0TiQZ61AjLlXAYz6M3r5c8UxFiRS5oZuP4vQkVrvTw8T chTWnVHILJanwXQSER3khgjy76KKMs21zP1Gi1ubiCDNfZ6wPulL51ksYTxAPHIOYCW+ 8rrg== X-Gm-Message-State: AOAM533cW5BQ2X86LofBCwEk1wTGRM3ng8oJM52zml40HjMVIg9afRlk 4nxII5Ycd/cyGDDORArgCJRQWNJpV0Q= X-Google-Smtp-Source: ABdhPJyhoCBxSZ78KfAXRL1vCHIkU/3TxPhR2oR2b1EZQg7JjcPgRbrYiftVi5G0tXXzEVlJaeHqFI3Eapk= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:d296:: with SMTP id j144mr32687925ybg.33.1614710763543; Tue, 02 Mar 2021 10:46:03 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:32 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-8-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 07/15] KVM: x86/mmu: Use '0' as the one and only value for an invalid PAE root From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use '0' to denote an invalid pae_root instead of '0' or INVALID_PAGE. Unlike root_hpa, the pae_roots hold permission bits and thus are guaranteed to be non-zero. Having to deal with both values leads to bugs, e.g. failing to set back to INVALID_PAGE, warning on the wrong value, etc... Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 3ef7fb2a9878..59b1709a55b4 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3191,11 +3191,14 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) { mmu_free_root_page(kvm, &mmu->root_hpa, &invalid_list); } else if (mmu->pae_root) { - for (i = 0; i < 4; ++i) - if (mmu->pae_root[i] != 0) - mmu_free_root_page(kvm, - &mmu->pae_root[i], - &invalid_list); + for (i = 0; i < 4; ++i) { + if (!mmu->pae_root[i]) + continue; + + mmu_free_root_page(kvm, &mmu->pae_root[i], + &invalid_list); + mmu->pae_root[i] = 0; + } } mmu->root_hpa = INVALID_PAGE; mmu->root_pgd = 0; @@ -3244,8 +3247,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_hpa = root; } else if (shadow_root_level == PT32E_ROOT_LEVEL) { for (i = 0; i < 4; ++i) { - WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i] && - VALID_PAGE(vcpu->arch.mmu->pae_root[i])); + WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i]); root = mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), i << 30, PT32_ROOT_LEVEL, true); @@ -3338,8 +3340,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) } for (i = 0; i < 4; ++i) { - WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i] && - VALID_PAGE(vcpu->arch.mmu->pae_root[i])); + WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i]); if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) { if (!(pdptrs[i] & PT_PRESENT_MASK)) { @@ -3412,7 +3413,7 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) for (i = 0; i < 4; ++i) { hpa_t root = vcpu->arch.mmu->pae_root[i]; - if (root && VALID_PAGE(root)) { + if (root && !WARN_ON_ONCE(!VALID_PAGE(root))) { root &= PT64_BASE_ADDR_MASK; sp = to_shadow_page(root); mmu_sync_children(vcpu, sp); @@ -5267,7 +5268,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) mmu->pae_root = page_address(page); for (i = 0; i < 4; ++i) - mmu->pae_root[i] = INVALID_PAGE; + mmu->pae_root[i] = 0; return 0; } From patchwork Tue Mar 2 18:45:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39DF8C43333 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1AB7F64EDF for ; Wed, 3 Mar 2021 07:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379939AbhCCHR6 (ORCPT ); Wed, 3 Mar 2021 02:17:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381232AbhCBS4i (ORCPT ); Tue, 2 Mar 2021 13:56:38 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8B98C0611BD for ; Tue, 2 Mar 2021 10:46:06 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id h126so17687863qkd.4 for ; Tue, 02 Mar 2021 10:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=pDw5a8xkwlwJrFheedicuJ7ZkUuVv2t+v7LNaWJm4Dw=; b=Ygk5jcCgeatzea90Rb8bzEwERdMBWJFL7meF/8n+9C1z76ccYNoc3o2KHJs49c9YS9 +rED1LRhDhhfZQrVJkSnnRdejrl1l/JXRTuDtcFlvwFXdsr6wWnOW/xNRcHG8+WBe4Yb WSUWTNcl7oNaDRe8m2fCdRaYBmA1vX0xKIJA/cCqU1oHZ8fnhVJPmR7yDURMfrk9M0z0 bTdi/oemt2W0JE3Qu6l2YHFXzLd5nJ84yxHKqXrQfB/seJYeJ1osrih0tN+q0gmF6i1r 1RzQGwI6nk4DVdJ6a1GDGTSLqYs/gkZ2qFom4DxeaJDP3wDuxXgXxo1D0vc0eT6FZ0+L byLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=pDw5a8xkwlwJrFheedicuJ7ZkUuVv2t+v7LNaWJm4Dw=; b=NRDSSjQ+LzyJ40Wp+yc1N39NVkkn2Od5bN0bdr8kicggPlEmX1zJ9XcOYH0Rhp0Tsi J9F7LfhazsCR3fvCNi/wdJgw4dkb86t/kdm5DssTA03uHUnAjcUOL3zp3Bvam3zdHQE3 hFEmfq9XHVqxon4XV6/PtzNmC9guzz6Q8l5qY1Pe/P8gyLMj0nb70K/G1lKcJp/TIflP 2m+8t4gGjFJezljBFoHivOZtEvf2+tKxGDKEVWALK7kbexwlR2tSkzMldwUApYlwnu+P /5MrcGA0ueQ1hHxbSc+J9EV9iIb4+Ikrw0SS1ko358BQYU/VjEv+PlB/kNhLY1XuAYHA XF1w== X-Gm-Message-State: AOAM533Pc1TJJme9l76q+jCMy3OihiTNOK9kJc9fu7BSDqIlfTWuOBKV id6ur8T8dB7qhYKyoB4fGQxP2W+VNRE= X-Google-Smtp-Source: ABdhPJy3aPjJhvDHE+DIXi6sA+MkP5OXW5n7T3/Tf9elJsyZ3fz2ZUT3gXs7k0/Itw25dW+LTYzmWA5qGC4= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:b7a1:: with SMTP id l33mr20824163qve.17.1614710766083; Tue, 02 Mar 2021 10:46:06 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:33 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-9-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 08/15] KVM: x86/mmu: Set the C-bit in the PDPTRs and LM pseudo-PDPTRs From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Set the C-bit in SPTEs that are set outside of the normal MMU flows, specifically the PDPDTRs and the handful of special cased "LM root" entries, all of which are shadow paging only. Note, the direct-mapped-root PDPTR handling is needed for the scenario where paging is disabled in the guest, in which case KVM uses a direct mapped MMU even though TDP is disabled. Fixes: d0ec49d4de90 ("kvm/x86/svm: Support Secure Memory Encryption within KVM") Cc: stable@vger.kernel.org Cc: Brijesh Singh Cc: Tom Lendacky Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 59b1709a55b4..ddf1845f072e 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3251,7 +3251,8 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root = mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), i << 30, PT32_ROOT_LEVEL, true); - vcpu->arch.mmu->pae_root[i] = root | PT_PRESENT_MASK; + vcpu->arch.mmu->pae_root[i] = root | PT_PRESENT_MASK | + shadow_me_mask; } vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); } else @@ -3303,7 +3304,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * or a PAE 3-level page table. In either case we need to be aware that * the shadow page table may be a PAE or a long mode page table. */ - pm_mask = PT_PRESENT_MASK; + pm_mask = PT_PRESENT_MASK | shadow_me_mask; if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) pm_mask |= PT_ACCESSED_MASK | PT_WRITABLE_MASK | PT_USER_MASK; From patchwork Tue Mar 2 18:45:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF6F7C28CC4 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9896F64ECE for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381237AbhCCHSk (ORCPT ); Wed, 3 Mar 2021 02:18:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581640AbhCBTBW (ORCPT ); Tue, 2 Mar 2021 14:01:22 -0500 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E4F3C0611BE for ; Tue, 2 Mar 2021 10:46:09 -0800 (PST) Received: by mail-qt1-x849.google.com with SMTP id p37so4315214qtb.1 for ; Tue, 02 Mar 2021 10:46:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=yssg349CIQcpusokJYUAbzu7BIQqlN4sh1bHcP1NJ8k=; b=ErDuGBDftepRR4OYZpswTC6tmPwDS8yB53aypu/42YKgFqbqnAGsFo+VNVcCZ98tS4 LKYeb3A15u0xi54RX3uLZK9VBam6kPhbLVfbO5EPrsWM/7FM7i3Yjgo+2XZ1bjBKijR+ QndDzLzFLHN1LcG72D2ixeApjOruIK06aBt6OZDWLC/cMvtAtrsOxRdIMSjkGC2+sfEg ToL/umXHkuxZv/yoymGi1+ngHX3ab3v4Eoa3T7M4Q0mPgm62xNCEpiwfqnQ5dFgF4yN8 m/Kl/jX9s+X2zlJCc8RtK5/NIVo7V+zLDPwoL/VxV9id8H/4vMppnofbMplq7I5JhWIG Kcbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=yssg349CIQcpusokJYUAbzu7BIQqlN4sh1bHcP1NJ8k=; b=n8pgQlAru6C2BRhsym42GCIoXuSQcoaO4xFStFsEiBhLd8UdB8cET5sw/andIT+4uI XOUNtSbO2kYodsXYNuAYsiRxkXOS625yEwKPb2tFxJfJqv8GsRo92l6cy7NjyMxW+aeU i3B3n7bu/5w3y23+NIKDGQq+vKNKKvVTR8cmxHUhgBXxnaNJKj/2+CRBh569CNIW2P3/ 3VcdhVDtl74LtyxE+9R6cPYOlbHB4zK1xP3L/MTummF7hStLCQypSMegxEqlMIqUQC5o r9iOIgQCI2ZNR8GkJT+t8olklncfh2XHelUtoG/IBRSxwJtWztm1pYkIRELgQr7vx5WO iTgg== X-Gm-Message-State: AOAM530ynBP7kYa9+6MnVppLtGONCcggj+nbTj6wPLNhl0Es+Um4800M jEYJKxMMtoLfjpNKjVlWDe59stMhanQ= X-Google-Smtp-Source: ABdhPJyrDbObIs1DThtN5Ro3gDtAE1iGh/GjGP6Ti2k30XVBdhgm3at2LNs7pWUWgjLMUTKAU1r92bsSnm0= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a0c:9ba4:: with SMTP id o36mr4947619qve.31.1614710768589; Tue, 02 Mar 2021 10:46:08 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:34 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-10-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 09/15] KVM: x86/mmu: Mark the PAE roots as decrypted for shadow paging From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Set the PAE roots used as decrypted to play nice with SME when KVM is using shadow paging. Explicitly skip setting the C-bit when loading CR3 for PAE shadow paging, even though it's completely ignored by the CPU. The extra documentation is nice to have. Note, there are several subtleties at play with NPT. In addition to legacy shadow paging, the PAE roots are used for SVM's NPT when either KVM is 32-bit (uses PAE paging) or KVM is 64-bit and shadowing 32-bit NPT. However, 32-bit Linux, and thus KVM, doesn't support SME. And 64-bit KVM can happily set the C-bit in CR3. This also means that keeping __sme_set(root) for 32-bit KVM when NPT is enabled is conceptually wrong, but functionally ok since SME is 64-bit only. Leave it as is to avoid unnecessary pollution. Fixes: d0ec49d4de90 ("kvm/x86/svm: Support Secure Memory Encryption within KVM") Cc: stable@vger.kernel.org Cc: Brijesh Singh Cc: Tom Lendacky Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 24 ++++++++++++++++++++++-- arch/x86/kvm/svm/svm.c | 7 +++++-- 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ddf1845f072e..45fe97b3b25d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include "trace.h" @@ -3313,8 +3314,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * tables are allocated and initialized at root creation as there is no * equivalent level in the guest's NPT to shadow. Allocate the tables * on demand, as running a 32-bit L1 VMM is very rare. Unlike 32-bit - * NPT, the PDP table doesn't need to be in low mem. Preallocate the - * pages so that the PAE roots aren't leaked on failure. + * NPT, the PDP table doesn't need to be in low mem, and doesn't need + * to be decrypted. Preallocate the pages so that the PAE roots aren't + * leaked on failure. */ if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL && (!vcpu->arch.mmu->pae_root || !vcpu->arch.mmu->lm_root)) { @@ -5234,6 +5236,8 @@ slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, static void free_mmu_pages(struct kvm_mmu *mmu) { + if (!tdp_enabled && mmu->pae_root) + set_memory_encrypted((unsigned long)mmu->pae_root, 1); free_page((unsigned long)mmu->pae_root); free_page((unsigned long)mmu->lm_root); } @@ -5271,6 +5275,22 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) for (i = 0; i < 4; ++i) mmu->pae_root[i] = 0; + /* + * CR3 is only 32 bits when PAE paging is used, thus it's impossible to + * get the CPU to treat the PDPTEs as encrypted. Decrypt the page so + * that KVM's writes and the CPU's reads get along. Note, this is + * only necessary when using shadow paging, as 64-bit NPT can get at + * the C-bit even when shadowing 32-bit NPT, and SME isn't supported + * by 32-bit kernels (when KVM itself uses 32-bit NPT). + */ + if (!tdp_enabled) + set_memory_decrypted((unsigned long)mmu->pae_root, 1); + else + WARN_ON_ONCE(shadow_me_mask); + + for (i = 0; i < 4; ++i) + mmu->pae_root[i] = 0; + return 0; } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 54610270f66a..4769cf8bf2fd 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3908,15 +3908,18 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long root, struct vcpu_svm *svm = to_svm(vcpu); unsigned long cr3; - cr3 = __sme_set(root); if (npt_enabled) { - svm->vmcb->control.nested_cr3 = cr3; + svm->vmcb->control.nested_cr3 = __sme_set(root); vmcb_mark_dirty(svm->vmcb, VMCB_NPT); /* Loading L2's CR3 is handled by enter_svm_guest_mode. */ if (!test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail)) return; cr3 = vcpu->arch.cr3; + } else if (vcpu->arch.mmu->shadow_root_level >= PT64_ROOT_4LEVEL) { + cr3 = __sme_set(root); + } else { + cr3 = root; } svm->vmcb->save.cr3 = cr3; From patchwork Tue Mar 2 18:45:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E8ECC43381 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8BF1B64ECE for ; Wed, 3 Mar 2021 07:23:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1380658AbhCCHSZ (ORCPT ); Wed, 3 Mar 2021 02:18:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581630AbhCBTA7 (ORCPT ); Tue, 2 Mar 2021 14:00:59 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3749C061A28 for ; Tue, 2 Mar 2021 10:46:11 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id i2so23705022ybl.16 for ; Tue, 02 Mar 2021 10:46:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=UoSJXw/PVEOjBFScqgNo0bQzSNuTFSWEQtCbZglmTq0=; b=lwgXT6FI8+/1UT8FGrjiTavPAr0JKYB0UWbkjGqt7EyPK3rhZ6rAiI6lBmAT7WA898 e0BEX/4xziKzMBSqkmjDozDuK5WnSHbDOG+eR8mInErGT+TbXD4at9EZnRInk/6rb9Cj BoWKjo0tzn+DwYSsG3imVzuTgoRlC9M73YKnNUSMR+8Dj0WsiVXdxUzAj8myq/vzRXr1 dlkurnV+eEAgGbSZqtYSHfu24HYZ4G8kXJNHvvkHlFaVk9DfSsXczRU+W1HualcuLUee z8dBZsxRkZ8lBY3j721lFPn5lJZj3po3y/bOCrwt906AIJwZK7pyl4iQRnxw+/q2k0VD 055w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=UoSJXw/PVEOjBFScqgNo0bQzSNuTFSWEQtCbZglmTq0=; b=O3kDAwVZTyA/Uqyih1Y5yocLa8Aa0RV28q66E2lKommjI+wQtF3Ak8Kaij7KWhvlfs G7KHH0f0JiAD7Bt10XHk9GcNfo25pDiLjQ9agui2r1cW9ycpZ/l8Tfgqd11+kLUhFP3w VqEotCOlOKdPgza9PCW4MxqvJkd6EaD75jD6gUg70r6FkbOY+OoNKYYTV42TYf1vIGhc GgMBPc/gYTV7PacF6nqwQ/dL1LCgOGAIpJBUJjdbcn0oJnGqXQo+/hdraaiSutVMHqEv EL3eKO6RVqW45TaI7sg9D0n/ckCzom9PzbQ++JIftmboC3I2E6dzCHU7nFPU8Z6VPI7V 7Ubw== X-Gm-Message-State: AOAM531wIRKFDdmd5BTQQt62YEli/dyB2IJXFuPA7a1GG9Gdy9B46oWv 215+n32YJfuw4JxZUYXIfYTpdrV3twA= X-Google-Smtp-Source: ABdhPJxXp9gj3TQbYMpbnDiuJU1nyT2DyJMbSHnnOW+armBYCgjzJRMfMkf1ZDQJr9aOpisAIIbUxejbTis= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:76c3:: with SMTP id r186mr33750361ybc.365.1614710770992; Tue, 02 Mar 2021 10:46:10 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:35 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-11-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 10/15] KVM: SVM: Don't strip the C-bit from CR2 on #PF interception From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Don't strip the C-bit from the faulting address on an intercepted #PF, the address is a virtual address, not a physical address. Fixes: 0ede79e13224 ("KVM: SVM: Clear C-bit from the page fault address") Cc: stable@vger.kernel.org Cc: Brijesh Singh Cc: Tom Lendacky Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 4769cf8bf2fd..dfc8fe231e8b 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1907,7 +1907,7 @@ static int pf_interception(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); - u64 fault_address = __sme_clr(svm->vmcb->control.exit_info_2); + u64 fault_address = svm->vmcb->control.exit_info_2; u64 error_code = svm->vmcb->control.exit_info_1; return kvm_handle_page_fault(vcpu, error_code, fault_address, From patchwork Tue Mar 2 18:45:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70F0FC4361B for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 30EC164EEB for ; Wed, 3 Mar 2021 07:23:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1380072AbhCCHSU (ORCPT ); Wed, 3 Mar 2021 02:18:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581590AbhCBS7P (ORCPT ); Tue, 2 Mar 2021 13:59:15 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D575C061A2B for ; Tue, 2 Mar 2021 10:46:14 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id y7so23426705ybh.20 for ; Tue, 02 Mar 2021 10:46:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=VvZxbsS7usA2XSAkfRC0sbz46vD+BnH1u7vFisa6mO0=; b=V1nMCm/qEh7jjrjHA5O0Kq+9vwKGhxQ0BuRI1m3H0kbtdE/n3BBMW8oK2oQr9bXjDt PlVkbEtvf6X7KthvohEdaYM5+qKB3tfWC2CPgUN7T50xS0/rXXJxB7k4/Gufjj41M4V8 35IencokX7qCTSsMr9RRQmdZkFBqdMuM6xkHBfy/Am0ikQDoXz19y01yRfgiyvkX2LwS jR8faVtHCxd1J/ek0BWuTSFEqB/mNoiCjh8ubc1pSXHPqkkU/gIrMevjKjalqfo97ZkQ QFSUYvzLJpV8YpwkK+kdQtAa7LC+ugUgOcaswbEby5TKIM7y31fFovUze0OUkvbQkIE5 YB8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=VvZxbsS7usA2XSAkfRC0sbz46vD+BnH1u7vFisa6mO0=; b=LfTzaHGLvngVLcfbd3MTOGS2JbuAJxBKcjdMxiy378hH0gQveh+zHJSThCv6fT756n REAApxhx5bqXwu0tKLIzjuPi46VSPF3AI09LIfkk5IVyYwAfGgz5ehwOupau3Ftfum0G 2XwKQBAX8RtM19XiHHbJ0lXBerCji1gNwIyw/M6oMCdKxX895XaGhY7K7Mhq8s8yy17s +BtbRi2PlVVisFpCPtxpmV1bi+yTX7PRn7+uUcaYRWxvK2bz9f8Lfl332k0v9Gc4E91U lF8/BZbRvVOnyLiTYbSSyWWNt1oozYADb7eFsj7of2O8J5TypM1IXjEL5vNglXAC45HA +phw== X-Gm-Message-State: AOAM530eQaNi+YCpx443FU3L15aDNftjZzYaC6KObR2cExMuOHMVjvf1 eEhsFeHX6lCaK3lhB/6tGqd0EKWxxzo= X-Google-Smtp-Source: ABdhPJzA/JVavTc1YLp1/c2Tzi52WiK+H305SJZmZc1wRoh9Mn2md0n0TSnsgq5eJ1jz5pb6Ia41rdwDf0E= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:d843:: with SMTP id p64mr31348287ybg.339.1614710773295; Tue, 02 Mar 2021 10:46:13 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:36 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-12-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 11/15] KVM: nVMX: Defer the MMU reload to the normal path on an EPTP switch From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Defer reloading the MMU after a EPTP successful EPTP switch. The VMFUNC instruction itself is executed in the previous EPTP context, any side effects, e.g. updating RIP, should occur in the old context. Practically speaking, this bug is benign as VMX doesn't touch the MMU when skipping an emulated instruction, nor does queuing a single-step #DB. No other post-switch side effects exist. Fixes: 41ab93727467 ("KVM: nVMX: Emulate EPTP switching for the L1 hypervisor") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/nested.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index fdd80dd8e781..81f609886c8b 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -5473,16 +5473,11 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, if (!nested_vmx_check_eptp(vcpu, new_eptp)) return 1; - kvm_mmu_unload(vcpu); mmu->ept_ad = accessed_dirty; mmu->mmu_role.base.ad_disabled = !accessed_dirty; vmcs12->ept_pointer = new_eptp; - /* - * TODO: Check what's the correct approach in case - * mmu reload fails. Currently, we just let the next - * reload potentially fail - */ - kvm_mmu_reload(vcpu); + + kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu); } return 0; From patchwork Tue Mar 2 18:45:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FE5CC43603 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CD8B164EE6 for ; Wed, 3 Mar 2021 07:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379990AbhCCHSN (ORCPT ); Wed, 3 Mar 2021 02:18:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581586AbhCBS7P (ORCPT ); Tue, 2 Mar 2021 13:59:15 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85DCEC061A2D for ; Tue, 2 Mar 2021 10:46:16 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id o9so23337417yba.18 for ; Tue, 02 Mar 2021 10:46:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=hUIu7sVBeZPHOKAKqkQEBvI6/Cyva1cSCDUAiAjyS7A=; b=U2m54mTmzJt31QJW9BmtK3SiG+FsiHpO3pN2uZt9Mw4OQZ5bvlOnQDJbtecM7GTN30 nFGCDx0rmiUX+THo6IaqwBTAr4uIgDycnai7BtIkHIveD7iIhDTxrQQatEK/mnedrwR0 C4Cy+o7VeCgOHBuvDGuGyf/1Il9Zq/BsWpIhq3pTzyqCaWUAcq8PRq/79spzOuiuLgNs 766OeInO7N3Hv1FTAsGqje1x2GBAuXDbSS4L7AA67n8XoGNTEELiZ6celCpyD26EqiGT WbTrJv+R9ITgernzX5ERoc9HPeuiSfFmo+ErL6IFKY9lQ/FHw0V8Z057E7YrfWL777Wf TbWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=hUIu7sVBeZPHOKAKqkQEBvI6/Cyva1cSCDUAiAjyS7A=; b=jY4rmj/ShcNJgBw2s1iyBTWcHbClRLJEUI+qWX1aJodeJ5bROuwgN6WYVmBtQpJqZ3 g/NVRdzSiSPZikLelrHIGArOIRY8g/KpaaBZwNt4bheB2m08W9R9LhDT9jqSGd3WKdM0 TzWDdgEynnBY0jPfYZnv0Er4bvSCCiJrVczD9dFYGCy1u8WIeiYXvwfrqjWFzXnwY/JC C+mlk0NalSjDFDip+F0uOdGiHXqPZnpmAVEuur9dZYDXXdLv4NeEUVZkQLwMwO55REKf ROznn5bhVBu9ZX6e2aiMg/Ch6ih9S3Qpp0B9fvZaH+vL1bPkhWQ3A0SKnE/2m3n6a82+ avYw== X-Gm-Message-State: AOAM533sFS+68iDvwDOjAGDVI/NAT7NT0baJuKgcvjqbMXaZ8SaEfU9k QWYy+fVJNnpwpNrVyoFWfvZyP2m0JQM= X-Google-Smtp-Source: ABdhPJy833CeQjsl9oNWyBsX/oLj6kjISj/tIBT30+qFHWc9PvBK68fihwd+i7l6jp/KwPZoZjK8k+yUeiI= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:310b:: with SMTP id x11mr8187618ybx.23.1614710775734; Tue, 02 Mar 2021 10:46:15 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:37 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-13-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 12/15] KVM: x86: Defer the MMU unload to the normal path on an global INVPCID From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Defer unloading the MMU after a INVPCID until the instruction emulation has completed, i.e. until after RIP has been updated. On VMX, this is a benign bug as VMX doesn't touch the MMU when skipping an emulated instruction. However, on SVM, if nrip is disabled, the emulator is used to skip an instruction, which would lead to fireworks if the emulator were invoked without a valid MMU. Fixes: eb4b248e152d ("kvm: vmx: Support INVPCID in shadow paging mode") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 828de7d65074..7b0adebec1ef 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11531,7 +11531,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva) fallthrough; case INVPCID_TYPE_ALL_INCL_GLOBAL: - kvm_mmu_unload(vcpu); + kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu); return kvm_skip_emulated_instruction(vcpu); default: From patchwork Tue Mar 2 18:45:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 355E5C4332E for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 74C2164EE1 for ; Wed, 3 Mar 2021 07:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379958AbhCCHSD (ORCPT ); Wed, 3 Mar 2021 02:18:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581572AbhCBS6x (ORCPT ); Tue, 2 Mar 2021 13:58:53 -0500 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9319C061A31 for ; Tue, 2 Mar 2021 10:46:18 -0800 (PST) Received: by mail-qt1-x84a.google.com with SMTP id w33so4941906qte.11 for ; Tue, 02 Mar 2021 10:46:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=YzlwVtABhaf0njzwLJlpB8Ftz/jd7QmttBOqgMTZo3A=; b=SeLHzq4Cio/vlmKsRAvq5kQSfdRJlLn2GUI+BOfuisLlBY/irxgPzGRMTMLdHWUzYv mP77Z3IyzSj9cPmibnzAFfsUJ72mWaSyRaH7I5JveSFb8ms5iarARutwqjjF3UmIEd0J DqMICED+eUvkTFxyB7raHdpfm953g3h3uSu3c+DeZ9ujlOfZnyUIhoZZOvMxQskn/jwY yaHIRjOujHbFpog+Fu+GL62qoel2BZHGqJ44cNAZUhjQI9R/XeN6qmC1Godp1LDwWpa7 QUvud7WWztAvJFKXEA5nmSbaK0JwGCstMOWaWna3hwDbF/LOYXaadLdDrBHiuwELxbhJ JaAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=YzlwVtABhaf0njzwLJlpB8Ftz/jd7QmttBOqgMTZo3A=; b=qtSXwue9w3ou9ySGTu0eSlgT41JLxL1iIQZWIPIzdXvC82iqfQ7q6sqQRd3CSCJNJ9 krwPB8pSYrIueCzmw2isqiGxpzZmtc9NC/KThpEOb+K8jLmSOYaR6pz2VtH0brqdipIq XQ/OOcVGeZM5kC9bD4E7kFjepBTFGn8NS2rPAS/ayP9uqxs8dq5Myc3MV+nPcRdD6H1H qczKzTFs1vX16aEf/KyF5T5xS1B6rlbKRjsxyW/EPWwWa+jnVS8KLQ+x0XTPG24x3fBQ 2kiyPWDtbPRgPbSxfIlmibDcOTe7lo2+djOeC3bnpY7mGsVC4kPUeLrPlOOd+znjaD3v /7Uw== X-Gm-Message-State: AOAM530zv6K1dmjqCQGyfS3SLwsTsiNSbRorDczS4h7n/0XvAe7tAyQu 9vfZPyt6Nj5MfFlvg7Wzoqs+w/5WzcQ= X-Google-Smtp-Source: ABdhPJzq1ZpQne4ImjhHgZA5rgE+d1F20qlPL09XKkXjdXZFxxMHkllbyy3mJ1z0hMVCHHwDEAKxHx4GtV8= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a05:6214:2262:: with SMTP id gs2mr5093265qvb.32.1614710778040; Tue, 02 Mar 2021 10:46:18 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:38 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-14-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 13/15] KVM: x86/mmu: Unexport MMU load/unload functions From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Unexport the MMU load and unload helpers now that they are no longer used (incorrectly) in vendor code. Opportunistically move the kvm_mmu_sync_roots() declaration into mmu.h, it should not be exposed to vendor code. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/mmu.h | 4 ++++ arch/x86/kvm/mmu/mmu.c | 2 -- 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 6db60ea8ee5b..2da6c9f5935a 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1592,9 +1592,6 @@ void kvm_update_dr7(struct kvm_vcpu *vcpu); int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn); void __kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu); -int kvm_mmu_load(struct kvm_vcpu *vcpu); -void kvm_mmu_unload(struct kvm_vcpu *vcpu); -void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, ulong roots_to_free); gpa_t translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u32 access, diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 72b0f66073dc..67e8c7c7a6ce 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -74,6 +74,10 @@ bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu); int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, u64 fault_address, char *insn, int insn_len); +int kvm_mmu_load(struct kvm_vcpu *vcpu); +void kvm_mmu_unload(struct kvm_vcpu *vcpu); +void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); + static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { if (likely(vcpu->arch.mmu->root_hpa != INVALID_PAGE)) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 45fe97b3b25d..86432d6a4092 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4827,7 +4827,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out: return r; } -EXPORT_SYMBOL_GPL(kvm_mmu_load); void kvm_mmu_unload(struct kvm_vcpu *vcpu) { @@ -4836,7 +4835,6 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu) kvm_mmu_free_roots(vcpu, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); WARN_ON(VALID_PAGE(vcpu->arch.guest_mmu.root_hpa)); } -EXPORT_SYMBOL_GPL(kvm_mmu_unload); static bool need_remote_flush(u64 old, u64 new) { From patchwork Tue Mar 2 18:45:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34F26C433E0 for ; Wed, 3 Mar 2021 07:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 02E2564EEC for ; Wed, 3 Mar 2021 07:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1380003AbhCCHSQ (ORCPT ); Wed, 3 Mar 2021 02:18:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581592AbhCBS7P (ORCPT ); Tue, 2 Mar 2021 13:59:15 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04531C0610CB for ; Tue, 2 Mar 2021 10:46:21 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id v196so23496808ybv.3 for ; Tue, 02 Mar 2021 10:46:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=5+PA/fudqiuRApGQPz15mK0wOSfWjZ1lIFOzJINKfHU=; b=bFc8khCJtym68Ud7GrqEZfDGnDW1noMU/UHyFuPtNmG1Yl+HtHLiwTfXLa3NAlFD6l xQrYzsRaQUMiVkc2yrJEIlbEu0Gh52AvTi3RoCcUnjaNjoHa1VKI//OcE4ICm5qMN7ef eDWuXOcMBkKH2WchlX5Er0y9ZURj5c678uLA1bPzRu7yxHugiThcqixuy9N/MZp4hLOR dXDi27l5EJ7oLACDTSDigecLeWV0u4GtEdSXVHV5bgbqEFe2F4iD0kDsH0Unv1sToZB5 bhk7EzMnK9NFowaiYNeFT4NFK8XY7XQk96gUOg6nc8SX9PfNAOXs8xu0ZjiRrk7W+rGT kLDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=5+PA/fudqiuRApGQPz15mK0wOSfWjZ1lIFOzJINKfHU=; b=YqUPyoNutFUFt6xfYTferGQmfrIUT8cVKOjaW+NEajvO6tJMgAGnHhj3MkBqZ/8bOa 0w54J2RCsAv9L8OxYKNuZazF8q7X0UtVj4q6DqBkGmldh2NqM65ZRSA20JZAZiM3O/JS 26hsqGtxD25PoPcqGc80yFudBy4knqVLsd0afyiP71K8VkXijJFgaoBdNlC3GaRM618a RrHZkNV7HDxW4PDROlQJT3BeikdQqFu2vuMKnOeNIJ2XyUrXIhkaSC6ejp4ygz1sUkqz IJ6RT/h85aMyEoHIsTm0lMNT/PieHwyMxtHrrR+NOCy6uMG1VkC5dD9ulAsOxUo1c0yq 7NPA== X-Gm-Message-State: AOAM533jwdBmXAHjF5Oy/slbu7kFTdvOJlTIiX85FosdhbfSIxX3BMW9 ZUq0U2DW7lsn1lXTc8gSQugAuTqcYYU= X-Google-Smtp-Source: ABdhPJwqo6WN5qp31P8PW9KH3y3esG2aSrP8Tl2w1rXS7TOvwfVB4PsGn0IioUSEZ+HCqKZaElgw8+9XVgY= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:abce:: with SMTP id v72mr34681758ybi.152.1614710780315; Tue, 02 Mar 2021 10:46:20 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:39 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-15-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 14/15] KVM: x86/mmu: Sync roots after MMU load iff load as successful From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org For clarity, explicitly skip syncing roots if the MMU load failed instead of relying on the !VALID_PAGE check in kvm_mmu_sync_roots(). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 86432d6a4092..34eeb39ee0f9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4818,10 +4818,11 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) else r = mmu_alloc_shadow_roots(vcpu); write_unlock(&vcpu->kvm->mmu_lock); + if (r) + goto out; kvm_mmu_sync_roots(vcpu); - if (r) - goto out; + kvm_mmu_load_pgd(vcpu); static_call(kvm_x86_tlb_flush_current)(vcpu); out: From patchwork Tue Mar 2 18:45:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12113037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D034C43381 for ; Wed, 3 Mar 2021 07:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E9E2F64EE6 for ; Wed, 3 Mar 2021 07:23:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379235AbhCCHRj (ORCPT ); Wed, 3 Mar 2021 02:17:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1577332AbhCBSuV (ORCPT ); Tue, 2 Mar 2021 13:50:21 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C47AC0610CE for ; Tue, 2 Mar 2021 10:46:23 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id s187so23586292ybs.22 for ; Tue, 02 Mar 2021 10:46:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=zBqYuQzYqXZuUz7wWVvo+o2pLB8f3Vunolmns9pwVP8=; b=KxmHKUHAYUpp0BZzBTJrdy95ud8ykZUG4XuLTzs7YG6WAIGg2Vb36WpSdPYtS2RG4M vcoXlGgljLFVnsvPO7uzRfaqqTTOIGQOfdrlBA8twvtUTebg5f95btqpBml0AoyXMdP1 /K5ONmYCXT8YDNNGGq66LoEZ0zqiDS+qsYsoJqplmxMSmAUJzi6HGQE7+7SzGJWMfHsQ qcKMnAwDUHHYNdwGojtbR/2hg924mlFeY58M5tbyO16Pll62nH9tUdzd3weJU2LFhYSk l94nDy2cezC+ouVLivg/uU1q1SZBdA2fkVmbUI9scbeF34FdFoIttdga6Ws4eejrAvUv QUpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=zBqYuQzYqXZuUz7wWVvo+o2pLB8f3Vunolmns9pwVP8=; b=NiflfNIQnv8Yj9QxcSB6uU426cmOtAkH6moPV86MDXrvEmfxlQgYgaa5zF5BpgOPCD okrtGn2rbtzHD7bL7ZO97//Xfs1JvfbVl+A8v2a6WjsJET8yyjy0EDQiDmam7QlfdRON D/9Q4heQv+wvziKaiP0y6PEZUUbzswORLtQA/JnNVsyLU6DBbUdhSjlH1FyqZfSv8j5u 3xJo6ffDnbY6UtQFthBc0cphl4VWE9LCeII2Qex8K9D9epdnvABoBIxS536Rt5JfgURX Tkb7/AY+VwF61NuH3yivU8NFppzTyT9tnedrxE2pUbuGa8VRAn29Y4onEbHBykBG8bsP 1fxg== X-Gm-Message-State: AOAM5314Wi8PXa8ooYZSVQQKIk2EVOOuMmt4FlxR+YLMeqcY7k6t4spy BboJFCYh2WrZjxpzP9wbO6djQN0yFLs= X-Google-Smtp-Source: ABdhPJyzdm9UNlhvQlwfULHP76PgQpjC8ZOw2uHTjdczOMWXqJX0skjLrOvOgr96OidyUOnWHZcla6JJ2CI= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:805d:6324:3372:6183]) (user=seanjc job=sendgmr) by 2002:a25:c689:: with SMTP id k131mr32210406ybf.515.1614710782684; Tue, 02 Mar 2021 10:46:22 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 2 Mar 2021 10:45:40 -0800 In-Reply-To: <20210302184540.2829328-1-seanjc@google.com> Message-Id: <20210302184540.2829328-16-seanjc@google.com> Mime-Version: 1.0 References: <20210302184540.2829328-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 15/15] KVM: x86/mmu: WARN on NULL pae_root and bad shadow root level From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org WARN if KVM is about to derference a NULL pae_root when loading an MMU, and convert the BUG() on a bad shadow_root_level into a WARN (now that errors are handled cleanly). With nested NPT, botching the level and sending KVM down the wrong path is all too easy, and the on-demand allocation of pae_root means bugs crash the host. Obviously, KVM could unconditionally allocate pae_root, but that's arguably a worse failure mode as it would potentially corrupt the guest instead of crashing it. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 34eeb39ee0f9..35f89bb1f205 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3247,6 +3247,9 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) true); vcpu->arch.mmu->root_hpa = root; } else if (shadow_root_level == PT32E_ROOT_LEVEL) { + if (WARN_ON_ONCE(!vcpu->arch.mmu->pae_root)) + return -EIO; + for (i = 0; i < 4; ++i) { WARN_ON_ONCE(vcpu->arch.mmu->pae_root[i]); @@ -3256,8 +3259,10 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) shadow_me_mask; } vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root); - } else - BUG(); + } else { + WARN_ONCE(1, "Bad TDP root level = %d\n", shadow_root_level); + return -EIO; + } /* root_pgd is ignored for direct MMUs. */ vcpu->arch.mmu->root_pgd = 0; @@ -3340,6 +3345,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) vcpu->arch.mmu->lm_root = lm_root; lm_root[0] = __pa(vcpu->arch.mmu->pae_root) | pm_mask; + } else if (WARN_ON_ONCE(!vcpu->arch.mmu->pae_root)) { + return -EIO; } for (i = 0; i < 4; ++i) {