From patchwork Wed Jun 23 23:05:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A0C9C49EA5 for ; Wed, 23 Jun 2021 23:06:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF146611AC for ; Wed, 23 Jun 2021 23:06:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229907AbhFWXIW (ORCPT ); Wed, 23 Jun 2021 19:08:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229818AbhFWXIT (ORCPT ); Wed, 23 Jun 2021 19:08:19 -0400 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AF76C061756 for ; Wed, 23 Jun 2021 16:06:01 -0700 (PDT) Received: by mail-qk1-x74a.google.com with SMTP id t144-20020a3746960000b02903ad9c5e94baso4288813qka.16 for ; Wed, 23 Jun 2021 16:06:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=oH77o4lz1NSzb78aMq4uAG+U/iFLzDNAUznM/bo62rM=; b=SHTgYbmg959e42qNZl0AWZ4dEw+/QJ0kmNAUoS+JotxoPZ67fTbBx9aztLXg0XppJR qX8eRksw5kNejjfxTs3qoTSm3JFG5mjA0vgts9XvdCPOqyXjQVpbBC4D1OxUb7QFFEcN lFV0n00kQCgZ0OMY+AOIPmdj23OHoYiMnaMfZ2YrJbgf0u8Pm/59bz6BwP5zydK8K6fF TEm1wHkuMa8K4j4Refjr5aoFTuf5x4DF9jHJBX8E0By3TnSSUH/d36n9fe5d1d9N/SOS Y1QJb0bpQ4dCh63xN9fh8NODwOxo0XhySnOa+WuOIVBzNrquX7sV1zA8+6LXPRRHjRdI FsXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=oH77o4lz1NSzb78aMq4uAG+U/iFLzDNAUznM/bo62rM=; b=OJmi5ZQT9cItBMHm93OSMloLJ/7KI7I949Qvwouvyr83JZQmU6XuKleDkuiGeUpjA8 Q7nIYrxMJZIZl8LBXCGLdVaKsJ/WZn5RMg1II1QSFGm+AJAbOXQn7LYwPQoqyFsX1dDH iaHLBMTxVuRvuq53Ql5Z4X6smoZ8TgtEHs9bBXVZsQandvBOtjaKv8ANHUF67jqOEPuL 6nl9WiYRmD0MweRZWBqOFtf14LOudtQ1CQdcUaYvD3wiRWgEqMIk5RXzYRr3gf/Pra4y xJ5ggmupH1X33J/oHodL6dhNOKBrefAoEqC1fcGOAqsIfj3ZRgjo9GObxP7IklxiCGvm cUhg== X-Gm-Message-State: AOAM532Arvtix00SmnSSrWE+VvVNmR2/iIXTWZvS28gnU6od2ES/X8Of E99cbFHaS4a8lVaqHHzwQksqtvoY0KA= X-Google-Smtp-Source: ABdhPJws+FaF+iL4+9O3U4PePeppdh6J+coIlouK26lbEh4WpKyXv5fC2xidL+TZqY/x+5e/gV0FkB8gM/Q= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:a25:cf03:: with SMTP id f3mr635411ybg.522.1624489560555; Wed, 23 Jun 2021 16:06:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:46 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-2-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 1/7] KVM: x86: Use guest MAXPHYADDR from CPUID.0x8000_0008 iff TDP is enabled From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Ignore the guest MAXPHYADDR reported by CPUID.0x8000_0008 if TDP, i.e. NPT, is disabled, and instead use the host's MAXPHYADDR. Per AMD'S APM: Maximum guest physical address size in bits. This number applies only to guests using nested paging. When this field is zero, refer to the PhysAddrSize field for the maximum guest physical address size. Fixes: 24c82e576b78 ("KVM: Sanitize cpuid") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/cpuid.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index b4da665bb892..4b2f8c6b41e8 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -940,8 +940,14 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) unsigned virt_as = max((entry->eax >> 8) & 0xff, 48U); unsigned phys_as = entry->eax & 0xff; - if (!g_phys_as) + /* + * Use bare metal's MAXPHADDR if the CPU doesn't report guest + * MAXPHYADDR separately, or if TDP (NPT) is disabled, as the + * guest version "applies only to guests using nested paging". + */ + if (!g_phys_as || !tdp_enabled) g_phys_as = phys_as; + entry->eax = g_phys_as | (virt_as << 8); entry->edx = 0; cpuid_entry_override(entry, CPUID_8000_0008_EBX); From patchwork Wed Jun 23 23:05:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2650FC49EAB for ; Wed, 23 Jun 2021 23:06:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0860060FE5 for ; Wed, 23 Jun 2021 23:06:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229924AbhFWXIX (ORCPT ); Wed, 23 Jun 2021 19:08:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229882AbhFWXIV (ORCPT ); Wed, 23 Jun 2021 19:08:21 -0400 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88DD7C061574 for ; Wed, 23 Jun 2021 16:06:03 -0700 (PDT) Received: by mail-qk1-x74a.google.com with SMTP id b125-20020a3799830000b02903ad1e638ccaso4321712qke.4 for ; Wed, 23 Jun 2021 16:06:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=jbfGWs931zA+JGWuuEGa9C1XdcBcxFB7u/E81/FmgGU=; b=q4witublxlQfQh9byhgfZ2JZsN+6vdXlBZi0aMSrSAjmRmYvx9rFDvbS8yFOPcM4LB /1NxO5D6/eZNco8/BabNsUhn0brbIh92TLER1gzpd13/O2Zy76WZxJCtulEKzpBuvW8I 988XdjGpKD6Bcr9HncDuGdCXh2LXROdtzD4VOqCu/2y55QLPQE7xBUTZweL+nFzXB24g pq1yIkl8TKm2Ia2OD8LDhSuqUA7TXbKcCoFPsTwJNu7HxvV/RRB8Ow0uEVbf5L8Hy2UQ IUMWRxhgRXtwlrhbbhWzCi+yckSL81UJn5vCW0JR7fdCZtUtz5KZSghBUQ4rW47zwXQm s2xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=jbfGWs931zA+JGWuuEGa9C1XdcBcxFB7u/E81/FmgGU=; b=L4KXO+W8g4DFymNrw6So8tk3INe5LFTUW1DA8tvoUwffNR2R45NLvxWsOG2wm7obG4 wyu7Fo5+grPpRHaIUQ5ZRJqmcyrX4IvvDPPUI0Ces0SzD+y4oV7DjxWiRBXY+i8jTTqA h2upxBbfMKdKNzaxrNwgakaPZubG0dmudTln5noxvXH3NbcDayX1E+8d8gTKoGYicvNX BTBg6IggVhriHlXAqOhVPEiQLVKgZzrvSEaMss6MjzRG+3Y5+p4nLiSubtKNAY5jgswO Bt1T6mOsNt0liEGxY7409KaKGTDkbYX+pGhDohpkvmC+XS4n4Wr5GbOEzdb2cUjzFHtC gBtg== X-Gm-Message-State: AOAM530/rWFdwVIq9CFX1RsxLCmfxrZmhQ8zFeNO0QzltqcawKGjsp4Q nYpF1Jdx02Sx2GSlXaDYqkfN2NA6WnA= X-Google-Smtp-Source: ABdhPJyWxbjbqPIzyPvxjQ1GE7fMSRSGs/DNU7B9f28W8AIL1w65sp4XTKNcI58saYVi4VCH5f0kO2HHF60= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:ad4:5a07:: with SMTP id ei7mr2045985qvb.46.1624489562735; Wed, 23 Jun 2021 16:06:02 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:47 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-3-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 2/7] KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use boot_cpu_data.x86_phys_bits instead of the raw CPUID information to enumerate the MAXPHYADDR for KVM guests when TDP is disabled (the guest version is only relevant to NPT/TDP). When using shadow paging, any reductions to the host's MAXPHYADDR apply to KVM and its guests as well, i.e. using the raw CPUID info will cause KVM to misreport the number of PA bits available to the guest. Unconditionally zero out the "Physical Address bit reduction" entry. For !TDP, the adjustment is already done, and for TDP enumerating the host's reduction is wrong as the reduction does not apply to GPAs. Fixes: 9af9b94068fb ("x86/cpu/AMD: Handle SME reduction in physical address size") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/cpuid.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 4b2f8c6b41e8..28878671d648 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -941,11 +941,18 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) unsigned phys_as = entry->eax & 0xff; /* - * Use bare metal's MAXPHADDR if the CPU doesn't report guest - * MAXPHYADDR separately, or if TDP (NPT) is disabled, as the - * guest version "applies only to guests using nested paging". + * If TDP (NPT) is disabled use the adjusted host MAXPHYADDR as + * the guest operates in the same PA space as the host, i.e. + * reductions in MAXPHYADDR for memory encryption affect shadow + * paging, too. + * + * If TDP is enabled but an explicit guest MAXPHYADDR is not + * provided, use the raw bare metal MAXPHYADDR as reductions to + * the HPAs do not affect GPAs. */ - if (!g_phys_as || !tdp_enabled) + if (!tdp_enabled) + g_phys_as = boot_cpu_data.x86_phys_bits; + else if (!g_phys_as) g_phys_as = phys_as; entry->eax = g_phys_as | (virt_as << 8); @@ -970,12 +977,18 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) case 0x8000001a: case 0x8000001e: break; - /* Support memory encryption cpuid if host supports it */ case 0x8000001F: - if (!kvm_cpu_cap_has(X86_FEATURE_SEV)) + if (!kvm_cpu_cap_has(X86_FEATURE_SEV)) { entry->eax = entry->ebx = entry->ecx = entry->edx = 0; - else + } else { cpuid_entry_override(entry, CPUID_8000_001F_EAX); + + /* + * Enumerate '0' for "PA bits reduction", the adjusted + * MAXPHYADDR is enumerated directly (see 0x80000008). + */ + entry->ebx &= ~GENMASK(11, 6); + } break; /*Add support for Centaur's CPUID instruction*/ case 0xC0000000: From patchwork Wed Jun 23 23:05:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D51EAC49EAF for ; Wed, 23 Jun 2021 23:06:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BBB3F6115A for ; Wed, 23 Jun 2021 23:06:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229955AbhFWXI0 (ORCPT ); Wed, 23 Jun 2021 19:08:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229940AbhFWXIZ (ORCPT ); Wed, 23 Jun 2021 19:08:25 -0400 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4FE2C061756 for ; Wed, 23 Jun 2021 16:06:05 -0700 (PDT) Received: by mail-qk1-x749.google.com with SMTP id 14-20020a37060e0000b02903aad32851d2so4356081qkg.1 for ; Wed, 23 Jun 2021 16:06:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=9ugzlw7yCqvttuGsYPFF9841OjnqtnS5Arl+jCDe6NI=; b=WLvvhmvbbq/Pomzk4a+6iHO6aZLiPMI63QIAbzByGdXZG7scEr1xttoW3fJ2B1jhhs p8LEAUJfwBzh9ifDiOogyAkKsxnbJr8ZzxXbv4fpkw8k1pON8H9cKHTMdaiZz/9SitHL NaB4cnHmu33ynS6YLB291L5PBlNFVc3x/9nCzr8Dj5E7uKGgmtaThvHeMXb9f4Bw+BR/ dtE4Z5zQnYFEsv3KFunnK3l4hD9JgJPOBXdaeCxZckC1BRHxtbOmkv06BXtrx6D4ksI6 DDwhcfeZFCF6HiMV3wKge2EgS38nshhI0XbV+np3uLuUZH/JbZjTX9DbYGiQ3/Gv1ROQ 49ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=9ugzlw7yCqvttuGsYPFF9841OjnqtnS5Arl+jCDe6NI=; b=q4bSonbQ/vYBaSY0+vJr5uso9xKqnlf8DnNgXiV0CLAIlD46mX3dd97U5ItV/T2UuH fxZMoRfL4qBpQteGOk3g5jPhq1lRBaV/UTCbJBzPDRGnRLCuSTgpvTe/vwMOa5rLmdPS KsP0Rb/LwZOJmOb+OGZYAG73hK6DU9x91uAuYOYqB073TyeohFc0XpLymUwschzAe/lz d/csn+9RK8JavSY9/zeOAKZP9zW34O0tUKUgeM7wWQS6viZDUGTRbsVpe9ZMttxmVvKu C/QaHpECUwWWsmkw7YXPA4MgPt9mXF/ihbnLA3hn70AF/c25YfFj9TdCregXTd/ECapj 6gbA== X-Gm-Message-State: AOAM530wEDvYF4C2EbQTmCUXPrRt6WnmR6IKKHtHHQYuz7v80DV9mMp5 dEelFnmiXBDKAjFZeUxvNTJqkJE7C0M= X-Google-Smtp-Source: ABdhPJxha8/+D066BvQrLuLO91KgGwWa5GXjlpzOHG7V2UU9DZqKCk6jwt6lncZgWeX4l/F0kNPwVj0CYN8= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:ad4:5f0e:: with SMTP id fo14mr2278544qvb.16.1624489564995; Wed, 23 Jun 2021 16:06:04 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:48 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-4-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 3/7] KVM: x86: Truncate reported guest MAXPHYADDR to C-bit if SEV is supported From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Enumerate "C-bit - 1" as the MAXPHYADDR for KVM_GET_SUPPORTED_CPUID if NPT is enabled, SEV is supported in hardware, and the C-bit overlaps the legal guest physical address space reported by hardware. I.e. advertise the C-bit and any PA bits above the C-bit as reserved. Reuse svm_adjust_mmio_mask()'s C-bit retrieval logic, but opportunistically switch to the "safe" RDMSR, e.g. the MSR may not be emulated if KVM is running nested. AMD's APM is a bit ambiguous as to the behavior of the C-bit for non-SEV guests running on SEV-capable hardware. In "15.34.6 Page Table Support", under "15.34 Secure Encrypted Virtualization", the APM does state that the guest MAXPHYADDR is not reduced unless the C-bit is stolen from the legal physical address space: Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit. What it doesn't clarify is whether or not that behavior applies to non-SEV guests, i.e. whether the C-bit is reserved or a legal GPA bit. Regardless of what the intended behavior is (more on this later), KVM is broken because it treats the C-bit as ignored for non-SEV. At first blush, it would appear the KVM treats the C-bit as a legal GPA bit for non-SEV guests, but the C-bit is stripped for host _and_ guest PTEs tables, as PT64_BASE_ADDR_MASK is defined to incorporate the adjusted physical_mask (CONFIG_DYNAMIC_PHYSICAL_MASK is forced for SME, and the C-bit is explicitly cleared). #ifdef CONFIG_DYNAMIC_PHYSICAL_MASK #define PT64_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1)) #else As a result, KVM reports the full guest MAXPHYADDR even though it (silently) drops the C-bit. If the APM's intent is that the C-bit is a GPA bit for non-SEV guests, then the correct fix would be to split PT64_BASE_ADDR_MASK into host and guest masks, with the guest mask being dynamic since SEV guests would still need to drop the C-bit. But, on AMD EPYC 7B12, a.k.a. Rome, the C-bit is reserved for non-SEV guests. E.g. the demand_paging_test selftest hits an unexpected #PF when running NPT due to using the highest possible GPAs. A dump of the #PF and guest page tables (via #PF interception) shows the CPU generates a reserved #PF, and clearing the C-bit in the test passes. The same dump also captures KVM's clearing of the C-bit in its final GPA calulcation. SVM: KVM: CPU #PF @ rip = 0x409ca4, cr2 = 0xc0000000, pfec = 0xb KVM: guest PTE = 0x181023 @ GPA = 0x180000, level = 4 KVM: guest PTE = 0x186023 @ GPA = 0x181000, level = 3 KVM: guest PTE = 0x187023 @ GPA = 0x186000, level = 2 KVM: guest PTE = 0xffffbffff003 @ GPA = 0x187000, level = 1 SVM: KVM: GPA = 0x7fffbffff000 Fixes: d0ec49d4de90 ("kvm/x86/svm: Support Secure Memory Encryption within KVM") Cc: stable@vger.kernel.org Cc: Peter Gonda Cc: Brijesh Singh Cc: Tom Lendacky Signed-off-by: Sean Christopherson --- arch/x86/kvm/cpuid.c | 11 +++++++++++ arch/x86/kvm/svm/svm.c | 37 +++++++++++++++++++++++++++++-------- arch/x86/kvm/x86.c | 3 +++ arch/x86/kvm/x86.h | 1 + 4 files changed, 44 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 28878671d648..e05a9eb0dd03 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -955,6 +955,17 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) else if (!g_phys_as) g_phys_as = phys_as; + /* + * The exception to the exception is if hardware supports SEV, + * in which case the C-bit is reserved for non-SEV guests and + * isn't a GPA bit for SEV guests. + * + * Note, KVM always reports '0' for the number of reduced PA + * bits (see 0x8000001F). + */ + if (tdp_enabled && sev_c_bit) + g_phys_as = min(g_phys_as, (unsigned int)sev_c_bit); + entry->eax = g_phys_as | (virt_as << 8); entry->edx = 0; cpuid_entry_override(entry, CPUID_8000_0008_EBX); diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 12c06ea28f5c..2549e80abf05 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -860,6 +860,26 @@ static void shrink_ple_window(struct kvm_vcpu *vcpu) } } +static __init u8 svm_get_c_bit(bool sev_only) +{ + unsigned int eax, ebx, ecx, edx; + u64 msr; + + if (cpuid_eax(0x80000000) < 0x8000001f) + return 0; + + if (rdmsrl_safe(MSR_K8_SYSCFG, &msr) || + !(msr & MSR_K8_SYSCFG_MEM_ENCRYPT)) + return 0; + + cpuid(0x8000001f, &eax, &ebx, &ecx, &edx); + + if (sev_only && !(eax & feature_bit(SEV))) + return 0; + + return ebx & 0x3f; +} + /* * The default MMIO mask is a single bit (excluding the present bit), * which could conflict with the memory encryption bit. Check for @@ -869,18 +889,13 @@ static void shrink_ple_window(struct kvm_vcpu *vcpu) static __init void svm_adjust_mmio_mask(void) { unsigned int enc_bit, mask_bit; - u64 msr, mask; - - /* If there is no memory encryption support, use existing mask */ - if (cpuid_eax(0x80000000) < 0x8000001f) - return; + u64 mask; /* If memory encryption is not enabled, use existing mask */ - rdmsrl(MSR_K8_SYSCFG, msr); - if (!(msr & MSR_K8_SYSCFG_MEM_ENCRYPT)) + enc_bit = svm_get_c_bit(false); + if (!enc_bit) return; - enc_bit = cpuid_ebx(0x8000001f) & 0x3f; mask_bit = boot_cpu_data.x86_phys_bits; /* Increment the mask bit if it is the same as the encryption bit */ @@ -1013,6 +1028,12 @@ static __init int svm_hardware_setup(void) kvm_configure_mmu(npt_enabled, get_max_npt_level(), PG_LEVEL_1G); pr_info("kvm: Nested Paging %sabled\n", npt_enabled ? "en" : "dis"); + /* + * The SEV C-bit location is needed to correctly enumeration guest + * MAXPHYADDR even if SEV is not fully supported. + */ + sev_c_bit = svm_get_c_bit(true); + /* Note, SEV setup consumes npt_enabled. */ sev_hardware_setup(); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 76dae88cf524..4479e67e5727 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -223,6 +223,9 @@ EXPORT_SYMBOL_GPL(host_xss); u64 __read_mostly supported_xss; EXPORT_SYMBOL_GPL(supported_xss); +u8 __read_mostly sev_c_bit; +EXPORT_SYMBOL_GPL(sev_c_bit); + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("pf_fixed", pf_fixed), VCPU_STAT("pf_guest", pf_guest), diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 521f74e5bbf2..a80cc63039c3 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -338,6 +338,7 @@ extern u64 host_xcr0; extern u64 supported_xcr0; extern u64 host_xss; extern u64 supported_xss; +extern u8 sev_c_bit; static inline bool kvm_mpx_supported(void) { From patchwork Wed Jun 23 23:05:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE708C49EA5 for ; Wed, 23 Jun 2021 23:06:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B17E161164 for ; Wed, 23 Jun 2021 23:06:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230015AbhFWXIa (ORCPT ); Wed, 23 Jun 2021 19:08:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229849AbhFWXI1 (ORCPT ); Wed, 23 Jun 2021 19:08:27 -0400 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 274E4C061574 for ; Wed, 23 Jun 2021 16:06:08 -0700 (PDT) Received: by mail-qk1-x749.google.com with SMTP id n6-20020a3759060000b02903ad3598ec02so4278413qkb.17 for ; Wed, 23 Jun 2021 16:06:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=M3ydmh5FXw/WwmZgZ0UhGBQS2xGblp3Q4P5QtrxRfkI=; b=vnK9fPSgKQWR5ZxNGiTOxjt1N7jAjbLXhEtcfyhBhy4nNsP17p5KlXGZRi/Dp8/x5D xoiUNjkDX5N7UtD7i4eBWfngirwXYukuiWD0nwwDZqmvrpwFJrWvvk9iwKF+XudfWMZv c8ev7fqKW/Hh8elAUTIGw2vSsPMAEq85vasEB0OKNSeWcwF5LYnQl/JTFgTyva/Fhqv3 ZYNkcuFedfXRVwt1GO6tW0a2NbPwsTs77yWMMrDtcEyVpczy6YOd/ngxJO0BS1rb8CLu RWwhbALIREUWdkbx4n9viMMkJ+JWT+jKgV/nDjjExtRRvIBp51E8DRMhxexgJzzo0Nj9 ysFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=M3ydmh5FXw/WwmZgZ0UhGBQS2xGblp3Q4P5QtrxRfkI=; b=I3tP+dRDTuistIpfPX8EAit0i4e3c/PxF5mquALeY9yOWA8Aiw2/hVB4tSX2lM5mvq 3QDR7t7CvrFdLh1TE9CdEW8u/rub6sF0PMpdKfJl+fC9t5w0W8LKvpFcTDOmLxt98eKg OZp8UBy3ImbfEQYN/w/L9Q3mM028Bvc+3Fr10Z7CMRo3yBQ1hJm7u7q689RWzmy2lIGQ RaC+RAyttgYEREk4ivVQemXuI8tCrDFqg5AkUbG/5r9DP2nczHpwGixg0BaXZ9Y80vtA wvUvZAoYNfV4En0KnySu6Z5kuzy5XgW5BabnK5V0cJVVZzeJcbv0c5YZWz6blBMP8hEm 7sZA== X-Gm-Message-State: AOAM530NgC+wB+4J+SL4EpiofnuQ0BXqYJWEBMs9rAeWWWBoTEMhFQv+ QgH2vTCa9iOqHQi1g3ApBEbtUaRfpck= X-Google-Smtp-Source: ABdhPJzu9sXF91Gb7/TRKPqNiTlFm2WN9pf0kU7l2qzE5IHlnU7bheDRkZQi1EvV/yMrscItkc57snYovTg= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:a25:6f55:: with SMTP id k82mr692122ybc.490.1624489567284; Wed, 23 Jun 2021 16:06:07 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:49 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-5-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 4/7] KVM: x86/mmu: Do not apply HPA (memory encryption) mask to GPAs From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Ignore "dynamic" host adjustments to the physical address mask when generating the masks for guest PTEs, i.e. the guest PA masks. The host physical address space and guest physical address space are two different beasts, e.g. even though SEV's C-bit is the same bit location for both host and guest, disabling SME in the host (which clears shadow_me_mask) does not affect the guest PTE->GPA "translation". For non-SEV guests, not dropping bits is the correct behavior. Assuming KVM and userspace correctly enumerate/configure guest MAXPHYADDR, bits that are lost as collateral damage from memory encryption are treated as reserved bits, i.e. KVM will never get to the point where it attempts to generate a gfn using the affected bits. And if userspace wants to create a bogus vCPU, then userspace gets to deal with the fallout of hardware doing odd things with bad GPAs. For SEV guests, not dropping the C-bit is technically wrong, but it's a moot point because KVM can't read SEV guest's page tables in any case since they're always encrypted. Not to mention that the current KVM code is also broken since sme_me_mask does not have to be non-zero for SEV to be supported by KVM. The proper fix would be to teach all of KVM to correctly handle guest private memory, but that's a task for the future. Fixes: d0ec49d4de90 ("kvm/x86/svm: Support Secure Memory Encryption within KVM") Cc: stable@vger.kernel.org Cc: Brijesh Singh Cc: Tom Lendacky Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/paging_tmpl.h | 17 +++++++++++++++-- arch/x86/kvm/mmu/spte.h | 6 ------ 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 823a5919f9fa..9df7e4b315a1 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -20,11 +20,24 @@ * so the code in this file is compiled twice, once per pte size. */ +/* Shadow paging constants/helpers that don't need to be #undef'd. */ +#ifndef __KVM_X86_PAGING_TMPL_COMMON_H +#define __KVM_X86_PAGING_TMPL_COMMON_H + +#define GUEST_PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) +#define PT64_LVL_ADDR_MASK(level) \ + (GUEST_PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \ + * PT64_LEVEL_BITS))) - 1)) +#define PT64_LVL_OFFSET_MASK(level) \ + (GUEST_PT64_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \ + * PT64_LEVEL_BITS))) - 1)) +#endif /* __KVM_X86_PAGING_TMPL_COMMON_H */ + #if PTTYPE == 64 #define pt_element_t u64 #define guest_walker guest_walker64 #define FNAME(name) paging##64_##name - #define PT_BASE_ADDR_MASK PT64_BASE_ADDR_MASK + #define PT_BASE_ADDR_MASK GUEST_PT64_BASE_ADDR_MASK #define PT_LVL_ADDR_MASK(lvl) PT64_LVL_ADDR_MASK(lvl) #define PT_LVL_OFFSET_MASK(lvl) PT64_LVL_OFFSET_MASK(lvl) #define PT_INDEX(addr, level) PT64_INDEX(addr, level) @@ -57,7 +70,7 @@ #define pt_element_t u64 #define guest_walker guest_walkerEPT #define FNAME(name) ept_##name - #define PT_BASE_ADDR_MASK PT64_BASE_ADDR_MASK + #define PT_BASE_ADDR_MASK GUEST_PT64_BASE_ADDR_MASK #define PT_LVL_ADDR_MASK(lvl) PT64_LVL_ADDR_MASK(lvl) #define PT_LVL_OFFSET_MASK(lvl) PT64_LVL_OFFSET_MASK(lvl) #define PT_INDEX(addr, level) PT64_INDEX(addr, level) diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index bca0ba11cccf..6925dfc38981 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -38,12 +38,6 @@ static_assert(SPTE_TDP_AD_ENABLED_MASK == 0); #else #define PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) #endif -#define PT64_LVL_ADDR_MASK(level) \ - (PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \ - * PT64_LEVEL_BITS))) - 1)) -#define PT64_LVL_OFFSET_MASK(level) \ - (PT64_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \ - * PT64_LEVEL_BITS))) - 1)) #define PT64_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \ | shadow_x_mask | shadow_nx_mask | shadow_me_mask) From patchwork Wed Jun 23 23:05:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB98AC49EAB for ; Wed, 23 Jun 2021 23:06:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA98D60FE5 for ; Wed, 23 Jun 2021 23:06:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229995AbhFWXId (ORCPT ); Wed, 23 Jun 2021 19:08:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230013AbhFWXIa (ORCPT ); Wed, 23 Jun 2021 19:08:30 -0400 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0169C061756 for ; Wed, 23 Jun 2021 16:06:10 -0700 (PDT) Received: by mail-qv1-xf4a.google.com with SMTP id 12-20020a05621420ecb02902766cc25115so4720339qvk.1 for ; Wed, 23 Jun 2021 16:06:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ECxHPd8ARt3g/FbXO6kaB2YAiLV+c0qQVpIxsjvN8ho=; b=sEtH99jwiQ93AFFZ+4qGZKG6dGPffpY3n63kic4KqeOVt+fqR+Uuaqjvl2aPaOs5LV +kAOO2CbASDeHPRmOqt/NzKURFkBm6luGWNTqkWADfEsQKkEDyk/v4fPH7zLfREIo8nS VlR9OensOnYHTVVaqLwGXPkGTdFY8ZsdPz0jRF0dSt11MlKK5XBqT0yKVrJBErLszjD5 l+tQEpYQELkuyN5LBtOKNsJ0S8HEGlQW1ZbIYrCKYdUA81ANlFmwvgmxlj/FPUs38ywk xHkha3JfyoMkZ93y5B7uADrAJtxUP/QrRQ5h9Q0ljV3nO7y3rKi7Ud8kNpOiz/xLIRQF dqjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ECxHPd8ARt3g/FbXO6kaB2YAiLV+c0qQVpIxsjvN8ho=; b=ELCSwdKwqnCW3A8GuJZcOHGaubWuxyh7SHk3GXoZYQ98im9fjKuOyozj5QD1yUpzjU hfzO3AK9AQy7DtGpSEVcWaFlYR/q0G23DPM0Mh2/go5fulwmcCSRYmtW/SXAdZW7+2hv Ch8xA5Esrtmb2skhKi0YLHTAEhiuVOod3ddWMunk2Wcxliclqk7Y24o7Aem6vUGTzoQO cE+vub468dL45fWH4h/67Cbb1S+M0JD6isiJhWD3Pe3jnz0LcXDPo8KkYzg5veQ3zHv9 Hrd2ZSGCecLAW0gY2sVp7KsnP2E7Yhkd96tRyxarUKAWMlesCcEYnvq+5txUrpympSbI 7Xlg== X-Gm-Message-State: AOAM531fZdI9w1xTycWmrCzBZLv9Ktuc9t24LVc4r9T4PG1HXAqFUvve ZBosXb49KkNlpbQVR+6cKoGBcOBVWzg= X-Google-Smtp-Source: ABdhPJwGGkYrzt9VXDp9VVzM6DTgXp7pCLDTPfh12+pCsTsoR+0WhHJK2RbX/l1BTJ0xgBwdpzwy9aOHv2Q= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:a05:6902:522:: with SMTP id y2mr730525ybs.12.1624489569847; Wed, 23 Jun 2021 16:06:09 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:50 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-6-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 5/7] KVM: VMX: Refactor 32-bit PSE PT creation to avoid using MMU macro From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Compute the number of PTEs to be filled for the 32-bit PSE page tables using the page size and the size of each entry. While using the MMU's PT32_ENT_PER_PAGE macro is arguably better in isolation, removing VMX's usage will allow a future namespacing cleanup to move the guest page table macros into paging_tmpl.h, out of the reach of code that isn't directly related to shadow paging. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index ab6f682645d7..f6fa922ca6e3 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3584,7 +3584,7 @@ static int init_rmode_identity_map(struct kvm *kvm) } /* Set up identity-mapping pagetable for EPT in real mode */ - for (i = 0; i < PT32_ENT_PER_PAGE; i++) { + for (i = 0; i < (PAGE_SIZE / sizeof(tmp)); i++) { tmp = (i << 22) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); if (__copy_to_user(uaddr + i * sizeof(tmp), &tmp, sizeof(tmp))) { From patchwork Wed Jun 23 23:05:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CDF1C48BC2 for ; Wed, 23 Jun 2021 23:06:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 43F896115A for ; Wed, 23 Jun 2021 23:06:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230083AbhFWXIg (ORCPT ); Wed, 23 Jun 2021 19:08:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229940AbhFWXIc (ORCPT ); Wed, 23 Jun 2021 19:08:32 -0400 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 319E4C061760 for ; Wed, 23 Jun 2021 16:06:13 -0700 (PDT) Received: by mail-qt1-x849.google.com with SMTP id c29-20020ac86e9d0000b0290247b267c8e4so4241556qtv.22 for ; Wed, 23 Jun 2021 16:06:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=JYOF6IU31rISPQS51fxGfY25qPgNWHWroepjRPbVyV0=; b=FhDEofVv7xZfuj0yEbanAeLh5izVcGy94ZW/HpIxrq+UTUfNz5932VE3qFDG90wFze HFSZ52k9mcrYSV+VtkUaHKwh3gr1GZ1Ux6JsrdeRDten3lWqA6Vort0o2Ru4mb83KcbJ DcPScXdoL4aIr5jIVVMySiGw9N38hgK1UH/v+ZLzinbYQHnrXzrPq7TRo8uAhHpzwfMP xOCYuFE4CmwTfay1XzlAnIfrVmFFAqRjzemp2IIvmDz9wK36cCpTohzIw2v2U1SYw7Fl 1nbrqBJ2K2sKUOcLPAeIvWHQe4IlrcafPG0y9zk2iCkrWkWDSQNMrIEw8YEBjmoNfx27 Zm2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=JYOF6IU31rISPQS51fxGfY25qPgNWHWroepjRPbVyV0=; b=LvfrBBxvV6fwuLYKfz1QgBtFIFpKD4k217oDXiKYt5P/6D2yjrSIyJYTheinP0dyO/ 2AR20LDwa0rbvM9cv5cs1ssEDvwICdhasuIypHEgyLUgPpwI98+gKxEQkOoZszdskLWJ Ue8RPvPMVitMDGkpcc4p7denZiiw4eg42gZniCWyDRvXsZJFyOakhT/NX5DiFHvWBCRf y9C6pIZbL3kZCFli/cHYVlC9DoOmCIE4UmRE9Ixt9NBMsh1b/yp7siJFD5StqQtXZ8Jc rS+bnVorD1bQgQSd1bjeYaOfpTS4WZQ0zza2jSY2hz4vycLBxb+nFHp1SPT/nCroBkVw tQug== X-Gm-Message-State: AOAM5307JLSCBWl8913fSOx2D4thEwI5tQ/WWNre2dwxGUgyU3Gng2L5 8e4TnASxPC7X0WGUo/wTKmI//IyvuDw= X-Google-Smtp-Source: ABdhPJzlnWyNDJVBCpEkhcYeS0AKEtJS2LIBFOouzwcvlNqNDRnX3IgykVtM5EsNA5pqC4TTIFHy/V56rSg= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:a25:b98d:: with SMTP id r13mr654971ybg.430.1624489572305; Wed, 23 Jun 2021 16:06:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:51 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-7-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 6/7] KVM: x86/mmu: Bury 32-bit PSE paging helpers in paging_tmpl.h From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move a handful of one-off macros and helpers for 32-bit PSE paging into paging_tmpl.h. Under no circumstance should anything but shadow paging care about 32-bit PSE paging. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu.h | 5 ----- arch/x86/kvm/mmu/mmu.c | 13 ------------- arch/x86/kvm/mmu/paging_tmpl.h | 18 ++++++++++++++++++ 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index bc11402df83b..2b9d08b080cc 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -34,11 +34,6 @@ #define PT_DIR_PAT_SHIFT 12 #define PT_DIR_PAT_MASK (1ULL << PT_DIR_PAT_SHIFT) -#define PT32_DIR_PSE36_SIZE 4 -#define PT32_DIR_PSE36_SHIFT 13 -#define PT32_DIR_PSE36_MASK \ - (((1ULL << PT32_DIR_PSE36_SIZE) - 1) << PT32_DIR_PSE36_SHIFT) - #define PT64_ROOT_5LEVEL 5 #define PT64_ROOT_4LEVEL 4 #define PT32_ROOT_LEVEL 2 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 84d48a33e38b..ef92717bff86 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -259,23 +259,10 @@ static gpa_t translate_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u32 access, return gpa; } -static int is_cpuid_PSE36(void) -{ - return 1; -} - static int is_nx(struct kvm_vcpu *vcpu) { return vcpu->arch.efer & EFER_NX; } - -static gfn_t pse36_gfn_delta(u32 gpte) -{ - int shift = 32 - PT32_DIR_PSE36_SHIFT - PAGE_SHIFT; - - return (gpte & PT32_DIR_PSE36_MASK) << shift; -} - #ifdef CONFIG_X86_64 static void __set_spte(u64 *sptep, u64 spte) { diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 9df7e4b315a1..a2dbea70ffda 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -31,6 +31,24 @@ #define PT64_LVL_OFFSET_MASK(level) \ (GUEST_PT64_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \ * PT64_LEVEL_BITS))) - 1)) + +#define PT32_DIR_PSE36_SIZE 4 +#define PT32_DIR_PSE36_SHIFT 13 +#define PT32_DIR_PSE36_MASK \ + (((1ULL << PT32_DIR_PSE36_SIZE) - 1) << PT32_DIR_PSE36_SHIFT) + +static inline int is_cpuid_PSE36(void) +{ + return 1; +} + +static inline gfn_t pse36_gfn_delta(u32 gpte) +{ + int shift = 32 - PT32_DIR_PSE36_SHIFT - PAGE_SHIFT; + + return (gpte & PT32_DIR_PSE36_MASK) << shift; +} + #endif /* __KVM_X86_PAGING_TMPL_COMMON_H */ #if PTTYPE == 64 From patchwork Wed Jun 23 23:05:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12340591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EC45C48BC2 for ; Wed, 23 Jun 2021 23:06:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B19561164 for ; Wed, 23 Jun 2021 23:06:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229864AbhFWXIl (ORCPT ); Wed, 23 Jun 2021 19:08:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230064AbhFWXIe (ORCPT ); Wed, 23 Jun 2021 19:08:34 -0400 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 662EFC061574 for ; Wed, 23 Jun 2021 16:06:15 -0700 (PDT) Received: by mail-qt1-x849.google.com with SMTP id z5-20020ac86c450000b029024e9a87714dso4321850qtu.2 for ; Wed, 23 Jun 2021 16:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=l9aUZrrzGLgepJinEz3O0tk6pl4pmaaVEKJ+sq+iUN0=; b=pmsPDc5hXdnA6yKgqqTSnoOMXjRgXzeAos39T6ctNGKcHpWPf7Alees6O7nfCeQsNW /dHyXaMxgREaFX2ycxlV8Dq3JyBW3bmjee6bjX1aMtJ+FN13ou5ju56cx7xOQXbl8Aj4 +/qWvBC3UdJe8AgSqH5Ag9gYkiWxiVL5sSO++/YTtv7duDUwn7tnV3H62jZc3v1bH/Md UNuIPU876HO90Bt9c9vBJdtAk439UEgm1cwx5ma6J+NL60QVfdc4k6R4BG5QpPCQRtVs A464WPCkdNCCL4IuDZDpXO0H1TXIyUffBWgH0YJT6yM10Ywyl4fxXEcUv+C/O/68OwqI OA0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=l9aUZrrzGLgepJinEz3O0tk6pl4pmaaVEKJ+sq+iUN0=; b=HJufIOgnu4zZvxg+TWpZ5Saw1HzxVXYO4ibr2K8c/KjnMx5nFhY4JJs2hJoqMZeau7 9irOfhAuhRoGIV4XCboxmI/GAH7oQFGM7abQwIgjCUgzbDvITHqRm4MeUNWI1/xIr31T fcD12cECRUphYCys6R2uhl6SLyUUesf3qEuYyPzySIqR390WxsugOE+Tttr7XwUzbpXQ ODky1TBPZ9CLCr0xh3mmaCMoOh5B2TXO6gQeQcLGn60JfpLf8Lz39bvRnjZJ4QywxCe4 nVvqge07CMrub0sstUkjFv9bonoyLgi22GFf3sCbhF8V886K7NllkoJGs6n0/7RSj1bT NqkQ== X-Gm-Message-State: AOAM530DjFtl3r/9ExDxaKsqodhH1y0HI9i6jIjBBSP4T2UX9NKZK0JJ RMo1f9BFFIZJG2YCLB6Y8UD3+tG90kg= X-Google-Smtp-Source: ABdhPJxpoyVkWTGtRZmaBZHSEw7M2IBz0+/aAH7SHg6Z4kChqFyLVaKo9txpCm+74+Sb/tCcuYn9/qpiyAs= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:e9e:5b86:b4f2:e3c9]) (user=seanjc job=sendgmr) by 2002:a25:bd84:: with SMTP id f4mr741765ybh.143.1624489574547; Wed, 23 Jun 2021 16:06:14 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 23 Jun 2021 16:05:52 -0700 In-Reply-To: <20210623230552.4027702-1-seanjc@google.com> Message-Id: <20210623230552.4027702-8-seanjc@google.com> Mime-Version: 1.0 References: <20210623230552.4027702-1-seanjc@google.com> X-Mailer: git-send-email 2.32.0.288.g62a8d224e6-goog Subject: [PATCH 7/7] KVM: x86/mmu: Use separate namespaces for guest PTEs and shadow PTEs From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Gonda , Brijesh Singh , Tom Lendacky Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Separate the macros for KVM's shadow PTEs (SPTE) from guest 64-bit PTEs (PT64). SPTE and PT64 are _mostly_ the same, but the few differences are quite critical, e.g. *_BASE_ADDR_MASK must differentiate between host and guest physical address spaces, and SPTE_PERM_MASK (was PT64_PERM_MASK) is very much specific to SPTEs. Add helper macros to deduplicate the 32-bit vs. 64-bit code, and to avoid additional duplication for SPTEs. Opportunistically move most guest macros into paging_tmpl.h to clearly associate them with shadow paging, and to make it more difficult to unintentionally use a guest macro in the MMU. Sadly, PT32_LEVEL_BITS is left behind in mmu.h because it's need for the quadrant calculation in kvm_mmu_get_page(), which is hot enough that adding a per-context helper is undesirable, and burying the computation in paging_tmpl.h with a forward declaration isn't exactly an improvement. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu.h | 8 ++---- arch/x86/kvm/mmu/mmu.c | 50 ++++++++++----------------------- arch/x86/kvm/mmu/mmu_audit.c | 6 ++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++++ arch/x86/kvm/mmu/paging_tmpl.h | 35 +++++++++++++++++------ arch/x86/kvm/mmu/spte.c | 2 +- arch/x86/kvm/mmu/spte.h | 28 ++++++++---------- arch/x86/kvm/mmu/tdp_iter.c | 6 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 9 files changed, 79 insertions(+), 72 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 2b9d08b080cc..0199c8c2222d 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -6,11 +6,6 @@ #include "kvm_cache_regs.h" #include "cpuid.h" -#define PT64_PT_BITS 9 -#define PT64_ENT_PER_PAGE (1 << PT64_PT_BITS) -#define PT32_PT_BITS 10 -#define PT32_ENT_PER_PAGE (1 << PT32_PT_BITS) - #define PT_WRITABLE_SHIFT 1 #define PT_USER_SHIFT 2 @@ -34,6 +29,9 @@ #define PT_DIR_PAT_SHIFT 12 #define PT_DIR_PAT_MASK (1ULL << PT_DIR_PAT_SHIFT) +/* The number of bits for 32-bit PTEs is to compute the quandrant. :-( */ +#define PT32_LEVEL_BITS 10 + #define PT64_ROOT_5LEVEL 5 #define PT64_ROOT_4LEVEL 4 #define PT32_ROOT_LEVEL 2 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ef92717bff86..cc93649f41cb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -113,26 +113,6 @@ module_param(dbg, bool, 0644); #define PTE_PREFETCH_NUM 8 -#define PT32_LEVEL_BITS 10 - -#define PT32_LEVEL_SHIFT(level) \ - (PAGE_SHIFT + (level - 1) * PT32_LEVEL_BITS) - -#define PT32_LVL_OFFSET_MASK(level) \ - (PT32_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \ - * PT32_LEVEL_BITS))) - 1)) - -#define PT32_INDEX(address, level)\ - (((address) >> PT32_LEVEL_SHIFT(level)) & ((1 << PT32_LEVEL_BITS) - 1)) - - -#define PT32_BASE_ADDR_MASK PAGE_MASK -#define PT32_DIR_BASE_ADDR_MASK \ - (PAGE_MASK & ~((1ULL << (PAGE_SHIFT + PT32_LEVEL_BITS)) - 1)) -#define PT32_LVL_ADDR_MASK(level) \ - (PAGE_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \ - * PT32_LEVEL_BITS))) - 1)) - #include /* make pte_list_desc fit well in cache line */ @@ -675,7 +655,7 @@ static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index) if (!sp->role.direct) return sp->gfns[index]; - return sp->gfn + (index << ((sp->role.level - 1) * PT64_LEVEL_BITS)); + return sp->gfn + (index << ((sp->role.level - 1) * SPTE_LEVEL_BITS)); } static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn) @@ -1706,7 +1686,7 @@ static int __mmu_unsync_walk(struct kvm_mmu_page *sp, continue; } - child = to_shadow_page(ent & PT64_BASE_ADDR_MASK); + child = to_shadow_page(ent & SPTE_BASE_ADDR_MASK); if (child->unsync_children) { if (mmu_pages_add(pvec, child, i)) @@ -1989,8 +1969,8 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, role.gpte_is_8_bytes = true; role.access = access; if (!direct_mmu && vcpu->arch.mmu->root_level <= PT32_ROOT_LEVEL) { - quadrant = gaddr >> (PAGE_SHIFT + (PT64_PT_BITS * level)); - quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1; + quadrant = gaddr >> (PAGE_SHIFT + (SPTE_LEVEL_BITS * level)); + quadrant &= (1 << ((PT32_LEVEL_BITS - SPTE_LEVEL_BITS) * level)) - 1; role.quadrant = quadrant; } @@ -2082,7 +2062,7 @@ static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterato iterator->shadow_addr = vcpu->arch.mmu->pae_root[(addr >> 30) & 3]; - iterator->shadow_addr &= PT64_BASE_ADDR_MASK; + iterator->shadow_addr &= SPTE_BASE_ADDR_MASK; --iterator->level; if (!iterator->shadow_addr) iterator->level = 0; @@ -2101,7 +2081,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator) if (iterator->level < PG_LEVEL_4K) return false; - iterator->index = SHADOW_PT_INDEX(iterator->addr, iterator->level); + iterator->index = SPTE_INDEX(iterator->addr, iterator->level); iterator->sptep = ((u64 *)__va(iterator->shadow_addr)) + iterator->index; return true; } @@ -2114,7 +2094,7 @@ static void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator, return; } - iterator->shadow_addr = spte & PT64_BASE_ADDR_MASK; + iterator->shadow_addr = spte & SPTE_BASE_ADDR_MASK; --iterator->level; } @@ -2153,7 +2133,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, * so we should update the spte at this point to get * a new sp with the correct access. */ - child = to_shadow_page(*sptep & PT64_BASE_ADDR_MASK); + child = to_shadow_page(*sptep & SPTE_BASE_ADDR_MASK); if (child->role.access == direct_access) return; @@ -2176,7 +2156,7 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, if (is_large_pte(pte)) --kvm->stat.lpages; } else { - child = to_shadow_page(pte & PT64_BASE_ADDR_MASK); + child = to_shadow_page(pte & SPTE_BASE_ADDR_MASK); drop_parent_pte(child, spte); /* @@ -2202,7 +2182,7 @@ static int kvm_mmu_page_unlink_children(struct kvm *kvm, int zapped = 0; unsigned i; - for (i = 0; i < PT64_ENT_PER_PAGE; ++i) + for (i = 0; i < SPTE_ENT_PER_PAGE; ++i) zapped += mmu_page_zap_pte(kvm, sp, sp->spt + i, invalid_list); return zapped; @@ -2580,7 +2560,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep, struct kvm_mmu_page *child; u64 pte = *sptep; - child = to_shadow_page(pte & PT64_BASE_ADDR_MASK); + child = to_shadow_page(pte & SPTE_BASE_ADDR_MASK); drop_parent_pte(child, sptep); flush = true; } else if (pfn != spte_to_pfn(*sptep)) { @@ -3134,7 +3114,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, if (!VALID_PAGE(*root_hpa)) return; - sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK); + sp = to_shadow_page(*root_hpa & SPTE_BASE_ADDR_MASK); if (is_tdp_mmu_page(sp)) kvm_tdp_mmu_put_root(kvm, sp, false); @@ -3494,7 +3474,7 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) hpa_t root = vcpu->arch.mmu->pae_root[i]; if (IS_VALID_PAE_ROOT(root)) { - root &= PT64_BASE_ADDR_MASK; + root &= SPTE_BASE_ADDR_MASK; sp = to_shadow_page(root); mmu_sync_children(vcpu, sp); } @@ -4927,11 +4907,11 @@ static bool need_remote_flush(u64 old, u64 new) return false; if (!is_shadow_present_pte(new)) return true; - if ((old ^ new) & PT64_BASE_ADDR_MASK) + if ((old ^ new) & SPTE_BASE_ADDR_MASK) return true; old ^= shadow_nx_mask; new ^= shadow_nx_mask; - return (old & ~new & PT64_PERM_MASK) != 0; + return (old & ~new & SPTE_PERM_MASK) != 0; } static u64 mmu_pte_write_fetch_gpte(struct kvm_vcpu *vcpu, gpa_t *gpa, diff --git a/arch/x86/kvm/mmu/mmu_audit.c b/arch/x86/kvm/mmu/mmu_audit.c index cedc17b2f60e..4b5335188d01 100644 --- a/arch/x86/kvm/mmu/mmu_audit.c +++ b/arch/x86/kvm/mmu/mmu_audit.c @@ -45,7 +45,7 @@ static void __mmu_spte_walk(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, !is_last_spte(ent[i], level)) { struct kvm_mmu_page *child; - child = to_shadow_page(ent[i] & PT64_BASE_ADDR_MASK); + child = to_shadow_page(ent[i] & SPTE_BASE_ADDR_MASK); __mmu_spte_walk(vcpu, child, fn, level - 1); } } @@ -71,7 +71,7 @@ static void mmu_spte_walk(struct kvm_vcpu *vcpu, inspect_spte_fn fn) hpa_t root = vcpu->arch.mmu->pae_root[i]; if (IS_VALID_PAE_ROOT(root)) { - root &= PT64_BASE_ADDR_MASK; + root &= SPTE_BASE_ADDR_MASK; sp = to_shadow_page(root); __mmu_spte_walk(vcpu, sp, fn, 2); } @@ -117,7 +117,7 @@ static void audit_mappings(struct kvm_vcpu *vcpu, u64 *sptep, int level) return; hpa = pfn << PAGE_SHIFT; - if ((*sptep & PT64_BASE_ADDR_MASK) != hpa) + if ((*sptep & SPTE_BASE_ADDR_MASK) != hpa) audit_printk(vcpu->kvm, "levels %d pfn %llx hpa %llx " "ent %llxn", vcpu->arch.mmu->root_level, pfn, hpa, *sptep); diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index 18be103df9d5..b9ef013a2202 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -20,6 +20,20 @@ extern bool dbg; #define MMU_WARN_ON(x) do { } while (0) #endif +/* Page table builder macros common to shadow (host) PTEs and guest PTEs. */ +#define __PT_LEVEL_SHIFT(level, bits_per_level) \ + (PAGE_SHIFT + ((level) - 1) * (bits_per_level)) +#define __PT_INDEX(address, level, bits_per_level) \ + (((address) >> __PT_LEVEL_SHIFT(level, bits_per_level)) & ((1 << (bits_per_level)) - 1)) + +#define __PT_LVL_ADDR_MASK(base_addr_mask, level, bits_per_level) \ + ((base_addr_mask) & ~((1ULL << (PAGE_SHIFT + (((level) - 1) * (bits_per_level)))) - 1)) + +#define __PT_LVL_OFFSET_MASK(base_addr_mask, level, bits_per_level) \ + ((base_addr_mask) & ((1ULL << (PAGE_SHIFT + (((level) - 1) * (bits_per_level)))) - 1)) + +#define __PT_ENT_PER_PAGE(bits_per_level) (1 << (bits_per_level)) + /* * Unlike regular MMU roots, PAE "roots", a.k.a. PDPTEs/PDPTRs, have a PRESENT * bit, and thus are guaranteed to be non-zero when valid. And, when a guest diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index a2dbea70ffda..caaeec848b12 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -24,13 +24,32 @@ #ifndef __KVM_X86_PAGING_TMPL_COMMON_H #define __KVM_X86_PAGING_TMPL_COMMON_H -#define GUEST_PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) +/* 64-bit guest PTEs, i.e. PAE paging and EPT. */ +#define PT64_LEVEL_BITS 9 +#define PT64_LEVEL_SHIFT __PT_LEVEL_SHIFT(level, PT64_LEVEL_BITS) +#define PT64_INDEX(address, level) __PT_INDEX(address, level, PT64_LEVEL_BITS) +#define PT64_ENT_PER_PAGE __PT_ENT_PER_PAGE(PT64_LEVEL_BITS) + +#define PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) #define PT64_LVL_ADDR_MASK(level) \ - (GUEST_PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \ - * PT64_LEVEL_BITS))) - 1)) + __PT_LVL_ADDR_MASK(PT64_BASE_ADDR_MASK, level, PT64_LEVEL_BITS) + #define PT64_LVL_OFFSET_MASK(level) \ - (GUEST_PT64_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \ - * PT64_LEVEL_BITS))) - 1)) + __PT_LVL_OFFSET_MASK(PT64_BASE_ADDR_MASK, level, PT64_LEVEL_BITS) + +/* 32-bit guest PTEs, i.e. non-PAE IA32 paging. */ +#define PT32_LEVEL_SHIFT(level) __PT_LEVEL_SHIFT(level, PT32_LEVEL_BITS) +#define PT32_INDEX(address, level) __PT_INDEX(address, level, PT32_LEVEL_BITS) +#define PT32_ENT_PER_PAGE __PT_ENT_PER_PAGE(PT32_LEVEL_BITS) + +#define PT32_BASE_ADDR_MASK PAGE_MASK +#define PT32_DIR_BASE_ADDR_MASK \ + (PAGE_MASK & ~((1ULL << (PAGE_SHIFT + PT32_LEVEL_BITS)) - 1)) +#define PT32_LVL_ADDR_MASK(level) \ + __PT_LVL_ADDR_MASK(PT32_BASE_ADDR_MASK, level, PT32_LEVEL_BITS) + +#define PT32_LVL_OFFSET_MASK(level) \ + __PT_LVL_OFFSET_MASK(PT32_BASE_ADDR_MASK, level, PT32_LEVEL_BITS) #define PT32_DIR_PSE36_SIZE 4 #define PT32_DIR_PSE36_SHIFT 13 @@ -55,7 +74,7 @@ static inline gfn_t pse36_gfn_delta(u32 gpte) #define pt_element_t u64 #define guest_walker guest_walker64 #define FNAME(name) paging##64_##name - #define PT_BASE_ADDR_MASK GUEST_PT64_BASE_ADDR_MASK + #define PT_BASE_ADDR_MASK PT64_BASE_ADDR_MASK #define PT_LVL_ADDR_MASK(lvl) PT64_LVL_ADDR_MASK(lvl) #define PT_LVL_OFFSET_MASK(lvl) PT64_LVL_OFFSET_MASK(lvl) #define PT_INDEX(addr, level) PT64_INDEX(addr, level) @@ -88,7 +107,7 @@ static inline gfn_t pse36_gfn_delta(u32 gpte) #define pt_element_t u64 #define guest_walker guest_walkerEPT #define FNAME(name) ept_##name - #define PT_BASE_ADDR_MASK GUEST_PT64_BASE_ADDR_MASK + #define PT_BASE_ADDR_MASK PT64_BASE_ADDR_MASK #define PT_LVL_ADDR_MASK(lvl) PT64_LVL_ADDR_MASK(lvl) #define PT_LVL_OFFSET_MASK(lvl) PT64_LVL_OFFSET_MASK(lvl) #define PT_INDEX(addr, level) PT64_INDEX(addr, level) @@ -1072,7 +1091,7 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) first_pte_gpa = FNAME(get_level1_sp_gpa)(sp); - for (i = 0; i < PT64_ENT_PER_PAGE; i++) { + for (i = 0; i < SPTE_ENT_PER_PAGE; i++) { unsigned pte_access; pt_element_t gpte; gpa_t pte_gpa; diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 66d43cec0c31..cc7feac12e26 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -200,7 +200,7 @@ u64 kvm_mmu_changed_pte_notifier_make_spte(u64 old_spte, kvm_pfn_t new_pfn) { u64 new_spte; - new_spte = old_spte & ~PT64_BASE_ADDR_MASK; + new_spte = old_spte & ~SPTE_BASE_ADDR_MASK; new_spte |= (u64)new_pfn << PAGE_SHIFT; new_spte &= ~PT_WRITABLE_MASK; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 6925dfc38981..719785eea2fe 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -34,12 +34,12 @@ static_assert(SPTE_TDP_AD_ENABLED_MASK == 0); #ifdef CONFIG_DYNAMIC_PHYSICAL_MASK -#define PT64_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1)) +#define SPTE_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1)) #else -#define PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) +#define SPTE_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) #endif -#define PT64_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \ +#define SPTE_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \ | shadow_x_mask | shadow_nx_mask | shadow_me_mask) #define ACC_EXEC_MASK 1 @@ -48,17 +48,13 @@ static_assert(SPTE_TDP_AD_ENABLED_MASK == 0); #define ACC_ALL (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK) /* The mask for the R/X bits in EPT PTEs */ -#define PT64_EPT_READABLE_MASK 0x1ull -#define PT64_EPT_EXECUTABLE_MASK 0x4ull +#define SPTE_EPT_READABLE_MASK 0x1ull +#define SPTE_EPT_EXECUTABLE_MASK 0x4ull -#define PT64_LEVEL_BITS 9 - -#define PT64_LEVEL_SHIFT(level) \ - (PAGE_SHIFT + (level - 1) * PT64_LEVEL_BITS) - -#define PT64_INDEX(address, level)\ - (((address) >> PT64_LEVEL_SHIFT(level)) & ((1 << PT64_LEVEL_BITS) - 1)) -#define SHADOW_PT_INDEX(addr, level) PT64_INDEX(addr, level) +#define SPTE_LEVEL_BITS 9 +#define SPTE_LEVEL_SHIFT(level) __PT_LEVEL_SHIFT(level, SPTE_LEVEL_BITS) +#define SPTE_INDEX(address, level) __PT_INDEX(address, level, SPTE_LEVEL_BITS) +#define SPTE_ENT_PER_PAGE __PT_ENT_PER_PAGE(SPTE_LEVEL_BITS) /* Bits 9 and 10 are ignored by all non-EPT PTEs. */ #define DEFAULT_SPTE_HOST_WRITEABLE BIT_ULL(9) @@ -71,8 +67,8 @@ static_assert(SPTE_TDP_AD_ENABLED_MASK == 0); * restored only when a write is attempted to the page. This mask obviously * must not overlap the A/D type mask. */ -#define SHADOW_ACC_TRACK_SAVED_BITS_MASK (PT64_EPT_READABLE_MASK | \ - PT64_EPT_EXECUTABLE_MASK) +#define SHADOW_ACC_TRACK_SAVED_BITS_MASK (SPTE_EPT_READABLE_MASK | \ + SPTE_EPT_EXECUTABLE_MASK) #define SHADOW_ACC_TRACK_SAVED_BITS_SHIFT 54 #define SHADOW_ACC_TRACK_SAVED_MASK (SHADOW_ACC_TRACK_SAVED_BITS_MASK << \ SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) @@ -269,7 +265,7 @@ static inline bool is_executable_pte(u64 spte) static inline kvm_pfn_t spte_to_pfn(u64 pte) { - return (pte & PT64_BASE_ADDR_MASK) >> PAGE_SHIFT; + return (pte & SPTE_BASE_ADDR_MASK) >> PAGE_SHIFT; } static inline bool is_accessed_spte(u64 spte) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index b3ed302c1a35..5c002c43cb3c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -11,7 +11,7 @@ static void tdp_iter_refresh_sptep(struct tdp_iter *iter) { iter->sptep = iter->pt_path[iter->level - 1] + - SHADOW_PT_INDEX(iter->gfn << PAGE_SHIFT, iter->level); + SPTE_INDEX(iter->gfn << PAGE_SHIFT, iter->level); iter->old_spte = READ_ONCE(*rcu_dereference(iter->sptep)); } @@ -113,8 +113,8 @@ static bool try_step_side(struct tdp_iter *iter) * Check if the iterator is already at the end of the current page * table. */ - if (SHADOW_PT_INDEX(iter->gfn << PAGE_SHIFT, iter->level) == - (PT64_ENT_PER_PAGE - 1)) + if (SPTE_INDEX(iter->gfn << PAGE_SHIFT, iter->level) == + (SPTE_ENT_PER_PAGE - 1)) return false; iter->gfn += KVM_PAGES_PER_HPAGE(iter->level); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index caac4ddb46df..3720d244e5a2 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -335,7 +335,7 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, tdp_ptep_t pt, tdp_mmu_unlink_page(kvm, sp, shared); - for (i = 0; i < PT64_ENT_PER_PAGE; i++) { + for (i = 0; i < SPTE_ENT_PER_PAGE; i++) { sptep = rcu_dereference(pt) + i; gfn = base_gfn + i * KVM_PAGES_PER_HPAGE(level);