From patchwork Sat Dec 30 17:23:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Roth X-Patchwork-Id: 13506971 X-Patchwork-Delegate: herbert@gondor.apana.org.au Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2069.outbound.protection.outlook.com [40.107.244.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F8AEBA56; Sat, 30 Dec 2023 17:30:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="iqsQuC3/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QdnmZp+rjVX79uTvC59GZwL2rC74qEC5mdwh47Erh0+Vfmag7o32pXTEtfVfRJnG+PA1U0DvFBvhzkvkkONPiewA+p1nPzWl3xSLph9TMHubZ5tbw6s3MyhYa7E2s9gAAp9HF5vvFkPRzMtQT5kfYflRzIeXJ7czVz2ubYP7ONcpNcc7ukmjFmspjbwexyOYD0clhrpMVI1IHlyCvJW+wRy04MbY2Hix6Xun+Nu8uOcowsUziOfNhBEkYy+ZUOwjaA4z6De85F1syUVr2thqBRBrrGtTmNylO4fDWHKvfQabCcbE0FjvYxONUpYJkr2ImmPDbjatBT++D4djAW4IGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KEvKLYGqaOkTf4/xmcGIj8QjD0aDrDfVjx+hNzCqkU8=; b=i1O99t4oIW1m3becIysqRY2f2hIeDaDhjKt7+W1XW4sq/w3TrfJTVR55RTXDENRv6El8TRxBwsnTzOOfpWxGomErm24+1xHtf+nAVs+FOFJAirNbDizXJS2KnBejLLBmvfCthDFOd8VozN2Jz8hwNU0HIhA/SYwNvFggU+buIfw3G1tKeTIIW5LgTaDyfSV7b0u1NToE61bSmtBxJQpi+bypKUAkrJbLxmF5EUSw6Za7z8+YV88T4QiZMAGLRs2gOtWN5kGpVtX4P91qDTGyk8FMRGjzjqQI11OmyWF24t5OvDs5nSP8kQwYO/VqFl9gXCPwFDx/m82BEU/aSVnR+g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=temperror (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=temperror action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KEvKLYGqaOkTf4/xmcGIj8QjD0aDrDfVjx+hNzCqkU8=; b=iqsQuC3/fr4vLJ6Ji8h9oVP11USGXwqsE6JxW/dk9lhKrQMJ3DrrFCZYaersEu2bMyFYfQT18pNQaQs0o1xrv/D7u2OAoDYUXSX1fzZ4b964Jjd8X26z22QoRy/trczoBHPIWyqThA3BvKtnDY1+PrIByu87U8gyWQLXlrUurm8= Received: from MW2PR2101CA0005.namprd21.prod.outlook.com (2603:10b6:302:1::18) by IA0PR12MB8373.namprd12.prod.outlook.com (2603:10b6:208:40d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7135.22; Sat, 30 Dec 2023 17:30:19 +0000 Received: from CO1PEPF000044F4.namprd05.prod.outlook.com (2603:10b6:302:1:cafe::7e) by MW2PR2101CA0005.outlook.office365.com (2603:10b6:302:1::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.7 via Frontend Transport; Sat, 30 Dec 2023 17:30:19 +0000 X-MS-Exchange-Authentication-Results: spf=temperror (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=temperror action=none header.from=amd.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of amd.com: DNS Timeout) Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044F4.mail.protection.outlook.com (10.167.241.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7159.9 via Frontend Transport; Sat, 30 Dec 2023 17:30:17 +0000 Received: from localhost (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34; Sat, 30 Dec 2023 11:30:15 -0600 From: Michael Roth To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Brijesh Singh Subject: [PATCH v11 24/35] KVM: SEV: Add support to handle RMP nested page faults Date: Sat, 30 Dec 2023 11:23:40 -0600 Message-ID: <20231230172351.574091-25-michael.roth@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231230172351.574091-1-michael.roth@amd.com> References: <20231230172351.574091-1-michael.roth@amd.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F4:EE_|IA0PR12MB8373:EE_ X-MS-Office365-Filtering-Correlation-Id: 285ee3f8-fadb-4691-6d4c-08dc095cfd91 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EaCjLEJeWTYavLlAVxsu+75/Z7WqTOheXiULOlWroaohQ9Ks7eV2RuD+6hFVzyJ/gdqWFzQTWFVTcEm2Tc83KcBj3kUtzAMX947ZrMnh/SRb9xPVIjGxOfP43sLKY7KzLb8k5bZz6IkVYTTkaz9/igQr+BLP1hjut6fhnujdNwdv0M/uxj3l98uuapqTrTXQnOMEEfSM63ZgTXJEM8MZHBqMrlPHcoLquxFez5VI2IG/qEmr5o2VZ91xPt7N6yeX8g+SdFCLnfYAWOmkb84Adf8zmtVWhkrxNr8UIsZ7BTNff5wR1UEhhsZXS+A1Tzc5B9KZ/rK4vQboIe4o9wStXoc/OuYyglAa77lMmh5XhFwq5L13MGXSQF+L/FWVuQt5RjRus+kgE+DuD012bmzOQIGCjHBrz6sV99CqWzVWXx/GfsOSplT/7aKXd96johvx9C1LcIaQ85P8hz8ss9JFSjkV5/+g4wqlAz/viuySST6PrHsOao6xhUJ0kCT+qB1aVlNkx/FrnJcKcD5/IMbBwhNBwmGuWcCKFhEGZCnf4+qL6EitWxw6LKeG8bHh5bYqGzL1Md8B5cx4fjvfObiOL/5NzXdAUPisyvs+IBDDB8QFD6YMgKDEyEowf+EbwsmDP8tuxzgpmEiSRrl9EuAprYZia3bKcuQiUUG2D7IwS7XGbkuUTIbtl21FqpUuM/UX48q/OdhBtKHyRHoWlZmT2dDhBnVrD/CCd7f2PsgpQbgVLmSZ9Xx/7plc5/hi/3eUoDOslc/VHYCR9ZszZi68eA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(396003)(39860400002)(136003)(376002)(346002)(230922051799003)(186009)(451199024)(1800799012)(64100799003)(82310400011)(36840700001)(40470700004)(46966006)(36860700001)(82740400003)(81166007)(356005)(41300700001)(36756003)(54906003)(63350400001)(44832011)(40480700001)(63370400001)(1076003)(70206006)(86362001)(336012)(8676002)(6916009)(83380400001)(8936002)(2616005)(16526019)(26005)(4326008)(426003)(70586007)(316002)(2906002)(47076005)(6666004)(5660300002)(478600001)(7416002)(7406005)(40460700003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Dec 2023 17:30:17.4012 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 285ee3f8-fadb-4691-6d4c-08dc095cfd91 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F4.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8373 From: Brijesh Singh When SEV-SNP is enabled in the guest, the hardware places restrictions on all memory accesses based on the contents of the RMP table. When hardware encounters RMP check failure caused by the guest memory access it raises the #NPF. The error code contains additional information on the access type. See the APM volume 2 for additional information. When using gmem, RMP faults resulting from mismatches between the state in the RMP table vs. what the guest expects via its page table result in KVM_EXIT_MEMORY_FAULTs being forwarded to userspace to handle. This means the only expected case that needs to be handled in the kernel is when the page size of the entry in the RMP table is larger than the mapping in the nested page table, in which case a PSMASH instruction needs to be issued to split the large RMP entry into individual 4K entries so that subsequent accesses can succeed. Signed-off-by: Brijesh Singh Co-developed-by: Michael Roth Signed-off-by: Michael Roth Signed-off-by: Ashish Kalra --- arch/x86/include/asm/sev.h | 3 ++ arch/x86/kvm/svm/sev.c | 92 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/svm/svm.c | 21 +++++++-- arch/x86/kvm/svm/svm.h | 1 + 4 files changed, 113 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index 435ba9bc4510..e84dd1d2d8ab 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -90,6 +90,9 @@ extern bool handle_vc_boot_ghcb(struct pt_regs *regs); /* RMUPDATE detected 4K page and 2MB page overlap. */ #define RMPUPDATE_FAIL_OVERLAP 4 +/* PSMASH failed due to concurrent access by another CPU */ +#define PSMASH_FAIL_INUSE 3 + /* RMP page size */ #define RMP_PG_SIZE_4K 0 #define RMP_PG_SIZE_2M 1 diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 8b6143110411..ad1aea7f6266 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3276,6 +3276,13 @@ static void set_ghcb_msr(struct vcpu_svm *svm, u64 value) svm->vmcb->control.ghcb_gpa = value; } +static int snp_rmptable_psmash(kvm_pfn_t pfn) +{ + pfn = pfn & ~(KVM_PAGES_PER_HPAGE(PG_LEVEL_2M) - 1); + + return psmash(pfn); +} + static int snp_complete_psc_msr(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); @@ -3835,3 +3842,88 @@ struct page *snp_safe_alloc_page(struct kvm_vcpu *vcpu) return p; } + +void handle_rmp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) +{ + struct kvm_memory_slot *slot; + struct kvm *kvm = vcpu->kvm; + int order, rmp_level, ret; + bool assigned; + kvm_pfn_t pfn; + gfn_t gfn; + + gfn = gpa >> PAGE_SHIFT; + + /* + * The only time RMP faults occur for shared pages is when the guest is + * triggering an RMP fault for an implicit page-state change from + * shared->private. Implicit page-state changes are forwarded to + * userspace via KVM_EXIT_MEMORY_FAULT events, however, so RMP faults + * for shared pages should not end up here. + */ + if (!kvm_mem_is_private(kvm, gfn)) { + pr_warn_ratelimited("SEV: Unexpected RMP fault, size-mismatch for non-private GPA 0x%llx\n", + gpa); + return; + } + + slot = gfn_to_memslot(kvm, gfn); + if (!kvm_slot_can_be_private(slot)) { + pr_warn_ratelimited("SEV: Unexpected RMP fault, non-private slot for GPA 0x%llx\n", + gpa); + return; + } + + ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &order); + if (ret) { + pr_warn_ratelimited("SEV: Unexpected RMP fault, no private backing page for GPA 0x%llx\n", + gpa); + return; + } + + ret = snp_lookup_rmpentry(pfn, &assigned, &rmp_level); + if (ret || !assigned) { + pr_warn_ratelimited("SEV: Unexpected RMP fault, no assigned RMP entry found for GPA 0x%llx PFN 0x%llx error %d\n", + gpa, pfn, ret); + goto out; + } + + /* + * There are 2 cases where a PSMASH may be needed to resolve an #NPF + * with PFERR_GUEST_RMP_BIT set: + * + * 1) RMPADJUST/PVALIDATE can trigger an #NPF with PFERR_GUEST_SIZEM + * bit set if the guest issues them with a smaller granularity than + * what is indicated by the page-size bit in the 2MB-aligned RMP + * entry for the PFN that backs the GPA. + * + * 2) Guest access via NPT can trigger an #NPF if the NPT mapping is + * smaller than what is indicated by the 2MB-aligned RMP entry for + * the PFN that backs the GPA. + * + * In both these cases, the corresponding 2M RMP entry needs to + * be PSMASH'd to 512 4K RMP entries. If the RMP entry is already + * split into 4K RMP entries, then this is likely a spurious case which + * can occur when there are concurrent accesses by the guest to a 2MB + * GPA range that is backed by a 2MB-aligned PFN who's RMP entry is in + * the process of being PMASH'd into 4K entries. These cases should + * resolve automatically on subsequent accesses, so just ignore them + * here. + */ + if (rmp_level == PG_LEVEL_4K) { + pr_debug_ratelimited("%s: Spurious RMP fault for GPA 0x%llx, error_code 0x%llx", + __func__, gpa, error_code); + goto out; + } + + pr_debug_ratelimited("%s: Splitting 2M RMP entry for GPA 0x%llx, error_code 0x%llx", + __func__, gpa, error_code); + ret = snp_rmptable_psmash(pfn); + if (ret && ret != PSMASH_FAIL_INUSE) + pr_err_ratelimited("SEV: Unable to split RMP entry for GPA 0x%llx PFN 0x%llx ret %d\n", + gpa, pfn, ret); + + kvm_zap_gfn_range(kvm, gfn, gfn + PTRS_PER_PMD); +out: + put_page(pfn_to_page(pfn)); +} diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 18d55df7fa5f..4367da074612 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -2051,15 +2051,28 @@ static int pf_interception(struct kvm_vcpu *vcpu) static int npf_interception(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); + int rc; u64 fault_address = svm->vmcb->control.exit_info_2; u64 error_code = svm->vmcb->control.exit_info_1; trace_kvm_page_fault(vcpu, fault_address, error_code); - return kvm_mmu_page_fault(vcpu, fault_address, error_code, - static_cpu_has(X86_FEATURE_DECODEASSISTS) ? - svm->vmcb->control.insn_bytes : NULL, - svm->vmcb->control.insn_len); + rc = kvm_mmu_page_fault(vcpu, fault_address, error_code, + static_cpu_has(X86_FEATURE_DECODEASSISTS) ? + svm->vmcb->control.insn_bytes : NULL, + svm->vmcb->control.insn_len); + + /* + * rc == 0 indicates a userspace exit is needed to handle page + * transitions, so do that first before updating the RMP table. + */ + if (error_code & PFERR_GUEST_RMP_MASK) { + if (rc == 0) + return rc; + handle_rmp_page_fault(vcpu, fault_address, error_code); + } + + return rc; } static int db_interception(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 2bee24017bae..fb98d88d8124 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -717,6 +717,7 @@ void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa); void sev_es_unmap_ghcb(struct vcpu_svm *svm); struct page *snp_safe_alloc_page(struct kvm_vcpu *vcpu); +void handle_rmp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code); /* vmenter.S */