From patchwork Tue Mar 8 04:38:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F54C433EF for ; Tue, 8 Mar 2022 04:40:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245213AbiCHElH (ORCPT ); Mon, 7 Mar 2022 23:41:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245130AbiCHElE (ORCPT ); Mon, 7 Mar 2022 23:41:04 -0500 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2041.outbound.protection.outlook.com [40.107.93.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B6762C647; Mon, 7 Mar 2022 20:40:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M1/KvOIV5bax8ukHSBSCrZ0PBG26Fixc1yXdUfEvMFQHvRfgk0igxCVRb7TCR3jcF+aeiPoKc59VS+VlS3hbTsU8MOMTbtUGjFQVD8p3bUIgd3tduONXTTj+ceZXxvBkZ3+ANqJM2AO8qs3ByXBceAFR/C92VP8MoQOesrIQ6R9KbFN2sa1Dcc8Ik22Rea26zp5T+FKt1LFsYmPUZsHcdZt2OwOIrRGYfnt6E9+z2xZQfCFbWBtTc5QXbshvwl069+R2lqwczdYAwoDJyDzEs8EThX1OA432ln8Q9ox2ZN3peCHB/r5BDG2Be+lcegOfcL94O9j6WCpIiDfdfFQZCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Uqz3I+koh45aexq7CPsIKKTkBL0kzdnWB4RnzMSqJek=; b=H2FvyUCSj+CbbTxTzsptmrxaarvHJ38tyHKb3NgltBzF9A7GBLndPRyvTdd/kcK8m6OGCbdAkzcwf4aUbIC/PZC04P0oPFEHhYE05ruByksq2nU89GC8KOcBSm206KVq+9TrB9XKaaaXQsKiAwJPpU/aDtOfANhfmEXDoKUqj3784kTeknjKCJ1ZujbR41rfn75i4O5uHrEdQtV0ZNr466aMGV89oZKlkb7nGZmr+EpVqv1RU8Go4UWceblVujurCCjy/pcdYjE6rOSWhPmx685RGh+32XEXwtcjpGSEnRPpgGDJcMDkXTBlwVbzGParjfVrtlJSdcivWZ67lqovzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Uqz3I+koh45aexq7CPsIKKTkBL0kzdnWB4RnzMSqJek=; b=4e+5Ws3rYoy0DxW/1AjrvgnRZfWSiXo8Fq8dRW7OpikBwZinEvD2BDOhhgttTN6CpPwmK6tEXD7HdUgLzF1UcwmVCNybEdwDHgQi/IfTibH+rcu7U4wc17hQnxLkleY+pUVQpOHz1j+CV3ph2FfDoj6Wq2HTL0c64UVB/2cXbXc= Received: from DS7PR03CA0006.namprd03.prod.outlook.com (2603:10b6:5:3b8::11) by DM6PR12MB4042.namprd12.prod.outlook.com (2603:10b6:5:215::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.15; Tue, 8 Mar 2022 04:40:07 +0000 Received: from DM6NAM11FT027.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b8:cafe::a5) by DS7PR03CA0006.outlook.office365.com (2603:10b6:5:3b8::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT027.mail.protection.outlook.com (10.13.172.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:07 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:01 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 1/9] KVM: Introduce pinning flag to hva_to_pfn* Date: Tue, 8 Mar 2022 10:08:49 +0530 Message-ID: <20220308043857.13652-2-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f147d715-128d-46b5-8825-08da00bdb893 X-MS-TrafficTypeDiagnostic: DM6PR12MB4042:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HGWU0fx3tvPn6Hm/1uhX+EuF+n3eq5s8qzeJHUK85cVFNoDReAOg6hqv4TirInRNaRcSMjiEnME18N0g6EHyQywi/D7MvQsFaX70DZdjU95Ae6p8fylWvMVXS/IZcO/vpA3FjSTLUx4caK1tfqEKzsGwJKGQLJmjZVI4S2cbtuyLNgVfikE7mm2Pn8o0bLoVkfsC6Q7rqC34tegmJRPZXlUagR7P7BXBjcJze+Pe2eQyqeQISw1JEiwfDb8Ld2BDspLe3Dne61BLXMxDffD7+L0ckKUdjTq5GeYZpaMnKkOIrjW9Isuw3uFvIQ16WNGApy8UgYu5JwlTTkjshhZUBJDCpfhZZ8aL8LBl5AB0/5MadHCLs1rGax/67h6fGXS/581JN4stZ9aTkZf1acAmDNnf4/h4F9TIwtj3rjIW8v6XRpXFNDAaqUuowfO5eUdJ+dBV0KA9onLVL5DL54S1pZYQFXc3rc9WNcBKRA2UA8bdPRlYP71lwvftv0I92suXJzsnb3EAtq0Sv2nDBOznE6kjGpb0xZVEe87KIKqmmUii1akVkTICwfmoE6wulaDAmjIwQ2q7+CWAyszmK4JXeBGsNGYKNqRdKpRu0jSJUGGOjmQtJ4xoWgwa2seT2zW612rXiqm52K+GxMU1mgPXc1I3If37lOwB61XnmAKKsxGKdfKyXhITjPVfw5GnpM5W0STjWAUlyTMfx3Rzp8jSwQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(36840700001)(40470700004)(46966006)(7696005)(36860700001)(16526019)(6916009)(54906003)(2906002)(26005)(186003)(316002)(82310400004)(356005)(40460700003)(81166007)(336012)(426003)(47076005)(8936002)(2616005)(508600001)(6666004)(83380400001)(1076003)(7416002)(36756003)(8676002)(4326008)(70206006)(5660300002)(70586007)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:07.0904 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f147d715-128d-46b5-8825-08da00bdb893 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT027.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4042 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org KVM allocates pages with get_user_pages_* (that use FOLL_GET). For long term pinning of guest pages pin_user_pages_* (that use FOLL_PIN) need to be used. Add a flag to hva_to_pfn* to allocate pinned pages when the memslot represents encrypted memory. Suggested-by: David Hildenbrand Signed-off-by: Nikunj A Dadhania --- include/linux/kvm_host.h | 6 ++++ virt/kvm/kvm_main.c | 63 ++++++++++++++++++++++++++++++---------- virt/kvm/kvm_mm.h | 2 +- virt/kvm/pfncache.c | 2 +- 4 files changed, 56 insertions(+), 17 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index f11039944c08..c23022960d51 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -55,6 +55,7 @@ * include/linux/kvm_h. */ #define KVM_MEMSLOT_INVALID (1UL << 16) +#define KVM_MEMSLOT_ENCRYPTED (1UL << 17) /* * Bit 63 of the memslot generation number is an "update in-progress flag", @@ -583,6 +584,11 @@ static inline unsigned long *kvm_second_dirty_bitmap(struct kvm_memory_slot *mem return memslot->dirty_bitmap + len / sizeof(*memslot->dirty_bitmap); } +static inline bool memslot_is_encrypted(const struct kvm_memory_slot *slot) +{ + return slot && (slot->flags & KVM_MEMSLOT_ENCRYPTED); +} + #ifndef KVM_DIRTY_LOG_MANUAL_CAPS #define KVM_DIRTY_LOG_MANUAL_CAPS KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 0afc016cc54d..c035fe6b39ec 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2381,9 +2381,10 @@ static inline int check_user_page_hwpoison(unsigned long addr) * only part that runs if we can in atomic context. */ static bool hva_to_pfn_fast(unsigned long addr, bool write_fault, - bool *writable, kvm_pfn_t *pfn) + bool *writable, kvm_pfn_t *pfn, bool use_pin) { struct page *page[1]; + bool ret; /* * Fast pin a writable pfn only if it is a write fault request @@ -2393,7 +2394,12 @@ static bool hva_to_pfn_fast(unsigned long addr, bool write_fault, if (!(write_fault || writable)) return false; - if (get_user_page_fast_only(addr, FOLL_WRITE, page)) { + if (!use_pin) + ret = get_user_page_fast_only(addr, FOLL_WRITE, page); + else + ret = pin_user_pages_fast_only(addr, 1, FOLL_WRITE | FOLL_LONGTERM, page); + + if (ret) { *pfn = page_to_pfn(page[0]); if (writable) @@ -2409,9 +2415,9 @@ static bool hva_to_pfn_fast(unsigned long addr, bool write_fault, * 1 indicates success, -errno is returned if error is detected. */ static int hva_to_pfn_slow(unsigned long addr, bool *async, bool write_fault, - bool *writable, kvm_pfn_t *pfn) + bool *writable, kvm_pfn_t *pfn, bool use_pin) { - unsigned int flags = FOLL_HWPOISON; + unsigned int flags = 0; struct page *page; int npages = 0; @@ -2422,20 +2428,41 @@ static int hva_to_pfn_slow(unsigned long addr, bool *async, bool write_fault, if (write_fault) flags |= FOLL_WRITE; - if (async) - flags |= FOLL_NOWAIT; - npages = get_user_pages_unlocked(addr, 1, &page, flags); + if (!use_pin) { + flags |= FOLL_HWPOISON; + if (async) + flags |= FOLL_NOWAIT; + + npages = get_user_pages_unlocked(addr, 1, &page, flags); + } else { + /* + * FOLL_LONGTERM is not supported in pin_user_pages_unlocked, + * use *_fast instead. + */ + flags |= FOLL_LONGTERM; + npages = pin_user_pages_fast(addr, 1, flags, &page); + } + if (npages != 1) return npages; /* map read fault as writable if possible */ if (unlikely(!write_fault) && writable) { struct page *wpage; + bool ret; + + if (!use_pin) + ret = get_user_page_fast_only(addr, FOLL_WRITE, &wpage); + else + ret = pin_user_pages_fast_only(addr, 1, FOLL_WRITE | FOLL_LONGTERM, &wpage); - if (get_user_page_fast_only(addr, FOLL_WRITE, &wpage)) { + if (ret) { *writable = true; - put_page(page); + if (!use_pin) + put_page(page); + else + unpin_user_page(page); page = wpage; } } @@ -2541,7 +2568,7 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, * whether the mapping is writable. */ kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async, - bool write_fault, bool *writable) + bool write_fault, bool *writable, bool use_pin) { struct vm_area_struct *vma; kvm_pfn_t pfn = 0; @@ -2550,13 +2577,13 @@ kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async, /* we can do it either atomically or asynchronously, not both */ BUG_ON(atomic && async); - if (hva_to_pfn_fast(addr, write_fault, writable, &pfn)) + if (hva_to_pfn_fast(addr, write_fault, writable, &pfn, use_pin)) return pfn; if (atomic) return KVM_PFN_ERR_FAULT; - npages = hva_to_pfn_slow(addr, async, write_fault, writable, &pfn); + npages = hva_to_pfn_slow(addr, async, write_fault, writable, &pfn, use_pin); if (npages == 1) return pfn; @@ -2616,7 +2643,7 @@ kvm_pfn_t __gfn_to_pfn_memslot(const struct kvm_memory_slot *slot, gfn_t gfn, } return hva_to_pfn(addr, atomic, async, write_fault, - writable); + writable, memslot_is_encrypted(slot)); } EXPORT_SYMBOL_GPL(__gfn_to_pfn_memslot); @@ -2788,8 +2815,14 @@ EXPORT_SYMBOL_GPL(kvm_release_page_clean); void kvm_release_pfn_clean(kvm_pfn_t pfn) { - if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn)) - put_page(pfn_to_page(pfn)); + if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn)) { + struct page *page = pfn_to_page(pfn); + + if (page_maybe_dma_pinned(page)) + unpin_user_page(page); + else + put_page(page); + } } EXPORT_SYMBOL_GPL(kvm_release_pfn_clean); diff --git a/virt/kvm/kvm_mm.h b/virt/kvm/kvm_mm.h index 34ca40823260..b1a5e379949b 100644 --- a/virt/kvm/kvm_mm.h +++ b/virt/kvm/kvm_mm.h @@ -25,7 +25,7 @@ #endif /* KVM_HAVE_MMU_RWLOCK */ kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async, - bool write_fault, bool *writable); + bool write_fault, bool *writable, bool use_pin); #ifdef CONFIG_HAVE_KVM_PFNCACHE void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index ce878f4be4da..44384f06c81b 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -135,7 +135,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, unsigned long uhva) smp_rmb(); /* We always request a writeable mapping */ - new_pfn = hva_to_pfn(uhva, false, NULL, true, NULL); + new_pfn = hva_to_pfn(uhva, false, NULL, true, NULL, false); if (is_error_noslot_pfn(new_pfn)) break; From patchwork Tue Mar 8 04:38:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AA3FC433F5 for ; Tue, 8 Mar 2022 04:40:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245270AbiCHElN (ORCPT ); Mon, 7 Mar 2022 23:41:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245130AbiCHElL (ORCPT ); Mon, 7 Mar 2022 23:41:11 -0500 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2046.outbound.protection.outlook.com [40.107.212.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED3452CCA2; Mon, 7 Mar 2022 20:40:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aHcB4mAttlyAfLapKjtaUtOLhfYTcKI6dTg0/WOmFRvh0TPtnNYJ85f9Vrcr0wlQP4sRfxLHukP2ZSE4OOScmEMalWiOAhIxQaofzMwC6lY4OP015TVtzh0xnAlSiXw4bPRM3f09WrM7atEc5bsnqM1WA+JrMjf2E2/HSpl/YRqOddF9/o2TmjJa+PxcaLPi4e8p+YcHJRFALbtSt+JquQ6aB6lpbVyxf7QgsO+V/uPfzZTU1dIPq9ATFt+R3AJCuCKUCpvFevhU1In120ilDEh8kjDXN8uhJ9/Mkz7TT53HktKG+y30dmeJBp5McikF+he2Rsm7QhKdf8JXj3dDYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AaLxSGDTdIod5jBkm0rL2cgPZM4JG2UMZfSouTwN0MU=; b=HiykpBLpbli+Q76sw9t8sO5gfNBzggLmnB46LzmpNELkA9ExhQCswMN22Zud4Sc+kuRvKfajm0q6tBtFWWZctSSUgXzkxuNy54s3/tnzziUnQDyzNnwHAJvw3oHAwR9r/8Wd15lxqWj4e9KhjLIOdJPCk2ZKpdkOgHR6/qIuzIMyDgukEU5x1Gt8K9S745QUwF8LnDL5gNJ2abbcbvcUjO/toBf7rsTTBKYH2wBTfPhH2baxKZcqr7SOF/2g2Jvo2cXlH7B+7p7/AwR1ty2jDCb+tw947E+qCCUVY8FN8qd9SFbUQ6GVAxuAnb/60wlwwUVqGdgArF23w9lxey1THg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AaLxSGDTdIod5jBkm0rL2cgPZM4JG2UMZfSouTwN0MU=; b=31HudVNOfaIwPCIp8XCAIBJkXZou+21qrdafNARVEwPgpbjMWNvAr3X9BxOlq2jf6zaszjbPNnTOhTBGRq68Qj790FKjh8pstbTTFqPB9JAPZaIXzXQRwwPykYErriQzZa5oAY+AABCDNxHbeHhM9XyVVqEcPjqan0n4YOHykz0= Received: from DM6PR06CA0010.namprd06.prod.outlook.com (2603:10b6:5:120::23) by DM6PR12MB4957.namprd12.prod.outlook.com (2603:10b6:5:20d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 04:40:13 +0000 Received: from DM6NAM11FT046.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::9c) by DM6PR06CA0010.outlook.office365.com (2603:10b6:5:120::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT046.mail.protection.outlook.com (10.13.172.121) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:12 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:06 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 2/9] KVM: x86/mmu: Move hugepage adjust to direct_page_fault Date: Tue, 8 Mar 2022 10:08:50 +0530 Message-ID: <20220308043857.13652-3-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 78621c51-c1a8-4ca4-b31a-08da00bdbbcb X-MS-TrafficTypeDiagnostic: DM6PR12MB4957:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AV3viyRPMl8fC3JzrFbrLV+2JyPPACQZzaGi5A7vACOxdlWspxE2weXPvenGJnlnriVw8LH3f2g7IKXJ2chdTQJ1yQla0hhVrnbtoPxDlFlz9zPtQTj3I6NrPQfm7AWYZgf/D88USB5SN1emJbPm+S6pW4acN6HRyi8rqj/gq9BONqz2zeoPiFaPfVD+2ZWH62YCeNfQJMXZUsx7+IVIoDFnp1fxxTistUKsu8+OB5B3jyQVn4IKjbWst4JZ4T9yIhvM1Mv+KCLCMGTz4jROJMTUhbHr4NXb1/xlCHBaT127L+AoBIy0YjvWm1XujbhIQIRx/nlGZm3Lk62qRGX3zlphDIyNcpYIin1kQMZkFOTaxIPIgoT6IvDSfe0Fx9r+E3KtPTdaAOTK7Iq3ZGcDBXCYtKtFExmlw7siw7Vxx3XbrL8MCqz9tCNiRA+VCs1r+MqEJug9LIL302BQH1hfWfCVt1SgPvcvrUFHFtdNHlAHCdqLcyTIcbXz/eaRJsE7SZfgLaypykspSJNhQATpibEev3xXne+MZzo6COFwkDQwD9q3v1mZYiP3BYc/ER1RXErTyJg2K6DxTMfuKYpoptAmtn2rONO6uM/15bSaQClea6EUYqiHwtLxc/OrnkCdje8Dk8AX9BxrrvJOXn8VRNRBjUWBptXl54NPeG4wYLHhLePNbXoWeChPRtdmm8RlKjzZhe6ZWrP3CmpR6uicJw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(6916009)(54906003)(316002)(40460700003)(186003)(70206006)(81166007)(70586007)(356005)(8676002)(4326008)(36756003)(36860700001)(508600001)(26005)(1076003)(83380400001)(8936002)(6666004)(2616005)(7416002)(5660300002)(47076005)(7696005)(426003)(16526019)(336012)(82310400004)(2906002)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:12.4760 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 78621c51-c1a8-4ca4-b31a-08da00bdbbcb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT046.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4957 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Both TDP MMU and legacy MMU do hugepage adjust in the mapping routine. Adjust the pfn early in the common code. This will be used by the following patches for pinning the pages. No functional change intended. Signed-off-by: Nikunj A Dadhania --- arch/x86/kvm/mmu/mmu.c | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 2 -- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 8e24f73bf60b..db1feecd6fed 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2940,8 +2940,6 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) int ret; gfn_t base_gfn = fault->gfn; - kvm_mmu_hugepage_adjust(vcpu, fault); - trace_kvm_mmu_spte_requested(fault); for_each_shadow_entry(vcpu, fault->addr, it) { /* @@ -4035,6 +4033,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault r = RET_PF_RETRY; + kvm_mmu_hugepage_adjust(vcpu, fault); + if (is_tdp_mmu_fault) read_lock(&vcpu->kvm->mmu_lock); else diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index bc9e3553fba2..e03bf59b2f81 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -959,8 +959,6 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) u64 new_spte; int ret; - kvm_mmu_hugepage_adjust(vcpu, fault); - trace_kvm_mmu_spte_requested(fault); rcu_read_lock(); From patchwork Tue Mar 8 04:38:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 342FAC433FE for ; Tue, 8 Mar 2022 04:40:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245261AbiCHEls (ORCPT ); Mon, 7 Mar 2022 23:41:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245429AbiCHElW (ORCPT ); Mon, 7 Mar 2022 23:41:22 -0500 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2050.outbound.protection.outlook.com [40.107.243.50]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0023B3BA53; Mon, 7 Mar 2022 20:40:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=deUshmlwhjHo4s5UluqRc3e/7pRMU4NnvhYM2MkM8g8gEbtJaKJMqhE787ffZprrHdHTB8P7inHFAihmPlnwGHW7O/Qk95rtB3OEqDNbbKQTvSXQT8ObYBzW6n5rB66GhRrZBvy8DguArFymckzMUqpQQRWQrUkvJbgvwACDuUFGRONO/SD1NwChG6O2/LnZQl86oVJW9VEIQ9BIsGm9bnmCI9LXQ9prFGW6yNCVWJE90pDbozivgcihLL2UfkyxQCS/iPNLCehut/H9uqTEsN/7mAfgln0m3Wz2/c5h3Tgs+/hSe8ARu//3/jtSzuV01vXKbT+cKOGDoTMzeAkbcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5qM2tLOnF3ZYpC6ARUyWqAGZBf1ntzZKkRbKsSdrjXY=; b=PZbm43/wwhdISPUgMjLYXSM19LlvZCemWMSDtbiAesOlFuNqMXG+zL2qDPx+o9s9biU6IVBFN4h/wvUB8jCbcIxSwrHHqk2/BoWLMvtHDji4zEeDT7XRd/P6fe3AGKP+sQt5rN118zw9K36hc4EuUXUJernYpYTuNgmLjBTMPO6IYv7NT7+MEsGfWGtWft4BFp+doRBAXlnjMg7+J4odv9j5U9byNmcw6YfyBb2RMvIWrI5QscGX0zBX+AE+kHSeXEma/CkU5iszB7H26S6WTpWm5uYJsTEVcK+HA+5qQSslEStCH/iqve53Wc09eYCdifezS9/qss2cQhh3SGk15g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5qM2tLOnF3ZYpC6ARUyWqAGZBf1ntzZKkRbKsSdrjXY=; b=c+yaMZjg9iuGT7DEQMHgi/e95qhGpdxoGnHY37WUnrM1qfXnSWY8k46eAGamt/pnPrvwjBljV4/Iac9bwB0eU1Oy2+baTOfWwWCRzKuUtXXcgDLq3aOpcPZ3h5ZaXCgnJmunIMMOWaSn2F2ZA1tMQHFZ/wJzWhlhcJ4ipo2MrJM= Received: from DM6PR11CA0038.namprd11.prod.outlook.com (2603:10b6:5:14c::15) by MN2PR12MB4454.namprd12.prod.outlook.com (2603:10b6:208:26c::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 04:40:18 +0000 Received: from DM6NAM11FT024.eop-nam11.prod.protection.outlook.com (2603:10b6:5:14c:cafe::1b) by DM6PR11CA0038.outlook.office365.com (2603:10b6:5:14c::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT024.mail.protection.outlook.com (10.13.172.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:17 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:11 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 3/9] KVM: x86/mmu: Add hook to pin PFNs on demand in MMU Date: Tue, 8 Mar 2022 10:08:51 +0530 Message-ID: <20220308043857.13652-4-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 36666c20-36ea-44dd-2e91-08da00bdbeb8 X-MS-TrafficTypeDiagnostic: MN2PR12MB4454:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8ZlkOeum9HXdoP8nmeLiupAmhnaiwhvNCFsO+on/c/CsBHqx2ujJKhiLxIuYaj6MDT8X506bSUlwa7K70mRM96+wrIvei3RQERyKlNuCOJvFpXVW1nPjIOGHx5WYTVRQF3C1Pn0p+TkjwH22//DN+5N2Ojw5vMgaumu6N6egCyI4YyF/tilAJC/tsMdftr/KRG+dm9HjpQwL4R87jm4ckyR4DhSmLGC82BZQsrBXTM7WUyRUJCt27jM4Sd1S4qKb956hwzK2kqw8B09Ky/Z3PT9WQTDPUHrkvo9u5T8p9euxLCa0zq7qzsUZJQsKgZ4VMCGROH+J3XmH29d4cXqP1xgR1ti2l4jv6l4v6bm1oEZKFqiZfchAb4liNQjhpSQH1iSDT2o4o9AKNnvMvXYJ+on2VpHSQRwuX68GvzyU44wnbs9y5+n645hXWy6woVR/LoX1e3CejHB8kN1eWmoKqXQkb/hlhiT6LZ9+PTUF0W/QGA0g1wgEB7w0jPS053hq59Pr9shGlzmmkB/0mGb6rW1wolSLirNEeFM61rrHliM7rT+Am7tCmYE7FKRthn3CHUaO7xy0VQNXKGsNab6pcCB2d6J/wPaNKdl6spDnDyWHhHYlHU/oIl5mHYw4bFTe7m/nja+zRlEx2B4VvsGFIJ6H/D7/UNHZBJCZPt92ut9RXsJg36Icg30D3gZH3wa5WA0wtrTQD5Pg0tueCCiPYQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(5660300002)(7416002)(7696005)(8936002)(508600001)(6666004)(8676002)(2906002)(1076003)(186003)(26005)(16526019)(40460700003)(4326008)(82310400004)(70206006)(70586007)(81166007)(47076005)(2616005)(356005)(36860700001)(316002)(426003)(336012)(54906003)(36756003)(6916009)(83380400001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:17.3813 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 36666c20-36ea-44dd-2e91-08da00bdbeb8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT024.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4454 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use vendor code via kvm_x86_ops hooks for pinning. Signed-off-by: Nikunj A Dadhania --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 15 +++++++++++++++ 3 files changed, 19 insertions(+) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index d39e0de06be2..8efb43d92eef 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -88,6 +88,7 @@ KVM_X86_OP(set_tss_addr) KVM_X86_OP(set_identity_map_addr) KVM_X86_OP(get_mt_mask) KVM_X86_OP(load_mmu_pgd) +KVM_X86_OP(pin_pfn) KVM_X86_OP_NULL(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index ec9830d2aabf..df11f1fb76de 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1418,6 +1418,9 @@ struct kvm_x86_ops { void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level); + bool (*pin_pfn)(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, + kvm_pfn_t pfn, hva_t hva, bool write, + enum pg_level level); bool (*has_wbinvd_exit)(void); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index db1feecd6fed..b94e5e71653e 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4001,6 +4001,16 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu, mmu_notifier_retry_hva(vcpu->kvm, mmu_seq, fault->hva); } +static bool kvm_pin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) +{ + if (is_error_noslot_pfn(fault->pfn) || kvm_is_reserved_pfn(fault->pfn) || + !kvm_x86_ops.pin_pfn) + return true; + + return kvm_x86_ops.pin_pfn(vcpu, fault->slot, fault->pfn, fault->hva, + fault->write, fault->goal_level); +} + static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { bool is_tdp_mmu_fault = is_tdp_mmu(vcpu->arch.mmu); @@ -4035,6 +4045,9 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault kvm_mmu_hugepage_adjust(vcpu, fault); + if (memslot_is_encrypted(fault->slot) && !kvm_pin_pfn(vcpu, fault)) + goto out_release; + if (is_tdp_mmu_fault) read_lock(&vcpu->kvm->mmu_lock); else @@ -4057,6 +4070,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault read_unlock(&vcpu->kvm->mmu_lock); else write_unlock(&vcpu->kvm->mmu_lock); + +out_release: kvm_release_pfn_clean(fault->pfn); return r; } From patchwork Tue Mar 8 04:38:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772950 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEA09C433F5 for ; Tue, 8 Mar 2022 04:40:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245490AbiCHElc (ORCPT ); Mon, 7 Mar 2022 23:41:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245415AbiCHElZ (ORCPT ); Mon, 7 Mar 2022 23:41:25 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2062.outbound.protection.outlook.com [40.107.220.62]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3016D3BBCE; Mon, 7 Mar 2022 20:40:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CXhIZflxipEgfh6UDXqcxFqgsLc3305bKE5nwh1lfbu5L5NgNXwAMqWC9RvcQSRvvVi/p3aNx5d69xqttfr58+qXDjcTcHldui2p64In4+OwWZ0+q7DlnWuJR2coIms50iusPq8dcGLJT2/Ka3t29CV8EnwU/GS0i3v/uYlR5GrJ8gQAH9EgOiSCwbm0kqzgUqU3VzHs7UFuKkPPDDLxZbYFJ1OJsxN7Q+WibahPlPPJnPAdBRgfCUeTYNOChcwNVigV9wPDl4IvvtoChEfYnpbpljHlifL4clSQXsbB05UefazZG9GvUCwC5OvwPnmlWUVSbU8eIxNfSFiVswRhpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lYtVtx3Zm2oBUMFae3DZxhD+8teRR1GNGqNCvIbmrFE=; b=ErjQfdBHTJDFlAkUhacj2AcXHdYXWVj8bJSD43K9YL9YcMShEH7jADnqHEXfpL3vhyy4aaYSDgjE/gvUfeFkPZRtAFnQQX3EaXM4K/cw7vx9H27bBYOlZXUWeiaWm2Q6ONYN7xV4ieQYGFESKYD4dy3EhAah1AWzqFFpfOrZHRtKwOHozZAhMR3kk5lbaS29Uqn+Yori1SkAKgDibLP/eso+PZ70/Jiymfy9BAM19cCDwpSUws6LDYHgeU5uXJQHk4BMsgikf5TfQhIzfS9geD5ozn3fAftkIxUlTI9CEvKaSIZRunTZx1fAG+Xb7Sy0AZlKrhogDZGTDgoSeepNeQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lYtVtx3Zm2oBUMFae3DZxhD+8teRR1GNGqNCvIbmrFE=; b=flZmzu5nu1a6gRu2hSG0iwufEsOtjpswaaaPA9CctpTXzyDOgQB3gNV4yFpCw9pqRty2sF9L6ej+fy3ZX6Az9agk2BK5Q9hmrBe/C/VLoBsCXWzf/trXmLd9Szx/IHoWG0+TQB9u8Ne6oy7QMFByJhYHl4eP09W4DdmEx4+IWFo= Received: from DM6PR02CA0062.namprd02.prod.outlook.com (2603:10b6:5:177::39) by CH0PR12MB5331.namprd12.prod.outlook.com (2603:10b6:610:d6::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.16; Tue, 8 Mar 2022 04:40:23 +0000 Received: from DM6NAM11FT033.eop-nam11.prod.protection.outlook.com (2603:10b6:5:177:cafe::91) by DM6PR02CA0062.outlook.office365.com (2603:10b6:5:177::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.13 via Frontend Transport; Tue, 8 Mar 2022 04:40:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT033.mail.protection.outlook.com (10.13.172.221) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:22 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:16 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 4/9] KVM: SVM: Add pinning metadata in the arch memslot Date: Tue, 8 Mar 2022 10:08:52 +0530 Message-ID: <20220308043857.13652-5-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae454688-23f7-4347-f81a-08da00bdc1c7 X-MS-TrafficTypeDiagnostic: CH0PR12MB5331:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: U5MfQZjYB4K7xhWBQxyCPwRp6WFpCA/v3HuFlaQ/+VCxwyE8102QT2LfBzq+7uKJtOE1zdXB0s409kACJrqnIW0OxloFexgZltR77NrOSm/e9w0HSZzOUmNkqNuAPILwi7oHFebDgrMdNhFlVHBrDndZ4MEwlfrEfdHkfgV/r6KeesLkVK6MAWlYQQ4aAmY+kCzdJnkmMbczNdpHZtW7L8x/TErLgUGBkobL6mgsVfG9nE9NxhrdnAk10RtxUCQvzpx1XlznND0hPqgQiN5wYZzRXRPfiQbKuQWd76XjmfYdCS4ykxp2JNnAvZCOutBG19oYMmlX+OSL+/Xs+y4v8AbHezlaREsxXGNReYXnQtnXAKyOC0GGMsvzhignBSXAyutzYSgydqXUmc8Da0g2Gpw3aGRoIQ2bQg/mhWdqd25ZIVxX0V0XqZVmK5AYUr8zp77OTJTCvce5Tews5k93FiwIuQfkcPZbCEWQJLslAwFZK+9HnSDNEYCdH8SFt6OazXrGuDTwBkDGK2cC5MqdH+C1xRMczIeFJvikUQuO+fLYinT1mXFmU+XB8ba1YYP0c4WLIZwL9ukzWvczj/cY/gaNKmg4pPPruosUyH+AqnLQ6YKv06GdORGm2NKv3DM0MQpBdKvvZacujYZtV8Cs3vPoVkWJ09nb2iGOsYhBCL2+aThfENCz8xHIt+2j0PHJ60CNvOG7xcJjkmb6AYXZYQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(26005)(36860700001)(2906002)(47076005)(4326008)(36756003)(6666004)(316002)(6916009)(508600001)(7696005)(356005)(1076003)(186003)(2616005)(426003)(81166007)(336012)(16526019)(7416002)(82310400004)(8676002)(70206006)(70586007)(8936002)(83380400001)(40460700003)(5660300002)(54906003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:22.5335 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ae454688-23f7-4347-f81a-08da00bdc1c7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT033.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5331 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org AMD SEV guest requires the guest's pages to be pinned in host physical memory. The memory encryption scheme uses the physical address of the memory being encrypted. If guest pages are moved, content decrypted would be incorrect, corrupting guest's memory. For SEV/SEV-ES guests, the hypervisor doesn't know which pages are encrypted and when the guest is done using those pages. Hypervisor should treat all the guest pages as encrypted until they are deallocated or the guest is destroyed. The KVM MMU needs to track the pages that are pinned and the corresponding pfns for unpinning them during the guest destroy path and deallocation path. Signed-off-by: Nikunj A Dadhania --- arch/x86/include/asm/kvm-x86-ops.h | 2 ++ arch/x86/include/asm/kvm_host.h | 7 +++++ arch/x86/kvm/svm/sev.c | 49 ++++++++++++++++++++++++++++++ arch/x86/kvm/svm/svm.c | 3 ++ arch/x86/kvm/svm/svm.h | 6 ++++ arch/x86/kvm/x86.c | 11 ++++++- 6 files changed, 77 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 8efb43d92eef..61ff8a636db6 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -89,6 +89,8 @@ KVM_X86_OP(set_identity_map_addr) KVM_X86_OP(get_mt_mask) KVM_X86_OP(load_mmu_pgd) KVM_X86_OP(pin_pfn) +KVM_X86_OP(alloc_memslot_metadata) +KVM_X86_OP(free_memslot) KVM_X86_OP_NULL(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index df11f1fb76de..eeb2c799b59f 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -926,6 +926,8 @@ struct kvm_arch_memory_slot { struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES]; struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1]; unsigned short *gfn_track[KVM_PAGE_TRACK_MAX]; + unsigned long *pinned_bitmap; + kvm_pfn_t *pfns; }; /* @@ -1421,6 +1423,11 @@ struct kvm_x86_ops { bool (*pin_pfn)(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, kvm_pfn_t pfn, hva_t hva, bool write, enum pg_level level); + int (*alloc_memslot_metadata)(struct kvm *kvm, + const struct kvm_memory_slot *old, + struct kvm_memory_slot *new); + void (*free_memslot)(struct kvm *kvm, + struct kvm_memory_slot *slot); bool (*has_wbinvd_exit)(void); diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 17b53457d866..bd7572517c99 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -2950,3 +2950,52 @@ void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) ghcb_set_sw_exit_info_2(svm->sev_es.ghcb, 1); } + +void sev_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) +{ + struct kvm_arch_memory_slot *aslot = &slot->arch; + + if (!sev_guest(kvm)) + return; + + if (aslot->pinned_bitmap) { + kvfree(aslot->pinned_bitmap); + aslot->pinned_bitmap = NULL; + } + + if (aslot->pfns) { + kvfree(aslot->pfns); + aslot->pfns = NULL; + } +} + +int sev_alloc_memslot_metadata(struct kvm *kvm, + const struct kvm_memory_slot *old, + struct kvm_memory_slot *new) +{ + struct kvm_arch_memory_slot *aslot = &new->arch; + unsigned long pinned_bytes = new->npages * sizeof(kvm_pfn_t); + + if (!sev_guest(kvm)) + return 0; + + if (old && old->arch.pinned_bitmap && old->arch.pfns) { + WARN_ON(old->npages != new->npages); + aslot->pinned_bitmap = old->arch.pinned_bitmap; + aslot->pfns = old->arch.pfns; + return 0; + } + + aslot->pfns = kvcalloc(new->npages, sizeof(*aslot->pfns), + GFP_KERNEL_ACCOUNT); + if (!aslot->pfns) + return -ENOMEM; + + aslot->pinned_bitmap = kvzalloc(pinned_bytes, GFP_KERNEL_ACCOUNT); + if (!aslot->pinned_bitmap) { + kvfree(aslot->pfns); + aslot->pfns = NULL; + return -ENOMEM; + } + return 0; +} diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index fd3a00c892c7..ec06421cb532 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4658,6 +4658,9 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .complete_emulated_msr = svm_complete_emulated_msr, .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, + + .alloc_memslot_metadata = sev_alloc_memslot_metadata, + .free_memslot = sev_free_memslot, }; /* diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index fa98d6844728..f00364020d7e 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -616,4 +616,10 @@ void sev_es_unmap_ghcb(struct vcpu_svm *svm); void __svm_sev_es_vcpu_run(unsigned long vmcb_pa); void __svm_vcpu_run(unsigned long vmcb_pa, unsigned long *regs); +int sev_alloc_memslot_metadata(struct kvm *kvm, + const struct kvm_memory_slot *old, + struct kvm_memory_slot *new); +void sev_free_memslot(struct kvm *kvm, + struct kvm_memory_slot *slot); + #endif diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 82a9dcd8c67f..95070aaa1636 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11796,6 +11796,7 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) } kvm_page_track_free_memslot(slot); + static_call_cond(kvm_x86_free_memslot)(kvm, slot); } int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages) @@ -11821,6 +11822,7 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages) } static int kvm_alloc_memslot_metadata(struct kvm *kvm, + const struct kvm_memory_slot *old, struct kvm_memory_slot *slot) { unsigned long npages = slot->npages; @@ -11873,8 +11875,15 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm, if (kvm_page_track_create_memslot(kvm, slot, npages)) goto out_free; + if (kvm_x86_ops.alloc_memslot_metadata && + static_call(kvm_x86_alloc_memslot_metadata)(kvm, old, slot)) + goto out_free_page_track; + return 0; +out_free_page_track: + kvm_page_track_free_memslot(slot); + out_free: memslot_rmap_free(slot); @@ -11907,7 +11916,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, enum kvm_mr_change change) { if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) - return kvm_alloc_memslot_metadata(kvm, new); + return kvm_alloc_memslot_metadata(kvm, old, new); if (change == KVM_MR_FLAGS_ONLY) memcpy(&new->arch, &old->arch, sizeof(old->arch)); From patchwork Tue Mar 8 04:38:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9D32C433F5 for ; Tue, 8 Mar 2022 04:40:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245130AbiCHEli (ORCPT ); Mon, 7 Mar 2022 23:41:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245315AbiCHEld (ORCPT ); Mon, 7 Mar 2022 23:41:33 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2080.outbound.protection.outlook.com [40.107.223.80]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FB3431504; Mon, 7 Mar 2022 20:40:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GhAghTPFnC0ICH+dbtsNb+4EHJmDX+K8ri2i3ndcR6y2BQGgUS3vpsSB8ViaAAfGVEO3o7g2sgZSbI5YgiUcgSLOnlFKrIipadN1xmVw3vsYNf8ESArd/IUgYgM6BH7ZO1krnrDJG1Y21Jlt1Q5Kg6ehmaD5ZnSNRdQgfX44vUZpq1T6+3XF60EJNJjnAnSH7DVxxatbUB6uGYgba3vLDXZiEpSLPr4qpHaV/t0BxRrDXWAVbIZuZc4gc3lmy9pC7e9p+fTKaIpoWCBh6O56Svaduh9uWAv3psMkpjEjwCtOiRisWFoRPfT+RIbkaA7/bGA25FNShVzCl/6DHrkLxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O737pXNU3JIqLDkkaKttX9X3S4g+ai2wSj7UWpHiHQY=; b=gdZYoziTxA9ulkAPBA0DhNK3SpUOX3/g36+pLu/HXQRr/C7pTYOFqxErKWH2v9TXi/vl80qR1Hz4zdcb0raB1YqdDj239TVzatSXPCnHwl3szObHPJ8GBB/qVU48Zsj8Rpw4PXUmi5T31wnEATDVBbhB951knLrer86aunlhTf/KlY8KPV5fxwjyd6GphFCN9EcZ4R6r3bexxEXKsbdup0revSoB954GNOZUNt4xabx8GWl48kjyM29Ku74KGqZ1dbutudKQliYilIKQkAUaRbq7Tb4KJMGF2ae+BLgzmJknAniG2Iqsh293oBKuiwY4LoHfFGOc7wA/a2COvIAh+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O737pXNU3JIqLDkkaKttX9X3S4g+ai2wSj7UWpHiHQY=; b=JoUN7nNvlC3Q90hQwQSx2Z24nKZm5FVpELIr/mHFK6cK0u+mBjrW+Gkb1a3GRQLKWHyINFmPvhAs2y3TBfFqM8Y1SsGFaTIa3rV4Gd76rZTjzEi0CuN0+EqCOsiE1V3q/2KHIsuR/ZVu9mZAG4qjHdNdjLNftMIp/sYKolugAvg= Received: from DM5PR2201CA0013.namprd22.prod.outlook.com (2603:10b6:4:14::23) by BYAPR12MB3013.namprd12.prod.outlook.com (2603:10b6:a03:a9::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.13; Tue, 8 Mar 2022 04:40:28 +0000 Received: from DM6NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:4:14:cafe::8) by DM5PR2201CA0013.outlook.office365.com (2603:10b6:4:14::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT049.mail.protection.outlook.com (10.13.172.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:27 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:22 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 5/9] KVM: SVM: Implement demand page pinning Date: Tue, 8 Mar 2022 10:08:53 +0530 Message-ID: <20220308043857.13652-6-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6414c5cd-fb29-4eb3-7bdf-08da00bdc4d6 X-MS-TrafficTypeDiagnostic: BYAPR12MB3013:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mmqvOumGxcAaNjIZbRw3uBTAbVqzK2OTT4xqC5qJbRie70F0lAPw+DpOAJjaKMNSw4R7Z+r1kuoGJTSpqEWCH4levoVLFhQ0TkHTyP2jeY82MSMy6H4fVkXXwX1Z26RugPYBWsEDU6BRi2KAXOFW5WrOBgjIWz921+0/pY6EmNk25dJpezzpvWmUoAhXPZsXi6XzfiAnAeO8AbBESXtAyvwlw45RRGbtwQoOr1Iro6xXoq9PHne492PwfUxXKybpRTFVl7DpxDLRVn5YwXhwfJO17yTZjbt7v7QT0juCfr65toTpPFEd0b22UcDr1bgce9+2+TyaisJLBcpQ9tLgaE0EPHWEVNgD4Vw2JvCnKLcFKufQ1HexKru2vx9MQgMM40HUPZCO/xGpQG48pmOT/btpZF9q2JW1EWJaZHVLUGgB9evp9caj2oAlioAFZ3DxXU+N0QBRkKsjWUTAXCLWkdNXtJQXVqAH0BeyR2HiyNZ/nJ/10w2gmCIwlD7ZasAa/bBOHkBSQfTdCSW5Xvs0vomSxpO7aYb7Htb6UE0T3s+XhuFsiqqbOzI5c4OkCZEhzV3VHGPmacPy4HC4efPgncspYYQLr8CkdCB7VsxqOOEPv/2QCovBMYwqvmvYRBPurHpzOPH5B/ww57g6a0SxIlHBaUB2jFoPLkJE5ZfprXXNHKACklbQSsxWTFj7IhBYlwhoi/szMroKVjGDRoyDoQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(47076005)(36860700001)(7696005)(36756003)(2616005)(2906002)(83380400001)(40460700003)(7416002)(5660300002)(316002)(54906003)(82310400004)(26005)(8936002)(81166007)(70206006)(30864003)(336012)(426003)(356005)(70586007)(186003)(1076003)(6916009)(8676002)(16526019)(6666004)(4326008)(508600001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:27.6462 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6414c5cd-fb29-4eb3-7bdf-08da00bdc4d6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3013 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use the memslot metadata to store the pinned data along with the pfns. This improves the SEV guest startup time from O(n) to a constant by deferring guest page pinning until the pages are used to satisfy nested page faults. The page reference will be dropped in the memslot free path or deallocation path. Reuse enc_region structure definition as pinned_region to maintain pages that are pinned outside of MMU demand pinning. Remove rest of the code which did upfront pinning, as they are no longer needed in view of the demand pinning support. Retain svm_register_enc_region() and svm_unregister_enc_region() with required checks for resource limit. Guest boot time comparison +---------------+----------------+-------------------+ | Guest Memory | baseline | Demand Pinning | | Size (GB) | (secs) | (secs) | +---------------+----------------+-------------------+ | 4 | 6.16 | 5.71 | +---------------+----------------+-------------------+ | 16 | 7.38 | 5.91 | +---------------+----------------+-------------------+ | 64 | 12.17 | 6.16 | +---------------+----------------+-------------------+ | 128 | 18.20 | 6.50 | +---------------+----------------+-------------------+ | 192 | 24.56 | 6.80 | +---------------+----------------+-------------------+ Signed-off-by: Nikunj A Dadhania --- arch/x86/kvm/svm/sev.c | 304 ++++++++++++++++++++++++++--------------- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/svm/svm.h | 6 +- 3 files changed, 200 insertions(+), 111 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index bd7572517c99..d0514975555d 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -66,7 +66,7 @@ static unsigned int nr_asids; static unsigned long *sev_asid_bitmap; static unsigned long *sev_reclaim_asid_bitmap; -struct enc_region { +struct pinned_region { struct list_head list; unsigned long npages; struct page **pages; @@ -257,7 +257,7 @@ static int sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp) if (ret) goto e_free; - INIT_LIST_HEAD(&sev->regions_list); + INIT_LIST_HEAD(&sev->pinned_regions_list); return 0; @@ -378,16 +378,34 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp) return ret; } +static bool rlimit_memlock_exceeds(unsigned long locked, unsigned long npages) +{ + unsigned long lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; + unsigned long lock_req; + + lock_req = locked + npages; + return (lock_req > lock_limit) && !capable(CAP_IPC_LOCK); +} + +static unsigned long get_npages(unsigned long uaddr, unsigned long ulen) +{ + unsigned long first, last; + + /* Calculate number of pages. */ + first = (uaddr & PAGE_MASK) >> PAGE_SHIFT; + last = ((uaddr + ulen - 1) & PAGE_MASK) >> PAGE_SHIFT; + return last - first + 1; +} + static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, unsigned long ulen, unsigned long *n, int write) { struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; + struct pinned_region *region; unsigned long npages, size; int npinned; - unsigned long locked, lock_limit; struct page **pages; - unsigned long first, last; int ret; lockdep_assert_held(&kvm->lock); @@ -395,15 +413,12 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, if (ulen == 0 || uaddr + ulen < uaddr) return ERR_PTR(-EINVAL); - /* Calculate number of pages. */ - first = (uaddr & PAGE_MASK) >> PAGE_SHIFT; - last = ((uaddr + ulen - 1) & PAGE_MASK) >> PAGE_SHIFT; - npages = (last - first + 1); + npages = get_npages(uaddr, ulen); - locked = sev->pages_locked + npages; - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - if (locked > lock_limit && !capable(CAP_IPC_LOCK)) { - pr_err("SEV: %lu locked pages exceed the lock limit of %lu.\n", locked, lock_limit); + if (rlimit_memlock_exceeds(sev->pages_to_lock, npages)) { + pr_err("SEV: %lu locked pages exceed the lock limit of %lu.\n", + sev->pages_to_lock + npages, + (rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT)); return ERR_PTR(-ENOMEM); } @@ -429,7 +444,19 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, } *n = npages; - sev->pages_locked = locked; + sev->pages_to_lock += npages; + + /* Maintain region list that is pinned to be unpinned in vm destroy path */ + region = kzalloc(sizeof(*region), GFP_KERNEL_ACCOUNT); + if (!region) { + ret = -ENOMEM; + goto err; + } + region->uaddr = uaddr; + region->size = ulen; + region->pages = pages; + region->npages = npages; + list_add_tail(®ion->list, &sev->pinned_regions_list); return pages; @@ -441,14 +468,43 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, return ERR_PTR(ret); } -static void sev_unpin_memory(struct kvm *kvm, struct page **pages, - unsigned long npages) +static void __sev_unpin_memory(struct kvm *kvm, struct page **pages, + unsigned long npages) { struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; unpin_user_pages(pages, npages); kvfree(pages); - sev->pages_locked -= npages; + sev->pages_to_lock -= npages; +} + +static struct pinned_region *find_pinned_region(struct kvm *kvm, + struct page **pages, + unsigned long n) +{ + struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; + struct list_head *head = &sev->pinned_regions_list; + struct pinned_region *i; + + list_for_each_entry(i, head, list) { + if (i->pages == pages && i->npages == n) + return i; + } + + return NULL; +} + +static void sev_unpin_memory(struct kvm *kvm, struct page **pages, + unsigned long npages) +{ + struct pinned_region *region; + + region = find_pinned_region(kvm, pages, npages); + __sev_unpin_memory(kvm, pages, npages); + if (region) { + list_del(®ion->list); + kfree(region); + } } static void sev_clflush_pages(struct page *pages[], unsigned long npages) @@ -551,8 +607,9 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) set_page_dirty_lock(inpages[i]); mark_page_accessed(inpages[i]); } - /* unlock the user pages */ - sev_unpin_memory(kvm, inpages, npages); + /* unlock the user pages on error */ + if (ret) + sev_unpin_memory(kvm, inpages, npages); return ret; } @@ -1059,7 +1116,8 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) set_page_dirty_lock(pages[i]); mark_page_accessed(pages[i]); } - sev_unpin_memory(kvm, pages, n); + if (ret) + sev_unpin_memory(kvm, pages, n); return ret; } @@ -1338,7 +1396,8 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) e_free_hdr: kfree(hdr); e_unpin: - sev_unpin_memory(kvm, guest_page, n); + if (ret) + sev_unpin_memory(kvm, guest_page, n); return ret; } @@ -1508,7 +1567,8 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) ret = sev_issue_cmd(kvm, SEV_CMD_RECEIVE_UPDATE_DATA, &data, &argp->error); - sev_unpin_memory(kvm, guest_page, n); + if (ret) + sev_unpin_memory(kvm, guest_page, n); e_free_trans: kfree(trans); @@ -1629,16 +1689,17 @@ static void sev_migrate_from(struct kvm_sev_info *dst, dst->active = true; dst->asid = src->asid; dst->handle = src->handle; - dst->pages_locked = src->pages_locked; + dst->pages_to_lock = src->pages_to_lock; dst->enc_context_owner = src->enc_context_owner; src->asid = 0; src->active = false; src->handle = 0; - src->pages_locked = 0; + src->pages_to_lock = 0; src->enc_context_owner = NULL; - list_cut_before(&dst->regions_list, &src->regions_list, &src->regions_list); + list_cut_before(&dst->pinned_regions_list, &src->pinned_regions_list, + &src->pinned_regions_list); } static int sev_es_migrate_from(struct kvm *dst, struct kvm *src) @@ -1862,8 +1923,7 @@ int svm_register_enc_region(struct kvm *kvm, struct kvm_enc_region *range) { struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; - struct enc_region *region; - int ret = 0; + unsigned long npages; if (!sev_guest(kvm)) return -ENOTTY; @@ -1875,101 +1935,35 @@ int svm_register_enc_region(struct kvm *kvm, if (range->addr > ULONG_MAX || range->size > ULONG_MAX) return -EINVAL; - region = kzalloc(sizeof(*region), GFP_KERNEL_ACCOUNT); - if (!region) + npages = get_npages(range->addr, range->size); + if (rlimit_memlock_exceeds(sev->pages_to_lock, npages)) { + pr_err("SEV: %lu locked pages exceed the lock limit of %lu.\n", + sev->pages_to_lock + npages, + (rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT)); return -ENOMEM; - - mutex_lock(&kvm->lock); - region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages, 1); - if (IS_ERR(region->pages)) { - ret = PTR_ERR(region->pages); - mutex_unlock(&kvm->lock); - goto e_free; } + sev->pages_to_lock += npages; - region->uaddr = range->addr; - region->size = range->size; - - list_add_tail(®ion->list, &sev->regions_list); - mutex_unlock(&kvm->lock); - - /* - * The guest may change the memory encryption attribute from C=0 -> C=1 - * or vice versa for this memory range. Lets make sure caches are - * flushed to ensure that guest data gets written into memory with - * correct C-bit. - */ - sev_clflush_pages(region->pages, region->npages); - - return ret; - -e_free: - kfree(region); - return ret; -} - -static struct enc_region * -find_enc_region(struct kvm *kvm, struct kvm_enc_region *range) -{ - struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; - struct list_head *head = &sev->regions_list; - struct enc_region *i; - - list_for_each_entry(i, head, list) { - if (i->uaddr == range->addr && - i->size == range->size) - return i; - } - - return NULL; -} - -static void __unregister_enc_region_locked(struct kvm *kvm, - struct enc_region *region) -{ - sev_unpin_memory(kvm, region->pages, region->npages); - list_del(®ion->list); - kfree(region); + return 0; } int svm_unregister_enc_region(struct kvm *kvm, struct kvm_enc_region *range) { - struct enc_region *region; - int ret; + struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; + unsigned long npages; /* If kvm is mirroring encryption context it isn't responsible for it */ if (is_mirroring_enc_context(kvm)) return -EINVAL; - mutex_lock(&kvm->lock); - - if (!sev_guest(kvm)) { - ret = -ENOTTY; - goto failed; - } - - region = find_enc_region(kvm, range); - if (!region) { - ret = -EINVAL; - goto failed; - } - - /* - * Ensure that all guest tagged cache entries are flushed before - * releasing the pages back to the system for use. CLFLUSH will - * not do this, so issue a WBINVD. - */ - wbinvd_on_all_cpus(); + if (!sev_guest(kvm)) + return -ENOTTY; - __unregister_enc_region_locked(kvm, region); + npages = get_npages(range->addr, range->size); + sev->pages_to_lock -= npages; - mutex_unlock(&kvm->lock); return 0; - -failed: - mutex_unlock(&kvm->lock); - return ret; } int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd) @@ -2018,7 +2012,7 @@ int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd) mirror_sev->fd = source_sev->fd; mirror_sev->es_active = source_sev->es_active; mirror_sev->handle = source_sev->handle; - INIT_LIST_HEAD(&mirror_sev->regions_list); + INIT_LIST_HEAD(&mirror_sev->pinned_regions_list); ret = 0; /* @@ -2038,8 +2032,9 @@ int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd) void sev_vm_destroy(struct kvm *kvm) { struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; - struct list_head *head = &sev->regions_list; + struct list_head *head = &sev->pinned_regions_list; struct list_head *pos, *q; + struct pinned_region *region; WARN_ON(sev->num_mirrored_vms); @@ -2072,8 +2067,14 @@ void sev_vm_destroy(struct kvm *kvm) */ if (!list_empty(head)) { list_for_each_safe(pos, q, head) { - __unregister_enc_region_locked(kvm, - list_entry(pos, struct enc_region, list)); + /* + * Unpin the memory that were pinned outside of MMU + * demand pinning + */ + region = list_entry(pos, struct pinned_region, list); + __sev_unpin_memory(kvm, region->pages, region->npages); + list_del(®ion->list); + kfree(region); cond_resched(); } } @@ -2951,13 +2952,96 @@ void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) ghcb_set_sw_exit_info_2(svm->sev_es.ghcb, 1); } +bool sev_pin_pfn(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, + kvm_pfn_t pfn, hva_t hva, bool write, enum pg_level level) +{ + unsigned int npages = KVM_PAGES_PER_HPAGE(level); + unsigned int flags = FOLL_LONGTERM, npinned; + struct kvm_arch_memory_slot *aslot; + struct kvm *kvm = vcpu->kvm; + gfn_t gfn_start, rel_gfn; + struct page *page[1]; + kvm_pfn_t old_pfn; + + if (!sev_guest(kvm)) + return true; + + if (WARN_ON_ONCE(!memslot->arch.pfns)) + return false; + + if (KVM_BUG_ON(level > PG_LEVEL_1G, kvm)) + return false; + + hva = ALIGN_DOWN(hva, npages << PAGE_SHIFT); + flags |= write ? FOLL_WRITE : 0; + + mutex_lock(&kvm->slots_arch_lock); + gfn_start = hva_to_gfn_memslot(hva, memslot); + rel_gfn = gfn_start - memslot->base_gfn; + aslot = &memslot->arch; + if (test_bit(rel_gfn, aslot->pinned_bitmap)) { + old_pfn = aslot->pfns[rel_gfn]; + if (old_pfn == pfn) + goto out; + + /* Flush the cache before releasing the page to the system */ + sev_flush_guest_memory(to_svm(vcpu), __va(old_pfn), + npages * PAGE_SIZE); + unpin_user_page(pfn_to_page(old_pfn)); + } + /* Pin the page, KVM doesn't yet support page migration. */ + npinned = pin_user_pages_fast(hva, 1, flags, page); + KVM_BUG(npinned != 1, kvm, "SEV: Pinning failed\n"); + KVM_BUG(pfn != page_to_pfn(page[0]), kvm, "SEV: pfn mismatch\n"); + + if (!this_cpu_has(X86_FEATURE_SME_COHERENT)) + clflush_cache_range(__va(pfn << PAGE_SHIFT), npages * PAGE_SIZE); + + WARN_ON(rel_gfn >= memslot->npages); + aslot->pfns[rel_gfn] = pfn; + set_bit(rel_gfn, aslot->pinned_bitmap); + +out: + mutex_unlock(&kvm->slots_arch_lock); + return true; +} + void sev_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) { struct kvm_arch_memory_slot *aslot = &slot->arch; + kvm_pfn_t *pfns; + gfn_t gfn; + int i; if (!sev_guest(kvm)) return; + if (!aslot->pinned_bitmap || !slot->arch.pfns) + goto out; + + pfns = aslot->pfns; + + /* + * Ensure that all guest tagged cache entries are flushed before + * releasing the pages back to the system for use. CLFLUSH will + * not do this, so issue a WBINVD. + */ + wbinvd_on_all_cpus(); + + /* + * Iterate the memslot to find the pinned pfn using the bitmap and drop + * the pfn stored. + */ + for (i = 0, gfn = slot->base_gfn; i < slot->npages; i++, gfn++) { + if (test_and_clear_bit(i, aslot->pinned_bitmap)) { + if (WARN_ON(!pfns[i])) + continue; + + unpin_user_page(pfn_to_page(pfns[i])); + } + } + +out: if (aslot->pinned_bitmap) { kvfree(aslot->pinned_bitmap); aslot->pinned_bitmap = NULL; @@ -2992,6 +3076,8 @@ int sev_alloc_memslot_metadata(struct kvm *kvm, return -ENOMEM; aslot->pinned_bitmap = kvzalloc(pinned_bytes, GFP_KERNEL_ACCOUNT); + new->flags |= KVM_MEMSLOT_ENCRYPTED; + if (!aslot->pinned_bitmap) { kvfree(aslot->pfns); aslot->pfns = NULL; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index ec06421cb532..463a90ed6f83 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4661,6 +4661,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .alloc_memslot_metadata = sev_alloc_memslot_metadata, .free_memslot = sev_free_memslot, + .pin_pfn = sev_pin_pfn, }; /* diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index f00364020d7e..2f38e793ead0 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -75,8 +75,8 @@ struct kvm_sev_info { unsigned int asid; /* ASID used for this guest */ unsigned int handle; /* SEV firmware handle */ int fd; /* SEV device fd */ - unsigned long pages_locked; /* Number of pages locked */ - struct list_head regions_list; /* List of registered regions */ + unsigned long pages_to_lock; /* Number of page that can be locked */ + struct list_head pinned_regions_list; /* List of pinned regions */ u64 ap_jump_table; /* SEV-ES AP Jump Table address */ struct kvm *enc_context_owner; /* Owner of copied encryption context */ unsigned long num_mirrored_vms; /* Number of VMs sharing this ASID */ @@ -621,5 +621,7 @@ int sev_alloc_memslot_metadata(struct kvm *kvm, struct kvm_memory_slot *new); void sev_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot); +bool sev_pin_pfn(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, + kvm_pfn_t pfn, hva_t hva, bool write, enum pg_level level); #endif From patchwork Tue Mar 8 04:38:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB9A5C433F5 for ; Tue, 8 Mar 2022 04:40:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232005AbiCHElw (ORCPT ); Mon, 7 Mar 2022 23:41:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343572AbiCHElt (ORCPT ); Mon, 7 Mar 2022 23:41:49 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2073.outbound.protection.outlook.com [40.107.220.73]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 789613BBD0; Mon, 7 Mar 2022 20:40:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Uz9jUe/tkRdm7PjFyeUAPVPgka8dI+APL41DbBbEKl0i+GLd/ypOmnpG++ZQLrzkw04us6ab7GRbcyJIQUSdXrVyISrFJ0c0p2lGh1KB3eeognW8jay8NuJvldAPctwExv+NrG5h8Y/D3i8K6IHeNqhsPjyQKy2eG0giUxa/fnUHLp4guluSireU86FvrBwcJNjtLNRCYyxJ1Lb5/jQR2S/sft+ClhWuq9Pzfo2ZsB37jxQ5sd2BZlQvCIyxZD9Zqx6CUeXWGfX6eo86DCdyuw8fphilaRzc4nwdxPsU4hNtgPfmHve6usnhRhKi5RWSAG0T4bjlgPvOu6UhI+WEsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GvKfnx5pDMJVWTqfkckvKFX5tRfjiqPceq5Mk2APk9A=; b=AZZDM/ciAK+4R6mx5jnBwjcewg9aDb51VHKYan9sEQd5Hg8qD6GgXu21fIe8tHHFu5H2YQAnZNUsW3f8dln70a3BJY+KBAvkiURUCJeTdaMCasganxdwcriHtIbp7y/2yFRqPtPkwwpa818KwpWLtimGEq/GwZUl9jMm8AqljWxv3lvH7yiao0/Lsw3S2IO+N+ih1DIXvvnyMni3IKZJ3id5+/tu+GKsjSUfdxCTUlfoke/ql7mTAC0R/59HnavkboUDWOR5qVic/Bh6S/PmdIXv49k/xuY4KLHAfpVNDXi49GF3jPnB+uEvgHrFSYH8nqmqcDpNnqpjuPHaWNKqoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GvKfnx5pDMJVWTqfkckvKFX5tRfjiqPceq5Mk2APk9A=; b=GpJz3+8KaJMb/MPeUBMAnztkAYTACbyfyM5Lp9eJD2CP2YVvNNhmKB8GjUpCfKFpXeaoO7kfHRsfxai1JQ1idAgAUPO9zPVdQTaO1+AQjtKid5Eq8i/wsT/ZRCh6yWERNbiBVBZGKxHxxfTtSaekLVpPrNM7Tkj3kHHCOlwvXIk= Received: from DM5PR10CA0020.namprd10.prod.outlook.com (2603:10b6:4:2::30) by LV2PR12MB5848.namprd12.prod.outlook.com (2603:10b6:408:173::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.15; Tue, 8 Mar 2022 04:40:33 +0000 Received: from DM6NAM11FT044.eop-nam11.prod.protection.outlook.com (2603:10b6:4:2:cafe::c) by DM5PR10CA0020.outlook.office365.com (2603:10b6:4:2::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT044.mail.protection.outlook.com (10.13.173.185) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:32 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:27 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 6/9] KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by SEV/TDX Date: Tue, 8 Mar 2022 10:08:54 +0530 Message-ID: <20220308043857.13652-7-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bf52e22b-5c41-4711-4829-08da00bdc804 X-MS-TrafficTypeDiagnostic: LV2PR12MB5848:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dphlwWKdQkXoxK98exJKSqVxlJFDTWEZqC2MvVH/wUVT7nZ+G6jXaHwdIPikoNzIB5Bm33hBO3X/3NgHG0hoeayFL0j3Mo1fIQXILDveKtgmLgTYcpF31QwDRddLqx/edWY01OKKdy/0X9QtVZCUidZXqvGl0v+KYSw7jXr317Fb41mtR0RotjdLrcCE4w7BvWf8SVE3vAkTKdKTEw+qhrEYXFXNHloVFeQeS03zpfTeAB9yci1I+C3r0hOYs9jJjC5szlys8LNX5g/Ap7Y2O/3Gcye//VGYAB+TLAv729wpddP2OQrhn7bSwrh6WSrSDf0TwLcHLTIxJpE1KmHWLziFsMR0hKvJLTEw7jIaWTmsz9Eb+F/oeakvJr2mOI9Bx0ndjc/KlJqOUpv/1heIfzdplmGbxdKLc+G4YAjAzQ0w02ETkU+gyVHEDbI4g76cks1Y4a/kr6SBuFv6o7gwOdlddZtl/qs3KmTCjXoaiHNC+F4c4exdPBz8Cu9oI1ThBisFBp4p/6R0ddYhxFXAvT4g4FJFkR5MAmNrECnsAn+NN2S4TCpwVz3c4jYPMXQZzed385Z5gQMYIor9z6xFMEI2+vZXyO/MqLWmYgTXTbsuZZfCF3a3eWTf/5HGAvXIEHGX6+yCSXCx7qaz/3jl9lA6lq00v6Bf2bOaBd9ipLnUPPeg3G2uDdmT0oW/CqAhrTQlr3+YhXVw5iiMScyORA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(356005)(81166007)(336012)(426003)(40460700003)(186003)(26005)(1076003)(16526019)(83380400001)(8936002)(5660300002)(36756003)(7416002)(2906002)(6916009)(54906003)(6666004)(4326008)(508600001)(70586007)(70206006)(7696005)(8676002)(2616005)(47076005)(316002)(82310400004)(36860700001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:32.9795 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bf52e22b-5c41-4711-4829-08da00bdc804 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT044.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5848 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Introduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows SEV/TDX to pin pages before booting the guest, provides the resulting pfn to vendor code if should be needed in the future, and allows the RET_PF_* enums to stay in mmu.c where they belong. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Signed-off-by: Nikunj A Dadhania --- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index e9fbb2c8bbe2..0595891dd834 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -202,6 +202,9 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, return vcpu->arch.mmu->page_fault(vcpu, &fault); } +kvm_pfn_t kvm_mmu_map_tdp_page(struct kvm_vcpu *vcpu, gpa_t gpa, + u32 error_code, int max_level); + /* * Currently, we have two sorts of write-protection, a) the first one * write-protects guest page to sync the guest modification, b) another one is diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b94e5e71653e..5f5da1a4e6be 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4134,6 +4134,44 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) return direct_page_fault(vcpu, fault); } +kvm_pfn_t kvm_mmu_map_tdp_page(struct kvm_vcpu *vcpu, gpa_t gpa, + u32 error_code, int max_level) +{ + int r; + struct kvm_page_fault fault = (struct kvm_page_fault) { + .addr = gpa, + .error_code = error_code, + .exec = error_code & PFERR_FETCH_MASK, + .write = error_code & PFERR_WRITE_MASK, + .present = error_code & PFERR_PRESENT_MASK, + .rsvd = error_code & PFERR_RSVD_MASK, + .user = error_code & PFERR_USER_MASK, + .prefetch = false, + .is_tdp = true, + .nx_huge_page_workaround_enabled = is_nx_huge_page_enabled(), + }; + + if (mmu_topup_memory_caches(vcpu, false)) + return KVM_PFN_ERR_FAULT; + + /* + * Loop on the page fault path to handle the case where an mmu_notifier + * invalidation triggers RET_PF_RETRY. In the normal page fault path, + * KVM needs to resume the guest in case the invalidation changed any + * of the page fault properties, i.e. the gpa or error code. For this + * path, the gpa and error code are fixed by the caller, and the caller + * expects failure if and only if the page fault can't be fixed. + */ + do { + fault.max_level = max_level; + fault.req_level = PG_LEVEL_4K; + fault.goal_level = PG_LEVEL_4K; + r = direct_page_fault(vcpu, &fault); + } while (r == RET_PF_RETRY && !is_error_noslot_pfn(fault.pfn)); + return fault.pfn; +} +EXPORT_SYMBOL_GPL(kvm_mmu_map_tdp_page); + static void nonpaging_init_context(struct kvm_mmu *context) { context->page_fault = nonpaging_page_fault; From patchwork Tue Mar 8 04:38:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 511E2C433FE for ; Tue, 8 Mar 2022 04:41:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245619AbiCHEl7 (ORCPT ); Mon, 7 Mar 2022 23:41:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245660AbiCHEls (ORCPT ); Mon, 7 Mar 2022 23:41:48 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2041.outbound.protection.outlook.com [40.107.220.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC2B93BA59; Mon, 7 Mar 2022 20:40:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Pl0U1PjURwddK3g4rwN+EZ+BGyTL9R48/WTZeeuDuWVjQH7lMcBk6ZeuGxp9t7/n7em7ygofeNoMd8nKKKzyRLn+5GijXqBV26oBgRdT59ZPgFm9aIxzSvSavhfV+O/DxYPotv2h9i/8KKUWGX2uesK2b94u2CQtUCZ1WmRqaCOKLmf8jdwkWW2+zlYtA7mr6PXvHCUle5aQ+RotJ744ryH6JlIyQSfXlvLLOqAy2I3Tdsq4qVZ6nt5SJk1sYS8olkAxwazbeO/lx6wANrXSjMBBfhfz1JThoOkR9oRlzDmx0JqUL/DS44Rh/TFapS7oga1VbpIFUnY+hDNaWIPlMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eqhbSOAP91+41B7UDti6ZhBGWVEsYfQPQAiPzO0wzXQ=; b=Wua3dptqYdfH9aUxTDN9GkTgaod5Xq8Qxd065ciiFqUqOAGq6b1VYRBdnw8iKHUpOY+Durz1WaYiiYzWklihjxCfYdNfBFsw/pbgob3aQAhIzb1J8yNIc9tAaWQ9j6FHC3w7uROr0C7zCMXE1BUJQZ6ICUM87RTiZg3wBVYQhLLVm6azCs1P3DMq86OM/B1AwZHZS60wzQdIXOvuKlEHzR1qp6sA6YuXXagGbcxg2fr0yb2Tq3PHJx+/d4k2wqNtm4tyrgWGP71Wr1fdZyNJ3Y/T9OfVc1D9kyQCjGuQQIBgq51abkTMKrejqA1JrTNJ5Nlk4Yl9fGQkvTV1jWomrg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eqhbSOAP91+41B7UDti6ZhBGWVEsYfQPQAiPzO0wzXQ=; b=ol9/rSZSgpzi1frpik3Cc2O5Wg2t0JWaZXmgY+pkwNUssT2iSDf02iMvO6gDu6beE2MNvGfKwHMblrUZcp6k9l++BV7dzPkRgNoj+JhRbB85g6yPAF2yZPvkCAiNCuLdG61JkyD3HphNTkEs3uSK8ky6/h+9gO1xioAu8XcocU4= Received: from DM5PR20CA0033.namprd20.prod.outlook.com (2603:10b6:3:13d::19) by PH7PR12MB5949.namprd12.prod.outlook.com (2603:10b6:510:1d8::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.17; Tue, 8 Mar 2022 04:40:40 +0000 Received: from DM6NAM11FT020.eop-nam11.prod.protection.outlook.com (2603:10b6:3:13d:cafe::ad) by DM5PR20CA0033.outlook.office365.com (2603:10b6:3:13d::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.13 via Frontend Transport; Tue, 8 Mar 2022 04:40:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT020.mail.protection.outlook.com (10.13.172.224) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:38 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:32 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 7/9] KVM: SEV: Carve out routine for allocation of pages Date: Tue, 8 Mar 2022 10:08:55 +0530 Message-ID: <20220308043857.13652-8-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9584ea4d-e4a0-4e4a-dd48-08da00bdcb68 X-MS-TrafficTypeDiagnostic: PH7PR12MB5949:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rw6rpe7fVhbdlIeEVJqUtjzW9I/zZV0hQ1RZKXlk38pIoIWrqNJNLsnlMkwWWChb9Ew6JWPkW4OA7cDZkVGSogCXBu1TgGQvPPv0mcXJRKexfESdZiFXsH/oYHEUJhqn8zDgV/4PF5C9YOfBPZHfn80J/pTzlzOwTBMJAMQ9EvISUgUqJK045qkdyuP5KjY+Sk3Pk48JSRRMtHa9Y1K7vsNv6YRt1DNTDEYOGefKpafTCR8oJ0xBv4MjWQv+xVaVst26O41b2iBRx41lVETOAZu/rXaIFv9JEnaITaF7uncY5SGE/ri8s/9nuRSNhxnrDK83gsT+d+FsX33floAjjREDTnmkfdOBA+YHc42Do30vqURj4Q4SMSERhmSlGw3BnVYV4YgwRLxW60BRp2BButG3Dv9Y+OJddkbr8fR0AMyUIz7Bqpd0XHQ8pzrxL5X3sTiorGIooquSA1x4ejp2IgaJlNNclJ639XKJkg2G7A2NwkYa7abcr9i++RkVgT5sE9uaC1guy+1mFH0TJ9+fiPHhTnVcuwr2LFdiB+ctH8wv8z2EsJWkelEeUBGGWDOWofffJzhxO4khkN48I3tdpUw6l7/T24FpVHpLHrVRXDd80wsXUNfEBOTiPbWg/qvN05mp+cyMhVXpmraIgSyNbgoP5TIqo5F4UE/+OsYtKg5RatyVd1J9fPvMlb1AkgxEOn8EuFLcJ5wsiUm0Qr+18A== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(81166007)(70206006)(336012)(316002)(426003)(82310400004)(4326008)(2616005)(1076003)(8676002)(70586007)(6666004)(7696005)(16526019)(508600001)(356005)(36860700001)(54906003)(6916009)(186003)(40460700003)(36756003)(47076005)(83380400001)(26005)(2906002)(5660300002)(7416002)(8936002)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:38.6875 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9584ea4d-e4a0-4e4a-dd48-08da00bdcb68 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT020.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5949 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Create a separate routine sev_alloc_pages() for allocating sev pages. This will be used in the following MMU based pinning. While at it, validate the number of pages before the RLIMIT check and use kzalloc instead of kmalloc. Signed-off-by: Nikunj A Dadhania --- arch/x86/kvm/svm/sev.c | 44 ++++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index d0514975555d..7e39320fc65d 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -397,43 +397,53 @@ static unsigned long get_npages(unsigned long uaddr, unsigned long ulen) return last - first + 1; } -static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, - unsigned long ulen, unsigned long *n, - int write) +static void *sev_alloc_pages(struct kvm_sev_info *sev, unsigned long uaddr, + unsigned long ulen, unsigned long *n) { - struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; - struct pinned_region *region; unsigned long npages, size; - int npinned; struct page **pages; - int ret; - - lockdep_assert_held(&kvm->lock); if (ulen == 0 || uaddr + ulen < uaddr) return ERR_PTR(-EINVAL); npages = get_npages(uaddr, ulen); + if (WARN_ON_ONCE(npages > INT_MAX)) + return ERR_PTR(-EINVAL); if (rlimit_memlock_exceeds(sev->pages_to_lock, npages)) { pr_err("SEV: %lu locked pages exceed the lock limit of %lu.\n", - sev->pages_to_lock + npages, - (rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT)); + sev->pages_to_lock + npages, + (rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT)); return ERR_PTR(-ENOMEM); } - if (WARN_ON_ONCE(npages > INT_MAX)) - return ERR_PTR(-EINVAL); - /* Avoid using vmalloc for smaller buffers. */ size = npages * sizeof(struct page *); if (size > PAGE_SIZE) pages = __vmalloc(size, GFP_KERNEL_ACCOUNT | __GFP_ZERO); else - pages = kmalloc(size, GFP_KERNEL_ACCOUNT); + pages = kzalloc(size, GFP_KERNEL_ACCOUNT); - if (!pages) - return ERR_PTR(-ENOMEM); + *n = pages ? npages : 0; + return pages; +} + +static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, + unsigned long ulen, unsigned long *n, + int write) +{ + struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; + struct pinned_region *region; + unsigned long npages; + struct page **pages; + int npinned; + int ret; + + lockdep_assert_held(&kvm->lock); + + pages = sev_alloc_pages(sev, uaddr, ulen, &npages); + if (IS_ERR(pages)) + return pages; /* Pin the user virtual address. */ npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages); From patchwork Tue Mar 8 04:38:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0192C4332F for ; Tue, 8 Mar 2022 04:41:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343554AbiCHEly (ORCPT ); Mon, 7 Mar 2022 23:41:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343649AbiCHElt (ORCPT ); Mon, 7 Mar 2022 23:41:49 -0500 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam08on2046.outbound.protection.outlook.com [40.107.102.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1497F3BBDD; Mon, 7 Mar 2022 20:40:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hW7uVqO0/ELvMhe3Kz4I/p6udTh2mu8SCq//JARvQBU3C//05TfgPK4VRfqRT3LF1IqxW4EI7xNRfbPJXnhdCsrT1S7XXH/w9X7BEw/9nwnl4b8UDOfQRGUYS7xvKJ5r7HJOKj9i3lRvL1L8cVOvMkXTQPDHnD87e8F1xc92uvxT9Uv2mzsBXRFVCyzWV1kdlAjn/Cj74LNe9hd3DVwaUHQCwP9f4xjrXCQkN0v1mZpKdMUlUnPRzop/D9hWujssnpOs2GTM39hsM2QsZY8TmHtFzVZXLrJ9XJwxG+Q5FLJdBYJM7YlDp2I+a1px/ZeScjM1bksg+Thi0QSGzdkOgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BySD8ekJIeFpP166KxGk5bnzjfRJ5WoLQobkj5r3ybU=; b=jK8sGjiCeMd5V+DH3vwWOAXtZxyuk2QgD5R30oGJ9Yxr1bjZbscy2M1zFipS4Yaq3y6c7BOeBreVRS4ikfUwtS7GuuTHP0QrzEhc/QH54V6xz8LZsQLs71f4/SrV+xSjk7j3APsB2kZMSs7+/YZ+6Drt1AmBFReMnaKsqgNVwAK4CcAT/AHKvLUihxD7Z5RwfvUvLyrVdvyapOEZ8iI+I0dn5KyUpZccznJTriVR4HDQdBA5Wz3dTSKbSy9SlOLhO7eRuNhLFaVbo9PfJzo65gacTON+84lbjXL69JZyKhtWFZQxWbvImLj3+PXubn4kSu82ZObXF2LcODPot3Cq+g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BySD8ekJIeFpP166KxGk5bnzjfRJ5WoLQobkj5r3ybU=; b=KvrJLpGUS9gYgzvnHpOUIyveNyAASA8vHofDQawj4Hydwae+hKmyBLUwRXMQkoRXrfkawilLd2gQOgyExdy2G1hlyPL1mHRy1zE4iMEJb2Quf8ToFJ3/X8qWsE7Q4tXS69Ar5L4ADkZOF0mbUWOKt7CCGiiX2k3ZVxTIvwDa8Z8= Received: from BN9PR03CA0271.namprd03.prod.outlook.com (2603:10b6:408:f5::6) by SA0PR12MB4399.namprd12.prod.outlook.com (2603:10b6:806:98::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 04:40:44 +0000 Received: from BN8NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f5:cafe::73) by BN9PR03CA0271.outlook.office365.com (2603:10b6:408:f5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT058.mail.protection.outlook.com (10.13.177.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:43 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:37 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 8/9] KVM: Move kvm_for_each_memslot_in_hva_range() to be used in SVM Date: Tue, 8 Mar 2022 10:08:56 +0530 Message-ID: <20220308043857.13652-9-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2eb7985d-ed0a-4c29-0af2-08da00bdce32 X-MS-TrafficTypeDiagnostic: SA0PR12MB4399:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EAwtEjWKykSCAm1MDYEYZaRLY9njk8mEw5ETcPp3ZnWNNoHCi73DvVdUAByiNWWU9rWCMnVzF7al9mhIlE8X+o7dlyCJUZ97+jkTVqKpmOgBy/L9kjOFBmDT4k574Ov7n7P403J0Giz4mAANyriM/oPlT7oDcQ+V+nsnCHlhu77fmyWh5xBZxTTPX+fQ1muV3kf/lcgQSNXVWb/KOGPa69EjApfsFiQOgg/OQKderk54Hat63exxX0EFEuULSZZ7xrH4b9aZTQ/G2wNyq0JgTkPr8j5cpJ+3lcRw4bTaOP+szYmFZn8ySsQUr87fjX1ovmjpL1EQ22pspR/HqkbatoSdIoXMTKLN6vXWUlBxkkRDWnAtfYbBXjwnwNiUd7sZ3u2Abn9s0tM6juEaOKCL0yQpspbFWW9JxQcUFLKfx1UL+yNBOyKOqiXx+0f/VYS0ArZIVcE6ecLpnWOAtYKeFWuLivA2pGzmIlgbBNux5DRcRibxuhQgXMdiZTH2bDLDt64/RVCRqcnjaGzEpTA7ONvJbhC51bb6Mlpu4ExJpZypuaB6v8FlnOnnSpDwYSVlDp+4lBOefQSUzbrMkzDcdTdYZznsGygxM1F5ixuZ1BoEmNlfplRC4a3i7Qrct0uyM3BCD2HsBw5yQ1R03FaiijaGOeJd94LJqHbIuUmbmStBm/4N6fSur6rxG7YlU8iXn1bpdFmB3ICNQCziGldrZg== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(83380400001)(40460700003)(336012)(47076005)(7416002)(508600001)(2906002)(426003)(36756003)(6666004)(36860700001)(7696005)(82310400004)(5660300002)(8936002)(26005)(186003)(16526019)(81166007)(356005)(70206006)(8676002)(4326008)(1076003)(70586007)(2616005)(54906003)(6916009)(316002)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:43.3770 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2eb7985d-ed0a-4c29-0af2-08da00bdce32 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4399 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the macro to kvm_host.h and make if visible for SVM to use. No functional change intended. Suggested-by: Maciej S. Szmigiero Signed-off-by: Nikunj A Dadhania --- include/linux/kvm_host.h | 6 ++++++ virt/kvm/kvm_main.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index c23022960d51..d72f692725d2 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1063,6 +1063,12 @@ static inline bool kvm_memslot_iter_is_valid(struct kvm_memslot_iter *iter, gfn_ kvm_memslot_iter_is_valid(iter, end); \ kvm_memslot_iter_next(iter)) +/* Iterate over each memslot intersecting [start, last] (inclusive) range */ +#define kvm_for_each_memslot_in_hva_range(node, slots, start, last) \ + for (node = interval_tree_iter_first(&slots->hva_tree, start, last); \ + node; \ + node = interval_tree_iter_next(node, start, last)) + /* * KVM_SET_USER_MEMORY_REGION ioctl allows the following operations: * - create a new memory slot diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index c035fe6b39ec..ff890f41c7ce 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -511,12 +511,6 @@ static void kvm_null_fn(void) } #define IS_KVM_NULL_FN(fn) ((fn) == (void *)kvm_null_fn) -/* Iterate over each memslot intersecting [start, last] (inclusive) range */ -#define kvm_for_each_memslot_in_hva_range(node, slots, start, last) \ - for (node = interval_tree_iter_first(&slots->hva_tree, start, last); \ - node; \ - node = interval_tree_iter_next(node, start, last)) \ - static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, const struct kvm_hva_range *range) { From patchwork Tue Mar 8 04:38:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikunj A. Dadhania" X-Patchwork-Id: 12772956 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31641C433F5 for ; Tue, 8 Mar 2022 04:41:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245438AbiCHEmN (ORCPT ); Mon, 7 Mar 2022 23:42:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245570AbiCHEmH (ORCPT ); Mon, 7 Mar 2022 23:42:07 -0500 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2058.outbound.protection.outlook.com [40.107.236.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB7CA3BA61; Mon, 7 Mar 2022 20:40:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=khCdZ6iLEOEKIAwcZwwNmR45o1Pvr0cZJxdWhg3Jv78K0fCi4jsK28FoZzbZNfH2dj0do2q3DnKbeS8XtTdAsAcJ9G5JdMm3M4iCL8c5ZkNkw3YTWhVrKySQleCHySE0IQTJ5fYxwuqVBfBtVbiIAmGCp2J/FOziNbgtrGhsU9WWWq1o8XyJx68/HbeHfD6kEW88oo1NifSXlbLsdDMKSAxxwVn1j84EcMdNFlyuJXexE/+U5aqSY7LD5gHnniQMh5a07rkIWerjq4Bce28f1Ua811ewPtMlDwHwQtSSEQI6RGl0pKnxTw0H9rU56hrB07oadvBQswfcL6628aWvdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7fS5KJk+FNalIXRvnvjm3wg4niRt0/uJiUGFXnkpgy8=; b=Dra1tjyRZwsnfMgovy4/OzWiL5JagYUBqTp1lhSLncSgwtlg7YQ0Qih9NeVlpOUU0N5x9YbvLy0e52txT+2wSskg44sJkPTTamvcwVjk03FGZPBknb5j1Nc6+ynP0Yc/BZsRzj10QshQDLvjHtXuegFnZOQCCj7ZNjGCzBnvO+EZ6TvG+QgcwwezuAsfs7MJvaurfiVmRNkYt0A/rS516paRAtBShIj3toRuYqgowe19sISotfmR0yODw7fSDmZ3srqOAkJLd5zghwBoQEfKnYZRWjFN2v60gzF6Brx/c44DQoIGcSUFjpsZAc82tB0xaAAwTErL0EC1djPsVFsMXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7fS5KJk+FNalIXRvnvjm3wg4niRt0/uJiUGFXnkpgy8=; b=f8I/mMMenmkmCG6F+5XVJtnrYwv2JQ78/WYSIIXszjdW9p5cgaT4vHivjRBUNJMYW0YBqTgIQRSuFxnbC0E6DDyTDYRvsHgNGKOzgnYOOze27Of3pB5MLg/LMhQDDpr3fGUOAZAnq4MwfM6lF99XUchnADKTxc3XzAVuNF6KWMc= Received: from BN8PR04CA0033.namprd04.prod.outlook.com (2603:10b6:408:70::46) by MW3PR12MB4348.namprd12.prod.outlook.com (2603:10b6:303:5f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.15; Tue, 8 Mar 2022 04:40:50 +0000 Received: from BN8NAM11FT052.eop-nam11.prod.protection.outlook.com (2603:10b6:408:70:cafe::f) by BN8PR04CA0033.outlook.office365.com (2603:10b6:408:70::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.13 via Frontend Transport; Tue, 8 Mar 2022 04:40:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT052.mail.protection.outlook.com (10.13.177.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5038.14 via Frontend Transport; Tue, 8 Mar 2022 04:40:49 +0000 Received: from gomati.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Mon, 7 Mar 2022 22:40:43 -0600 From: Nikunj A Dadhania To: Paolo Bonzini CC: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Brijesh Singh , Tom Lendacky , Peter Gonda , Bharata B Rao , "Maciej S . Szmigiero" , Mingwei Zhang , "David Hildenbrand" , , , Nikunj A Dadhania Subject: [PATCH RFC v1 9/9] KVM: SVM: Pin SEV pages in MMU during sev_launch_update_data() Date: Tue, 8 Mar 2022 10:08:57 +0530 Message-ID: <20220308043857.13652-10-nikunj@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220308043857.13652-1-nikunj@amd.com> References: <20220308043857.13652-1-nikunj@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fce8d315-bc92-4a8e-a3e3-08da00bdd1e5 X-MS-TrafficTypeDiagnostic: MW3PR12MB4348:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kb2H5I24lENDweXkJ1K0+gLPsvArhbgHQpnaI1n9jzr/R5XHVaRXKjgUMrIwnD4cmRG4gfL/7vSzR0g4MGFx6CLb5xCnamL71vgU3vyzOtEB3zYDYId4IktnD1sr+KaCTsHu9gDv/MKFYN1/P3MTXB/0Nnn7VDnECMeiS5mukhk1/EFHKE0XbQOdzeThH3smdjA9IPTEUn9oXGFWHuwEdc94GR5L41xPKdWdlWZLLE6F8tvDrdFwlEGUdgLmAfuzMcBGxKGqxJ6VIu+2fBI0OwqFa1tYGAieXSdHn3fEnSOctkEwGdel/emfaQE473sYcl6xpNIscwS2QeeEclF0M4lEJrGGUSzNs09kBWy1TVTDrK4wdIIe+IjFIwqwNlhAAQ8NSiQsIrUUADwsfcrQymZRcHE0/r3hfpCjNB2O/l0FelITC4zqbZT6Xyb68GN5QjmVMU+FSIJ7AyFyQnAQ0RGNnu3tZUuoSOt8DB0SftRKY9rx3BqYxtUeetN7q/9yfmYX3Bz4F1K8RG0p3SiQQdBadXpz9LU2ewDzFGVsgAGYsbyIau1LN0NRqOSsd271t4gILEc6WvGPZqgcLA5l6SCaXuIF1X3hcI4ziFj3md//bNUOqfQUiADBQJdz4uPIxk7sLzdq+9W58mwHktLX/b4Npsy2KITy1Hqk8PnUQLBINNA/qbUzaHiX6QzbTxLir2Jan+Gk9cC+IbcZAmoQ9Q== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(7416002)(8676002)(4326008)(70586007)(70206006)(7696005)(47076005)(2616005)(5660300002)(8936002)(6916009)(316002)(40460700003)(54906003)(36756003)(2906002)(186003)(1076003)(426003)(36860700001)(16526019)(26005)(6666004)(82310400004)(508600001)(81166007)(83380400001)(336012)(356005)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 04:40:49.6028 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fce8d315-bc92-4a8e-a3e3-08da00bdd1e5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT052.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4348 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Pin the memory for the data being passed to launch_update_data() because it gets encrypted before the guest is first run and must not be moved which would corrupt it. Signed-off-by: Sean Christopherson [ * Use kvm_for_each_memslot_in_hva_range() to find slot and iterate * Updated sev_pin_memory_in_mmu() error handling. * As pinning/unpining pages is handled within MMU, removed {get,put}_user(). ] Signed-off-by: Nikunj A Dadhania --- arch/x86/kvm/svm/sev.c | 146 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 134 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 7e39320fc65d..1c371268934b 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -22,6 +22,7 @@ #include #include +#include "mmu.h" #include "x86.h" #include "svm.h" #include "svm_ops.h" @@ -428,9 +429,93 @@ static void *sev_alloc_pages(struct kvm_sev_info *sev, unsigned long uaddr, return pages; } +#define SEV_PFERR_RO (PFERR_USER_MASK) +#define SEV_PFERR_RW (PFERR_WRITE_MASK | PFERR_USER_MASK) + +static struct page **sev_pin_memory_in_mmu(struct kvm *kvm, unsigned long addr, + unsigned long size, + unsigned long *npages) +{ + unsigned long hva_start, hva_end, uaddr, end, slot_start, slot_end; + struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; + struct interval_tree_node *node; + struct kvm_memory_slot *slot; + struct kvm_memslots *slots; + int idx, ret = 0, i = 0; + struct kvm_vcpu *vcpu; + struct page **pages; + kvm_pfn_t pfn; + u32 err_code; + gfn_t gfn; + + pages = sev_alloc_pages(sev, addr, size, npages); + if (IS_ERR(pages)) + return pages; + + vcpu = kvm_get_vcpu(kvm, 0); + if (mutex_lock_killable(&vcpu->mutex)) { + kvfree(pages); + return ERR_PTR(-EINTR); + } + + vcpu_load(vcpu); + idx = srcu_read_lock(&kvm->srcu); + + kvm_mmu_load(vcpu); + + end = addr + (*npages << PAGE_SHIFT); + slots = kvm_memslots(kvm); + + kvm_for_each_memslot_in_hva_range(node, slots, addr, end) { + slot = container_of(node, struct kvm_memory_slot, + hva_node[slots->node_idx]); + slot_start = slot->userspace_addr; + slot_end = slot_start + (slot->npages << PAGE_SHIFT); + hva_start = max(addr, slot_start); + hva_end = min(end, slot_end); + + err_code = (slot->flags & KVM_MEM_READONLY) ? + SEV_PFERR_RO : SEV_PFERR_RW; + + for (uaddr = hva_start; uaddr < hva_end; uaddr += PAGE_SIZE) { + if (signal_pending(current)) { + ret = -ERESTARTSYS; + break; + } + + if (need_resched()) + cond_resched(); + + /* + * Fault in the page and sev_pin_page() will handle the + * pinning + */ + gfn = hva_to_gfn_memslot(uaddr, slot); + pfn = kvm_mmu_map_tdp_page(vcpu, gfn_to_gpa(gfn), + err_code, PG_LEVEL_4K); + if (is_error_noslot_pfn(pfn)) { + ret = -EFAULT; + break; + } + pages[i++] = pfn_to_page(pfn); + } + } + + kvm_mmu_unload(vcpu); + srcu_read_unlock(&kvm->srcu, idx); + vcpu_put(vcpu); + mutex_unlock(&vcpu->mutex); + + if (!ret) + return pages; + + kvfree(pages); + return ERR_PTR(ret); +} + static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, unsigned long ulen, unsigned long *n, - int write) + int write, bool mmu_usable) { struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; struct pinned_region *region; @@ -441,6 +526,10 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, lockdep_assert_held(&kvm->lock); + /* Use MMU based pinning if possible. */ + if (mmu_usable) + return sev_pin_memory_in_mmu(kvm, uaddr, ulen, n); + pages = sev_alloc_pages(sev, uaddr, ulen, &npages); if (IS_ERR(pages)) return pages; @@ -558,6 +647,7 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) { unsigned long vaddr, vaddr_end, next_vaddr, npages, pages, size, i; struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; + bool mmu_usable = atomic_read(&kvm->online_vcpus) > 0; struct kvm_sev_launch_update_data params; struct sev_data_launch_update_data data; struct page **inpages; @@ -574,15 +664,18 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) vaddr_end = vaddr + size; /* Lock the user memory. */ - inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1); + inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1, mmu_usable); if (IS_ERR(inpages)) return PTR_ERR(inpages); /* * Flush (on non-coherent CPUs) before LAUNCH_UPDATE encrypts pages in * place; the cache may contain the data that was written unencrypted. + * Flushing is automatically handled if the pages can be pinned in the + * MMU. */ - sev_clflush_pages(inpages, npages); + if (!mmu_usable) + sev_clflush_pages(inpages, npages); data.reserved = 0; data.handle = sev->handle; @@ -617,9 +710,14 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) set_page_dirty_lock(inpages[i]); mark_page_accessed(inpages[i]); } - /* unlock the user pages on error */ + /* + * unlock the user pages on error, else pages will be unpinned either + * during memslot free path or vm destroy path + */ if (ret) sev_unpin_memory(kvm, inpages, npages); + else if (mmu_usable) + kvfree(inpages); return ret; } @@ -1001,11 +1099,11 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec) int len, s_off, d_off; /* lock userspace source and destination page */ - src_p = sev_pin_memory(kvm, vaddr & PAGE_MASK, PAGE_SIZE, &n, 0); + src_p = sev_pin_memory(kvm, vaddr & PAGE_MASK, PAGE_SIZE, &n, 0, false); if (IS_ERR(src_p)) return PTR_ERR(src_p); - dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, 1); + dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, 1, false); if (IS_ERR(dst_p)) { sev_unpin_memory(kvm, src_p, n); return PTR_ERR(dst_p); @@ -1057,6 +1155,7 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec) static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) { + bool mmu_usable = atomic_read(&kvm->online_vcpus) > 0; struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; struct sev_data_launch_secret data; struct kvm_sev_launch_secret params; @@ -1071,15 +1170,18 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) if (copy_from_user(¶ms, (void __user *)(uintptr_t)argp->data, sizeof(params))) return -EFAULT; - pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, 1); + pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, 1, mmu_usable); if (IS_ERR(pages)) return PTR_ERR(pages); /* * Flush (on non-coherent CPUs) before LAUNCH_SECRET encrypts pages in * place; the cache may contain the data that was written unencrypted. + * Flushing is automatically handled if the pages can be pinned in the + * MMU. */ - sev_clflush_pages(pages, n); + if (!mmu_usable) + sev_clflush_pages(pages, n); /* * The secret must be copied into contiguous memory region, lets verify @@ -1126,8 +1228,15 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) set_page_dirty_lock(pages[i]); mark_page_accessed(pages[i]); } + /* + * unlock the user pages on error, else pages will be unpinned either + * during memslot free path or vm destroy path + */ if (ret) sev_unpin_memory(kvm, pages, n); + else if (mmu_usable) + kvfree(pages); + return ret; } @@ -1358,7 +1467,7 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) /* Pin guest memory */ guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK, - PAGE_SIZE, &n, 0); + PAGE_SIZE, &n, 0, false); if (IS_ERR(guest_page)) return PTR_ERR(guest_page); @@ -1406,6 +1515,10 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) e_free_hdr: kfree(hdr); e_unpin: + /* + * unlock the user pages on error, else pages will be unpinned either + * during memslot free path or vm destroy path + */ if (ret) sev_unpin_memory(kvm, guest_page, n); @@ -1512,6 +1625,7 @@ static int sev_receive_start(struct kvm *kvm, struct kvm_sev_cmd *argp) static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) { + bool mmu_usable = atomic_read(&kvm->online_vcpus) > 0; struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; struct kvm_sev_receive_update_data params; struct sev_data_receive_update_data data; @@ -1555,7 +1669,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) /* Pin guest memory */ guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK, - PAGE_SIZE, &n, 1); + PAGE_SIZE, &n, 1, mmu_usable); if (IS_ERR(guest_page)) { ret = PTR_ERR(guest_page); goto e_free_trans; @@ -1564,9 +1678,11 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) /* * Flush (on non-coherent CPUs) before RECEIVE_UPDATE_DATA, the PSP * encrypts the written data with the guest's key, and the cache may - * contain dirty, unencrypted data. + * contain dirty, unencrypted data. Flushing is automatically handled if + * the pages can be pinned in the MMU. */ - sev_clflush_pages(guest_page, n); + if (!mmu_usable) + sev_clflush_pages(guest_page, n); /* The RECEIVE_UPDATE_DATA command requires C-bit to be always set. */ data.guest_address = (page_to_pfn(guest_page[0]) << PAGE_SHIFT) + offset; @@ -1577,8 +1693,14 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) ret = sev_issue_cmd(kvm, SEV_CMD_RECEIVE_UPDATE_DATA, &data, &argp->error); + /* + * unlock the user pages on error, else pages will be unpinned either + * during memslot free path or vm destroy path + */ if (ret) sev_unpin_memory(kvm, guest_page, n); + else if (mmu_usable) + kvfree(guest_page); e_free_trans: kfree(trans);