From patchwork Fri Jan 28 17:17:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brijesh Singh X-Patchwork-Id: 12728786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BD12C433F5 for ; Fri, 28 Jan 2022 17:18:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9EBC6B00DB; Fri, 28 Jan 2022 12:18:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E294C6B00DD; Fri, 28 Jan 2022 12:18:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C538D6B00DE; Fri, 28 Jan 2022 12:18:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id B0CA66B00DB for ; Fri, 28 Jan 2022 12:18:55 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 762928A3F8 for ; Fri, 28 Jan 2022 17:18:55 +0000 (UTC) X-FDA: 79080355830.04.78A1C5A Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2083.outbound.protection.outlook.com [40.107.237.83]) by imf03.hostedemail.com (Postfix) with ESMTP id D325020008 for ; Fri, 28 Jan 2022 17:18:54 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gASTemb+A5VAYL1KhqGJKZ4YtCZJ9UDPzxwNdC6+uA9kK84H7qcIkGmZX4Np++wVtoYtnpfq244yMhoRxdU8R1BWiiSj3uAkr5KwTbPmoKyL/yPPMWdLq29X2JSLmRbgSj4dEXzgZSq6wRkO6UoVm5OEYshQicLrjFtHLxcapSlSEcvQN9iyQP2hED1Fpk/kfv0Ea06o2A/zjO78Nnk09KzNW/bKh+PIgb9Z5ovSpGlu6v3IbbME10NGS0FlINaAfk4zd74f6nV5ZTSZQSkalVQ66xyZzL1J4HGkjOrVsfhXiXaQjlsKTd0g3pGLeRd1RGNkAGSbbK5JkIpjIcf7sA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+FRdp+JeaGof7FQxSxNOdm/gARZoxPI1qmu7S/zX04E=; b=LcRG0zfIsnjQeGdH07HB9hGlqecPFcnmT5536pVkLWdc8yXKlgJWLdNn8I+aPNwv3UNGbSyI/h9CwNGt3XEO8m6KczvCascEOv61ipLj3JlehnYYjwsjgOjtc6/v6EzSH8OKi0w4X2OJIrKG+2Od5k0X5kSz/toTbKMTJGeQWyNZLugJj1TCTixfjDlYuOaQ5OCXp74gCkM1h4BYHtMBl0B7HBAqvN6oa7ASX3dJTkvc0U3fAUdn4DlQlbXVsHyYHYJtsDcuycom5rIprap9Ry422nSBxfn2CnTook9sQDQ2E6gMVDfxj17aSwvfZKHLCTn+rsOs3YiI5O0/nqxbwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+FRdp+JeaGof7FQxSxNOdm/gARZoxPI1qmu7S/zX04E=; b=IRGjSzZZkz1cSV2jkCl0T0egXHCQ6PTcSlxs1YknJKpl64tcMSfVRO/h/+VgvDudBaiXcQoi9yNBbC/34ooT49JKaj1Ul3FvbUTc9Uw5yS1DN0rycDgmTz1ef2QE4f8U6xk4+r/dEThlKsla9knux6ExgSQwUNGXejIMY9aXm1k= Received: from DM5PR19CA0069.namprd19.prod.outlook.com (2603:10b6:3:116::31) by BN9PR12MB5114.namprd12.prod.outlook.com (2603:10b6:408:137::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.15; Fri, 28 Jan 2022 17:18:52 +0000 Received: from DM6NAM11FT036.eop-nam11.prod.protection.outlook.com (2603:10b6:3:116:cafe::47) by DM5PR19CA0069.outlook.office365.com (2603:10b6:3:116::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.19 via Frontend Transport; Fri, 28 Jan 2022 17:18:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT036.mail.protection.outlook.com (10.13.172.64) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4930.15 via Frontend Transport; Fri, 28 Jan 2022 17:18:52 +0000 Received: from sbrijesh-desktop.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Fri, 28 Jan 2022 11:18:48 -0600 From: Brijesh Singh To: , , , , , , CC: Thomas Gleixner , Ingo Molnar , Joerg Roedel , Tom Lendacky , "H. Peter Anvin" , Ard Biesheuvel , Paolo Bonzini , Sean Christopherson , "Vitaly Kuznetsov" , Jim Mattson , "Andy Lutomirski" , Dave Hansen , Sergio Lopez , Peter Gonda , "Peter Zijlstra" , Srinivas Pandruvada , David Rientjes , Dov Murik , Tobin Feldman-Fitzthum , Borislav Petkov , Michael Roth , Vlastimil Babka , "Kirill A . Shutemov" , Andi Kleen , "Dr . David Alan Gilbert" , , , , , Brijesh Singh Subject: [PATCH v9 19/43] x86/mm: Add support to validate memory when changing C-bit Date: Fri, 28 Jan 2022 11:17:40 -0600 Message-ID: <20220128171804.569796-20-brijesh.singh@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220128171804.569796-1-brijesh.singh@amd.com> References: <20220128171804.569796-1-brijesh.singh@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 083eebe5-3fa2-4373-c646-08d9e28241b5 X-MS-TrafficTypeDiagnostic: BN9PR12MB5114:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iSFFFlouI0sqbsJwr6B7JNuPSGakI7yj0ntF3p3tc2mJGUscU7niGsnwIxkguQAFDJdQWW/V5SBqnUzWUOz9QE5tcXMRnoPMBDrgKcnabgp0PbYVWqHvsTk2roK3mCx8PFm0yNUA3sYs53TDysWm7r+SV46GRHPDlKu6K9pZfk8SPpKApTRMpfGPY7EPnI+8sVpAaka4qnvnRJvSq1qlu44BRCv5RwEnE52P0ktS1QQ1ZNy+tCT1MSbDzd2Q+EqypNvD2vvuTvgRb/jDPg3I1+owvP9Yu9DyG6BRGRkMt4EILCigIZemOZrR6M0qkBzdr/AxcRc1yBpAFrVvEBZAYXTPuiL9jxKZSTp4LaemNETVKZQtiY+n5kOXX3Wm2BzrrFCxUlZKysGmvf7WEkT7tZ1/UTggEi7pcowISncGZzNY60KlsSn5ikT5i5ObAVZKyf8oYxyXdoHQjOd/ybT/TDTdaRaNzi0c3stc43LYOWYN/URBiHDQM2Pi+3GQc7hvwjISYTl90Oc2qT89wsTBbx+4iNX5tPtOn8rjAJ5CnHKyPhA7kB97iAE+W0rEKGvwvFNFmZl0OIaMe52zNHPs3hly5AhuBPGO61507j72B0uU3iruAglviAR4SV4Inv2U6Ag9uJN0GLzkuHdZ8/Fi0SjPgTUKE5+nq+AnJ2pnpo13u1B643zPmdo3KwhSsu1io8xa0xjvKwp60irj8SEzre1U4Ow+L+4383NWwetbc2A= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(54906003)(110136005)(426003)(336012)(47076005)(1076003)(316002)(86362001)(186003)(26005)(36860700001)(16526019)(83380400001)(2616005)(82310400004)(508600001)(44832011)(6666004)(356005)(40460700003)(30864003)(15650500001)(36756003)(2906002)(7696005)(7406005)(7416002)(5660300002)(8676002)(4326008)(81166007)(8936002)(70586007)(70206006)(2101003)(36900700001)(20210929001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jan 2022 17:18:52.4599 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 083eebe5-3fa2-4373-c646-08d9e28241b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT036.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5114 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D325020008 X-Stat-Signature: 4irbxcrafgsfo6shwzxmgg83wgyy6n8d Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=IRGjSzZZ; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf03.hostedemail.com: domain of brijesh.singh@amd.com designates 40.107.237.83 as permitted sender) smtp.mailfrom=brijesh.singh@amd.com X-Rspam-User: nil X-HE-Tag: 1643390334-459050 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The set_memory_{encrypted,decrypted}() are used for changing the pages from decrypted (shared) to encrypted (private) and vice versa. When SEV-SNP is active, the page state transition needs to go through additional steps done by the guest. If the page is transitioned from shared to private, then perform the following after the encryption attribute is set in the page table: 1. Issue the page state change VMGEXIT to add the memory region in the RMP table. 2. Validate the memory region after the RMP entry is added. To maintain the security guarantees, if the page is transitioned from private to shared, then perform the following before encryption attribute is removed from the page table: 1. Invalidate the page. 2. Issue the page state change VMGEXIT to remove the page from RMP table. To change the page state in the RMP table, use the Page State Change VMGEXIT defined in the GHCB specification. The GHCB specification provides the flexibility to use either 4K or 2MB page size in during the page state change (PSC) request. For now use the 4K page size for all the PSC until RMP page size tracking is supported in the kernel. Signed-off-by: Brijesh Singh --- arch/x86/include/asm/sev-common.h | 22 ++++ arch/x86/include/asm/sev.h | 4 + arch/x86/include/uapi/asm/svm.h | 2 + arch/x86/kernel/sev.c | 167 ++++++++++++++++++++++++++++++ arch/x86/mm/pat/set_memory.c | 15 +++ 5 files changed, 210 insertions(+) diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-common.h index 891d03408f93..956b8b49528a 100644 --- a/arch/x86/include/asm/sev-common.h +++ b/arch/x86/include/asm/sev-common.h @@ -105,6 +105,28 @@ enum psc_op { #define GHCB_HV_FT_SNP BIT_ULL(0) +/* SNP Page State Change NAE event */ +#define VMGEXIT_PSC_MAX_ENTRY 253 + +struct psc_hdr { + u16 cur_entry; + u16 end_entry; + u32 reserved; +} __packed; + +struct psc_entry { + u64 cur_page : 12, + gfn : 40, + operation : 4, + pagesize : 1, + reserved : 7; +} __packed; + +struct snp_psc_desc { + struct psc_hdr hdr; + struct psc_entry entries[VMGEXIT_PSC_MAX_ENTRY]; +} __packed; + #define GHCB_MSR_TERM_REQ 0x100 #define GHCB_MSR_TERM_REASON_SET_POS 12 #define GHCB_MSR_TERM_REASON_SET_MASK 0xf diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index f65d257e3d4a..feeb93e6ec97 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -128,6 +128,8 @@ void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long padd void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, unsigned int npages); void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op); +void snp_set_memory_shared(unsigned long vaddr, unsigned int npages); +void snp_set_memory_private(unsigned long vaddr, unsigned int npages); #else static inline void sev_es_ist_enter(struct pt_regs *regs) { } static inline void sev_es_ist_exit(void) { } @@ -142,6 +144,8 @@ early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned static inline void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, unsigned int npages) { } static inline void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op) { } +static inline void snp_set_memory_shared(unsigned long vaddr, unsigned int npages) { } +static inline void snp_set_memory_private(unsigned long vaddr, unsigned int npages) { } #endif #endif diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h index b0ad00f4c1e1..0dcdb6e0c913 100644 --- a/arch/x86/include/uapi/asm/svm.h +++ b/arch/x86/include/uapi/asm/svm.h @@ -108,6 +108,7 @@ #define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005 #define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0 #define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1 +#define SVM_VMGEXIT_PSC 0x80000010 #define SVM_VMGEXIT_HV_FEATURES 0x8000fffd #define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff @@ -219,6 +220,7 @@ { SVM_VMGEXIT_NMI_COMPLETE, "vmgexit_nmi_complete" }, \ { SVM_VMGEXIT_AP_HLT_LOOP, "vmgexit_ap_hlt_loop" }, \ { SVM_VMGEXIT_AP_JUMP_TABLE, "vmgexit_ap_jump_table" }, \ + { SVM_VMGEXIT_PSC, "vmgexit_page_state_change" }, \ { SVM_VMGEXIT_HV_FEATURES, "vmgexit_hypervisor_feature" }, \ { SVM_EXIT_ERR, "invalid_guest_state" } diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index 5ee0fbd98d0d..b7ae741a8c66 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -655,6 +655,173 @@ void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op WARN(1, "invalid memory op %d\n", op); } +static int vmgexit_psc(struct snp_psc_desc *desc) +{ + int cur_entry, end_entry, ret = 0; + struct snp_psc_desc *data; + struct ghcb_state state; + struct es_em_ctxt ctxt; + unsigned long flags; + struct ghcb *ghcb; + + /* + * __sev_get_ghcb() needs to run with IRQs disabled because it is using + * a per-CPU GHCB. + */ + local_irq_save(flags); + + ghcb = __sev_get_ghcb(&state); + if (unlikely(!ghcb)) + return 1; + + /* Copy the input desc into GHCB shared buffer */ + data = (struct snp_psc_desc *)ghcb->shared_buffer; + memcpy(ghcb->shared_buffer, desc, min_t(int, GHCB_SHARED_BUF_SIZE, sizeof(*desc))); + + /* + * As per the GHCB specification, the hypervisor can resume the guest + * before processing all the entries. Check whether all the entries + * are processed. If not, then keep retrying. Note, the hypervisor + * will update the data memory directly to indicate the status, so + * reference the data->hdr everywhere. + * + * The strategy here is to wait for the hypervisor to change the page + * state in the RMP table before guest accesses the memory pages. If the + * page state change was not successful, then later memory access will + * result in a crash. + */ + cur_entry = data->hdr.cur_entry; + end_entry = data->hdr.end_entry; + + while (data->hdr.cur_entry <= data->hdr.end_entry) { + ghcb_set_sw_scratch(ghcb, (u64)__pa(data)); + + /* This will advance the shared buffer data points to. */ + ret = sev_es_ghcb_hv_call(ghcb, true, &ctxt, SVM_VMGEXIT_PSC, 0, 0); + + /* + * Page State Change VMGEXIT can pass error code through + * exit_info_2. + */ + if (WARN(ret || ghcb->save.sw_exit_info_2, + "SEV-SNP: PSC failed ret=%d exit_info_2=%llx\n", + ret, ghcb->save.sw_exit_info_2)) { + ret = 1; + goto out; + } + + /* Verify that reserved bit is not set */ + if (WARN(data->hdr.reserved, "Reserved bit is set in the PSC header\n")) { + ret = 1; + goto out; + } + + /* + * Sanity check that entry processing is not going backwards. + * This will happen only if hypervisor is tricking us. + */ + if (WARN(data->hdr.end_entry > end_entry || cur_entry > data->hdr.cur_entry, +"SEV-SNP: PSC processing going backward, end_entry %d (got %d) cur_entry %d (got %d)\n", + end_entry, data->hdr.end_entry, cur_entry, data->hdr.cur_entry)) { + ret = 1; + goto out; + } + } + +out: + __sev_put_ghcb(&state); + local_irq_restore(flags); + + return ret; +} + +static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, + unsigned long vaddr_end, int op) +{ + struct psc_hdr *hdr; + struct psc_entry *e; + unsigned long pfn; + int i; + + hdr = &data->hdr; + e = data->entries; + + memset(data, 0, sizeof(*data)); + i = 0; + + while (vaddr < vaddr_end) { + if (is_vmalloc_addr((void *)vaddr)) + pfn = vmalloc_to_pfn((void *)vaddr); + else + pfn = __pa(vaddr) >> PAGE_SHIFT; + + e->gfn = pfn; + e->operation = op; + hdr->end_entry = i; + + /* + * Current SEV-SNP implementation doesn't keep track of the RMP + * page size so use 4K for simplicity. + */ + e->pagesize = RMP_PG_SIZE_4K; + + vaddr = vaddr + PAGE_SIZE; + e++; + i++; + } + + if (vmgexit_psc(data)) + sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC); +} + +static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) +{ + unsigned long vaddr_end, next_vaddr; + struct snp_psc_desc *desc; + + desc = kmalloc(sizeof(*desc), GFP_KERNEL_ACCOUNT); + if (!desc) + panic("SEV-SNP: failed to allocate memory for PSC descriptor\n"); + + vaddr = vaddr & PAGE_MASK; + vaddr_end = vaddr + (npages << PAGE_SHIFT); + + while (vaddr < vaddr_end) { + /* + * Calculate the last vaddr that can be fit in one + * struct snp_psc_desc. + */ + next_vaddr = min_t(unsigned long, vaddr_end, + (VMGEXIT_PSC_MAX_ENTRY * PAGE_SIZE) + vaddr); + + __set_pages_state(desc, vaddr, next_vaddr, op); + + vaddr = next_vaddr; + } + + kfree(desc); +} + +void snp_set_memory_shared(unsigned long vaddr, unsigned int npages) +{ + if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) + return; + + pvalidate_pages(vaddr, npages, false); + + set_pages_state(vaddr, npages, SNP_PAGE_STATE_SHARED); +} + +void snp_set_memory_private(unsigned long vaddr, unsigned int npages) +{ + if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) + return; + + set_pages_state(vaddr, npages, SNP_PAGE_STATE_PRIVATE); + + pvalidate_pages(vaddr, npages, true); +} + int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { u16 startup_cs, startup_ip; diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index b4072115c8ef..1bc15b9d15f3 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -32,6 +32,7 @@ #include #include #include +#include #include "../mm_internal.h" @@ -2012,8 +2013,22 @@ static int __set_memory_enc_pgtable(unsigned long addr, int numpages, bool enc) */ cpa_flush(&cpa, !this_cpu_has(X86_FEATURE_SME_COHERENT)); + /* + * To maintain the security guarantees of SEV-SNP guest invalidate the + * memory before clearing the encryption attribute. + */ + if (!enc) + snp_set_memory_shared(addr, numpages); + ret = __change_page_attr_set_clr(&cpa, 1); + /* + * Now that memory is mapped encrypted in the page table, validate it + * so that is consistent with the above page state. + */ + if (!ret && enc) + snp_set_memory_private(addr, numpages); + /* * After changing the encryption attribute, we need to flush TLBs again * in case any speculative TLB caching occurred (but no need to flush