From patchwork Fri Jan 28 17:17:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brijesh Singh X-Patchwork-Id: 12728787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41109C43217 for ; Fri, 28 Jan 2022 17:19:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B217E6B00E2; Fri, 28 Jan 2022 12:18:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A1F856B00E0; Fri, 28 Jan 2022 12:18:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E69D8D0002; Fri, 28 Jan 2022 12:18:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 6FAED6B00DD for ; Fri, 28 Jan 2022 12:18:56 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3B9F383EBB for ; Fri, 28 Jan 2022 17:18:56 +0000 (UTC) X-FDA: 79080355872.06.A8A469B Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2082.outbound.protection.outlook.com [40.107.236.82]) by imf07.hostedemail.com (Postfix) with ESMTP id 4D4184000E for ; Fri, 28 Jan 2022 17:18:54 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rd+ImEBAJZgHObn1bkrJ1k4OdAoQLL75Z+ZrkTOeu/kxqtNp7eYNF701ffgyFBxkvJhkhr1myB+osJpy2BIGjMf/oerN7FdB2mteQKZCrNVhgWb/gH+A3Hqf8KRk35MhdT4Vvok4T974lX3HkQB5G9JQ3J/LNtwtfS/mg0y551pQHs0EneRIhmgxwRl1BeAf1XnfdjiY63kHhYVVRA1uEvf6G4J85G5AwZ1Aw+mNlVnpFKWCEzjAIqic5XirNfQ/cjTTCv2enLMNOE31tphTd2VTzCnS51L1EBV7eYT1tbr2dDwNudS4TErPz4xe0Y7VS4bECgkgA1DAoFFuy6RLAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7Dx7EiVfBXl88l7VPonmOi21nBvy9NFaXS1dL6fpY7o=; b=IK/AfkrJZOYdV2DKDWfcQDP8e3+oCUqlRPS6oTpulynUX2g0GBvEHgG3Ztnfy/GY5niMt8u8fKf8SHHVfOj4Bov/p/VzIJq1BdML5IUkPthxRzlZdgLBgOrMEckH+gFDVsrjysOzfDOadUWKa1T5F9xONY4ouI6DgTkrugT4RHp3L2Ghagw+qTSLgb4j6jFossRObNfTbrN5uZkvkIjN5LvI6T5HBf95SvPMdQ3Vy+HtvsyN0eNgve/PDQe/8uV258Rk7/bWlrexd6Pu+5CGmQQSVjefdKh/jmo0QUICHNCzcw7RtZ+FcBL45rzy4WcWRhviDyQfp9J0XSGVO+fKJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7Dx7EiVfBXl88l7VPonmOi21nBvy9NFaXS1dL6fpY7o=; b=Ymk76YMh3Nt7dIP9ot1Jef7pIfch+1qtwXbHoNzvuknQSCLG+bmVLreu30NU15yNt7AT9zcfO1UE+hPLZT6oRja/AvtyFx5GsEDpVJXC+hlunVNDGnwhVw1qG5Zi3DCLzebz2BuuCp+E3XfRflwYu9pO/cRg2I+H2scFBQxQDl0= Received: from DM5PR19CA0057.namprd19.prod.outlook.com (2603:10b6:3:116::19) by DM6PR12MB2652.namprd12.prod.outlook.com (2603:10b6:5:41::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.17; Fri, 28 Jan 2022 17:18:51 +0000 Received: from DM6NAM11FT036.eop-nam11.prod.protection.outlook.com (2603:10b6:3:116:cafe::62) by DM5PR19CA0057.outlook.office365.com (2603:10b6:3:116::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.18 via Frontend Transport; Fri, 28 Jan 2022 17:18:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT036.mail.protection.outlook.com (10.13.172.64) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4930.15 via Frontend Transport; Fri, 28 Jan 2022 17:18:51 +0000 Received: from sbrijesh-desktop.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Fri, 28 Jan 2022 11:18:43 -0600 From: Brijesh Singh To: , , , , , , CC: Thomas Gleixner , Ingo Molnar , Joerg Roedel , Tom Lendacky , "H. Peter Anvin" , Ard Biesheuvel , Paolo Bonzini , Sean Christopherson , "Vitaly Kuznetsov" , Jim Mattson , "Andy Lutomirski" , Dave Hansen , Sergio Lopez , Peter Gonda , "Peter Zijlstra" , Srinivas Pandruvada , David Rientjes , Dov Murik , Tobin Feldman-Fitzthum , Borislav Petkov , Michael Roth , Vlastimil Babka , "Kirill A . Shutemov" , Andi Kleen , "Dr . David Alan Gilbert" , , , , , Brijesh Singh , Venu Busireddy Subject: [PATCH v9 16/43] x86/sev: Add helper for validating pages in early enc attribute changes Date: Fri, 28 Jan 2022 11:17:37 -0600 Message-ID: <20220128171804.569796-17-brijesh.singh@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220128171804.569796-1-brijesh.singh@amd.com> References: <20220128171804.569796-1-brijesh.singh@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2fec3972-0fc9-422a-bb0a-08d9e2824126 X-MS-TrafficTypeDiagnostic: DM6PR12MB2652:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2657; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Za4ouy4dDhI+HLievGKix7FnbUj7pZroqgcvJmzxosAorA51HaXhgYWwmdsYRMWRL9P16dGA4Ju+WlluUKiXFSjnqyQYn3wTeeDwjH8AHseZeu/I0bcVCe8kyuXIXkG9Teo0nJ/CgaI4kXDGGgWuLsuuOH3txB1NWhtLHnNUNqtbyxbU6NucoFMau+eEzInwz6kkHqNGSZhA43RwTtn9Vn5kBsbR3LOnfGntVlVLsKoMQRfI+kmgxPqJbWJ3m54xaWJg8M5KniX5gIb3OFnTIVcUrHOGTvX9WrlNpAUzX6tz8z5CGz4fJbFiQQlNW3OZvBgDUQRbpsBdO/iYB0reZ5QtvObZatb9HZYtrgM15+5d5Qjdmj7WndtjwVn4/EQfRXB0xwXXJc4pUy9/cYDvS/8mC9dWnf4ncEwyMQytxP9uiRKAJLdoKd/g+wyAqJZlWlhHahwOvmPcRCRKU5GOgPNZKj3fhru9aLM++5w0ylq9HvbcLIIzazjZT1YS+83+fmjQZrCD+IJSdGLxyCaJJzPNYeq1/UoRe77X1IJmff5Cjf8oBdaYrKi87i88G3XaqVLKm8CH2gphWvQwUNsTLaL3MQ/hx4h3S1QXsswHQeX8q4mz9/i9L34T4EFauYye5Y8JdYJkqLzhNiGWUpuTx388MqmBb1bM0wOIQjRyeG6eXddXIpw3OrWZf1UeQIedLHskDoROzVR110EHNsSQrmC9lyvfYCa7Rof/2GzwmQiCOgqJA3D/9hK/VStjE+txQ9Q+0jo9/ljiP6rn9kUdKV0kIOqMSVowYOxxhluzeNsaa8nO3aporX+7x64JwCvm X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(4636009)(40470700004)(36840700001)(46966006)(86362001)(7416002)(36756003)(26005)(2616005)(2906002)(70206006)(8676002)(8936002)(186003)(36860700001)(110136005)(426003)(70586007)(54906003)(336012)(40460700003)(16526019)(7406005)(83380400001)(508600001)(44832011)(356005)(47076005)(82310400004)(5660300002)(4326008)(1076003)(316002)(7696005)(81166007)(6666004)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jan 2022 17:18:51.5225 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2fec3972-0fc9-422a-bb0a-08d9e2824126 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT036.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2652 X-Rspam-User: nil X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4D4184000E X-Stat-Signature: 7wuaqk3mzg8rdmgb96wahfbdfr97adhx Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=Ymk76YMh; spf=pass (imf07.hostedemail.com: domain of brijesh.singh@amd.com designates 40.107.236.82 as permitted sender) smtp.mailfrom=brijesh.singh@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com X-HE-Tag: 1643390334-592168 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The early_set_memory_{encrypted,decrypted}() are used for changing the page from decrypted (shared) to encrypted (private) and vice versa. When SEV-SNP is active, the page state transition needs to go through additional steps. If the page is transitioned from shared to private, then perform the following after the encryption attribute is set in the page table: 1. Issue the page state change VMGEXIT to add the page as a private in the RMP table. 2. Validate the page after its successfully added in the RMP table. To maintain the security guarantees, if the page is transitioned from private to shared, then perform the following before clearing the encryption attribute from the page table. 1. Invalidate the page. 2. Issue the page state change VMGEXIT to make the page shared in the RMP table. The early_set_memory_{encrypted,decrypted} can be called before the GHCB is setup, use the SNP page state MSR protocol VMGEXIT defined in the GHCB specification to request the page state change in the RMP table. While at it, add a helper snp_prep_memory() which will be used in probe_roms(), in a later patch. Reviewed-by: Venu Busireddy Signed-off-by: Brijesh Singh --- arch/x86/include/asm/sev.h | 10 ++++ arch/x86/kernel/sev.c | 99 +++++++++++++++++++++++++++++++++++ arch/x86/mm/mem_encrypt_amd.c | 58 ++++++++++++++++++-- 3 files changed, 163 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index 48df02713ee0..f65d257e3d4a 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -123,6 +123,11 @@ static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) return rc; } void setup_ghcb(void); +void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, + unsigned int npages); +void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, + unsigned int npages); +void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op); #else static inline void sev_es_ist_enter(struct pt_regs *regs) { } static inline void sev_es_ist_exit(void) { } @@ -132,6 +137,11 @@ static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; } static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) { return 0; } static inline int rmpadjust(unsigned long vaddr, bool rmp_psize, unsigned long attrs) { return 0; } static inline void setup_ghcb(void) { } +static inline void __init +early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned int npages) { } +static inline void __init +early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, unsigned int npages) { } +static inline void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op) { } #endif #endif diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index b86b48b66a44..5ee0fbd98d0d 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -556,6 +556,105 @@ static u64 get_jump_table_addr(void) return ret; } +static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool validate) +{ + unsigned long vaddr_end; + int rc; + + vaddr = vaddr & PAGE_MASK; + vaddr_end = vaddr + (npages << PAGE_SHIFT); + + while (vaddr < vaddr_end) { + rc = pvalidate(vaddr, RMP_PG_SIZE_4K, validate); + if (WARN(rc, "Failed to validate address 0x%lx ret %d", vaddr, rc)) + sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PVALIDATE); + + vaddr = vaddr + PAGE_SIZE; + } +} + +static void __init early_set_pages_state(unsigned long paddr, unsigned int npages, enum psc_op op) +{ + unsigned long paddr_end; + u64 val; + + paddr = paddr & PAGE_MASK; + paddr_end = paddr + (npages << PAGE_SHIFT); + + while (paddr < paddr_end) { + /* + * Use the MSR protocol because this function can be called before + * the GHCB is established. + */ + sev_es_wr_ghcb_msr(GHCB_MSR_PSC_REQ_GFN(paddr >> PAGE_SHIFT, op)); + VMGEXIT(); + + val = sev_es_rd_ghcb_msr(); + + if (WARN(GHCB_RESP_CODE(val) != GHCB_MSR_PSC_RESP, + "Wrong PSC response code: 0x%x\n", + (unsigned int)GHCB_RESP_CODE(val))) + goto e_term; + + if (WARN(GHCB_MSR_PSC_RESP_VAL(val), + "Failed to change page state to '%s' paddr 0x%lx error 0x%llx\n", + op == SNP_PAGE_STATE_PRIVATE ? "private" : "shared", + paddr, GHCB_MSR_PSC_RESP_VAL(val))) + goto e_term; + + paddr = paddr + PAGE_SIZE; + } + + return; + +e_term: + sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC); +} + +void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, + unsigned int npages) +{ + if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) + return; + + /* + * Ask the hypervisor to mark the memory pages as private in the RMP + * table. + */ + early_set_pages_state(paddr, npages, SNP_PAGE_STATE_PRIVATE); + + /* Validate the memory pages after they've been added in the RMP table. */ + pvalidate_pages(vaddr, npages, true); +} + +void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, + unsigned int npages) +{ + if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) + return; + + /* Invalidate the memory pages before they are marked shared in the RMP table. */ + pvalidate_pages(vaddr, npages, false); + + /* Ask hypervisor to mark the memory pages shared in the RMP table. */ + early_set_pages_state(paddr, npages, SNP_PAGE_STATE_SHARED); +} + +void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op) +{ + unsigned long vaddr, npages; + + vaddr = (unsigned long)__va(paddr); + npages = PAGE_ALIGN(sz) >> PAGE_SHIFT; + + if (op == SNP_PAGE_STATE_PRIVATE) + early_snp_set_memory_private(vaddr, paddr, npages); + else if (op == SNP_PAGE_STATE_SHARED) + early_snp_set_memory_shared(vaddr, paddr, npages); + else + WARN(1, "invalid memory op %d\n", op); +} + int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { u16 startup_cs, startup_ip; diff --git a/arch/x86/mm/mem_encrypt_amd.c b/arch/x86/mm/mem_encrypt_amd.c index 2b2d018ea345..b59be6aac97b 100644 --- a/arch/x86/mm/mem_encrypt_amd.c +++ b/arch/x86/mm/mem_encrypt_amd.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -47,6 +48,36 @@ EXPORT_SYMBOL(sme_me_mask); /* Buffer used for early in-place encryption by BSP, no locking needed */ static char sme_early_buffer[PAGE_SIZE] __initdata __aligned(PAGE_SIZE); +/* + * SNP-specific routine which needs to additionally change the page state from + * private to shared before copying the data from the source to destination and + * restore after the copy. + */ +static inline void __init snp_memcpy(void *dst, void *src, size_t sz, + unsigned long paddr, bool decrypt) +{ + unsigned long npages = PAGE_ALIGN(sz) >> PAGE_SHIFT; + + if (decrypt) { + /* + * @paddr needs to be accessed decrypted, mark the page shared in + * the RMP table before copying it. + */ + early_snp_set_memory_shared((unsigned long)__va(paddr), paddr, npages); + + memcpy(dst, src, sz); + + /* Restore the page state after the memcpy. */ + early_snp_set_memory_private((unsigned long)__va(paddr), paddr, npages); + } else { + /* + * @paddr need to be accessed encrypted, no need for the page state + * change. + */ + memcpy(dst, src, sz); + } +} + /* * This routine does not change the underlying encryption setting of the * page(s) that map this memory. It assumes that eventually the memory is @@ -95,8 +126,13 @@ static void __init __sme_early_enc_dec(resource_size_t paddr, * Use a temporary buffer, of cache-line multiple size, to * avoid data corruption as documented in the APM. */ - memcpy(sme_early_buffer, src, len); - memcpy(dst, sme_early_buffer, len); + if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) { + snp_memcpy(sme_early_buffer, src, len, paddr, enc); + snp_memcpy(dst, sme_early_buffer, len, paddr, !enc); + } else { + memcpy(sme_early_buffer, src, len); + memcpy(dst, sme_early_buffer, len); + } early_memunmap(dst, len); early_memunmap(src, len); @@ -318,14 +354,28 @@ static void __init __set_clr_pte_enc(pte_t *kpte, int level, bool enc) clflush_cache_range(__va(pa), size); /* Encrypt/decrypt the contents in-place */ - if (enc) + if (enc) { sme_early_encrypt(pa, size); - else + } else { sme_early_decrypt(pa, size); + /* + * ON SNP, the page state in the RMP table must happen + * before the page table updates. + */ + early_snp_set_memory_shared((unsigned long)__va(pa), pa, 1); + } + /* Change the page encryption mask. */ new_pte = pfn_pte(pfn, new_prot); set_pte_atomic(kpte, new_pte); + + /* + * If page is set encrypted in the page table, then update the RMP table to + * add this page as private. + */ + if (enc) + early_snp_set_memory_private((unsigned long)__va(pa), pa, 1); } static int __init early_set_memory_enc_dec(unsigned long vaddr,