From patchwork Tue Apr 18 21:18:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Lendacky X-Patchwork-Id: 9686359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 24692602C2 for ; Tue, 18 Apr 2017 21:18:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 134BE26E4D for ; Tue, 18 Apr 2017 21:18:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 07F382787C; Tue, 18 Apr 2017 21:18:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 341F226E4D for ; Tue, 18 Apr 2017 21:18:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757994AbdDRVSl (ORCPT ); Tue, 18 Apr 2017 17:18:41 -0400 Received: from mail-co1nam03on0053.outbound.protection.outlook.com ([104.47.40.53]:35006 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1757967AbdDRVSd (ORCPT ); Tue, 18 Apr 2017 17:18:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=NnVKtwbzilyTBYlZr4JJVI1zLm/U/RQmgKo2oDOmiZM=; b=nGrIlbWw8gHbQVRUbE9EyU9MmVsYov08QSpBi6veGs5IPq6IODt4Vm8cGnBcyUolXI3pOdDB3SKEwLQkrBkH97wEF5f3awh30EqP6z5dIrrmfUdRFEAicgEtC3QkJZO8x9YPmwrq5tM33T3XXAN3ITLbA4xQ4MpoKxCvkMsTpic= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from tlendack-t1.amdoffice.net (165.204.77.1) by CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1034.10; Tue, 18 Apr 2017 21:18:24 +0000 From: Tom Lendacky Subject: [PATCH v5 12/32] x86/mm: Insure that boot memory areas are mapped properly To: , , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Toshimitsu Kani , Arnd Bergmann , Jonathan Corbet , Matt Fleming , "Michael S. Tsirkin" , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Larry Woodman , Brijesh Singh , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , Dave Young , Thomas Gleixner , Dmitry Vyukov Date: Tue, 18 Apr 2017 16:18:22 -0500 Message-ID: <20170418211822.10190.67435.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170418211612.10190.82788.stgit@tlendack-t1.amdoffice.net> References: <20170418211612.10190.82788.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: DM5PR20CA0033.namprd20.prod.outlook.com (10.171.161.147) To CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ce0d4803-e0b1-44b5-b399-08d486a0735d X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(201703131423075)(201703031133081); SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 3:Vjgpuu7M0izMRwmwMisxsvD21ZtuwP4uIqnxm3BwOGyAbO7H/VLojY5cQTfshq2mvhaM3qFrbP7ms+3PruF/wWI83FYTAkPrKB0MsdibiO33O9+V1Pg1nK6TAmHv0rXBirHC4+ucFx4XQRWNeK1SOfYGrEEJdJATEBac34j2FM1u/efbfJ1IL7F/PQ90BIwse3Fb+FNmGqHIRzk0yMLIpepqGNDb0TM7PHE5N8whcCrvoW5BfNHyjsx+NZM++RySg6J+E1HKKBDQtVOtyMdtAS8QNLG/Cm8Zt+r7yjprNeTszCjM8srn7aajLj15Bj8vhDIjIDCZjzTJWgpZO00/6aULJzKLX3+v77ehUGO2+qY=; 25:by3V+ZdMVmHvUsqDdP/H1poWO3o+6auXterEC9E89ltcsW9yWyH+D9c5WdzvlgAVAljmNT9lgC0WPACHG5w9gKKC2hxPMJp7zSZum7bZPeX/ThiSKQpvm4sqS25r6Dut9u0B567iTr+sct+H8lm5EfjIXNWV7bD7JbKqKgwJ8w0GbyYbInvkMXOQlG3pnAoWmezRyp/NuwsofcX+DS8+995vBwitmz9inJcPSSaIwPkdQL8d8ZnOI7QUQYXrdTwyMcXl7G7cU1tBsHLSpNmR8djaYkoWzYHn3kyU3Th0rwyB8a18ohK/pAFVYVC3KuM4ZkKi4D3dFhfW8gl0R0A3FjutvAV4vl961tLnP/7cQYhcNwdmaVaOUef6Zz8oEU6SDKyHm+fNiONRBPMu3GJqCxyMUtZgkBq0PrPB8e40EpWpfiW3/tLBP8QJvaSnW9ppEkjiF0OzPaSmJoLo86grIg== X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 31:OMsLi6zT0W05C3OudKUKW2vzED2c9bucUxl9x7np0SLJKOmdfMHy3PY6ryWlpV9h4NWHcmdtx6DRZ9VHPob7DZIdZajZRqIW2bkjpbRwBFPe00747wGNXtGMzbC3tgAZFDj+iYhnSN2chBiMi5E4rRK2WBGIylSshMov2FLaQvXx+xVVZWwlAP+F14+0DM7dIUHWiAYKfnJVhcyhkXY+nB4B/yMK0z5HDqYz8n9DSQ0=; 20:QcguLJkzCy3GMM9wdyOdtwanFR3c7aY+6uhJ21LPGHmSc+0aJFR4Oq2jZgLgiNGUzsWecXoiuhxIFDQPTMsvx26hSdn95DatrIOCnsdNfCr1KOP5VTau/9Wj14uHZVDR5S+MET3SIOcc0bj7KQKXlnTYLQqR+QrRpPTz00HN/2o4CKe7umSVIfhMb0Osu18Dvb9Ip6iyczzpJCGVZTkSoCkzSEkxJ+Uwq3T+1f7VXOIQD2VHrMln0FmyyqfKmWOcR8ovuxxB1XsLCAGv70eROLUGRz/KXkuXrqJWZCa4vustSNSaDJWoIqQSkoGM0etlMMCgLQHwVoFHZWvZVZAtIzWBI1T6rljMEoQ65df4SO98N/SDt1qPzja1WR15RrLwd/FfGjGrdAdUTQODwxJHVOqplzIFhNa/X5HBkDHthZKwIGXZv1d6vFMw0VRv1tzycyjNnD2fhuwnWSquZjluinXy6MnKUPVFsXricALbA/CcG0k8lD+Sg6IshivFZit3 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(3002001)(6055026)(6041248)(20161123562025)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(20161123555025)(6072148); SRVR:CY4PR12MB1141; BCL:0; PCL:0; RULEID:; SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 4:KL7Lq3aeFN7bB0oARGWJhhPOiuO9foXI4NxQ340GELmUZ0mcGf7pwUXTITS73zzwD8TNZgNXrhL43ahPGvuYUyQymexkN+syDi6RDP5gHAQrJDS7SaMzYXui5+vFzD8kXiQzhWyF6ouhEFArnub+VP5j8fFJlZTxqALeUK2OjCSOi3JAcR/cgJkpatzqMvCS0ncy1J2I2RwmjDML2Hyu3PYMYsyUwhz8h0E5R2SFq8yUay9JwPSMF9xDUY6NczuJjn94b2trCxCpbKI+JVcRnMEkA2BYTyLgMF7TzienLnLiICcDE0VMzdz0qgnzJHCmfzZRRD7HE3UWKEBW1I9VMIHC5zD/uXIk0tT4yEKKT13L4pepkrpOCb4Dhm57Z8GswessvBXYgqhP+uGoZh5s34PQRLAAXzpLFhBHLVZy8H7ClZzHLsF3DWWPcPwcKe/6MsmknatxFBJ8JU2H10qxruc0YXrZh5M9xamXI+E3Z+ZUqrk6q4egEFnH8rKVhbMWAmWr/c5xNyJ16wK3advNHVPZQUCVzKzHqq7O8RDA27BOGB7JQ1xDGlGbDAaGciXyIBkx1OaKOC/jKK8HoB8BXx8zpqwdNwAeZJfK5uttBa3hP3nBtZE8qaL6c+y5pTbN4EpCRROvPS/1tEWppsHoZB0HWru1GOK42zv96rC5WLM0CTDNqq4O10d3rZo+CWJjrfMf7Dsz2ONiqRGlGvLBinYHxedErV9qybpVTjyJ8lghLgzfltOfGHrJ162wSTVzCNsjJVKkMA4NxeL5LDOb8Q== X-Forefront-PRVS: 028166BF91 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(39450400003)(39860400002)(39850400002)(39840400002)(39400400002)(39410400002)(42186005)(3846002)(305945005)(25786009)(7406005)(50466002)(7736002)(38730400002)(2906002)(4001350100001)(230700001)(53416004)(86362001)(2950100002)(5660300001)(103116003)(23676002)(6116002)(7416002)(2201001)(33646002)(1076002)(50986999)(54356999)(81166006)(55016002)(76176999)(97746001)(83506001)(66066001)(189998001)(9686003)(4326008)(6506006)(47776003)(53936002)(54906002)(8676002)(921003)(71626007)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR12MB1141; H:tlendack-t1.amdoffice.net; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzIzOnY3MzFSVjFtNktSaUZVUExBWGxiajlMdW1v?= =?utf-8?B?bzJQK3NkRDM3VDBKRzNHVjJRZ1QyMzhSODBJbElWREJOaFpzRFpCc05jdlNZ?= =?utf-8?B?akM4QW5uRWpFeWMrK09jK1JJSTlpQlBiWHBBSmtyMDNvdEUwVWVBTkpqZjdn?= =?utf-8?B?M0lUb0srQjB2RnZwalhKN2owYys3RjcvZ281Z1JLcTVNR2xLT1J2cERldWVw?= =?utf-8?B?a0NXeDBTaWNZNm44L3pTdFM4SXh6d1FPWFNOelJKUWNTdG5QNWprL2FPOVZG?= =?utf-8?B?WVlNam43cVZlZi9tTEZLTVptTlAwUjJveU5hWnpZR1c0aTc0NmZqSlZGT3Vu?= =?utf-8?B?ZExJbXV0LzFYMG40bXpVQXhhYXBtMlU4K1dLeFowY1lqbSt1UEVpUDJKbnZR?= =?utf-8?B?SlJBOWlET0dtdk85UEJTR21QcWNkRDd6RmRUN0V2OTRGM25BdGcxWjFYV2xx?= =?utf-8?B?ZUhpaDRWU0VsR2tGbythY3lhRkhnd3Z1a0prL05nVUlUa0JxejUralhzZi9J?= =?utf-8?B?MllpQVFqQ2JJejF1VGlialFtNFFtOFd5SGRKU2FvS0IzQWx5TlEzZW1LUCtT?= =?utf-8?B?YXJoTXp3MU40YjNIVkliNEVEMmtqWUM3MXl4T0p0cVhxRUNkUXpYOU9Sbkto?= =?utf-8?B?bjlQZzFRbHVmd2FLTlJGOWQyanpIUUtwL2doTmxpVXRNNFR5Rm5tOVJ3bTg2?= =?utf-8?B?WW5wbllsdW9qTW5zeEEwcW1Hckg4N2szWEFqejkyc25HeTVLMHByZDlmQTlS?= =?utf-8?B?eVgxSzZXaTN1czRVeGErQ1FxSEIwWTRlMCtHSngxTE9zaHdkTEJCelFaT1Ay?= =?utf-8?B?Q1BOSU4vUWdkc1NpRjlrV2ViOHY5WjJWeTYxSXNQVTVvTUFPdUFIWVJwZWc3?= =?utf-8?B?SUMySEphL1dndVJXK04vSU1jL000MkMybWtoS2lTbFM1U05KQTY3ZSs4UGdk?= =?utf-8?B?Y0R2T3NmVTFtWWZqRTQzZmNqM1FKOWhUcS9QZ3lQa3owMHRpdGtGOUhxVTNE?= =?utf-8?B?OEdPWXZVMWFtTzczUUkvOTdCNDhTQ0Z6Ym80NGZ3cFRoNmZDMDFFNmxYVVR5?= =?utf-8?B?RWQrTnhDVjljQjVPYUl5bHpYaGpSa2JTVHpvUlVSc3NjR2diT3c1Q2Q1UWx5?= =?utf-8?B?cDZVeFpPanJqZGphQzZyMnp2dGgxcVJPREV3SmhsVWN4WHFweTFLYzFMVkxq?= =?utf-8?B?YjZJN1VRejFBVmtDaEV2R0o5MnkyNHhwZ0hibzA1enBBdU00NU9UVjJNOGs4?= =?utf-8?B?QkhQSXR3cEhBYXJsR1JUZG1JMDgwSlJiTjVGaWJOcDdqWVorTDZPRnNBS25p?= =?utf-8?B?WWVWTGJyclZZaEErSU5QaFRzZUVVc294WWNOVFlJS2M3bjU3SkNkeGNSeS9u?= =?utf-8?B?ZmtUQ3MzSmo5NUVvY2ZiTWJsQWJkdTN5VmFGUit0dHhmdENCeWhMSG5qejRm?= =?utf-8?B?L1BINVVTdzEycGhOMkRwSzlRV1RiOWhFSm5xTHd1RlNTcndFZ2hGTG5uVEVm?= =?utf-8?B?SzdIeFRhc0xaZXRLenRZbWZDdlBIYlZ3K2pPN2t0OXJnYTNDV2hIV3JESmNu?= =?utf-8?B?cFcvc0EvdWZ3WnhkTlpkQml3TUJjbzBPOFIxL2RzRWt3R0lpa0xja2MrMTdv?= =?utf-8?Q?EVxKTuGoS25vmLBot2sX?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 6:HZ07DExmeqZh/VIzga1fCZro+v1oWG20ckjbZEq6nSm6kgvjzW/cdT3Ja1xj5UARyYu/fGGkgqyKTVayU74dJrzuLPteuru2ngJ/93d7JsoENQ/ZZGeba33o9Ed6rgbqplc2E9StV3BBGhpp73CDnL35PA0GzGhF5m5Qt3g0ZRaZGMl1KMtHWdCkkWEOYkzaIAz1ORW4RVUAruPJ1GvURRGUDTLZT9AtgHAUURimheoo5F8QqDqLIxI+fJ8J/F3oIV14V98rr1ETV/NPt8kAJ2mNcqK58PDBQTteQxDA0BwRRZ1XsTPZXe3Q2WdCnCaanuXVdtaPYpLYY1u7F6kMb4Z664OolYkH8c6dh3ppLdopJfUNEuK931S+KIihPjGOSR2Wew2va2elRhPtXsO4MmTlQwaarO/eBYphPAOvpeHE89S3I/10yxlLzbpGvpPx9cHsH54Y5fMUKUdhbVCnDioRn7s0hJy6o3WmwKAeQ+E=; 5:HTdhh8akdFuvMx7PiW88VjKVYc8xVj7PZmdlyCHMtsbiuecBkva86vG3JMbHMeJOm2zEMpvw5cQl6eBQ2zv07EvrMV8jRJmCHulNx4L6iHWq9Pv8l0t7kJVjoSNd3EBchvy5n4r65u8ncQ/803hbBQ==; 24:bLPclEdpxTX4ezPkpNrRo0YI9daqQHDTrpJRqVHew8wM9sGLWvcUhfchM8qX5X3ZE386mnfabBO5Te0fFyEXrFHiyW5tU/NAosFrLiBYp2o= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 7:21WPLhDrTlYaRWeQQH/6dU2sUWqGcXJKk650pyZShl3iuWya4kZufqZZneWTdkDHQ1B7T6gHDriKkoznsDnIXgZWzGDEujmyb4HUgXaST7aRMy1IzEfjDnA7jutGGLdAa0Lu1oTonBLIgeq6tmIwmhFEmLFzEbwA7hvm7RIxC0XfkUB9CUhK9pb+C69R1tJVywS1ZwDE/Mkl9hDoS0jsZmI65vEvuFBnLhJHv0KZGp/SFQUNan19uW06RgeFhrKg5jZCyJsgTY6He9kYzoLmCBINXeXlQAEmgfD+Os4b6o0ux0ZFMyNLkkUx5jzibBNH2JejYVWPzdPEz5CyWhveFg==; 20:V2N3i27J+1OeOh1ApvrM6F9+qYN7YWk/6E0sk5thZNAi6aQ2+wGSEzcpwgFve1M374wlirz/VVDmMBQF82t4YyWwxSOs+cMDsjEEmmBtxMpBot8POmoyVSYQSlqRMq0vJq3G2HKLowsbzHTwz8iOPm0At7y3BZk80fMTufTJN36jVV1smEZkuz3BzjoKJPsgSi3wBhvQybJFpobtxKpSUwN6b0vmp3X/hc9I6qlV/ZTjDcCiYeeboHasVGO0l4gK X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Apr 2017 21:18:24.2062 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1141 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The boot data and command line data are present in memory in a decrypted state and are copied early in the boot process. The early page fault support will map these areas as encrypted, so before attempting to copy them, add decrypted mappings so the data is accessed properly when copied. For the initrd, encrypt this data in place. Since the future mapping of the initrd area will be mapped as encrypted the data will be accessed properly. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/mem_encrypt.h | 11 +++++ arch/x86/include/asm/pgtable.h | 3 + arch/x86/kernel/head64.c | 30 ++++++++++++-- arch/x86/kernel/setup.c | 10 +++++ arch/x86/mm/mem_encrypt.c | 77 ++++++++++++++++++++++++++++++++++++ 5 files changed, 127 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 4021203..130d7fe 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -31,6 +31,9 @@ void __init sme_early_encrypt(resource_size_t paddr, void __init sme_early_decrypt(resource_size_t paddr, unsigned long size); +void __init sme_map_bootdata(char *real_mode_data); +void __init sme_unmap_bootdata(char *real_mode_data); + void __init sme_early_init(void); #else /* !CONFIG_AMD_MEM_ENCRYPT */ @@ -54,6 +57,14 @@ static inline void __init sme_early_decrypt(resource_size_t paddr, { } +static inline void __init sme_map_bootdata(char *real_mode_data) +{ +} + +static inline void __init sme_unmap_bootdata(char *real_mode_data) +{ +} + static inline void __init sme_early_init(void) { } diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 1f9a2c4..1611bb5 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -23,6 +23,9 @@ #ifndef __ASSEMBLY__ #include +extern pgd_t early_level4_pgt[PTRS_PER_PGD]; +int __init __early_make_pgtable(unsigned long address, pmdval_t pmd); + void ptdump_walk_pgd_level(struct seq_file *m, pgd_t *pgd); void ptdump_walk_pgd_level_checkwx(void); diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 9056cf9..e789e14 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -34,7 +34,6 @@ /* * Manage page tables very early on. */ -extern pgd_t early_level4_pgt[PTRS_PER_PGD]; extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt = 2; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); @@ -48,12 +47,12 @@ static void __init reset_early_page_tables(void) } /* Create a new PMD entry */ -int __init early_make_pgtable(unsigned long address) +int __init __early_make_pgtable(unsigned long address, pmdval_t pmd) { unsigned long physaddr = address - __PAGE_OFFSET; pgdval_t pgd, *pgd_p; pudval_t pud, *pud_p; - pmdval_t pmd, *pmd_p; + pmdval_t *pmd_p; /* Invalid address or early pgt is done ? */ if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt)) @@ -95,12 +94,21 @@ int __init early_make_pgtable(unsigned long address) memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); *pud_p = (pudval_t)pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; } - pmd = (physaddr & PMD_MASK) + early_pmd_flags; pmd_p[pmd_index(address)] = pmd; return 0; } +int __init early_make_pgtable(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pmdval_t pmd; + + pmd = (physaddr & PMD_MASK) + early_pmd_flags; + + return __early_make_pgtable(address, pmd); +} + /* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss(void) @@ -123,6 +131,12 @@ static void __init copy_bootdata(char *real_mode_data) char * command_line; unsigned long cmd_line_ptr; + /* + * If SME is active, this will create decrypted mappings of the + * boot data in advance of the copy operations. + */ + sme_map_bootdata(real_mode_data); + memcpy(&boot_params, real_mode_data, sizeof boot_params); sanitize_boot_params(&boot_params); cmd_line_ptr = get_cmd_line_ptr(); @@ -130,6 +144,14 @@ static void __init copy_bootdata(char *real_mode_data) command_line = __va(cmd_line_ptr); memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); } + + /* + * The old boot data is no longer needed and won't be reserved, + * freeing up that memory for use by the system. If SME is active, + * we need to remove the mappings that were created so that the + * memory doesn't remain mapped as decrypted. + */ + sme_unmap_bootdata(real_mode_data); } asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data) diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 603a166..a95800b 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -115,6 +115,7 @@ #include #include #include +#include /* * max_low_pfn_mapped: highest direct mapped pfn under 4GB @@ -374,6 +375,15 @@ static void __init reserve_initrd(void) !ramdisk_image || !ramdisk_size) return; /* No initrd provided by bootloader */ + /* + * If SME is active, this memory will be marked encrypted by the + * kernel when it is accessed (including relocation). However, the + * ramdisk image was loaded decrypted by the bootloader, so make + * sure that it is encrypted before accessing it. + */ + if (sme_active()) + sme_early_encrypt(ramdisk_image, ramdisk_end - ramdisk_image); + initrd_start = 0; mapped_size = memblock_mem_size(max_pfn_mapped); diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 18c0887..2321f05 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -19,6 +19,8 @@ #include #include +#include +#include /* * Since SME related variables are set early in the boot process they must @@ -101,6 +103,81 @@ void __init sme_early_decrypt(resource_size_t paddr, unsigned long size) __sme_early_enc_dec(paddr, size, false); } +static void __init sme_early_pgtable_flush(void) +{ + write_cr3(__sme_pa_nodebug(early_level4_pgt)); +} + +static void __init __sme_early_map_unmap_mem(void *vaddr, unsigned long size, + bool map) +{ + unsigned long paddr = (unsigned long)vaddr - __PAGE_OFFSET; + pmdval_t pmd_flags, pmd; + + /* Use early_pmd_flags but remove the encryption mask */ + pmd_flags = __sme_clr(early_pmd_flags); + + do { + pmd = map ? (paddr & PMD_MASK) + pmd_flags : 0; + __early_make_pgtable((unsigned long)vaddr, pmd); + + vaddr += PMD_SIZE; + paddr += PMD_SIZE; + size = (size <= PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init __sme_map_unmap_bootdata(char *real_mode_data, bool map) +{ + struct boot_params *boot_data; + unsigned long cmdline_paddr; + + __sme_early_map_unmap_mem(real_mode_data, sizeof(boot_params), map); + boot_data = (struct boot_params *)real_mode_data; + + /* + * Determine the command line address only after having established + * the decrypted mapping. + */ + cmdline_paddr = boot_data->hdr.cmd_line_ptr | + ((u64)boot_data->ext_cmd_line_ptr << 32); + + if (cmdline_paddr) + __sme_early_map_unmap_mem(__va(cmdline_paddr), + COMMAND_LINE_SIZE, map); +} + +void __init sme_unmap_bootdata(char *real_mode_data) +{ + /* If SME is not active, the bootdata is in the correct state */ + if (!sme_active()) + return; + + /* + * The bootdata and command line aren't needed anymore so clear + * any mapping of them. + */ + __sme_map_unmap_bootdata(real_mode_data, false); + + sme_early_pgtable_flush(); +} + +void __init sme_map_bootdata(char *real_mode_data) +{ + /* If SME is not active, the bootdata is in the correct state */ + if (!sme_active()) + return; + + /* + * The bootdata and command line will not be encrypted, so they + * need to be mapped as decrypted memory so they can be copied + * properly. + */ + __sme_map_unmap_bootdata(real_mode_data, true); + + sme_early_pgtable_flush(); +} + void __init sme_early_init(void) { unsigned int i;