From patchwork Tue Jun 27 15:14:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Lendacky X-Patchwork-Id: 9812631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9A7F16020A for ; Tue, 27 Jun 2017 15:32:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A43328595 for ; Tue, 27 Jun 2017 15:32:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7DC6628683; Tue, 27 Jun 2017 15:32:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4358228595 for ; Tue, 27 Jun 2017 15:31:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751974AbdF0Pbw (ORCPT ); Tue, 27 Jun 2017 11:31:52 -0400 Received: from mail-dm3nam03on0080.outbound.protection.outlook.com ([104.47.41.80]:1173 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753139AbdF0PO0 (ORCPT ); Tue, 27 Jun 2017 11:14:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=4F2Fxw9wEFCaJfSTYIjOmYXJPcamsF29a+HBtRs9ehE=; b=PLfIO55OjuZkX7iJ9oG0ODuT5WnkT2GBXROXddQEDa/QGUNIZDRsCg6D+jh4PFgl234uv/pD4LbwVtNJDMdleL7SprgcO/2dZQIAwaq8ZwFIZ9XpXnANQc1KEniEAZU/w36v2f2pdwxRYPCl+rujnN/93n8gfz19brvMT8eJE9o= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from tlendack-t1.amdoffice.net (165.204.77.1) by CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1199.15; Tue, 27 Jun 2017 15:14:18 +0000 From: Tom Lendacky Subject: [PATCH v8 RESEND 35/38] x86/mm: Add support to encrypt the kernel in-place To: linux-arch@vger.kernel.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, x86@kernel.org, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, xen-devel@lists.xen.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Brijesh Singh , Toshimitsu Kani , Radim =?utf-8?b?S3LEjW3DocWZ?= , Matt Fleming , Alexander Potapenko , "H. Peter Anvin" , Larry Woodman , Jonathan Corbet , Joerg Roedel , "Michael S. Tsirkin" , Ingo Molnar , Andrey Ryabinin , Dave Young , Rik van Riel , Arnd Bergmann , Konrad Rzeszutek Wilk , Borislav Petkov , Andy Lutomirski , Boris Ostrovsky , Dmitry Vyukov , Juergen Gross , Thomas Gleixner , Paolo Bonzini Date: Tue, 27 Jun 2017 10:14:16 -0500 Message-ID: <20170627151415.17428.76460.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170627150718.17428.81813.stgit@tlendack-t1.amdoffice.net> References: <20170627150718.17428.81813.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: DM5PR1101CA0001.namprd11.prod.outlook.com (10.174.246.11) To CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 096ec854-495b-408f-0ade-08d4bd6f2f35 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(48565401081)(300000503095)(300135400095)(201703131423075)(201703031133081)(300000504095)(300135200095)(300000505095)(300135600095); SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 3:9Ndl2B5MfQaxnadoQ7FgF/XI09agVOl+SDp7FPY+yS0XKxZuin6/qEesvGV+5HYPBZSSUHbcFpCww25iYbMzZbgBe3ff2fWO5TqTMebOBGhkbvutPtiqJF1yhm93K9agcc9o70hwcoMb/ag6joCExLi23wZq59f5p2ZeZcnugO4PUPXA2RlI9UHbAH0J5GREgMq51dDsBocV915CM+uxZEn7f5w3yKV16irxc7sduGrO1RuwfcBqXl1CN440a85fIc+z9EeRhjFT0uMAYFVDPkCI4re5AnIe0YrVJ/n5+XGgmfOU+eo+wCE2FJA7wcQ9GbkcE3gz6GpRBYFkeNfE9JAFEgGMHcq3nDlhXhQh/e/Ached7n165xcxf8a8PsaPJv91ZWbzjQL1iqzac4rChjHk/iNWaJYTxvs13S0q0e9PnoC8QY/XpNk+TTfgIlHHWRUrOTdeWtuw2ifdYpvEjoxDAGHR2ZRNJDzbsrMcSntxWL1p2HcdiljkLtzNpaYpNl/1NxBdedAZT4Tetp/+Rp4WleC98LGHu9VI3hI9ufjnYwNs/ysuD8K/YG5Fcix0xnxIpsWif7cK3DYXav8SGzYH+3lanFvSHpMMbxG2Sr5qp++TtfuaQP9M1q4TxqCkvB61IU75EHVXgXCr/xMLKSrIO+yLJq3G6dGSxBSrIi4/BpJkPx0kHPM8TQtg5Oj5Q2TLlZLtunw4v00ImIpD1A== X-MS-TrafficTypeDiagnostic: CY4PR12MB1141: X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 25:r+cZ0S0mANMQypBo0v50aW1R5EO6sgvVNTi+FFcP6U+Gh16stnrBV9wKknD/G2nbdIb8rsicxgAXn3GCnTTRtVV8NFtPcNir6FafSkTn8dyDVuiWz0q4JwWrT02051iI5iba9QmchcVx56ks9kCwvfwb6oCC8hqZ329hjr/St9RpY/hF+c6TBF/LkHqoFvoOwNuKYuqbkql9ZHMEsYX0JozmkvIha8+FHq7RXa5HfSz9t2Nz5P+xQAzUXunCaNr3pqExHYYSzzWixZ5jUNGYCOb7N1O6FDLKZq8tqFXFWFqEQFVaxEVfsSiU8N68oZ+WBt4eo/iZRfYub6eagJIlDja1sXCPlcr8qu5LFIJ629ZL4pD3MnYmKaqyIJLUre7HsdTvUaDskccahezHEbeIntC+jBnLa8kXS99ny9Uu+q3F2JskRl6OIgwiMc6+kYlomyDBGCdFRkU1l5tN+xnVuLiGvNIp2Z1xk+YLPkCMEss1uiK3daQa9XRCOmYbbFyPvYn/xWUxE4dVXsUdkLDrOasqTgTsnUhEM7W/n93cVu8xB433UKCRfEnQ2C13EcY7TBiXCVs0F3z300mlS9Xhcla1igvIpMp1U9KHBoxb91jofIx/gpRs0X469c7BIGXzf5MgotQdnwRjMbJXo1AwmDt1b7dLojCaOtk2ip7iNBi24UFwPjfA/En8RPeEIcvw0yeZcKZapouk3wxZsUurm+pxO5FF0RIMVzmrnX5QFRI0DtybAjNmqBmesRkm8evksWPq+/PCmn9VNf6Rx/kwvNxBTXP1E3POyMd2r4oQ4pQIgOwVdTewA0hwvmzL0o4YZ5cxJFHPA6p6YdAXryKSqc1mijSqTJ89vOVu7rJPIkOioITbCNwVfNHRby8J7mEdN8Pbq/uc8xM8MOUjLB5G342adukUdYIOIaKdN0E26 lg= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 31:HpJG3vhYX5vy72BUOaQl68+9G5bg2vu2tS8ICecuvOC1UFOoTe6tLwQeyFKoIzV7+zcQbxJZDNs9FZ/75rRG1hiTpCjhF+TTmBHF5p1ml4RI6iHcUd2jUnq3ciaOAxZ39tM1FKMGAGiZF3+1zyOpNWYD+JE5RRhSBQSsKAsND5OcKuuVdK88gkzkXu8a9QF8DTorSO5p+Iu9qwYLkS62WAVLWkosZd4Y856p7EFjsOK+84l4KDk4R9WPLC86Xj5wsKinOK67PZeo1lvVB+Ra2pAjQ2XG04715X6vRAm1PPF8Rth4n/ZnEBqVEhttZ5lzMXkdaEerGcRlXNYA8ilPFHXAf2Q2nqxvzJVdi2k/pplXlDnVn+jUsmmtOFxOJMnWUeeYGkxzcS9vaA3E63CUy7yJBymepx+eopBxOWGyEouQxQ+T4EyVRScVidAalVpS3CinTJ/nugT0aCImQM2pqteXy2XLrFucWnBt2Eem9JVxiVVxRZuqXsTug4JoJNlLg897+Ap0f2//wuCPKlW6L+fpe+APzNuvv55Y+lhsP6Hu8Sev6IVhQgN9qTXJmP5tXw7UciNcAiUiJwuqksZgU8nZw9Ah8EFLNU2Gneag5GTlnqvgOoYwKLPkbZ5B3vAKU2d8fVzYDylkYi+szbWCb7sM1PLwvT3Mzp/q07wXa94= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 20:Z2VBsANEaTZkVgso7om+wZGtAsX6I+cg/OEusGqTFMw5b8NueQeOZL3pXhM0FbQb5sfzcf0Zxy+oX5wpr4GXQk6UFzVMZA4cHP73hnMGyNQnILKeD44ycpBzGMzERGZX4nPQRzm03uQZGXRXDXipFK/dHwnnzInSK3Q1bVSyUeHZUdfBXcInWzKUgImhVX0bexqLbuxvns9CDSdqK5p4UfqmXJtJmunzua/eiidmUO3F4s/ydH6NthmvGCXuoUYaZQ/sFYjmcgePYNkFtLje7caWQuldKpsOhGQRCbxvq6BCecfP1y3xlr0TVvwELx8wlkRwFmZakEgRBSqPNEfRHqKzhtmCUiDJGTdkiJfBCecME5zvWsOLRtJU/aUE2Nz/RL8fhZmL8vfJjbR9ApqM1kIZ4k0Fay3zbEucq4u8BsOYUlvCvLhA1FUOD4iG92WuDpFJp1vLyviO3NfP0GgCDqWp0JDGPUiGRTmhkoxWPRBeILyYXkKz+uKwueYgx+cS X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(133145235818549)(236129657087228)(767451399110)(148574349560750)(247924648384137); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(3002001)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(6055026)(6041248)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY4PR12MB1141; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzQ6eGllK2tBZ2VqanQwNjExWWRkelBDalZ6NmU4?= =?utf-8?B?UDcvVFA4eGxGZGtnbFFPd1pGZUNUTXJBM3JNUW8zbUN5ZnN0eVdEdXJoWUVX?= =?utf-8?B?K1U2aTUxcWVDZjE3ZXZYQW1sK3FCT2RibCtubjM1MVRUaWdBN3plQktXNXRS?= =?utf-8?B?VTA4SWJWMjl6dXIwenFwVXJ3bmxwODdRZk4xdUpIMGRYRTVVWmNFUGJtZm4w?= =?utf-8?B?aHpPVVdBbXRnRGpmMmF0bGZ5ZHJJK1J1UTgzRzZjQ1RJVEhyWjdKcVkwcHdB?= =?utf-8?B?b1Y2L1dMU2FxN2FwcGJLK2I0WVA1T215cW5PdFkzbEFIUzdseEZoT1R3OG92?= =?utf-8?B?QXhYNGFSVERaQXd1N1VjQzRlb2g1eTNhSnVELzdqY0dLQXdROVhLYkEvYkJW?= =?utf-8?B?MCtJYVB0clI3bEdrZmNQd1RHdnhSeUFFRGJBYjVrYmluYVZtYUxmRkN0bmUr?= =?utf-8?B?Z3VUSWtPcjJndTNsV2ZsSmNRRy9NKzFZWHZjOUhZTlh3ZkRxY2licW1UQkx2?= =?utf-8?B?QzNNaElKVFlkZmQ0S2szMU1kN0VhK0wrQTI3UFY4OEdHMXpKQ2tUdjkxdGJp?= =?utf-8?B?a0FWVkVXNVNvRm5pQ2MzSnZmQXVBMmZNYVVXbTEyTjJ5U0tCQmRZRU5jOUhT?= =?utf-8?B?QVZiTVRMVzJHa0ljV3htWGZvWmtrODJMdy9Mb2FxdkljT09HK3JHUzJ3NmpE?= =?utf-8?B?WFlGSWE3LzVrUlZobG5PVDBxTzYrc0cwVithZTJTUFl5MktLcCtGdjVZWDk4?= =?utf-8?B?OTJMSmtwUXpXejFodGNVQ3FRYlNKY2xUaWVGbjJGZzRsdm43aWZWTml0YXl0?= =?utf-8?B?dnZxa0hHL2ZjTjRxRGhkRmRtaVllQUo5Wk1uU1MxSHB0cHpEb01DM3h0ZFoy?= =?utf-8?B?M2dNWHlxcWxNVmlqTUQ4OW5keWhlTU84SVBnQkNOYzRoTWZkSFNDdElra3I4?= =?utf-8?B?V1Q0SllrdmlRN3d6TFNtZTVMdlJucGVrcDZzaWloZGdwVXVRdHIzUWJFWXI2?= =?utf-8?B?WEpiTVVxSXJ2ZjFTMGk5amY0cTRqN2c1ZWgwdUM5K2h5KzIrbENTUlAxMTg1?= =?utf-8?B?eVdCb1lXZERDclMvMDNsQkJQbmtiN1BRVy9uUzNVMWQ0S2xZbjUxN0E1bGs1?= =?utf-8?B?T3dYMkMzRXpJVlZQcnRHSzdUVjF6aFBtb05rU29UODc1NEZkekMxS05LUm8x?= =?utf-8?B?NXUvcDdzT2libTRoVTAzRkdkSVExVEZqZVFJZnNKM3REeVlqZDA1elVWR3ZI?= =?utf-8?B?RWwxOXhOeGRRc1JFazA3cVJHczQrWjExN2hUelBJWU11TDVwUUtWU21hRER2?= =?utf-8?B?YUhXak9vTVNuMEs3bVBYV1VQR1VWc05YWTdRR0tmeDViRHJyTXFHR2Y0TTlH?= =?utf-8?B?Q2NycEpSK01vRjJBU0VIYlFRRnB6Qjd4cFprL3A3djI5bHZHVnBGalFRZE8x?= =?utf-8?B?aG9yYThRZFp6VmhBR0JhS1VsZTZnbU1RT0J1bDh3YXByZFRnb1VMdnJFWEY2?= =?utf-8?B?eFJlbmVHWDdOalcxSmgxYlFvZnVxQWxoL2xGVVlQQUF0eWduSDc3djVZL2pB?= =?utf-8?B?bzAvMzVlNDd3SjdmS05GUmxmSGNTdzh3WStVa0hxVzhxNmFBd0wyVjhmenNo?= =?utf-8?B?TlpLMGRMTVdPRXFmczNIZUxQYWVrcmhERjB6TkhiYlgzcEloUDF4eUg5Sm9S?= =?utf-8?B?S3N0K2lPRCt1WmlpV0tXVTluWS8zM1hObFZWTHIxOG5jNWVkMStiK0ZxV1k1?= =?utf-8?B?blM0Y3l5QUlzV1RvVVUrcDhLeUJEdWwwU1p3RG1UeHQ1cjg4eHBleHZOMXVP?= =?utf-8?Q?/e5xw4qwH6yp?= X-Forefront-PRVS: 0351D213B3 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(39840400002)(39400400002)(39450400003)(39410400002)(39850400002)(39860400002)(103116003)(53416004)(97746001)(47776003)(42186005)(66066001)(50466002)(72206003)(81166006)(33646002)(7736002)(86362001)(7416002)(7406005)(6506006)(5660300001)(305945005)(23676002)(8676002)(53936002)(9686003)(55016002)(2950100002)(6116002)(54906002)(110136004)(38730400002)(76176999)(2906002)(54356999)(83506001)(3846002)(50986999)(4326008)(1076002)(25786009)(189998001)(230700001)(4001350100001)(478600001)(921003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR12MB1141; H:tlendack-t1.amdoffice.net; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzIzOnBkamFmdmZxQXF2VVRkNHZPZDB0Vjh4NDdp?= =?utf-8?B?UEFsdXhBN1hvVUNuRmFxRWpLVnFVSjAyTVdNajZrWkkzS2Q5eEJmUW5IWnd3?= =?utf-8?B?WTBueHExbnpQN1lQUG4vNEpIM0d0eGlJdUpmK3VHUDBabTlMWWY3MXgyaGpp?= =?utf-8?B?TTJDMUtKNzVCbkZrcGR1N0JReFNvSXM5cjZjMG9lM3JFSGdLQW56Z2NZd1ox?= =?utf-8?B?QjNiR3VtYU5sNGtmdnV2L0pIckgvMWtzQkVheVo0VGx0emJVSnQ1b0swY0pH?= =?utf-8?B?emVwM08yYmpOb2VWZkxxbjZwTXFwMFJHRXV4bk14RmFMQWM0alo2TzFjOTlq?= =?utf-8?B?RnEyc3hWcmVaOEpSVG40QVhVL0hkeGcwNCtjYkJIaWhtQnhOSXdaUmhXWDc2?= =?utf-8?B?bXVwdW8rRmJ6NWVxSlVzcFdYSmpCWmd4ZTgra0g4YmFKNjZGUWtrbXA0VXRI?= =?utf-8?B?aGVoWDIzM1pVR3pGQytYY1BsZFlKWTNHVmdEMlpzVEMyZHNpMFFRSy83L0gy?= =?utf-8?B?OVlaZWxMOWpwNDh6RTZBVkRpYmx2alNGY2E3RGdVSnBaTjNSbjU0ZzVPZ3pQ?= =?utf-8?B?V3ZFQ2lnWENyV2JWZzBQaTJDelZteGI2QnkrNEJxRmZvZGwxUEJ3UWxRandz?= =?utf-8?B?c3JCT3FObExMZkpoaUtTcnRvakFoU2w0d3hLejN6Z0laNGtoVHNxdkMwdnla?= =?utf-8?B?RGhXZmNzMjZ1ZFBWK1EweTYvRXlzVnZtWVQvWHd3c3ZxMDB6V016bGZrY0VP?= =?utf-8?B?dWdzcm9RYkxKQ09WMEVMWENVa3YyYUVYdGtWVFNpREZMT1BpakN0Mmc2Sjdv?= =?utf-8?B?U0tZb0NWeHJDVUxFNGI5dm9lMmo0aWV1MDVreTIwY250RENpdlphZkcrazhE?= =?utf-8?B?bmtDaWN0MlZOYXRCMUZwd0U5dnZlV3U3K2FMVk5DS3lKY3l0NEd0QzdaUDQ5?= =?utf-8?B?RWw1TDlhK3htSWpMY3pYdXpTL2tKekNDRDhRV3dzdnE1V1dHanV4b05UWHoz?= =?utf-8?B?UUJBUG9WUFdmLzRPd3BJSWxYc2daT2Zhc2IyS2FEL2o5WkhQQWJmdW9lMzhZ?= =?utf-8?B?WG1ZZi9CNlBHQUIwc29JV3RocHowTGp6S3NKTWtOU2c4YlRrU01VNlFsSmN3?= =?utf-8?B?aWd0eFlOamkwZEtKRHNoM0gwRjUweFZZYVJjN2poR1AvUXJFd1dsaHhUQ0tK?= =?utf-8?B?aUlLZ0FiVERmQzZpWERpUm8zd3hERDhBUVFSR0U0QWdDeEtYSUFMWE94eVQ0?= =?utf-8?B?bklMOTJ3V1lSNWlFc09TTktiMWVjMmxPSU40TmxjVXhBZUVnQ3ZjWVlBdjFP?= =?utf-8?B?ekNZdCt5ZW5zWDlzVm5wTlRsUFR3aDVObTFiVHpNbGttQ3dMTkxENXczRG9P?= =?utf-8?B?NW5XRm8yVHZiSzIzSVNjbXFtMFY1UU53NFJGK3BZUWVsZkNZRmJOaGpYT3pI?= =?utf-8?B?RklHNXZpNEVnL3RrV01LQjUvSjNVYTNUcnI5K1JuclZWalp3b25YSTdBV3Q0?= =?utf-8?B?RUJ0NmFBWEoweXl6M2RBekNMZVNuMEc2b3RrajErRDJjZXJaN3o4QzNONUIz?= =?utf-8?B?VzJXcFlNMHF3emYrWm1KazZkT2ZnWmNGQzdBaHFTYnlWV1pPVGNyRXNscDdS?= =?utf-8?B?UFk4Z1RzQ0lwMklZRGpIQnpiZ1c5L2xneDNsRmFTSWx3ZU5RenNRNWJ6SDFX?= =?utf-8?Q?kTyBPgibgn0GoxVmoY=3D?= X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzY6aEVDMkovS1lwVk5yY09uYStMZEhPRkZyc0xU?= =?utf-8?B?c0MwWHpyaTdPdHovZ0RCWldQSDBSNG9RRE5pVk00OGZneVFxcy8rL0lMb1A1?= =?utf-8?B?WDh4NlJLaytpZ0xzT1lyVnNLcURTc2FOb0F1eUxuNVlvMnpoWUZqSDRxM2FW?= =?utf-8?B?RDh3bDhoR2xuSUhkQk5SbHRYeWphRVdlRGpiQndWUnkzQnZZcis5SDNIbWQ1?= =?utf-8?B?akVHa01yVG9JVzkvK1VjOS9SRUo1aXh1SFdoU01ranFtSno2S3ZCR21Vd1NL?= =?utf-8?B?d0JVc0VDOTVWLzNxOGRMMUZFTnlhMjRSZXIrVG1xdDRiMkd6akpKcVZNa3RI?= =?utf-8?B?Y1B5YyszanhlNmtWbGFwdkJ6Y2NGRGF6UHk2bWJ6N3pJbTVZRXNmeUx5VmdU?= =?utf-8?B?dHRJUjlPb1NaYzFVYzNMaDdrZm41MDh3V3hDNkNxay93cmZsc3dYbEkvNHBu?= =?utf-8?B?QnpUQXBjaDQ0NjViK3A3SHJQL2lDUG5oZllxSDg0aGdXaklOVjBRTVBBUGxs?= =?utf-8?B?TXIxcWVoajBnMjdaL0ZJZ3o5elEyVVhWc05JUUFGVlpjN0t3VHBDOHVMRHR6?= =?utf-8?B?WHZyT2h4d3dRdU5uZmdpVHZGczgybDBRQkZtUS9RM08wYmhXb0RBUTd1eFNU?= =?utf-8?B?SGlnaVlJaGlMMnhMaGROMTdVUEc5anlvdlUyOHl2UGFNUXdic2lCY1BhZmp6?= =?utf-8?B?YmprRHg5bkRISEl5VXZKMVo5c0x5dTFtdW1IbDRqaVJyRHhTaE1DWUFQbFlP?= =?utf-8?B?cHpJNmV4a0oya3dUcHJ1MkluUG9DZFVFZVVtb1FQNmcwNXZsZ2lrZXc3Mlhn?= =?utf-8?B?V1hRancza242eXN0Mi9neVVQTkJSWXltdkJMdEJqeWs2ZHdjQlVqcUtOa01r?= =?utf-8?B?dnFEeVJyUHY2RTF2cllnSzlTQW1BUURUQnlUREtRdXF6VHMyRGRpNjk2NUJV?= =?utf-8?B?aGNKaVh5VGFNZEFER1I5QWljU3NhSnZ4RHUxS201OWRMdENKU0dMM2NPOE5Z?= =?utf-8?B?SGxYOGtnTWVFV0VWRmtBTlF0cVJCQk9BYkhxNE5ycGllblZFOHJsK3pzTUM0?= =?utf-8?B?MnlYLzVnaGlSajJrNkpxVG1kVnlQenJiNEM3WERtbEkyUTZhRHJXUUlPSmhh?= =?utf-8?B?NW9LeGJJeENGdWF4NlZMdGd6Y0I1dVVCbGVrMzFHQ2VVV0UyL2dhOW83SC9y?= =?utf-8?B?MEhwTnZjTVpBWEpQQWFJb3ZjTzRMdVkyVG1jcktTdW9Pd1pVUis4UjZ4NEMw?= =?utf-8?B?S1lNdXpHTmJkMlJYS2N3RjNuZkYxamNuTTdwWEMvMFF6eW9HSkNwY0l0TE01?= =?utf-8?B?eDNNKy9rUlIvbzR4Qnp6dnZFczNiYzZlL3h1b3VqZmY3TlJlUUQ5NkhvWE9k?= =?utf-8?Q?qC6tl7o?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 5:7UXQYFXXLqBvxDjzx3Fl6PCno1Z7OAeUcMYZPajxmJYwVGFOOOxRbWlF8nB9PB3JYc/1aDNHwvIRLU81uRozuO8D/pFX77XoWfDdouREksvR4BSghlqJN3tGKDfS5cJsTB5HQcgq2uonWlgzlRiT6942/Y3MSnA1bW6bHXqL4rZ3MRVJ1GYgu1SicTjghVihTaPEPOajiPyxcujTXhRqPvhJt/EjTubcdYvkeO5Kt+HfHBlrt1JtZLCsH2ftXh0fdyR7E4jd+jJQhJ+sh8u22iS2AIAeHiVcU0OMlftpkwrx9EjTrhg2U7y6ixeSUxWHCH4pd0bcXIdx69g9RJefDWgCEV2xL4KN/iVnBTu2HVSrwEB79Iv9RIF6rXmSnS7XXAKuYbbOEyoIWcM7OfvUMP1XqkXrjHPJSAZ0ief2bMFNcvWl70xY6z0P8tNWDyQPL6O1TSrvhEdi0lO8fd1gLMkFk75UzaPXxfMgEzl0uky+jmDfyepjg0/zStDW83fr; 24:JFrjhfBDmnuFFtUkkQk+MI3Lfu49TzuvqDETlcMJA7eiJePQXWvOsS39A0wsORudiRD/VNrkDdikUhSpIElZ31Ge1+I+GwXT4rsVBko5mP8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 7:j+b2FaYtQ6EV+2jT8/YoLE5A7/sghJQhV6ZOHG9RChOjpf72GsL/TA81TWX16hyH7JWpU+gfCHFa7kQ/SirlHuJsJX7ACR9W8Y5nUcv0kOzLBhDtmyx627ySpydO99gLMXW8Ps2fnHq5+5y9WZ5SNQuFmcYK6Xfp6WLnDLKs6WLHoY+XRS2J98VS4uloDnkT1bcxPZPWJjx2vkEJQtcNoGKFYS8vqLmnds41DzZhBv2A+2/wLiIqr/CcQT9aEcq+98bD2TwUXK/Ib9PUtUWHE0gH9ntJ27IqhevvyT7+q3+UF3B3ls3ZRh7xbVQAwaJK+eDIDirKExBPYczXWiMQVul0gxktsyETROe49a0/4EtJSJrc7Z/S9+ZlnYuSrCh4XWtTPU59xP+QSbLoOKRCzJ6Csa8hfToluD0bD2ZAwA/F05JtRGAgAL8QEoqyHfyprXfR44Tx/7ZwNIX0xWc7Qff/7Q6K0CZk4TDOs4KltV8VSFI6rCpmA5AdEHlwLrYz6cdxpzBH2jATXRJhwf58NAhg0Mf8yI/SdA1wGlIiKRGjsAyB9U7nzsYCLTiMEvTkwhQ9zu3ER4i9TXD3AUeKjjq9EHtZIDt+XMtHHKOFC2IO4QtIXvjRPiBJ3V29Fwi5O8qnA6juoEDU6a7P5/QMCNn+N+O85BLlUiLI7G0Fu7nstah4J4zOVkUmYg9rKoTaqaFX/B0iZyygZGQUEHZLgb4/adEo/Z5NAmOpdobmELE5wkOS4fwhqg2VWeTu8i1Wpzr1T4h6Y3bS6YwonoZb5zV+AH33DzMC/jEAvLpy9hw= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 20:Xi3DLseQ1PjIqOMJbrRnJbsj8FbYcJTKgTpJseLr6TRvssV8/8fX+w9rTNILPgN3DIJzF8EZf8VwAWE8fqvMziE+p1AmlSbzLmGwb9cG8LHvS+1I6fj4h/AGok39yesm7b4qM6vUZkVI5n4n1Hla+66XDji0uts5UEte9D3+68WmUgW8cIXesB7II8iRnFtLZ5rq+TahYfYz9B+LSnvIdr0bdIq9YvT3mtrr3A9AedEuUy/FGxXrjhzaymTGQYif X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2017 15:14:18.3513 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1141 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add the support to encrypt the kernel in-place. This is done by creating new page mappings for the kernel - a decrypted write-protected mapping and an encrypted mapping. The kernel is encrypted by copying it through a temporary buffer. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/mem_encrypt.h | 6 + arch/x86/mm/Makefile | 1 arch/x86/mm/mem_encrypt.c | 310 ++++++++++++++++++++++++++++++++++++ arch/x86/mm/mem_encrypt_boot.S | 150 +++++++++++++++++ 4 files changed, 467 insertions(+) create mode 100644 arch/x86/mm/mem_encrypt_boot.S diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 70e55f6..7122c36 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -21,6 +21,12 @@ extern unsigned long sme_me_mask; +void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr, + unsigned long decrypted_kernel_vaddr, + unsigned long kernel_len, + unsigned long encryption_wa, + unsigned long encryption_pgd); + void __init sme_early_encrypt(resource_size_t paddr, unsigned long size); void __init sme_early_decrypt(resource_size_t paddr, diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index a94a7b6..72bf8c0 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -40,3 +40,4 @@ obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index a7400ec..e5d5439 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -21,6 +21,8 @@ #include #include #include +#include +#include /* * Since SME related variables are set early in the boot process they must @@ -199,8 +201,316 @@ void swiotlb_set_mem_attributes(void *vaddr, unsigned long size) set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT); } +static void __init sme_clear_pgd(pgd_t *pgd_base, unsigned long start, + unsigned long end) +{ + unsigned long pgd_start, pgd_end, pgd_size; + pgd_t *pgd_p; + + pgd_start = start & PGDIR_MASK; + pgd_end = end & PGDIR_MASK; + + pgd_size = (((pgd_end - pgd_start) / PGDIR_SIZE) + 1); + pgd_size *= sizeof(pgd_t); + + pgd_p = pgd_base + pgd_index(start); + + memset(pgd_p, 0, pgd_size); +} + +#define PGD_FLAGS _KERNPG_TABLE_NOENC +#define P4D_FLAGS _KERNPG_TABLE_NOENC +#define PUD_FLAGS _KERNPG_TABLE_NOENC +#define PMD_FLAGS (__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL) + +static void __init *sme_populate_pgd(pgd_t *pgd_base, void *pgtable_area, + unsigned long vaddr, pmdval_t pmd_val) +{ + pgd_t *pgd_p; + p4d_t *p4d_p; + pud_t *pud_p; + pmd_t *pmd_p; + + pgd_p = pgd_base + pgd_index(vaddr); + if (native_pgd_val(*pgd_p)) { + if (IS_ENABLED(CONFIG_X86_5LEVEL)) + p4d_p = (p4d_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); + else + pud_p = (pud_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); + } else { + pgd_t pgd; + + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_p = pgtable_area; + memset(p4d_p, 0, sizeof(*p4d_p) * PTRS_PER_P4D); + pgtable_area += sizeof(*p4d_p) * PTRS_PER_P4D; + + pgd = native_make_pgd((pgdval_t)p4d_p + PGD_FLAGS); + } else { + pud_p = pgtable_area; + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); + pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; + + pgd = native_make_pgd((pgdval_t)pud_p + PGD_FLAGS); + } + native_set_pgd(pgd_p, pgd); + } + + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_p += p4d_index(vaddr); + if (native_p4d_val(*p4d_p)) { + pud_p = (pud_t *)(native_p4d_val(*p4d_p) & ~PTE_FLAGS_MASK); + } else { + p4d_t p4d; + + pud_p = pgtable_area; + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); + pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; + + p4d = native_make_p4d((pudval_t)pud_p + P4D_FLAGS); + native_set_p4d(p4d_p, p4d); + } + } + + pud_p += pud_index(vaddr); + if (native_pud_val(*pud_p)) { + if (native_pud_val(*pud_p) & _PAGE_PSE) + goto out; + + pmd_p = (pmd_t *)(native_pud_val(*pud_p) & ~PTE_FLAGS_MASK); + } else { + pud_t pud; + + pmd_p = pgtable_area; + memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); + pgtable_area += sizeof(*pmd_p) * PTRS_PER_PMD; + + pud = native_make_pud((pmdval_t)pmd_p + PUD_FLAGS); + native_set_pud(pud_p, pud); + } + + pmd_p += pmd_index(vaddr); + if (!native_pmd_val(*pmd_p) || !(native_pmd_val(*pmd_p) & _PAGE_PSE)) + native_set_pmd(pmd_p, native_make_pmd(pmd_val)); + +out: + return pgtable_area; +} + +static unsigned long __init sme_pgtable_calc(unsigned long len) +{ + unsigned long p4d_size, pud_size, pmd_size; + unsigned long total; + + /* + * Perform a relatively simplistic calculation of the pagetable + * entries that are needed. That mappings will be covered by 2MB + * PMD entries so we can conservatively calculate the required + * number of P4D, PUD and PMD structures needed to perform the + * mappings. Incrementing the count for each covers the case where + * the addresses cross entries. + */ + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_size = (ALIGN(len, PGDIR_SIZE) / PGDIR_SIZE) + 1; + p4d_size *= sizeof(p4d_t) * PTRS_PER_P4D; + pud_size = (ALIGN(len, P4D_SIZE) / P4D_SIZE) + 1; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } else { + p4d_size = 0; + pud_size = (ALIGN(len, PGDIR_SIZE) / PGDIR_SIZE) + 1; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } + pmd_size = (ALIGN(len, PUD_SIZE) / PUD_SIZE) + 1; + pmd_size *= sizeof(pmd_t) * PTRS_PER_PMD; + + total = p4d_size + pud_size + pmd_size; + + /* + * Now calculate the added pagetable structures needed to populate + * the new pagetables. + */ + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_size = ALIGN(total, PGDIR_SIZE) / PGDIR_SIZE; + p4d_size *= sizeof(p4d_t) * PTRS_PER_P4D; + pud_size = ALIGN(total, P4D_SIZE) / P4D_SIZE; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } else { + p4d_size = 0; + pud_size = ALIGN(total, PGDIR_SIZE) / PGDIR_SIZE; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } + pmd_size = ALIGN(total, PUD_SIZE) / PUD_SIZE; + pmd_size *= sizeof(pmd_t) * PTRS_PER_PMD; + + total += p4d_size + pud_size + pmd_size; + + return total; +} + void __init sme_encrypt_kernel(void) { + unsigned long workarea_start, workarea_end, workarea_len; + unsigned long execute_start, execute_end, execute_len; + unsigned long kernel_start, kernel_end, kernel_len; + unsigned long pgtable_area_len; + unsigned long paddr, pmd_flags; + unsigned long decrypted_base; + void *pgtable_area; + pgd_t *pgd; + + if (!sme_active()) + return; + + /* + * Prepare for encrypting the kernel by building new pagetables with + * the necessary attributes needed to encrypt the kernel in place. + * + * One range of virtual addresses will map the memory occupied + * by the kernel as encrypted. + * + * Another range of virtual addresses will map the memory occupied + * by the kernel as decrypted and write-protected. + * + * The use of write-protect attribute will prevent any of the + * memory from being cached. + */ + + /* Physical addresses gives us the identity mapped virtual addresses */ + kernel_start = __pa_symbol(_text); + kernel_end = ALIGN(__pa_symbol(_end), PMD_PAGE_SIZE); + kernel_len = kernel_end - kernel_start; + + /* Set the encryption workarea to be immediately after the kernel */ + workarea_start = kernel_end; + + /* + * Calculate required number of workarea bytes needed: + * executable encryption area size: + * stack page (PAGE_SIZE) + * encryption routine page (PAGE_SIZE) + * intermediate copy buffer (PMD_PAGE_SIZE) + * pagetable structures for the encryption of the kernel + * pagetable structures for workarea (in case not currently mapped) + */ + execute_start = workarea_start; + execute_end = execute_start + (PAGE_SIZE * 2) + PMD_PAGE_SIZE; + execute_len = execute_end - execute_start; + + /* + * One PGD for both encrypted and decrypted mappings and a set of + * PUDs and PMDs for each of the encrypted and decrypted mappings. + */ + pgtable_area_len = sizeof(pgd_t) * PTRS_PER_PGD; + pgtable_area_len += sme_pgtable_calc(execute_end - kernel_start) * 2; + + /* PUDs and PMDs needed in the current pagetables for the workarea */ + pgtable_area_len += sme_pgtable_calc(execute_len + pgtable_area_len); + + /* + * The total workarea includes the executable encryption area and + * the pagetable area. + */ + workarea_len = execute_len + pgtable_area_len; + workarea_end = workarea_start + workarea_len; + + /* + * Set the address to the start of where newly created pagetable + * structures (PGDs, PUDs and PMDs) will be allocated. New pagetable + * structures are created when the workarea is added to the current + * pagetables and when the new encrypted and decrypted kernel + * mappings are populated. + */ + pgtable_area = (void *)execute_end; + + /* + * Make sure the current pagetable structure has entries for + * addressing the workarea. + */ + pgd = (pgd_t *)native_read_cr3_pa(); + paddr = workarea_start; + while (paddr < workarea_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + + /* Flush the TLB - no globals so cr3 is enough */ + native_write_cr3(__native_read_cr3()); + + /* + * A new pagetable structure is being built to allow for the kernel + * to be encrypted. It starts with an empty PGD that will then be + * populated with new PUDs and PMDs as the encrypted and decrypted + * kernel mappings are created. + */ + pgd = pgtable_area; + memset(pgd, 0, sizeof(*pgd) * PTRS_PER_PGD); + pgtable_area += sizeof(*pgd) * PTRS_PER_PGD; + + /* Add encrypted kernel (identity) mappings */ + pmd_flags = PMD_FLAGS | _PAGE_ENC; + paddr = kernel_start; + while (paddr < kernel_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* + * A different PGD index/entry must be used to get different + * pagetable entries for the decrypted mapping. Choose the next + * PGD index and convert it to a virtual address to be used as + * the base of the mapping. + */ + decrypted_base = (pgd_index(workarea_end) + 1) & (PTRS_PER_PGD - 1); + decrypted_base <<= PGDIR_SHIFT; + + /* Add decrypted, write-protected kernel (non-identity) mappings */ + pmd_flags = (PMD_FLAGS & ~_PAGE_CACHE_MASK) | (_PAGE_PAT | _PAGE_PWT); + paddr = kernel_start; + while (paddr < kernel_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr + decrypted_base, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* Add decrypted workarea mappings to both kernel mappings */ + paddr = workarea_start; + while (paddr < workarea_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + PMD_FLAGS); + + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr + decrypted_base, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + + /* Perform the encryption */ + sme_encrypt_execute(kernel_start, kernel_start + decrypted_base, + kernel_len, workarea_start, (unsigned long)pgd); + + /* + * At this point we are running encrypted. Remove the mappings for + * the decrypted areas - all that is needed for this is to remove + * the PGD entry/entries. + */ + sme_clear_pgd(pgd, kernel_start + decrypted_base, + kernel_end + decrypted_base); + + sme_clear_pgd(pgd, workarea_start + decrypted_base, + workarea_end + decrypted_base); + + /* Flush the TLB - no globals so cr3 is enough */ + native_write_cr3(__native_read_cr3()); } void __init sme_enable(void) diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S new file mode 100644 index 0000000..7720b00 --- /dev/null +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -0,0 +1,150 @@ +/* + * AMD Memory Encryption Support + * + * Copyright (C) 2016 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include + + .text + .code64 +ENTRY(sme_encrypt_execute) + + /* + * Entry parameters: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - length of kernel + * RCX - virtual address of the encryption workarea, including: + * - stack page (PAGE_SIZE) + * - encryption routine page (PAGE_SIZE) + * - intermediate copy buffer (PMD_PAGE_SIZE) + * R8 - physcial address of the pagetables to use for encryption + */ + + push %rbp + push %r12 + + /* Set up a one page stack in the non-encrypted memory area */ + movq %rsp, %rbp /* Save current stack pointer */ + movq %rcx, %rax /* Workarea stack page */ + movq %rax, %rsp /* Set new stack pointer */ + addq $PAGE_SIZE, %rsp /* Stack grows from the bottom */ + addq $PAGE_SIZE, %rax /* Workarea encryption routine */ + + movq %rdi, %r10 /* Encrypted kernel */ + movq %rsi, %r11 /* Decrypted kernel */ + movq %rdx, %r12 /* Kernel length */ + + /* Copy encryption routine into the workarea */ + movq %rax, %rdi /* Workarea encryption routine */ + leaq __enc_copy(%rip), %rsi /* Encryption routine */ + movq $(.L__enc_copy_end - __enc_copy), %rcx /* Encryption routine length */ + rep movsb + + /* Setup registers for call */ + movq %r10, %rdi /* Encrypted kernel */ + movq %r11, %rsi /* Decrypted kernel */ + movq %r8, %rdx /* Pagetables used for encryption */ + movq %r12, %rcx /* Kernel length */ + movq %rax, %r8 /* Workarea encryption routine */ + addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ + + call *%rax /* Call the encryption routine */ + + movq %rbp, %rsp /* Restore original stack pointer */ + + pop %r12 + pop %rbp + + ret +ENDPROC(sme_encrypt_execute) + +ENTRY(__enc_copy) +/* + * Routine used to encrypt kernel. + * This routine must be run outside of the kernel proper since + * the kernel will be encrypted during the process. So this + * routine is defined here and then copied to an area outside + * of the kernel where it will remain and run decrypted + * during execution. + * + * On entry the registers must be: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - address of the pagetables to use for encryption + * RCX - length of kernel + * R8 - intermediate copy buffer + * + * RAX - points to this routine + * + * The kernel will be encrypted by copying from the non-encrypted + * kernel space to an intermediate buffer and then copying from the + * intermediate buffer back to the encrypted kernel space. The physical + * addresses of the two kernel space mappings are the same which + * results in the kernel being encrypted "in place". + */ + /* Enable the new page tables */ + mov %rdx, %cr3 + + /* Flush any global TLBs */ + mov %cr4, %rdx + andq $~X86_CR4_PGE, %rdx + mov %rdx, %cr4 + orq $X86_CR4_PGE, %rdx + mov %rdx, %cr4 + + /* Set the PAT register PA5 entry to write-protect */ + push %rcx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + push %rdx /* Save original PAT value */ + andl $0xffff00ff, %edx /* Clear PA5 */ + orl $0x00000500, %edx /* Set PA5 to WP */ + wrmsr + pop %rdx /* RDX contains original PAT value */ + pop %rcx + + movq %rcx, %r9 /* Save kernel length */ + movq %rdi, %r10 /* Save encrypted kernel address */ + movq %rsi, %r11 /* Save decrypted kernel address */ + + wbinvd /* Invalidate any cache entries */ + + /* Copy/encrypt 2MB at a time */ +1: + movq %r11, %rsi /* Source - decrypted kernel */ + movq %r8, %rdi /* Dest - intermediate copy buffer */ + movq $PMD_PAGE_SIZE, %rcx /* 2MB length */ + rep movsb + + movq %r8, %rsi /* Source - intermediate copy buffer */ + movq %r10, %rdi /* Dest - encrypted kernel */ + movq $PMD_PAGE_SIZE, %rcx /* 2MB length */ + rep movsb + + addq $PMD_PAGE_SIZE, %r11 + addq $PMD_PAGE_SIZE, %r10 + subq $PMD_PAGE_SIZE, %r9 /* Kernel length decrement */ + jnz 1b /* Kernel length not zero? */ + + /* Restore PAT register */ + push %rdx /* Save original PAT value */ + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + pop %rdx /* Restore original PAT value */ + wrmsr + + ret +.L__enc_copy_end: +ENDPROC(__enc_copy)