From patchwork Tue Apr 18 21:21:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Lendacky X-Patchwork-Id: 9686411 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2F87A602C2 for ; Tue, 18 Apr 2017 21:22:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F7622094F for ; Tue, 18 Apr 2017 21:22:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1372C26E4D; Tue, 18 Apr 2017 21:22:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE4B12094F for ; Tue, 18 Apr 2017 21:22:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758117AbdDRVWP (ORCPT ); Tue, 18 Apr 2017 17:22:15 -0400 Received: from mail-bl2nam02on0087.outbound.protection.outlook.com ([104.47.38.87]:54736 "EHLO NAM02-BL2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1757967AbdDRVWG (ORCPT ); Tue, 18 Apr 2017 17:22:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=adsn/SLy0FIKVpPgHHkF+K3Mlh90UIM+h0Cf/9cjJxQ=; b=DesB9flW6lEynhOJnwXR/tArnbzeWhFwzSgYeV+KrtQDhpj5GN1bwvOt95PBMuBvTifnLuTvJK36IEyN0MLENUtBZqBFz/qzI2au2pw3veuCBCNweR5Y4xC+WlSUXLdb0x0fTL/p6Kqw1xc7FFqhtan7PYftCa1JVFji+tPGwuo= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from tlendack-t1.amdoffice.net (165.204.77.1) by CY4PR12MB1142.namprd12.prod.outlook.com (10.168.163.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1034.10; Tue, 18 Apr 2017 21:21:52 +0000 From: Tom Lendacky Subject: [PATCH v5 29/32] x86/mm: Add support to encrypt the kernel in-place To: , , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Toshimitsu Kani , Arnd Bergmann , Jonathan Corbet , Matt Fleming , "Michael S. Tsirkin" , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Larry Woodman , Brijesh Singh , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , Dave Young , Thomas Gleixner , Dmitry Vyukov Date: Tue, 18 Apr 2017 16:21:49 -0500 Message-ID: <20170418212149.10190.70894.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170418211612.10190.82788.stgit@tlendack-t1.amdoffice.net> References: <20170418211612.10190.82788.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: MWHPR10CA0016.namprd10.prod.outlook.com (10.172.48.26) To CY4PR12MB1142.namprd12.prod.outlook.com (10.168.163.150) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5ae00b7e-f798-41be-2982-08d486a0f080 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(201703131423075)(201703031133081); SRVR:CY4PR12MB1142; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1142; 3:oWlMlYTIc9xNttVjRy9sQRyeABZHD/0XyQEtbqQveltyvmwTKTebOXTCb3UcAjhXXv5z7q7TgvDmDzaRgkijOnXjqYbIv73RxvoNbGuCf255HP3D5W/uFnGk1/RPUJnOXjp5Smj+bqqBtP2nvDH8F/hyPwR0goh1Ra1H98ZreX1XNHELOdolX+rR/UK1VqSRYaqBUBIhoHMhDI6du8GyyEkVqNKT3q8R+GGKaZd65BwEUj4OHysqQNpjVdjJjDhmtVHtP3YY4wVdepxXiLSoP+BU7I5mzSRtu+tX48WuGDItBVdJjxzKh9LXwu3csZjXYG3H8P1j8ugvIp/sgrNh6Idv/aBxBjkuzNoZoBqr9xk=; 25:9lFxeMktf0GS2fnbdjrq4r1WUzJkqOyUYUPCFjkMtW11JrCEWkSXNlp0Y34Gcu5Ccg8qkVOKxnrqQbiakIknrLM83/OS0/1Z3ved5iDpMq35/+cp4ZXxoCySF4nf9q3b5kz32qucqCdM8F6PNtrCrchJ0uvFmDa1b/JlVGn2HhyZZK8mh+nzXGkmk6G4qFw+HAHsg08veQEtWufbdzQu1ecRMblIEaVG4zFxWYYV4ing/PlzwJ4g6Jd+kPLXylk+DNa8caqUs/3AoI1R9OvcS6L6o8YtT8u4G+aQaSy4KppyGYNTBeuiXG77Nd7YgHAOOGIJI7cIJgMpsCFhCrwIQomrHq/OAfdTwwubImaH6VWulxZDNIFeHQzurKva+bzUwWAGbVpr80cWqgGYzznO+uwG74nmB9vWOi8X31h7JjWl4FYYMhtOCEmFNMnY3RbmrrtyJiw84rghsLOOiOU3jg== X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1142; 31:z58SkSTcTGyZp8rFglYsxevB/N0scCEQzPwXAm7tvkxqA4WanYjUnWMGaKrtyuK9XhBbFnQpTaq6zA+YhTeFZ2mRUNeZFwj7+AtGJAbtWAiZdF0JcKlcqvQOOp6L3XWBZUIK9XZ3z9RJrCijLkCPcg8Gyx+p4Ya+StzXXkQuV8/ejUpdD0dcaDuQ9k0uCIBpD0CiPt6ZxHBlR9MgfjsSckrHtr/PkmrORgvb1prykhBdADQsSvCiLezgWfQGSnwG; 20:aKBBuxBz5I9rUlEjqCiTvflnE5uNZsax79tYgiuyy588gGbcyybqOHDyPTQpKFSC9IWNDFiU6T8qQFz94Ld/9oFgzAeNhabcA+Xv9qknNhJgwtfFa1vQQmMYf6gPs3sfYVv8NoZvqjfJuSuGVOBjm63v/So7+o/irXU/mRJSeO7AgnOPEYv7MvJzKwSEjq1Oi5Kg3cdkay8D8B/fkleu2eqT53O7G6HoNu2ckln7EmrB0QcTvSgMMr9w31ncj37pJKaJ74voRLT+QMjxwpx2ELHsQcccjP48j7xJv61/zaYCx8CjrD7WDEBsiugqHcnpeheTKOdDSWTYDWdi77te63WQ8WJM5r0Kc+uU+u6pDk14vOrRf8wdgBZ4D964NYwG6eMixRP7ZOJLvAUhHLmwwtuOvZVjUSmeeL3PvxJvz0sSija34tVgI94VusRV2EJ2uxebP36rDck777JncJvdnoSF+B4cgorqAyicz20TQGVnRGvlUcEMkC5hpLDOK3N4 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(93006095)(93001095)(3002001)(10201501046)(6055026)(6041248)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(20161123560025)(20161123555025)(6072148); SRVR:CY4PR12MB1142; BCL:0; PCL:0; RULEID:; SRVR:CY4PR12MB1142; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1142; 4:ROlflOQkaOSho4Kkd0ZLDqH7ilWYg2uVoyn9icbIFWeJ4CyUMa0/w8kS+wW5MMWxJTP3KqBkmRR+beUbgwwtODEBxvCeUM4hq2yPh1VhUkbAzZ60b6INXbQVykK8isuHkSeuc6qr64icBTUi+YVzHUQWgzbE+Kd0Er+wbAh0zhsgHiEX20WIQCduuW8YSIsxs44B2afA5gAb30GjyH3lBkeBZ213lIBQfzeGWxpfcpTDYBrwxskeDZo6SfRSADsHWL61s/NTw7zhH1AmggCuRwFngIOlcb8RVVQuMC1v8YdE2agoU+yi8zFp5uqYxs1zgg6Iyf4shxOnZelDKroYgPYpnT9I0rSOhughvM5s3oM7fpOskGnc9lpWeSkLlpnF7KlRBpA5jy8cbVSQz8Y+iHBKFfHXcP7YQtcj81SwTiMD7Yyy2RT5aUFAzbPHtkNRo64Y/Svy9+AdCZOpBjmqgy4Etan5Pja16z7TOVFy3HYVbCjzTETYLbo+FJNgV7Wp7Dg5LSOETQqTXJ/qrtSie/ye4OR6l9TJxCjrQhftmJ1pdLSpKUEYsrSTeKsB2RwCDGt42Ke8pC/Rqk+44Lp1WDA7+U6tS6ZAjQk1+tSLd3lnl6gaWEvkZOkvCWycpyfOJKm+qYoo07ZqF682I5OFUcVntUM4TKT4SGETsyaSy0u1XtWjP1FxEq60i7gzyvqordGjy0cydFhhAlY5KpAtDKWz+QHgehD0fgto7vrxrR1WFT6ML3ssSSaiO77VV8Z0H62Mez4dQAebn+9/g4cyIg== X-Forefront-PRVS: 028166BF91 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(39410400002)(39400400002)(39850400002)(39860400002)(39840400002)(39450400003)(33646002)(81166006)(7416002)(5660300001)(8676002)(7406005)(189998001)(50466002)(103116003)(3846002)(2906002)(25786009)(4326008)(23676002)(66066001)(53416004)(6116002)(6506006)(54356999)(9686003)(230700001)(1076002)(50986999)(54906002)(2201001)(53936002)(76176999)(7736002)(305945005)(4001350100001)(83506001)(38730400002)(86362001)(42186005)(2950100002)(6666003)(55016002)(97746001)(47776003)(2004002)(921003)(71626007)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR12MB1142; H:tlendack-t1.amdoffice.net; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQyOzIzOmhRNlQxc1p1UzVkaVZybWlsZnN2YjhZNDU3?= =?utf-8?B?UTJJSlhCbjdvV012T1JPQzNRN3ltRyt1SmpzSGVBVHUrNGxySnF3cUYzR20z?= =?utf-8?B?ZitnVTRrTzBpcE1IcTZXSGdNTk1haEpyREdFZ2lVUGFOd05ObHVtVmhZOTMv?= =?utf-8?B?SXYvZStKUUNPWUZhZFZ4MnAzTkROSGV6a0JzQXZxVnNZanRxbEtqTTRFSFAy?= =?utf-8?B?c0REMWdnV0hqeHcyLzlmSlBhRTZFR0Y2WTFPUU4veVZyL09GOWQrVVhzOTNw?= =?utf-8?B?TkZOZXUwVVp3ZGppNjl5L0I2dzV4NTU4K01yS0pod3QwQ1NpeFlyaEs1K1VP?= =?utf-8?B?ZHJLcERUMHFnVkNxcnZHQ0dQYSsvMW1Pb2xNVWp6N3NmekJNTzBqcUM4Mmlh?= =?utf-8?B?SUhiUmZVNCtRVGVESGc5SU5nQXNvT2FUbTBpdFhLNWx6cVpXZEZRL1FINUNn?= =?utf-8?B?ZE5kVWZaRWNJcXg4dDlwT2Qwclo5Njk5UVBTanZrZTNGdVhlYm5KcWMvUm8r?= =?utf-8?B?WWpQYlBZazRNU0lPcnhwY3loZGFOazhYb09NWVErZW5td2o5cVpzSGJJamJt?= =?utf-8?B?R3NkeGRKaGgxc1ZXdk9ZaEdpMGtZSUE1ckE0eVl0RVVKdnZacjZBbnhXM3Jq?= =?utf-8?B?QmJhRzdyZ0NFWXdnOGQ5Ykt3TUJyRVRsN3oxUW9XTkRrZ2JiajBPMFdUYUNI?= =?utf-8?B?a0xraGpHNkJkZExYWWtBQWU0aFhTaFMzTk5WM1NOcG9vZlhUK2FlM0lyNGtq?= =?utf-8?B?RWNBVVN6bWQ3QWhzOE9qbEVnUHJ4RFBCcGo1T0JHR3R1TEYrWVpsenY0VG5B?= =?utf-8?B?RjYxeGJzWDYrUDFYZ2EvZnczTFhPTUJuUUt2NWFEeFdxd29YUStteHNrOHls?= =?utf-8?B?UjE2eVlrbXcyTjVYc1ROV3YzV1hLcVBIcjk4bXdPNHhCNFlLY1F6bWp6dWMz?= =?utf-8?B?UE9oZ0wyS2ZHTzZmbHpDcVJNNXgra2k5a3dHL2xMZEc3RG1ZdEJWUmVnWUxF?= =?utf-8?B?UytXUEtaekUya2xLK3BLY1ZOMEIxSGJQbFJPYVFHSnJCWnVKT29XaUpxUWNZ?= =?utf-8?B?c0V6RTMyc245U2hrWEpjQVcvUXRHMWNZbW9KMTNNTUxMZEcvV1gxN0FyUnM1?= =?utf-8?B?eTBYcXVRWkd2WWcwdnd0bU5BUjVrVGMxRDJkZzhoZmVuNzZuWjdxanVrSkxG?= =?utf-8?B?YUFRYWRhUmZGaVRsSkt1N05WY0VuUmtneGxLcVRsN3hraDJoRk1yMFp1MWNy?= =?utf-8?B?ZlN1eWNHQmJyUkM0RDF3YW9LNTRVN2JuNmU4U2JYd2FEeFFuWnZKWGxSQkdE?= =?utf-8?B?LzZkNjdvMWt2Vzd3Ykw2ekVlWUZ3Y3pSM2hVSzlBUjBaZXJpZlhZTDVzaUN0?= =?utf-8?B?UUFnbnJjbE1yY05NQUt2eFhPNlZpMnhRRW5wNHBHSHF4L2lBVWZyVmJhdVVX?= =?utf-8?B?THUzNFdQOEFZaHpjSkVLQjNBNUZUYmExUXV5SFF4U0F1cHhOTWtNeW9MY1ZR?= =?utf-8?B?TElBZUlLck03VWdFL2dHZFdmTWU5ZXNqbEVEVWJZOWJOK01LN1doVi9kd3dK?= =?utf-8?B?Rmc4MWQxSFZTZ251VTBNcW1xV0pIVWpBT0RLdFhaZElPL0hWZDVsVkhwSkd4?= =?utf-8?B?SDlMT2ZJWkNqWTNXbUdnblJkQTRyMkhwVk41RTUySG9xekxRQ1VnWWc0cm1C?= =?utf-8?Q?/olb1grKv0sb4zHUZY=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1142; 6:keusiv2yW7cDup4L/PA0YVeYDbswVuvBBdi5nAojzl70+R83cCR7D5h00LDOgZHW/jcmKBwRzSJSEiQE6HPYKpxE80pst/nPizumNQztkr1BxZNEqeWLU+cawBtopfI+4v0J7cbqtxPOU4XU2QQWg7w78m+1z0BWEvmbfi6Fe6HGi4AyKTTM8seRic+ld4EoNOSu9mo6X4liub21AkKV7AnDz3MNgKRuoQNjGKUNDzO3Q1vM/iwRV6qQiTG9q5gNHm4pSg1PU4liM6265e2QDt1NvhabEjxazsIszd8ZL/xlMt3rsSOzIV2zu8nSLbmHR4Ln3FoVFhfV2VmSOTWEWmeRwdZEAv3uw4eWKIDr6Gdd+6BcP3z9H3S3ojt55eUm4WFtb4kX5EUFRVhbaXsyEOoD1+yE8mAjKuA+Qa68AORYa1/PKH1HkINecVeGX5f2Hkk3ber772/RxvF0NzCwrB0eh6WSmG8gqyKlAKN9qvs=; 5:GvCmwGcmB/p2/r6L7db8W85g+I26vDYTHnxBwWA9Mbvx5sjiqfiqSh9BrUu3yyNYTKnp25Pp2BEY4R1xlj13+koc4a8PKA79slwJZ4X6qfHUCR4k45MsZm9w9a9E3rVClI7oOapQ9eSPcUOFbad6hQ==; 24:/sZlQxb3jyuKpzwmM6NOMyKu++MDgzUEYc3NbKpjnLftmqCBDHCSn1LUHA8IWDgbmKxX2WkErEMTXVUe4kWY4F7k9gKakp/inOVQEsP2GZY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1142; 7:gv5YP5YzGcAMfbfv9ErzUBJyYcDfhKrAqBtNApuD8jiJgveM0h5Tq5VDpvWopAOR8k080gfpMVyanDDt+Eg0t8yyairuhlpR7jkxUm1ceT4p5IhHRAgmIGMLc7LiB6OJrRMlzZ2rVFb926RihEXjk4Co30TiTZVpt6/hvh7TyQocYxrYf0i3GjPKKJSs2//Qri0MgsV+26SlQEzEeFzE+OWF+pBdR5yfqcIfgrqCjSp0KuMWV0ueyLudtB9SIpWcRMDmA5Pa3/S2Pf5LXHIyQnmysQ2ZZuBXBzBOCWq2nGyfTApKw0yIG0BwIlwFMfaaBL5icX8Juw7PwRTBH7PlmQ==; 20:DrOsM2/J8cGcSEWRnXiMt8uMOraKSP8TWPija6WzEfKxZ9Mgpd95x7sA6+40JFF5Brz73SkyOhlAGepi5VsHi+Qj1tox8WpeneaWx2Fw0o6Ko58XauLalJow7ps7cWjUlkHE6FyBmQCPAVBrSKLvogF+FgirdGltdpWvWxJ2EAO9QnqO+Z9i3GxXnut4YxEAS9/8+78ww0+rYFeoxdgut5/yQuVK6W63GRd4z0W3MxtU/ASZ8bCZJ7NyW0bdd72U X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Apr 2017 21:21:52.7611 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1142 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add the support to encrypt the kernel in-place. This is done by creating new page mappings for the kernel - a decrypted write-protected mapping and an encrypted mapping. The kernel is encrypted by copying it through a temporary buffer. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/mem_encrypt.h | 6 + arch/x86/mm/Makefile | 2 arch/x86/mm/mem_encrypt.c | 262 ++++++++++++++++++++++++++++++++++++ arch/x86/mm/mem_encrypt_boot.S | 151 +++++++++++++++++++++ 4 files changed, 421 insertions(+) create mode 100644 arch/x86/mm/mem_encrypt_boot.S diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b406df2..8f6f9b4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -31,6 +31,12 @@ static inline u64 sme_dma_mask(void) return ((u64)sme_me_mask << 1) - 1; } +void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr, + unsigned long decrypted_kernel_vaddr, + unsigned long kernel_len, + unsigned long encryption_wa, + unsigned long encryption_pgd); + void __init sme_early_encrypt(resource_size_t paddr, unsigned long size); void __init sme_early_decrypt(resource_size_t paddr, diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 9e13841..0633142 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -38,3 +38,5 @@ obj-$(CONFIG_NUMA_EMU) += numa_emulation.o obj-$(CONFIG_X86_INTEL_MPX) += mpx.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o + +obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 30b07a3..0ff41a4 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -24,6 +24,7 @@ #include #include #include +#include /* * Since SME related variables are set early in the boot process they must @@ -216,8 +217,269 @@ void swiotlb_set_mem_attributes(void *vaddr, unsigned long size) set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT); } +void __init sme_clear_pgd(pgd_t *pgd_base, unsigned long start, + unsigned long end) +{ + unsigned long addr = start; + pgdval_t *pgd_p; + + while (addr < end) { + unsigned long pgd_end; + + pgd_end = (addr & PGDIR_MASK) + PGDIR_SIZE; + if (pgd_end > end) + pgd_end = end; + + pgd_p = (pgdval_t *)pgd_base + pgd_index(addr); + *pgd_p = 0; + + addr = pgd_end; + } +} + +#define PGD_FLAGS _KERNPG_TABLE_NOENC +#define PUD_FLAGS _KERNPG_TABLE_NOENC +#define PMD_FLAGS (__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL) + +static void __init *sme_populate_pgd(pgd_t *pgd_base, void *pgtable_area, + unsigned long vaddr, pmdval_t pmd_val) +{ + pgdval_t pgd, *pgd_p; + pudval_t pud, *pud_p; + pmdval_t pmd, *pmd_p; + + pgd_p = (pgdval_t *)pgd_base + pgd_index(vaddr); + pgd = *pgd_p; + if (pgd) { + pud_p = (pudval_t *)(pgd & ~PTE_FLAGS_MASK); + } else { + pud_p = pgtable_area; + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); + pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; + + *pgd_p = (pgdval_t)pud_p + PGD_FLAGS; + } + + pud_p += pud_index(vaddr); + pud = *pud_p; + if (pud) { + if (pud & _PAGE_PSE) + goto out; + + pmd_p = (pmdval_t *)(pud & ~PTE_FLAGS_MASK); + } else { + pmd_p = pgtable_area; + memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); + pgtable_area += sizeof(*pmd_p) * PTRS_PER_PMD; + + *pud_p = (pudval_t)pmd_p + PUD_FLAGS; + } + + pmd_p += pmd_index(vaddr); + pmd = *pmd_p; + if (!pmd || !(pmd & _PAGE_PSE)) + *pmd_p = pmd_val; + +out: + return pgtable_area; +} + +static unsigned long __init sme_pgtable_calc(unsigned long len) +{ + unsigned long pud_tables, pmd_tables; + unsigned long total = 0; + + /* + * Perform a relatively simplistic calculation of the pagetable + * entries that are needed. That mappings will be covered by 2MB + * PMD entries so we can conservatively calculate the required + * number of PUD and PMD structures needed to perform the mappings. + * Incrementing the count for each covers the case where the + * addresses cross entries. + */ + pud_tables = ALIGN(len, PGDIR_SIZE) / PGDIR_SIZE; + pud_tables++; + pmd_tables = ALIGN(len, PUD_SIZE) / PUD_SIZE; + pmd_tables++; + + total += pud_tables * sizeof(pud_t) * PTRS_PER_PUD; + total += pmd_tables * sizeof(pmd_t) * PTRS_PER_PMD; + + /* + * Now calculate the added pagetable structures needed to populate + * the new pagetables. + */ + pud_tables = ALIGN(total, PGDIR_SIZE) / PGDIR_SIZE; + pmd_tables = ALIGN(total, PUD_SIZE) / PUD_SIZE; + + total += pud_tables * sizeof(pud_t) * PTRS_PER_PUD; + total += pmd_tables * sizeof(pmd_t) * PTRS_PER_PMD; + + return total; +} + void __init sme_encrypt_kernel(void) { + pgd_t *pgd; + void *pgtable_area; + unsigned long kernel_start, kernel_end, kernel_len; + unsigned long workarea_start, workarea_end, workarea_len; + unsigned long execute_start, execute_end, execute_len; + unsigned long pgtable_area_len; + unsigned long decrypted_base; + unsigned long paddr, pmd_flags; + + if (!sme_active()) + return; + + /* + * Prepare for encrypting the kernel by building new pagetables with + * the necessary attributes needed to encrypt the kernel in place. + * + * One range of virtual addresses will map the memory occupied + * by the kernel as encrypted. + * + * Another range of virtual addresses will map the memory occupied + * by the kernel as decrypted and write-protected. + * + * The use of write-protect attribute will prevent any of the + * memory from being cached. + */ + + /* Physical addresses gives us the identity mapped virtual addresses */ + kernel_start = __pa_symbol(_text); + kernel_end = ALIGN(__pa_symbol(_end), PMD_PAGE_SIZE); + kernel_len = kernel_end - kernel_start; + + /* Set the encryption workarea to be immediately after the kernel */ + workarea_start = kernel_end; + + /* + * Calculate required number of workarea bytes needed: + * executable encryption area size: + * stack page (PAGE_SIZE) + * encryption routine page (PAGE_SIZE) + * intermediate copy buffer (PMD_PAGE_SIZE) + * pagetable structures for the encryption of the kernel + * pagetable structures for workarea (in case not currently mapped) + */ + execute_start = workarea_start; + execute_end = execute_start + (PAGE_SIZE * 2) + PMD_PAGE_SIZE; + execute_len = execute_end - execute_start; + + /* + * One PGD for both encrypted and decrypted mappings and a set of + * PUDs and PMDs for each of the encrypted and decrypted mappings. + */ + pgtable_area_len = sizeof(pgd_t) * PTRS_PER_PGD; + pgtable_area_len += sme_pgtable_calc(execute_end - kernel_start) * 2; + + /* PUDs and PMDs needed in the current pagetables for the workarea */ + pgtable_area_len += sme_pgtable_calc(execute_len + pgtable_area_len); + + /* + * The total workarea includes the executable encryption area and + * the pagetable area. + */ + workarea_len = execute_len + pgtable_area_len; + workarea_end = workarea_start + workarea_len; + + /* + * Set the address to the start of where newly created pagetable + * structures (PGDs, PUDs and PMDs) will be allocated. New pagetable + * structures are created when the workarea is added to the current + * pagetables and when the new encrypted and decrypted kernel + * mappings are populated. + */ + pgtable_area = (void *)execute_end; + + /* + * Make sure the current pagetable structure has entries for + * addressing the workarea. + */ + pgd = (pgd_t *)native_read_cr3(); + paddr = workarea_start; + while (paddr < workarea_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + native_write_cr3((unsigned long)pgd); + + /* + * A new pagetable structure is being built to allow for the kernel + * to be encrypted. It starts with an empty PGD that will then be + * populated with new PUDs and PMDs as the encrypted and decrypted + * kernel mappings are created. + */ + pgd = pgtable_area; + memset(pgd, 0, sizeof(*pgd) * PTRS_PER_PGD); + pgtable_area += sizeof(*pgd) * PTRS_PER_PGD; + + /* Add encrypted kernel (identity) mappings */ + pmd_flags = PMD_FLAGS | _PAGE_ENC; + paddr = kernel_start; + while (paddr < kernel_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* + * A different PGD index/entry must be used to get different + * pagetable entries for the decrypted mapping. Choose the next + * PGD index and convert it to a virtual address to be used as + * the base of the mapping. + */ + decrypted_base = (pgd_index(workarea_end) + 1) & (PTRS_PER_PGD - 1); + decrypted_base <<= PGDIR_SHIFT; + + /* Add decrypted, write-protected kernel (non-identity) mappings */ + pmd_flags = (PMD_FLAGS & ~_PAGE_CACHE_MASK) | (_PAGE_PAT | _PAGE_PWT); + paddr = kernel_start; + while (paddr < kernel_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr + decrypted_base, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* Add decrypted workarea mappings to both kernel mappings */ + paddr = workarea_start; + while (paddr < workarea_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + PMD_FLAGS); + + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr + decrypted_base, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + + /* Perform the encryption */ + sme_encrypt_execute(kernel_start, kernel_start + decrypted_base, + kernel_len, workarea_start, (unsigned long)pgd); + + /* + * At this point we are running encrypted. Remove the mappings for + * the decrypted areas - all that is needed for this is to remove + * the PGD entry/entries. + */ + sme_clear_pgd(pgd, kernel_start + decrypted_base, + kernel_end + decrypted_base); + + sme_clear_pgd(pgd, workarea_start + decrypted_base, + workarea_end + decrypted_base); + + /* Flush the TLB - no globals so cr3 is enough */ + native_write_cr3(native_read_cr3()); } unsigned long __init sme_enable(void) diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S new file mode 100644 index 0000000..fb58f9f --- /dev/null +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -0,0 +1,151 @@ +/* + * AMD Memory Encryption Support + * + * Copyright (C) 2016 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include + + .text + .code64 +ENTRY(sme_encrypt_execute) + + /* + * Entry parameters: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - length of kernel + * RCX - virtual address of the encryption workarea, including: + * - stack page (PAGE_SIZE) + * - encryption routine page (PAGE_SIZE) + * - intermediate copy buffer (PMD_PAGE_SIZE) + * R8 - physcial address of the pagetables to use for encryption + */ + + push %rbp + push %r12 + + /* Set up a one page stack in the non-encrypted memory area */ + movq %rsp, %rbp /* Save current stack pointer */ + movq %rcx, %rax /* Workarea stack page */ + movq %rax, %rsp /* Set new stack pointer */ + addq $PAGE_SIZE, %rsp /* Stack grows from the bottom */ + addq $PAGE_SIZE, %rax /* Workarea encryption routine */ + + movq %rdi, %r10 /* Encrypted kernel */ + movq %rsi, %r11 /* Decrypted kernel */ + movq %rdx, %r12 /* Kernel length */ + + /* Copy encryption routine into the workarea */ + movq %rax, %rdi /* Workarea encryption routine */ + leaq .Lenc_start(%rip), %rsi /* Encryption routine */ + movq $(.Lenc_stop - .Lenc_start), %rcx /* Encryption routine length */ + rep movsb + + /* Setup registers for call */ + movq %r10, %rdi /* Encrypted kernel */ + movq %r11, %rsi /* Decrypted kernel */ + movq %r8, %rdx /* Pagetables used for encryption */ + movq %r12, %rcx /* Kernel length */ + movq %rax, %r8 /* Workarea encryption routine */ + addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ + + call *%rax /* Call the encryption routine */ + + movq %rbp, %rsp /* Restore original stack pointer */ + + pop %r12 + pop %rbp + + ret +ENDPROC(sme_encrypt_execute) + +.Lenc_start: +ENTRY(sme_enc_routine) +/* + * Routine used to encrypt kernel. + * This routine must be run outside of the kernel proper since + * the kernel will be encrypted during the process. So this + * routine is defined here and then copied to an area outside + * of the kernel where it will remain and run decrypted + * during execution. + * + * On entry the registers must be: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - address of the pagetables to use for encryption + * RCX - length of kernel + * R8 - intermediate copy buffer + * + * RAX - points to this routine + * + * The kernel will be encrypted by copying from the non-encrypted + * kernel space to an intermediate buffer and then copying from the + * intermediate buffer back to the encrypted kernel space. The physical + * addresses of the two kernel space mappings are the same which + * results in the kernel being encrypted "in place". + */ + /* Enable the new page tables */ + mov %rdx, %cr3 + + /* Flush any global TLBs */ + mov %cr4, %rdx + andq $~X86_CR4_PGE, %rdx + mov %rdx, %cr4 + orq $X86_CR4_PGE, %rdx + mov %rdx, %cr4 + + /* Set the PAT register PA5 entry to write-protect */ + push %rcx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + push %rdx /* Save original PAT value */ + andl $0xffff00ff, %edx /* Clear PA5 */ + orl $0x00000500, %edx /* Set PA5 to WP */ + wrmsr + pop %rdx /* RDX contains original PAT value */ + pop %rcx + + movq %rcx, %r9 /* Save kernel length */ + movq %rdi, %r10 /* Save encrypted kernel address */ + movq %rsi, %r11 /* Save decrypted kernel address */ + + wbinvd /* Invalidate any cache entries */ + + /* Copy/encrypt 2MB at a time */ +1: + movq %r11, %rsi /* Source - decrypted kernel */ + movq %r8, %rdi /* Dest - intermediate copy buffer */ + movq $PMD_PAGE_SIZE, %rcx /* 2MB length */ + rep movsb + + movq %r8, %rsi /* Source - intermediate copy buffer */ + movq %r10, %rdi /* Dest - encrypted kernel */ + movq $PMD_PAGE_SIZE, %rcx /* 2MB length */ + rep movsb + + addq $PMD_PAGE_SIZE, %r11 + addq $PMD_PAGE_SIZE, %r10 + subq $PMD_PAGE_SIZE, %r9 /* Kernel length decrement */ + jnz 1b /* Kernel length not zero? */ + + /* Restore PAT register */ + push %rdx /* Save original PAT value */ + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + pop %rdx /* Restore original PAT value */ + wrmsr + + ret +ENDPROC(sme_enc_routine) +.Lenc_stop: