From patchwork Thu Nov 10 00:36:10 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Lendacky X-Patchwork-Id: 9420441 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 056496048E for ; Thu, 10 Nov 2016 00:53:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E50CC2914F for ; Thu, 10 Nov 2016 00:53:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D96152939E; Thu, 10 Nov 2016 00:53:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C80C2914F for ; Thu, 10 Nov 2016 00:53:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754662AbcKJAw1 (ORCPT ); Wed, 9 Nov 2016 19:52:27 -0500 Received: from mail-bl2nam02on0079.outbound.protection.outlook.com ([104.47.38.79]:50208 "EHLO NAM02-BL2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752796AbcKJAwV (ORCPT ); Wed, 9 Nov 2016 19:52:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ZnOu1ZVLQmvx193lUJjhIbcSfndebFFXryogSpKJxjU=; b=AIYQcwcfvsdPvA9dLlKKOurFlYfVaCCXUWSnID9+CeBN1u/1wkzzKZoVBXShuu9eQBSxxowTryvZlJU98ql2ZVFdA9ECuAND/ElBO8nu62PU/m5iQcK0zfunRiCPrDdvZxx1BQ2R4VL7Uzbtdi1z4ucyqHkatMvpS8302oATYRw= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; Received: from tlendack-t1.amdoffice.net (165.204.77.1) by CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.6; Thu, 10 Nov 2016 00:36:13 +0000 From: Tom Lendacky Subject: [RFC PATCH v3 08/20] x86: Add support for early encryption/decryption of memory To: , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , "Paolo Bonzini" , Larry Woodman , "Ingo Molnar" , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov Date: Wed, 9 Nov 2016 18:36:10 -0600 Message-ID: <20161110003610.3280.22043.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> References: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: CY1PR03CA0004.namprd03.prod.outlook.com (10.174.128.14) To CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) X-MS-Office365-Filtering-Correlation-Id: a44f8cf3-3817-4b5d-f853-08d4090193ef X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 2:uF/AgZWsrY/68VgIQuUFYxdvYDisVo6RRMA97g5FeSDb322E4PwUHKOkHqUZLbKSdKYNGJ215QzxHwU5qqM1ZOIKkSGv4to+9Qnv/zR9WR5LsaymeBicYRnt5d0R5FNi0j6Yntrrr/qi8E6yDyp2EIjaNj3n82zfejepaynD1WZAJBuJpyl3iTZTWyJu0Yy9eax6CMLW2BV7kt5gJvsJhg==; 3:dC9LU1uAN2aLZZaG9p4mnCFrD8uRIPrAsrPYLB9jrILTSt7tQroTM+LRb9ao9t+hJA63gc6bupmBqTnkzYUS5AlT5IDSpFVYBwV60+G54RDeDD/BSsa6a4RcgTmhsAgQG8VV32wxj5JPsBj/vWLY6w== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 25:OJyv1TBGCZ6E0JGrlbNsIvgW5tPHXcRQjWHY1hrxq6tbEcaK0RvHoTSRM+HY26JyNwM+kfPrpemrOJ8FHX6YKYJnRkpPRB4W7eZr4YVfy/w6720VLsSBNvxU9O7FVgkR5gCafPPk0h8mhxudtKmMDhMQaMRkCoUZ6iOZOz4SzL3CWYCoBFgr0IZhbNmrPRlpfgS/F71aS3No9NZm5Wo4nMnl3ctPyu6k0VAOima87MoSYRV1SyMcTyiJRZyrcvgzUfBI4VgDh6BELyx2NJ5vqaqF8BjOHIw+xZ2FDQdULyB+AP/8fTU9KQyuyA5vzz8fn1F0KHv3ZX1Xc/yZ0l6eSThRHyevU9g2rIrmAizddRSKivVx9AiPSxTSEGSZlL1G9qdJksceaV6iy+SHFtrlRvNYORLhR2mrtrDwyFcnCxMxpDQJkcIz/qC1p+ZqcTjdQwx1xrrUtXBBySJOstqj4KHjVptvGXS6guB48s5OEgg3D4PFf4GbRitAImdOHzuU8DaQe+dDnxIiPphXOULkRq+6EQTPZVcFzqR1GdZNrHpEqWDSs+wEoxiU+eucLVuxAs6z5ZJMt0d/QkseQqr90OCIm9N7ZebergE0/vSzlwGYBhVUnq3v7E6vuKcCmr5+I5Wd6OMAKl2MsUfjZsiJA/cAgXODf2JpizXBMJGmPmSu1BvIZ1yohYANFy0dOk8jKUOI+9KOIVxpAIQc396Wpw== X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 31:hg/3CZ6seDdi7GJzgHHcYOW/5kL0MZQZQC6zr6XFXK22geKB1Dr9b0qRqU7kHQHQlaa4kLGewM2M/Jou35r0//yrJddwgYek90D1XhWIcUc/7Uh6FJ7yxA8yztUCnqV98nHYCoCAQRRlORzvHmyqzhKsYxkNfZL2Z1RJ0MbVhjcfwwKmf4iS8usAQrRBUaHmGgwVJ55SvJAyckKYHStaVyZyr5tZQvkXmAEfp9gXJLHb+CYYDp+SFJcYAM+5RDjzPxLVOZYF/DE02U8CmwR6HQ==; 20:VG8Tw2zKqbscoZs2Y4HoceBgEZLjvWysSYG0JBDPDP486IdoQGIy04ERbrUGfQvROTOB+POAFLdpVh/3QLX+lX4uMTOBgS4ViDq0i1yt5xg58Pv4ijINtZDedf47s5hJ4ODJL5o8OGzPKcewVgGbjJppuYE68y7iGvTQ4bR6FcRW9DW/WbglgRGbZYdby8qBT58F8N66dMJ4qhH6D++ZQjhrhCM3ZK+mLIgpKgCvuo61LyfxwKELQJ5A6gez8J4UOxwqgAYtIzleyxNGIwNwaNzTgWLJLwG2xn5xzEheAlwqHxJ2AK+TkXN7NT73GIR003XQNqt1iJWzTIl62LJoZNPemjOv+9bzkgIX1aOZRN1VEF+bs3Fcjx0uwwAgmNCxlckIrgUp9s/4a9XnlooOfLsv54oyA6RA2BCKyZ/jN9whqplIV49OUfSb7vcYnFMLBo9vj4ZEvlAjqJPEX6O5CXoTHAFL9i0CWVC9cP245sOOMWWmXI3YhPpfCa9Xretv X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026); SRVR:CY4PR12MB1141; BCL:0; PCL:0; RULEID:; SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 4:czak/qwX8DDWhO4CybV9O62KaWN/MAS0OQCyT0TeQHa7gK540yafbkdosmDf8IwnB8ECI4J5hj0IfipQMXP3SLqWjuRtITxpdK58RnvHn4AU7WqurpYP5t//gp/WebNXB1evLu3G72SLeqUXjDeiBzVPTq/dc8EIQsnOep1wlICFkZYr79cwJyaQU5srHCgYSM3uDz/Vx3tU0jYo9iJKgTj5MZtVMxNkFwnhiyY0uo/emxU/lPt7pR2hRAeC0kz3GFLleYndGQH6J3HMLLVsidzkPIl0YfrXcX7TFwpHWIDnFlbtE1cV6XeoXXhibtDQbAO8NKmdMgggKcaDaDjqmx7rZ5bd8u6sJIg5Vi8d+9zygB2VzsSD1RegE/m98SoQuNwRR1B91NHHQuZ5INxldTCgA5C1QslJG9qsUmIsI/E+kilCRHAJhTwhj1aQWXZLp6ZHs8sXUGRZy15qo5S+HQ== X-Forefront-PRVS: 01221E3973 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(189002)(103116003)(92566002)(47776003)(23676002)(8676002)(7736002)(69596002)(50466002)(189998001)(81166006)(66066001)(9686002)(305945005)(5660300001)(77096005)(7846002)(101416001)(86362001)(2201001)(2950100002)(5001770100001)(50986999)(4001350100001)(76176999)(54356999)(97736004)(1076002)(81156014)(53416004)(105586002)(42186005)(4326007)(83506001)(7416002)(97746001)(6116002)(230700001)(3846002)(586003)(68736007)(106356001)(33646002)(2906002)(71626007)(217873001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR12MB1141; H:tlendack-t1.amdoffice.net; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzIzOk4zRXpiRlBhS21jRFdVVUNVOHQxaEhzQXZ4?= =?utf-8?B?REtKSUlhUkpEV2MySmdoaEkxb0k3Mkh3U3QxdTBEZW9JTktHT3h4Sk1vcjIz?= =?utf-8?B?NU8veHE3c0hIT242K2gvUVdibkRWR1VGRElNMG5iZHJXMjRzUHRKcVhHRVBh?= =?utf-8?B?aE1QRGFLUVd5MlNhMDBmNVhyYmswQWtJWVJZYUc2SERuQjl3cVkvQy9UVnBC?= =?utf-8?B?dHRhVEFIT0kwSzlZd1U1TkhHTlQ5VUxnMHVQMDRWb3h4c04wUUlHRXNRZ2cy?= =?utf-8?B?eWdiUml2djNYSG1mNFk5MWJXekF5Q0I3TDZXZzB6TDdHUXczbXhMaHl6Q0NH?= =?utf-8?B?cEErb3htSFgwNkpUaUFBVExuS0EyVDRWdXhuTWFtM2h2NlBoSFNFWko0TFRn?= =?utf-8?B?ckRnT0VWdTlMcm9rVUdxQ3lqN2FBYWtQSWVQbk5qVnVXRDFFRFpXQ1lOVmN6?= =?utf-8?B?QjIreXFSd3Fyb3hsbW1mZlppU2J2a0NlSnFDb09hVHRhb1lueWFMTE1ENmw3?= =?utf-8?B?UHRJT2kwdXFKaitsRmtvNXNQVm5vdzRnSnJLblA5ZFJka0sxQ2Yxb0dCdXN5?= =?utf-8?B?eUNWdXE2MXp5Rkp3K1k3U0FkaHdIY2VnZ2xJNDRaV1NCWDFYSVIrREhYa2xE?= =?utf-8?B?T1p6cXB4bytoVVphQ05QaWZNTVYrZDBFT21VZXhLN1IwUlBESm1yOVdyc0Nw?= =?utf-8?B?WWEyT21wVnhadkMvdElLQ2U5MnhCNXY4TkhiZU40Z05BbjdPeTFLbW5hUGFw?= =?utf-8?B?N0JZUzFBMUVLcHp5dDBpMEl1Si9UeUNsVWlTV2lzS2VScEdqeEVnY2tCcEdE?= =?utf-8?B?eXY4NG9YZUZlMFJaV05qaTR3OVpWSVA2Y2VSVmlHbkx2VnJsTEpXdTNMR01E?= =?utf-8?B?bzJCZk5SVEVXNExZSXJSTGlsRW1oejVaTzQrU05jWSthWWpLMnd6VGUvVGhO?= =?utf-8?B?VnBkVTlna3IxMGJaTm9RdVVJSVZFOXNPWHY5RCtJU2t1NDFVSWtYdm02cUNt?= =?utf-8?B?UGNSZkI5bklSVnY0ZUVnRmlJd1pjVTZUS0dzc1dIR0dIN2N4SEMxL3BiYkhU?= =?utf-8?B?c0FJY1FWaEZlaXYwSWxyZXgvNUs3TVF3YnFPVENlVEd2a1RBc0kwK0ZCTHFV?= =?utf-8?B?aVZ3SXp3alVNcTRxdDR0Nk0vNURwaVE2K3kvZlh4Vzh0dU1tbDBHUFh1YmhF?= =?utf-8?B?SFJnRHhNWG5ibWxYZnYrd2s0Z1U2Um1YbVBGVXNjRDlIV0lCczJmWHpGMUVV?= =?utf-8?B?WEZEZ0F2ZjZIa3EvOWVxcEFtVklmTUE1SWxydkFiUVhYOFdraGhmLzF4WXBB?= =?utf-8?B?RG1yTGNTZ00rNmtJVmhsUXFLeFlHK1FrV2VvRG9VQzQzdU9PYW9ibld1Wi9R?= =?utf-8?B?cDRkdStTWHZwaGl0b01iQkxlSnYraVhUYXlSNjRzWGQ5YVdIYUIvZ1hXMEN6?= =?utf-8?B?RXBRYThtS1MxZ3pLdyszeVRlZ3dCM0JRYi84d292NDdTeDFPUENrbkE0NlhN?= =?utf-8?B?N1lCU2V1NDBqZVg0QTg3WG02d240WStFVEt3eC95bHFnM214Q2V5UEZ2QWlH?= =?utf-8?B?My9meU5nOHNrblFpMVBmWDZFVldQUnA4NEt0L01vdVl1REZJTVZWODV6WXRy?= =?utf-8?B?eUJoUklTK052azlySllzSWdUcWRMcUdyZG1XUWJhQUJURGlDRWJpa0t3PT0=?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 6:+0x68ZjHVtdmm8QJMY6Z+4G6mdSd0Qt0j7o5rZEzqxJbTGeigxuXorsm0G/L39ERvUCQHLG+9TUviQzB/06j3C0s9CPnGIIOSMDUDQG0eDS/rES0hdcmnJMInBNMgRBzuT66X9DESiSF1R6uGCBz/fOmuluwAdpC4N6KbnYSmyQAlKB2yX2RoLyt6etuXIjpY+g24tenO0xfmvhQSCTbxjDCCHFupq376QdtHFcfzXmuC+FQvGb7kqDdlaH8bdpvPqmnUrFRqLxgxDQE2hpfeHgMPq1xzcVye74bSBcqlt4Z7Ok60R5aOe3wp91WOYP6XJV5VaQMdIVV/Yxtc0/bZA==; 5:2CxnuRnNA+mXK3D1kKTV6hphTW/ZsdYzzCXdEyXDsOpDxCXNCo7GHIoptNHDz+vFpWVYjpmFIkToVkiDlF31J7Ah0fUJR9M58whu0+3Yn2bP/9vy6UYDHuH0cJNOReJmRgJ2MB2LtYeJg9+Hp54F3w==; 24:NtIuPu6H4BJATVOwtEBWDgMsG7KXRzGr3ckoQcYGdnl1lakRrRh5RuFC3exX7bF4b7sNhRbA8eV1DbmU/PaunoKb3JufIjr9e9no6MkEwLQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1141; 7:+O7zlSdxODc0pWejm5TIx2upyEu/5R0vniIgnHJQxvJK9qsireoyO5O3UWznwsGEyX89FKZ5H7baaYLMbKnIN0r3dxuUVP4yWQvJrzgVid+1eu6Z9c9aJTcpP2A25YBDCuc29eOCbd3NAf29iAC2feZ8rNDatIKee6JS4vMQbkJ137UnZmq8FSpUpcZzDrt02m50wdMG5hvP5P/31PapohTg9gQA7yzVRo8zyVOrrqul/i3vmAMowk0B82RYufCy690eA9fDsbxef88AAqEazNjBSujV/+76CH4H12fmwjP0r6isGHIwadpvj/k88GIkVfu9UVEdhlqU9+xT56jyPpw9p2aJMQHdn02D6rr9eRI=; 20:5PiMNn5g/G8m+uS6AWOaq7x/4VYS6HsIy7UZP8jCevnCakovsrDc6NnkW5vdxA9EZTvtSq3v9kDQYLbImAuqWOC7Wv0RFm3JRbZRkx3/5AHwWNOgp9/x/gaLXi4IifoQ2o7SdEKvlkLk1MyopAIfK09Tn50BXZ0NeWCCewI1NeluP9GbaUT87MHZIgm9TbMMRSg/UCa5NJK9FNuM8h3BfMYevVGBet9vq1gY03eWJoXSmyixeY3aXqIlyCKHcZnf X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Nov 2016 00:36:13.8719 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1141 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add support to be able to either encrypt or decrypt data in place during the early stages of booting the kernel. This does not change the memory encryption attribute - it is used for ensuring that data present in either an encrypted or un-encrypted memory area is in the proper state (for example the initrd will have been loaded by the boot loader and will not be encrypted, but the memory that it resides in is marked as encrypted). The early_memmap support is enhanced to specify encrypted and un-encrypted mappings with and without write-protection. The use of write-protection is necessary when encrypting data "in place". The write-protect attribute is considered cacheable for loads, but not stores. This implies that the hardware will never give the core a dirty line with this memtype. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/fixmap.h | 9 +++ arch/x86/include/asm/mem_encrypt.h | 15 +++++ arch/x86/include/asm/pgtable_types.h | 8 +++ arch/x86/mm/ioremap.c | 28 +++++++++ arch/x86/mm/mem_encrypt.c | 102 ++++++++++++++++++++++++++++++++++ include/asm-generic/early_ioremap.h | 2 + mm/early_ioremap.c | 15 +++++ 7 files changed, 179 insertions(+) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h index 83e91f0..4d41878 100644 --- a/arch/x86/include/asm/fixmap.h +++ b/arch/x86/include/asm/fixmap.h @@ -160,6 +160,15 @@ static inline void __set_fixmap(enum fixed_addresses idx, */ #define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_IO_NOCACHE +void __init *early_memremap_enc(resource_size_t phys_addr, + unsigned long size); +void __init *early_memremap_enc_wp(resource_size_t phys_addr, + unsigned long size); +void __init *early_memremap_dec(resource_size_t phys_addr, + unsigned long size); +void __init *early_memremap_dec_wp(resource_size_t phys_addr, + unsigned long size); + #include #define __late_set_fixmap(idx, phys, flags) __set_fixmap(idx, phys, flags) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 5f1976d..2a8e186 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -21,6 +21,11 @@ extern unsigned long sme_me_mask; +void __init sme_early_mem_enc(resource_size_t paddr, + unsigned long size); +void __init sme_early_mem_dec(resource_size_t paddr, + unsigned long size); + void __init sme_early_init(void); #define __sme_pa(x) (__pa((x)) | sme_me_mask) @@ -30,6 +35,16 @@ void __init sme_early_init(void); #define sme_me_mask 0UL +static inline void __init sme_early_mem_enc(resource_size_t paddr, + unsigned long size) +{ +} + +static inline void __init sme_early_mem_dec(resource_size_t paddr, + unsigned long size) +{ +} + static inline void __init sme_early_init(void) { } diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index cbfb83e..c456d56 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -161,6 +161,7 @@ enum page_cache_mode { #define _PAGE_CACHE_MASK (_PAGE_PAT | _PAGE_PCD | _PAGE_PWT) #define _PAGE_NOCACHE (cachemode2protval(_PAGE_CACHE_MODE_UC)) +#define _PAGE_CACHE_WP (cachemode2protval(_PAGE_CACHE_MODE_WP)) #define PAGE_NONE __pgprot(_PAGE_PROTNONE | _PAGE_ACCESSED) #define PAGE_SHARED __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | \ @@ -189,6 +190,7 @@ enum page_cache_mode { #define __PAGE_KERNEL_VVAR (__PAGE_KERNEL_RO | _PAGE_USER) #define __PAGE_KERNEL_LARGE (__PAGE_KERNEL | _PAGE_PSE) #define __PAGE_KERNEL_LARGE_EXEC (__PAGE_KERNEL_EXEC | _PAGE_PSE) +#define __PAGE_KERNEL_WP (__PAGE_KERNEL | _PAGE_CACHE_WP) #define __PAGE_KERNEL_IO (__PAGE_KERNEL) #define __PAGE_KERNEL_IO_NOCACHE (__PAGE_KERNEL_NOCACHE) @@ -202,6 +204,12 @@ enum page_cache_mode { #define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ _PAGE_DIRTY | _PAGE_ENC) +#define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC) +#define __PAGE_KERNEL_ENC_WP (__PAGE_KERNEL_WP | _PAGE_ENC) + +#define __PAGE_KERNEL_DEC (__PAGE_KERNEL) +#define __PAGE_KERNEL_DEC_WP (__PAGE_KERNEL_WP) + #define PAGE_KERNEL __pgprot(__PAGE_KERNEL | _PAGE_ENC) #define PAGE_KERNEL_RO __pgprot(__PAGE_KERNEL_RO | _PAGE_ENC) #define PAGE_KERNEL_EXEC __pgprot(__PAGE_KERNEL_EXEC | _PAGE_ENC) diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 7aaa263..ff542cd 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -418,6 +418,34 @@ void unxlate_dev_mem_ptr(phys_addr_t phys, void *addr) iounmap((void __iomem *)((unsigned long)addr & PAGE_MASK)); } +/* Remap memory with encryption */ +void __init *early_memremap_enc(resource_size_t phys_addr, + unsigned long size) +{ + return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_ENC); +} + +/* Remap memory with encryption and write-protected */ +void __init *early_memremap_enc_wp(resource_size_t phys_addr, + unsigned long size) +{ + return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_ENC_WP); +} + +/* Remap memory without encryption */ +void __init *early_memremap_dec(resource_size_t phys_addr, + unsigned long size) +{ + return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_DEC); +} + +/* Remap memory without encryption and write-protected */ +void __init *early_memremap_dec_wp(resource_size_t phys_addr, + unsigned long size) +{ + return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_DEC_WP); +} + static pte_t bm_pte[PAGE_SIZE/sizeof(pte_t)] __page_aligned_bss; static inline pmd_t * __init early_ioremap_pmd(unsigned long addr) diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index d642cc5..06235b4 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -14,6 +14,9 @@ #include #include +#include +#include + extern pmdval_t early_pmd_flags; /* @@ -24,6 +27,105 @@ extern pmdval_t early_pmd_flags; unsigned long sme_me_mask __section(.data) = 0; EXPORT_SYMBOL_GPL(sme_me_mask); +/* Buffer used for early in-place encryption by BSP, no locking needed */ +static char sme_early_buffer[PAGE_SIZE] __aligned(PAGE_SIZE); + +/* + * This routine does not change the underlying encryption setting of the + * page(s) that map this memory. It assumes that eventually the memory is + * meant to be accessed as encrypted but the contents are currently not + * encrypted. + */ +void __init sme_early_mem_enc(resource_size_t paddr, unsigned long size) +{ + void *src, *dst; + size_t len; + + if (!sme_me_mask) + return; + + local_flush_tlb(); + wbinvd(); + + /* + * There are limited number of early mapping slots, so map (at most) + * one page at time. + */ + while (size) { + len = min_t(size_t, sizeof(sme_early_buffer), size); + + /* Create a mapping for non-encrypted write-protected memory */ + src = early_memremap_dec_wp(paddr, len); + + /* Create a mapping for encrypted memory */ + dst = early_memremap_enc(paddr, len); + + /* + * If a mapping can't be obtained to perform the encryption, + * then encrypted access to that area will end up causing + * a crash. + */ + BUG_ON(!src || !dst); + + memcpy(sme_early_buffer, src, len); + memcpy(dst, sme_early_buffer, len); + + early_memunmap(dst, len); + early_memunmap(src, len); + + paddr += len; + size -= len; + } +} + +/* + * This routine does not change the underlying encryption setting of the + * page(s) that map this memory. It assumes that eventually the memory is + * meant to be accessed as not encrypted but the contents are currently + * encrypted. + */ +void __init sme_early_mem_dec(resource_size_t paddr, unsigned long size) +{ + void *src, *dst; + size_t len; + + if (!sme_me_mask) + return; + + local_flush_tlb(); + wbinvd(); + + /* + * There are limited number of early mapping slots, so map (at most) + * one page at time. + */ + while (size) { + len = min_t(size_t, sizeof(sme_early_buffer), size); + + /* Create a mapping for encrypted write-protected memory */ + src = early_memremap_enc_wp(paddr, len); + + /* Create a mapping for non-encrypted memory */ + dst = early_memremap_dec(paddr, len); + + /* + * If a mapping can't be obtained to perform the decryption, + * then un-encrypted access to that area will end up causing + * a crash. + */ + BUG_ON(!src || !dst); + + memcpy(sme_early_buffer, src, len); + memcpy(dst, sme_early_buffer, len); + + early_memunmap(dst, len); + early_memunmap(src, len); + + paddr += len; + size -= len; + } +} + void __init sme_early_init(void) { unsigned int i; diff --git a/include/asm-generic/early_ioremap.h b/include/asm-generic/early_ioremap.h index 734ad4d..2edef8d 100644 --- a/include/asm-generic/early_ioremap.h +++ b/include/asm-generic/early_ioremap.h @@ -13,6 +13,8 @@ extern void *early_memremap(resource_size_t phys_addr, unsigned long size); extern void *early_memremap_ro(resource_size_t phys_addr, unsigned long size); +extern void *early_memremap_prot(resource_size_t phys_addr, + unsigned long size, unsigned long prot_val); extern void early_iounmap(void __iomem *addr, unsigned long size); extern void early_memunmap(void *addr, unsigned long size); diff --git a/mm/early_ioremap.c b/mm/early_ioremap.c index 6d5717b..d71b98b 100644 --- a/mm/early_ioremap.c +++ b/mm/early_ioremap.c @@ -226,6 +226,14 @@ early_memremap_ro(resource_size_t phys_addr, unsigned long size) } #endif +void __init * +early_memremap_prot(resource_size_t phys_addr, unsigned long size, + unsigned long prot_val) +{ + return (__force void *)__early_ioremap(phys_addr, size, + __pgprot(prot_val)); +} + #define MAX_MAP_CHUNK (NR_FIX_BTMAPS << PAGE_SHIFT) void __init copy_from_early_mem(void *dest, phys_addr_t src, unsigned long size) @@ -267,6 +275,13 @@ early_memremap_ro(resource_size_t phys_addr, unsigned long size) return (void *)phys_addr; } +void __init * +early_memremap_prot(resource_size_t phys_addr, unsigned long size, + unsigned long prot_val) +{ + return (void *)phys_addr; +} + void __init early_iounmap(void __iomem *addr, unsigned long size) { }