From patchwork Thu Feb 16 15:43:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Lendacky X-Patchwork-Id: 9577511 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EEB3960244 for ; Thu, 16 Feb 2017 15:44:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E09182861B for ; Thu, 16 Feb 2017 15:44:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D4F912861D; Thu, 16 Feb 2017 15:44:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4BFDC2861C for ; Thu, 16 Feb 2017 15:44:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932120AbdBPPoT (ORCPT ); Thu, 16 Feb 2017 10:44:19 -0500 Received: from mail-co1nam03on0047.outbound.protection.outlook.com ([104.47.40.47]:51952 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932374AbdBPPoI (ORCPT ); Thu, 16 Feb 2017 10:44:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=9DSXnTuixmPM+xi5+5evyQsSQCqNdZvoFKIv9A/dXSM=; b=SZLsduiZgnNt7TXDr8YEbH1P1nx8Qc4g0IBdPsVVoVPp7hrL9KCcVKEab56C3pApgJp+PiCG59xZqhLTEMynL90F66owjMRfasgwCVzeD6fqAvvMToLRY7VGSwDBmHuCr1mG/OPQcpdw4kgvqMZE5k6GpI1IrnXuGwHo7Tv3S20= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; Received: from tlendack-t1.amdoffice.net (165.204.77.1) by DM5PR12MB1147.namprd12.prod.outlook.com (10.168.236.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.888.16; Thu, 16 Feb 2017 15:44:01 +0000 From: Tom Lendacky Subject: [RFC PATCH v4 09/28] x86: Add support for early encryption/decryption of memory To: , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Toshimitsu Kani , Arnd Bergmann , Jonathan Corbet , Matt Fleming , "Michael S. Tsirkin" , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Brijesh Singh , Ingo Molnar , Alexander Potapenko , Andy Lutomirski , "H. Peter Anvin" , Borislav Petkov , Andrey Ryabinin , Thomas Gleixner , Larry Woodman , Dmitry Vyukov Date: Thu, 16 Feb 2017 09:43:58 -0600 Message-ID: <20170216154358.19244.6082.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net> References: <20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: CY1PR04CA0004.namprd04.prod.outlook.com (10.166.187.14) To DM5PR12MB1147.namprd12.prod.outlook.com (10.168.236.142) X-MS-Office365-Filtering-Correlation-Id: 9b223a74-ceee-481d-e573-08d45682a202 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081); SRVR:DM5PR12MB1147; X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1147; 3:0N19dJUbTOxQAW3o3aNefgHv5AKBZMJRcH4+Aq6ABXhdUCoIut9V9vhwRw+bw9xua67dPJFIvSnJfIy3ys2PregjrSYgtjv4AM6g86OhGmcboJJMBX+3yQiBkXmAcSl8M++LbLyGSy7CPp7VTB0WKLDxYHF62FaEXH8fm/2Td0BEsvU8lt6zAZ1hnhkAqpWzdxDH0nLRxAz704UYqarqiqLqiEUOx/JVLfa0z/o1qaUA/XBXY88Z3HNNDtzvm2O25pM5jSflYtNdWef9wP5T4vjdY8+SJJ+aMD26LIhmy6g=; 25:z2gGWJVfba0yyUSAMDyLadlF9Z4u4bn3kKWDBhtmUzW3dozINkq2mJkRrkIi4jBlVsBVbw84IhIMDFEx0c3RuZyY6cnykPOd1Y6gnwMHkRA6V7gyeKa9fi7GMFkvSNOWpcOez1CdOZaBZ6HWk20YfJjYW8aBdq5M86asnnfCt/SIKnpZkGf+Oqhy02ycYezqWN35NqjCub5LzXKu6hoySPBYOqen+rPPQY9SiMHChwABx1BRM4hz208edLXBjfznW1H+lmoSdQS4ONFb24yaRN7PruXow2ZUfy0MP9fEBU6gSvkbJHj3gIrMMaoTHWfFGoPzvEQQ4xf/oEMvLOorlixygYj60Zbt0D2v1xF2+EcQi2jpKsl+BDO35n24tW5NW7ZuW9PZfPbjghNS11hTmZlp0VkvAK4CKEA/w0xeWDeykNMGyKC/PVfQvfNAsR18ibeCC9mqSfcNqwJoprPiGQ== X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1147; 31:GnOV7oRTfZYnF1qybQbXjxP/N1vM43XAEABuT91tTT5gsSRjphh2CMvGol5EbVsPwsLKsPY23vFjrMIKzZOzsOf3fwHoAsyhGcFOu8u7Zye4tSsTtRyavdzu8HW1sbHJFeTPUcgii9FivUx70aU9VYnNoCPShifIv4GCNxenrRzBBanIuHY4p8SRtQMhl9YniEOL4Sd/Obm/IDCwSc+HLAzoUOyVyV3qGnadIh4I5xrzv0sGj0Lcj0mZYh0i30qY; 20:QACyb2VrhlRdtBiJWGz+f7sR2UdWIjz7JR8opVjEVpY8OvTkhDUib17Qa6pqHmnP09n/dEUV6gHf/CruqmoNtTxqha2KgjLJRFZloMricym//K/SADtnydM3MXtQP8Si7QmWHkTBECyUwMZWj2VlL5k4KwjJYQad4ovWzTUsz1DxPUGI81wBW6K3kbqhLQX5c5xMe7JNUdr7ogVYCtuKNM8hF+/Pl4JEk+mk3TD5b9yi0mitx3ZQOk7bZo20u3iUHBBocM8mKoO//OaXk1ugOShBgVix9Q4CE+QWdsfqcuwymNsuzDAlu2va1jZkKyJ8sWVam5WTF066ldkGxL5qjwo3cxECdavPwxohk+TWJXiBz7EG+2+T/ffPOwnR8nSSLsAYt3dOTahmmV0MX3fXp2LsCkvK4JvidW5NIPjnzgLDH5TMJhbgDr9scLgBrZEeRZ0q7pqRWZJPSvOa4CzK0gyILZqsubWXbRw4HgwiPp9HDa/drY26PKQZqVy/hY6g X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6041248)(20161123564025)(20161123558025)(20161123562025)(20161123555025)(20161123560025)(6072148); SRVR:DM5PR12MB1147; BCL:0; PCL:0; RULEID:; SRVR:DM5PR12MB1147; X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1147; 4:aqa73coDmjObnv0aIX7lrCfFsljSFn4ghSE792FAmuC/nMQJXwsvLhRfJNaSmrwjlp/dZmOit+NRz0tmfPyx+Ok9xy5L32G9zlG5m7AxKGdO4d+xUM2K1E9tTU3Ej/5FqR6T8+oQu4v1K59AVp3B5n2hlfmXrmFclqBpVj57CtD1EyfG2075FWDvdZUz6/f476MNPovSSEQakXpt5Y0NI0IZH+j9adwmZSI9lx2NUq0fLyzxo0gNW2zEwpLUvTMLIpp+OEPydCh14jM8xVaM1uIYVhBeTAF6LZ+YuK94MGXkYoN00hqIZtnLrkoZeaNg1pnCmGQvGmljM7dCAxvyg5eq7l+dncNWr0rSlr+x9EbBoqm1WQ2O5gKUYR7lCwmdTB/PYb0NhuFUSUGnn1B1nm29kbXzKxTVWqQ9d5fSWU2C5SlQ/i3Eu3kB4Wy0KGDrtvu/mAp0+Dh5vVLhVYOqpj13g7I3ixJf7efhL3hkJxaQ85K316JJ7rLja0fgt2OVxFGXMZuJBGBq54/cvsQrC9v629g8gvxFZ4Po2KbnlvocQqMeSBroORUrl6+fczdENOlsrMB+0dbFrtVSGQzNNdA8KC7TSHwY9/6nLGMTUvfxPL23PMYyYRDEmsN6//2EA050ycTUbQoPQazgRT1onFMwBceteEMUKladmUGgssc= X-Forefront-PRVS: 0220D4B98D X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(39850400002)(39450400003)(39410400002)(39860400002)(39840400002)(199003)(189002)(55016002)(50466002)(54906002)(83506001)(3846002)(6116002)(97746001)(2950100002)(25786008)(9686003)(97736004)(389900003)(53936002)(189998001)(4001350100001)(66066001)(47776003)(68736007)(33646002)(5660300001)(92566002)(1076002)(69596002)(103116003)(7736002)(2201001)(86362001)(230700001)(4326007)(305945005)(7416002)(2906002)(54356999)(50986999)(76176999)(23676002)(81166006)(81156014)(8676002)(38730400002)(101416001)(6666003)(106356001)(53416004)(42186005)(105586002)(6506006)(71626007)(217873001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR12MB1147; H:tlendack-t1.amdoffice.net; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMTQ3OzIzOmZUWWNucml1bGVnbDlsZEtBcCtML2JDcWI0?= =?utf-8?B?c2hEV2FsY3ZpQTZxVitwbnEydkNpM2VLZnZQVk1WeVdJUFlwaVhQeWVuNXNl?= =?utf-8?B?citEb2ZST08zQklZM0xJYU9ZVFlZNUVSTFdjOVR5enBUQUIwYWZVR1RtQjZw?= =?utf-8?B?ejV5L0lwN2NYbFd5YmtiYTAzVllIeDkyeXFKK2NWSXFWNnY2dytYNkFISVF0?= =?utf-8?B?b2Z4RFcwTjk3TkpCVU5CeGk1UDhBNVdRQkdLYkxneG5RM0RhVWhaTEZsbDRG?= =?utf-8?B?ak1qbUlmOHI0bUdSaGxmUXFXRkFScmIzMFBOR2VNc3AvTEo4V3g1VXpBVVhy?= =?utf-8?B?bHlRZFN5a2Z0T2ROdVdTOG5TVXZCbnF5a2l4eDdIeElwcnhyTXIzRVRFZDI0?= =?utf-8?B?NGNaNkdiWUkwcm41eGFmMGozQ0Z4OVpKclJMdFpBbXdjL1lURmd0bndWWkUw?= =?utf-8?B?ckJsZ25VMW4vODdzNCtLdnNYbk5vTjZBaDJIb1RUZVNiZXZhT0FQZ1JBcUFK?= =?utf-8?B?bkRyYVRTV0RFeWlYN2ZmQnl2UkMyUGFYMEFJeHI3aEJFYXRvdGM1TGlzSmVS?= =?utf-8?B?RTNHNVNRTUFzcXdROW9Rd21hVEQ0TFlLaWs3MGZZVisrOW9aMDZBd2pNMjV5?= =?utf-8?B?T0gzeWtCdXVZNjE1L2pobWkweHpOLzVhKzZQRFVhcjV2MnAwOVdVTzM3ejAv?= =?utf-8?B?WUFqSFprRlJ5ejJIREh2ZXNEODc0MGxid2tDdWUraFBVcHVmTkpIYTE4cEVY?= =?utf-8?B?bkk4dk1wcGllQk9lOEFhT3JQSTNZMm5BOUJ0b2dGNStQR1c5aGVOUFFPSDJO?= =?utf-8?B?eDhUWUlrL1A4d09YaHNjUG9BdHRTSkxEanFQbGYvaHF2SFVZNFhDN1ptdkJI?= =?utf-8?B?ZHJidEFueW9GNHE5cDJ5R1VhSWsxeFhBYkJtT2t2WGFVeWNiMnpTTmJGQWR1?= =?utf-8?B?Zm9aNnR4N2IvZUZjS3BtOHYzVEduSlErTE9tUFY0UFAycHphcHZkU1ZoSTNW?= =?utf-8?B?RVNWQ2M5aVJzVFZIWTdHcXByZkJGQWUvUEN1cjU2Mm5TemlCUHhkM3NyNGxz?= =?utf-8?B?c2RCZ3AwNlJSaTl2MjZVMlZweHFlTjJvOWJDZlVmM2prYW1rY2w5YnBlRUp0?= =?utf-8?B?TlJrVkt1aDh0SE5qN3ZEZXhMK2s5aDJIY3VDdERkZE1GSW9JRDVnc0R6TUhk?= =?utf-8?B?T2FtVklRSEtHVkhyV3FzZTRybTQzV2ZNQ05EMTNtQko2enFKOHFlek5CYit5?= =?utf-8?B?NUxVOVBUWlRrTUtDYnVFeXNYOUhQSnA0a2cyV2lVV0ZPNElyWDBaOTNsNk9i?= =?utf-8?B?UnVCN1A3MDNBUHpYUDhicjdpQ01qN2ZBODNxOUczNmlORHNlOHMxbXE1YUU5?= =?utf-8?B?MXRsWUlyWFlYNGFLbVVpRU1JNGhYdGl1cytCSFlVTGx6bmVhRWJIckN3MHV3?= =?utf-8?B?cVlaS1J5MkwzM3VXTS90TXQ2RSsvd0V0VkFhMjQvSWE0ZnZyRWlrbENUQ3Bw?= =?utf-8?B?b0x4WklhZUtiVS92d3pCNGhkSEFRTmpabGVpU0hhdEhZbFBxUWxWVWtQakJU?= =?utf-8?B?L0JTaHlBK29JQ1ZlaEhyRFNFSkdhU2VSbWh2RzBvYmpJUU5IdUI4UHlBZUc2?= =?utf-8?B?QStlWjQzdks0NjNmeFQrUzZWOGxiNi9QU0t2alNlVy9ncWY1azB4MXd1OVpS?= =?utf-8?B?WStZeG4wL1h6eXJ3bW8wUldXSmNORDA2V28vVXdHN0NnbW5rZTlFZFdxT1Vq?= =?utf-8?B?Qm5SM04vR3hMejBwSDJqYVhZOXQrdHV0SGhTSnVaVnM5blF4N2ZiNFROTUFE?= =?utf-8?B?SE0xR0hFay9Ic2JhVEpvTFlNZlRHU3pMeG5WOFRES0M4MVFZc3FaeURnL3hB?= =?utf-8?B?Y1lYZzhQK2VaWVVrUTVrS3JBcGlnZXRuM0tPdTh4Qkd0bWxxdUJ3djhNZytr?= =?utf-8?B?Q0NIT21RQllRPT0=?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1147; 6:GQ+mg+P8Jeyo+JZNhsjiDb6vM5kwUDkS2t4tVxUa8amK4j0Jpx7nkRWDuk1fSHBDhiSjfiXqSkMh25+Uta86TkcvJmlyCNfm8U/GKHtRH8JxqiduTQ2DxiC/zwjGK9dUROrzfj17ZtbdQ/Ufc5fzyikKdY70n/33aIhX6ltqUeNJw6+NDPWbR9eYLOjUcI3254O8wA0duYi1smpeIyAc34UdQ2B7ng7wZ+8IRN8QeGNHIsIA9ko2EMoraKj7hgT3V3rY+8G351b87n8HZAg/fiwt3+cJKILyqqUjqOzYCeQDmle2S/IcEJdJxYmMf+q6IVYHO0asR9JmY8JuuFMMq8F41rWvJcehBMJP6CqFzsSQLA3rrV/989lfOl/expNUVGQx1yUGEUslbhNOMvuEf97+Ro6iqJqQTQPi6qCufiU=; 5:ncezX7cOnHY7n5a1QokMlAKIzHjqW8NC+D89AKUyDR4hAOsl8NgJj1M7GYOUQHw4k5ZYdbBIXTicpOcMdTX6/679XhjaU+LQQIcLJYS6sPvRyAqWRqFWVBRZfHMgwZrOVgwcFaE37obhcFqAxBf2RA==; 24:NjWzYfx34bO7HsGryYWz7mANxf9ZItaaEuZmHxrNjCmJEcDpDeuniWl13HjEnDrUx7iygkcYfWBOqrLr2DkHHR1MHwuheP+u62muVZmunC8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1147; 7:6bs55pvgIoCyreYHGjLoyIKzCDJ0uwK7dkoZZbdGM7Xb2O38mKWivftr4ihT7J68uJgQEufZbIY4je0Qn70jeppxK/56i1uZZXX46bYqeIIswd2ExBe4eX9Oxu9OfkwUJupkNgDXuufpR5mbBCjdgHwBc+W5P54oz2uDsQ8EtQjAEhW77/9G2d9rqrnygY2vp/6UCe5HSY6NbO9J9AwVFQUKLNsONOg2WcdNPt/sgnyueXskM1LLGwsDN7uDnfMAajgplWhC1Dk0v8oeXa7lZbJ4XvKWaAs6kGPD+y3tC+5tpGC/2p6WUp8Nq3w5ADcImI49g+nz29IWnOBKtlbXFA==; 20:ytmFj02sk3uBpl8fvxpHNNJxwwSxgvul7kQCmHnMdmQ4WsdfVVsoZXlTNeyonvgEihNfzYmWASnOTT6W5Uqh8yE+HkOV4/PELdu1MZOGvtqg20skXZc0zCLxmNpiVMbVTBNCAdDokSO8hCnzE1M16p77Ji5L9IuxHB6IOkhHRKbIi4NYKfR2irYnIN3ERH1BeQBDht1IP28cLf9954pV/3AmRJiA1raAKwUu2yuUg10FVhHfA0WOyqFnXrKEAjYZ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2017 15:44:01.5486 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1147 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add support to be able to either encrypt or decrypt data in place during the early stages of booting the kernel. This does not change the memory encryption attribute - it is used for ensuring that data present in either an encrypted or decrypted memory area is in the proper state (for example the initrd will have been loaded by the boot loader and will not be encrypted, but the memory that it resides in is marked as encrypted). The early_memmap support is enhanced to specify encrypted and decrypted mappings with and without write-protection. The use of write-protection is necessary when encrypting data "in place". The write-protect attribute is considered cacheable for loads, but not stores. This implies that the hardware will never give the core a dirty line with this memtype. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/mem_encrypt.h | 15 +++++++ arch/x86/mm/mem_encrypt.c | 79 ++++++++++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 547989d..3c9052c 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -26,6 +26,11 @@ static inline bool sme_active(void) return (sme_me_mask) ? true : false; } +void __init sme_early_encrypt(resource_size_t paddr, + unsigned long size); +void __init sme_early_decrypt(resource_size_t paddr, + unsigned long size); + void __init sme_early_init(void); #define __sme_pa(x) (__pa((x)) | sme_me_mask) @@ -42,6 +47,16 @@ static inline bool sme_active(void) } #endif +static inline void __init sme_early_encrypt(resource_size_t paddr, + unsigned long size) +{ +} + +static inline void __init sme_early_decrypt(resource_size_t paddr, + unsigned long size) +{ +} + static inline void __init sme_early_init(void) { } diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index d71df97..ac3565c 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -14,6 +14,9 @@ #include #include +#include +#include + extern pmdval_t early_pmd_flags; /* @@ -24,6 +27,82 @@ unsigned long sme_me_mask __section(.data) = 0; EXPORT_SYMBOL_GPL(sme_me_mask); +/* Buffer used for early in-place encryption by BSP, no locking needed */ +static char sme_early_buffer[PAGE_SIZE] __aligned(PAGE_SIZE); + +/* + * This routine does not change the underlying encryption setting of the + * page(s) that map this memory. It assumes that eventually the memory is + * meant to be accessed as either encrypted or decrypted but the contents + * are currently not in the desired stated. + * + * This routine follows the steps outlined in the AMD64 Architecture + * Programmer's Manual Volume 2, Section 7.10.8 Encrypt-in-Place. + */ +static void __init __sme_early_enc_dec(resource_size_t paddr, + unsigned long size, bool enc) +{ + void *src, *dst; + size_t len; + + if (!sme_me_mask) + return; + + local_flush_tlb(); + wbinvd(); + + /* + * There are limited number of early mapping slots, so map (at most) + * one page at time. + */ + while (size) { + len = min_t(size_t, sizeof(sme_early_buffer), size); + + /* + * Create write protected mappings for the current format + * of the memory. + */ + src = enc ? early_memremap_decrypted_wp(paddr, len) : + early_memremap_encrypted_wp(paddr, len); + + /* + * Create mappings for the desired format of the memory. + */ + dst = enc ? early_memremap_encrypted(paddr, len) : + early_memremap_decrypted(paddr, len); + + /* + * If a mapping can't be obtained to perform the operation, + * then eventual access of that area will in the desired + * mode will cause a crash. + */ + BUG_ON(!src || !dst); + + /* + * Use a temporary buffer, of cache-line multiple size, to + * avoid data corruption as documented in the APM. + */ + memcpy(sme_early_buffer, src, len); + memcpy(dst, sme_early_buffer, len); + + early_memunmap(dst, len); + early_memunmap(src, len); + + paddr += len; + size -= len; + } +} + +void __init sme_early_encrypt(resource_size_t paddr, unsigned long size) +{ + __sme_early_enc_dec(paddr, size, true); +} + +void __init sme_early_decrypt(resource_size_t paddr, unsigned long size) +{ + __sme_early_enc_dec(paddr, size, false); +} + void __init sme_early_init(void) { unsigned int i;