From patchwork Thu Feb 16 15:46:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Lendacky X-Patchwork-Id: 9577545 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EB21460244 for ; Thu, 16 Feb 2017 15:46:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD21C2861B for ; Thu, 16 Feb 2017 15:46:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D19912861D; Thu, 16 Feb 2017 15:46:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED8722861B for ; Thu, 16 Feb 2017 15:46:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932781AbdBPPq2 (ORCPT ); Thu, 16 Feb 2017 10:46:28 -0500 Received: from mail-co1nam03on0081.outbound.protection.outlook.com ([104.47.40.81]:16416 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932454AbdBPPqW (ORCPT ); Thu, 16 Feb 2017 10:46:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=jeusKACVvSgLyQnrMOjCMWl7AvaUfS1sHGfoqkbrM3c=; b=yFFfji/E6/2i9y6rP56Zd+rFyVwYj4bwDELH4FkyoaYnpNjAeOHEVN2tAP+nzrE2qdepu7OVIlMT64DQ7p2FJySzglGfekSet423Wn5Nb7HtRO/dVv6Keg/hcboMh88fJxmGV98o5rzIpkLWJfB80OCDVu6rMI7jFT/JJclW0sE= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; Received: from tlendack-t1.amdoffice.net (165.204.77.1) by DM5PR12MB1148.namprd12.prod.outlook.com (10.168.236.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.888.16; Thu, 16 Feb 2017 15:46:08 +0000 From: Tom Lendacky Subject: [RFC PATCH v4 18/28] x86: DMA support for memory encryption To: , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Toshimitsu Kani , Arnd Bergmann , Jonathan Corbet , Matt Fleming , "Michael S. Tsirkin" , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Brijesh Singh , Ingo Molnar , Alexander Potapenko , Andy Lutomirski , "H. Peter Anvin" , Borislav Petkov , Andrey Ryabinin , Thomas Gleixner , Larry Woodman , Dmitry Vyukov Date: Thu, 16 Feb 2017 09:46:04 -0600 Message-ID: <20170216154604.19244.69522.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net> References: <20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: MWHPR21CA0023.namprd21.prod.outlook.com (10.173.47.33) To DM5PR12MB1148.namprd12.prod.outlook.com (10.168.236.143) X-MS-Office365-Filtering-Correlation-Id: 5eec0880-73b5-43ad-ff52-08d45682eec2 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081); SRVR:DM5PR12MB1148; X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1148; 3:nHo4Fo6avJxrvZBgTMucst5Ed+URHs3Io4r7Ffd7DhHG+8yohiMROpBd8KZ62O0yz5W+u/98zcSJLli/jpVo6fq7DZzSf68Fo6zEU+DHA23eWvITWLX5cJPa8zukcLifa3GIHGOkDaTPGKf7NfJmDy1FlaCElphkYY3yttxaxAau308gAyDwLDlFAsbJqGn9BWP89/YfEUOMC5VNNSgYpKlhoAgrUa6eYpSzYOW6OmoKGMKH4z7jRrbu6J8vR59/Ilzk4Xt8qKzJSkBIVqy5HfP8G1SffD4V/Q5CCahR8DU=; 25:g8Kjl41HWgyXMnWirW492Ug8IW8qwwLg7QIkGq3wriiUWneNdqZV6Z/Cy37AZiUz7wy5PiqBQg1Osjzf0QYpUOlR9lCxgKFgJnYo1ng3l2E7MPEvfzhTQz2/RDPKHImCF92DGfI9LsCthbkeCeshV+pCzbyC4DNkWrkxofbGrCTSXGTtgHMZC+BGyRb+HA4M9h2zATz+zPTauQaorss5IcLOwMeABSEVwmLMVNjMQdqgazp3IRDHlIj9SlTttzMUNStzUxRFTNcTuSM/C4yk5beSUg50oTsd0Mz98xEmizhGPTxRKNPJ3QkABgLW2U6PbNihcSYpeOpln0OYJ4EJD30ecFkGgCo0lwtNWV8XvO9HTAyU57xqoXNupO0klaF4yK7TQdwfQBygc5Vhdo55KezchnNob024n9lDegKPQl4ryyg2J5YOJeQSTaR/n6+GiHCpPF04W5L3locESTYChw== X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1148; 31:uZ1WrFy6cmP/aZoKlrWPoPtxvDxRFkAhkYNFErDbm3WLI6riDJNdftYNqJ/7ZOJcA1GHay50CCMGFQFLrb4BFaGZbasMUoC22L/jScNB2x/R0V8QOlqf1g+vIUi7pbUjKIurudiuhRrjMMX+KXK14dC1KZYBnJBiK8atEPgM6tjJD2zUyBkCOh6PjAEWssnZ+tVqnjNDgeXF/yxWYM6nVM4wZmrg6RnBUMwk1x1BNdo=; 20:Voa8kHvDrqD/dpK7sUQw26f08nf+kimCnlGqofGHyvLazsME9gIN+2IyvCq/DcrEYSSMrRaZ8YG8Aj4u6sjGZ6C0L/Do+k38RDCXBp6iO/DT+7aLZC5MdBA7SpDtRNfAtmXco/a6eMVSKLxu+2qYMX9iR7sErCU8aIeJV6MFZXYvZyNRBmJ3FTHB/cxVjNiZC8W3XpXgNlR5Oc/pung1Jp4ZAYiGFLjqpi4gBjTK4g91W+XK9yMqGDn85M0YQMsNUXdyanD7N27Eahy5Zwt4PTg5B1BRCAhP1OHg3+mHMw7oZ1o+Y3EUFW+xie5yBqsmofHU7aJzRHwsVyuwiAaDTAfIlpRZZzfe9LGgTpUp/dB1oSoFEPGdRXP39frD8l1/wqd5t3jJnVKMTbe+rbFNY+ExObQKmVnB9qIOv6yrqWpTPUutSlVwNIE6wZtTkF3NaxAYw4WTA5xY7h8i+5acsHwCsghVVP7LkylZpNWP2vSBqD0XQnjmsHTWKDRvbWrD X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6055026)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123558025)(20161123560025)(6072148); SRVR:DM5PR12MB1148; BCL:0; PCL:0; RULEID:; SRVR:DM5PR12MB1148; X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1148; 4:ueDgRfNcBsEILlP3qCcOHUVfzYurKujITNrFB19rSamUKEUwH6+jn5zGO4jYGcLQkCDJ3rRAWQ9e9m8j/M3VUPXaYlftlildnUN9CSGy3uDNDsDHmHj1s8MMM7jWHH0fLuI4PPVWVunostVkOsUu9AopdkZvTb1PSII3SsWs/SCbyCAJPly91bZyidg1EgH9r8yDi/g6zOUJxGVHKATUNos8DjEwbKAQH+DzNMtam7aMUlU1TknQhjN8RUqknuuFHCyztpK47X65FSlw21oOLUMQwW4cWyzIewhP6LMweG9BO7DuUjb232WAGDRMWAMZOi6tb1SEfrSZimL4/stYqvaAiRpAmuEN1Db6Uu2I6fFAjOxQ9vLwdrIXqzw+/AJkQojj8tBCKkvTvCQMvJFYWI7FRoPuRzUqQ9KROXc92NSuR5TyBCZhvAcRYcV2QwsHFtcBW2DqNZAXYccxZqtoDGh0ra0Y9rkBF7Q4ZYcOL5rFoikWrTsvXZZp7yUqvPKr9m3peRMHRRZkcIkPNvOV6OToZnoNiiDEqnT+tp/u16QQJyejNnLRZQ2BLH5581Dn4zfjDoT9/xrIx3zvZzwkqveiEaryErxO5Uy7pQE/QqsV/pibiMlEDMmOOWdeOtzN5f3lp6VatslPRT8dCUZHck8JARbmy7mvcTTlT1+NovU= X-Forefront-PRVS: 0220D4B98D X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(39860400002)(39410400002)(39450400003)(39850400002)(39840400002)(199003)(189002)(50466002)(2201001)(103116003)(66066001)(33646002)(53936002)(86362001)(23676002)(7416002)(105586002)(106356001)(69596002)(6506006)(50986999)(389900003)(92566002)(97746001)(81166006)(68736007)(81156014)(8676002)(25786008)(101416001)(1076002)(54356999)(47776003)(42186005)(7736002)(305945005)(189998001)(6666003)(83506001)(230700001)(3846002)(4326007)(2906002)(53416004)(5660300001)(54906002)(9686003)(4001350100001)(76176999)(2950100002)(55016002)(6116002)(97736004)(38730400002)(71626007)(217873001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR12MB1148; H:tlendack-t1.amdoffice.net; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMTQ4OzIzOlVXTzM5YnRHQUEzUGd5YnhHU2I1d1paYThk?= =?utf-8?B?T1dIUFNieldHOXRHUHFmZDVoOEt5Uy9jbWNmYkNWVFVSdFdCcnIzS0VMS1Fs?= =?utf-8?B?UDQxZStqSmZWRmQ3WnVUMFBTdlFVVnVGSFhRVHZVdDEyM3I0K2E3U2J2N2xV?= =?utf-8?B?Tm9wTjVGRUFZNnd6blloUjJrTmJjcHdNK3pDYm1LZFpaNWhIUjBTRElLREdU?= =?utf-8?B?WVYzdlpMZG1QMTlvV1p5QmNnSCtFN0xxVEhBbER5UnNua2VZb1pRQ1RpOXRo?= =?utf-8?B?bmppWnpxL01RYTZnZWRsbzJPU2JqUnNsRVA5NU4xOVNEc1hEekhjeTVoZzB6?= =?utf-8?B?QlJMM0xOczgwMTBJa0hyUjJwNjhrQy8wR2NueDM2bG56UDZUaWMwZUxZT2ln?= =?utf-8?B?NTk4cXRHK3NVUlMrVFZjSGpYeDVBSDE3K1NBOXZRbkc5SXdPTTZqUlk0V01x?= =?utf-8?B?dHhadittNHBhb05uWFRFeXlGZjhHMHowVGxxR3lUSXV6dDB5Ui9nVUtiWFM4?= =?utf-8?B?Y0lhV1BFcU1CY3V4NnBaS1pybWJXcE5oZE0wTDgyZHFhWWxVUmsxMWNSVkV2?= =?utf-8?B?ckNvOE5CYVdqMlh0a3g4ZHc4NWJXRlNhZDZ6QkM4RCt4S3dwcFE0TnZ3bS9E?= =?utf-8?B?TEZZQVRMSG9WeEZQeEVaYjNxaDRCOHdvcDBLeG13TDNuWjB6S3hkUHJrd2FK?= =?utf-8?B?S3RlMDFJWW1yeXhoa2liOHU3NnUwOXVpNHhScGJuM0pST01xN0JEWnRwalM1?= =?utf-8?B?dnRPb0xXQ3FMRFFtQTJCUE8rdVdURmNCdStRZGRCWEgvWGFzK1JVK0EydG1P?= =?utf-8?B?NE9SQVI0QkZCeXJWNlNvMnBuRnlweURxQ28rajQ5RHV2czBxdXNkTGV0OW4w?= =?utf-8?B?dENTL05mNmZTZStLMjVDeXNYcHR6NmdFVmJDTHpZR0dVY25WTmlVYklZK0d5?= =?utf-8?B?MUdPR1JVOFJtKytnbVc3QTBPSG9yb2JQYWJUM3hydXR0YXg2aXliLy8yTE4r?= =?utf-8?B?NkI4aXhGRzdCNEFvK0E5cC9wRktpTWxBZTF0cVVOMHF5ZGk2OTRJNmVPRWlq?= =?utf-8?B?N2d1bGdLbWd1RUhHeThUMkNSZWJUdVRQWXhsNkh1WlkwUEczc1JrMzFOTk9h?= =?utf-8?B?SWV0YkxKbk1qSFJmZ0pkUXh6bmhqSHBzMzN1Z2lRQ3hYWC9oc0xyaHhqeHVj?= =?utf-8?B?WEV4Mzd4RU5XRy9LMklHU2lEMmZFTGd6NE9uYlJCaDNtQUpDUDBQdU1CcUFR?= =?utf-8?B?aUdMVy96S0lxYUkvblBDYm84TlZwbHc4ZXRWZ1BidUFoVDJtbnhEUlQyTGsz?= =?utf-8?B?VUdOaFNPTDBnYkVtbzByZHZqelZ0bFdTQzN3eU5ZTkdOL2Zvb0pqN3c0QVZR?= =?utf-8?B?eWF4eEVOOWJuMjM4bHlNWFVNQVRCbmhDZDIvNjdOWXVhRlR3R1NrS2F4L1g4?= =?utf-8?B?MFhEM3RMbGtydUNMMFV1VTAzZEtDTndJZC9CRUZycnNLcEdDUG5lRi84VVdj?= =?utf-8?B?NGxUQU02MVh4ZnhuM2FJN0xPZ3o1SzhHQ3dHVStnZWt5b2dDcm9keG50Tk1o?= =?utf-8?B?VVBEY3R4bG9CZGwyc0dnQXJYQVg4TmFrcWo5Rnkycnc1SnlCRWM1OXpLWEls?= =?utf-8?B?OEEzcUk2K3pwRWVxV1FBWmlXRHZpRWQ5Ykdxa3dNQm05L01WK3E4dmhXMTRx?= =?utf-8?B?eDJLUTZRcTJ3Y010NGJIMEpmTlZzRVpKMGJTU2NDbzhmWHFnU3BERXhpek1z?= =?utf-8?B?RHVIRlo2WGVHM1JKeWM2cmVGYXdVTVppU01DT3RONmxOakFsaFk3TW5pU1pt?= =?utf-8?B?ZkVBOERqVUJvRUFVU0F3L1ZzUHpwQ0RuMVNKSUFJOEZ4ekVCWGJTL01lNVh6?= =?utf-8?B?bzRQTDgwYkdzdXdvSjlKa2tVK2dzNnc5b0hacjZiQS9NZDR2bTdGdVUwd2dT?= =?utf-8?B?YWNkYlpUNXBBPT0=?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1148; 6:IJ1SrWU3Xw4I8h/bPVrixDfUnfJKYTwNj5FD5Ddpzer+AAl37C+5vj7X1w4ye7pivKf+HFSHgmRq2iwTG/2CZ74kwnDkK3vervj834nkLjJQLN1VKh6xfpqzZsWDhi34hZdRmlhRXGkBu/pfjn7wDC+2zrK8ulaU1FNxG4KaczjhIG/omFiHmNZCIC9f0qW+stvOESodGKDSyq3L3CQfP8xEFIKD0efYre2i1/Kl+d27m9pFqzIU+B+qGcT+GtpvPVhqasM4Bc7qRl1MByoRUtpBxeEdqKfGLoB+WDR3XXuwLRA8uXssVd/8/6/kOPdrroTsNfnCj+CQMmLP04ZBanPTH2/Y+RZBlXHczAW75OagzwofZMiWJZy6IAEdtmLjNRyNJtZJzJSzLtLnjkl3bHSkLWSvkqctZmki2JU1/so=; 5:tghDEj8xc1Dd9iHn0raBPCK9DQx0HjUWUkS9e9kEX7+CJctpDV/OH7rekDDospoxucMnagvlszvowYVjvYcB25MTSQq4dw29gU7I7OZBLI6mnTWAj+ntvkRlW2RAqaBjbG5YVy9dDFdYlawzj5ZvOQ==; 24:+OFDoLkkeJk/iv8vJwh4AOWHWqdCck3vJYbeogDl/lt3xqfZawrPEidJKwD8PCg/Tn0rI5HS+Gv2GCmYm4P0qJvTeYfF3rVs/eLLI38M1Nw= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM5PR12MB1148; 7:CNyPemjIot0c9oFH2YGwE9MMFUZ770E1FP04tIBpqxl18eCTrUCAJBknty5L00n1vqd1djAPis1BFhD2S2z9dPVB97gy46q7cn2TmhcbgqiTEZS28Rti/LPoJD99dXSSibZ7/OqdkIWL3M9yJ3MgFh7hhiJUVloL4f5PAOGsoEddDrJjP9USBSoMqxr/QqR1KgHS52INwK5e1nAL2OOimhgq7ixLcUuKjxiFj5/AROw54NmWWDEJmfm3fycswXTaluVokGGI31hLntxXqnxZ8DR3v/J5LIPX0wbv2dzHhiOjoB8hz6nglTVgq73Uw99DmcHajSdI3wmUYDm6B/QfNQ==; 20:8/B3ob4swhOYRqRkKekYPA3D5UD4nlWhV2qdhrmq9F4AVPTQaMGjL0WtsdaOX6aqd6JFg+wf/qqNtNOPtFOXqqvrVY2v5PgOoOoUQOkkfy+X1FFUmf2CxSNc1RC+zYD0VGGcJYKkIYYuQvQvinPxpm1rOTTyguAkVW66ygK/LayCJUZYcaVg1uReZhivz2K4+7Ww1wmB/7sL8eAYw3398LqeOgOxzFKEmdMBwRciXdCKbvyjOLJX4Omvea2iXbXT X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2017 15:46:08.8472 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1148 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create decrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 5 +++ arch/x86/kernel/pci-dma.c | 11 +++++-- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 ++++- arch/x86/mm/mem_encrypt.c | 22 ++++++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 13 ++++++++ lib/swiotlb.c | 56 +++++++++++++++++++++++++++++++----- 9 files changed, 106 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 4446162..c9cdcae 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -69,12 +70,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index e2b7364..87e816f 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -36,6 +36,11 @@ void __init sme_early_decrypt(resource_size_t paddr, void __init sme_early_init(void); +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +void swiotlb_set_mem_attributes(void *vaddr, unsigned long size); + #define __sme_pa(x) (__pa((x)) | sme_me_mask) #define __sme_pa_nodebug(x) (__pa_nodebug((x)) | sme_me_mask) diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index d30c377..0ce28df 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index 00e71ce..922c10d 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index 410efb2..a0677a9 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,11 +66,13 @@ void x86_swiotlb_free_coherent(struct device *dev, size_t size, * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { - if (swiotlb_force == SWIOTLB_FORCE) + if ((swiotlb_force == SWIOTLB_FORCE) || sme_active()) swiotlb = 1; return swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index ec548e9..a46bcf4 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -13,11 +13,14 @@ #include #include #include +#include +#include #include #include #include #include +#include extern pmdval_t early_pmd_flags; int __init __early_make_pgtable(unsigned long, pmdval_t); @@ -192,3 +195,22 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = pgprot_encrypted(protection_map[i]); } + +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Call into SWIOTLB to update the SWIOTLB DMA buffers */ + swiotlb_update_mem_attributes(); +} + +void swiotlb_set_mem_attributes(void *vaddr, unsigned long size) +{ + WARN(PAGE_ALIGN(size) != size, + "size is not page aligned (%#lx)\n", size); + + /* Make the SWIOTLB buffer area decrypted */ + set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT); +} diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 4ee479f..15e7160 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -35,6 +35,7 @@ enum swiotlb_force { extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_update_mem_attributes(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index 8222caa..ba13f8f 100644 --- a/init/main.c +++ b/init/main.c @@ -466,6 +466,10 @@ void __init __weak thread_stack_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -614,6 +618,15 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + /* + * This needs to be called before any devices perform DMA + * operations that might use the swiotlb bounce buffers. + * This call will mark the bounce buffers as decrypted so + * that their usage will not cause "plain-text" data to be + * decrypted when accessed. + */ + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index a8d74a7..c463067 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -155,6 +156,17 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +void __weak swiotlb_set_mem_attributes(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~sme_me_mask; +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -183,6 +195,31 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * Early SWIOTLB allocation may be to early to allow an architecture to + * perform the desired operations. This function allows the architecture to + * call SWIOTLB when the operations are possible. This function needs to be + * called before the SWIOTLB memory is used. + */ +void __init swiotlb_update_mem_attributes(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_attributes(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_attributes(vaddr, bytes); + memset(vaddr, 0, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -320,6 +357,7 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + swiotlb_set_mem_attributes(tlb, bytes); memset(tlb, 0, bytes); /* @@ -330,6 +368,8 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) if (!v_overflow_buffer) goto cleanup2; + swiotlb_set_mem_attributes(v_overflow_buffer, io_tlb_overflow); + memset(v_overflow_buffer, 0, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -581,7 +621,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, return SWIOTLB_MAP_ERROR; } - start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir, attrs); } @@ -702,7 +742,7 @@ void swiotlb_tbl_sync_single(struct device *hwdev, phys_addr_t tlb_addr, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -812,10 +852,10 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir, attrs); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (dma_capable(dev, dev_addr, size)) @@ -824,7 +864,7 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, attrs |= DMA_ATTR_SKIP_CPU_SYNC; swiotlb_tbl_unmap_single(dev, map, size, dir, attrs); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } EXPORT_SYMBOL_GPL(swiotlb_map_page); @@ -958,7 +998,7 @@ void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -1026,7 +1066,7 @@ void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -1039,6 +1079,6 @@ void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported);