From patchwork Mon Apr 10 22:59:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206818 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2CC9C77B70 for ; Mon, 10 Apr 2023 23:11:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229778AbjDJXLn (ORCPT ); Mon, 10 Apr 2023 19:11:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229711AbjDJXLl (ORCPT ); Mon, 10 Apr 2023 19:11:41 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 966C9211E; Mon, 10 Apr 2023 16:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168300; x=1712704300; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=mh7LCZtK7qkc4vQH6KmOZnQFKL0O2wmnvhs6j7yyDS8=; b=C0JE0NVkNGEb+cCe3ou21/2FHVsGxTgg3sMl0BuGuoPsg+7ntQPQyx8+ ioT0L8a+hqPkFEmSL7QJwTgZ0ASmN952ysE8N2i5KEJQLl03sgrMNg5vF j1cSdvL7yLvJUJp2vlOSZY6QG5/1hcNfLsc5lv6c7oWsY5Wk1N74CoJPw s5wxOGdeb8zH+hQBWGGpfbaDkJN/ry72KSR9Lp43+SY9cdObnUjEUClLB PjIqFgwxtcteItwJPMPqLKDzgksqXddNP68UwNu4Cp9kkRXosUtedQ90F 28KXl4xL7fhugSfYn4/EGrAgAPqpBxeDZQLAfL9yGghlJGoXgeG50Sw9Z w==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962491" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962491" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757607990" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757607990" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:38 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Jonathan Corbet , linux-doc@vger.kernel.org Subject: [PATCH v6 01/12] Documentation/x86: Document Key Locker Date: Mon, 10 Apr 2023 15:59:25 -0700 Message-Id: <20230410225936.8940-2-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Document the overview of the feature along with relevant consideration when provisioning dm-crypt volumes with AES-KL instead of AES-NI. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Jonathan Corbet Cc: x86@kernel.org Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from v5: * Fix a typo: 'feature feature' -> 'feature' Changes from RFC v2: * Add as a new patch. The preview is available here: https://htmlpreview.github.io/?https://github.com/intel-staging/keylocker/kdoc/x86/keylocker.html --- Documentation/x86/index.rst | 1 + Documentation/x86/keylocker.rst | 98 +++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) create mode 100644 Documentation/x86/keylocker.rst diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index 8ac64d7de4dc..669c239c009f 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -43,3 +43,4 @@ x86-specific Documentation features elf_auxvec xstate + keylocker diff --git a/Documentation/x86/keylocker.rst b/Documentation/x86/keylocker.rst new file mode 100644 index 000000000000..3b405fade7d8 --- /dev/null +++ b/Documentation/x86/keylocker.rst @@ -0,0 +1,98 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +x86 Key Locker +============== + +Introduction +============ + +Key Locker is a CPU feature to reduce key exfiltration opportunities +while maintaining a programming interface similar to AES-NI. It +converts the AES key into an encoded form, called the 'key handle'. +The key handle is a wrapped version of the clear-text key where the +wrapping key has limited exposure. Once converted, all subsequent data +encryption using new AES instructions (AES-KL) uses this key handle, +reducing the exposure of private key material in memory. + +Internal Wrapping Key (IWKey) +============================= + +The CPU-internal wrapping key is an entity in a software-invisible CPU +state. On every system boot, a new key is loaded. So the key handle that +was encoded by the old wrapping key is no longer usable on system shutdown +or reboot. + +And the key may be lost on the following exceptional situation upon wakeup: + +IWKey Restore Failure +--------------------- + +The CPU state is volatile with the ACPI S3/4 sleep states. When the system +supports those states, the key has to be backed up so that it is restored +on wake up. The kernel saves the key in non-volatile media. + +The event of an IWKey restore failure upon resume from suspend, all +established key handles become invalid. In flight dm-crypt operations +receive error results from pending operations. In the likely scenario that +dm-crypt is hosting the root filesystem the recovery is identical to if a +storage controller failed to resume from suspend, reboot. If the volume +impacted by an IWKey restore failure is a data-volume then it is possible +that I/O errors on that volume do not bring down the rest of the system. +However, a reboot is still required because the kernel will have +soft-disabled Key Locker. Upon the failure, the crypto library code will +return -ENODEV on every AES-KL function call. The Key Locker implementation +only loads a new IWKey at initial boot, not any time after like resume from +suspend. + +Use Case and Non-use Cases +========================== + +Bare metal disk encryption is the only intended use case. + +Userspace usage is not supported because there is no ABI provided to +communicate and coordinate wrapping-key restore failure to userspace. For +now, key restore failures are only coordinated with kernel users. But the +kernel can not prevent userspace from using the feature's AES instructions +('AES-KL') when the feature has been enabled. So, the lack of userspace +support is only documented, not actively enforced. + +Key Locker is not expected to be advertised to guest VMs and the kernel +implementation ignores it even if the VMM enumerates the capability. The +expectation is that a guest VM wants private IWKey state, but the +architecture does not provide that. An emulation of that capability, by +caching per VM IWKeys in memory, defeats the purpose of Key Locker. The +backup / restore facility is also not performant enough to be suitable for +guest VM context switches. + +AES Instruction Set +=================== + +The feature accompanies a new AES instruction set. This instruction set is +analogous to AES-NI. A set of AES-NI instructions can be mapped to an +AES-KL instruction. For example, AESENC128KL is responsible for ten rounds +of transformation, which is equivalent to nine times AESENC and one +AESENCLAST in AES-NI. + +But they have some notable differences: + +* AES-KL provides a secure data transformation using an encrypted key. + +* If an invalid key handle is provided, e.g. a corrupted one or a handle + restriction failure, the instruction fails with setting RFLAGS.ZF. The + crypto library implementation includes the flag check to return an error + code. Note that the flag is also set when the internal wrapping key is + changed because of missing backup. + +* AES-KL implements support for 128-bit and 256-bit keys, but there is no + AES-KL instruction to process an 192-bit key. But there is no AES-KL + instruction to process a 192-bit key. The AES-KL cipher implementation + logs a warning message with a 192-bit key and then falls back to AES-NI. + So, this 192-bit key-size limitation is only documented, not enforced. It + means the key will remain in clear-text in memory. This is to meet Linux + crypto-cipher expectation that each implementation must support all the + AES-compliant key sizes. + +* Some AES-KL hardware implementation may have noticeable performance + overhead when compared with AES-NI instructions. + From patchwork Mon Apr 10 22:59:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206819 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20ED8C77B74 for ; Mon, 10 Apr 2023 23:11:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229766AbjDJXLm (ORCPT ); Mon, 10 Apr 2023 19:11:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229721AbjDJXLl (ORCPT ); Mon, 10 Apr 2023 19:11:41 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECF99212E; Mon, 10 Apr 2023 16:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168300; x=1712704300; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=KV4DRDJjZKL8mUaHgawndnJfeklSCTHqnQ39yJrSK6Q=; b=HTIRaEzqbFd/sWyMRO2bdpKE8X6X2uwUVVRZCQH7B3NITV2KprK1t1uo pPmdcxNm7n4PF8jOB0kpCOqtTe/j/lZ/oQOsA54i0NUL0j6qYSOI1OtA6 0SPKcRkKZpy2UEfgcizHDxsr3aeaZ/00c+/CY4195wBrqiVaMgLfky6w5 S5Ucj6sdMBjbgwMqXqhY67pGNIeoWomTETQbb2JYjDP3fS29xtSW0CKfv 8MQGDY4Rpfesr3sj0jxt4206NRakxzYsWXVaTVjPeho/v69Q4fHCOlv+g AsxKGMwd0IuNoAQBeyytGd4MsCYy+r6WVJkNzf6BEAbqVwAdWP3g2Ooj7 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962502" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962502" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757607993" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757607993" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:39 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Peter Zijlstra Subject: [PATCH v6 02/12] x86/cpufeature: Enumerate Key Locker feature Date: Mon, 10 Apr 2023 15:59:26 -0700 Message-Id: <20230410225936.8940-3-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker is a CPU feature to minimize exposure of clear-text key material. An encoded form, called 'key handle', is referenced for data encryption or decryption instead of accessing the clear text key. A wrapping key loaded in the CPU's software-inaccessible state is used to transform a user key into a key handle. It supports Advanced Encryption Standard (AES) cipher algorithm with new SIMD instruction set like its predecessor (AES-NI). So a new AES implementation will follow in the kernel's crypto library. Here add it to enumerate the hardware capability, but it will not be shown in /proc/cpuinfo as userspace usage is not supported. Make the feature depend on XMM2 as it comes with AES SIMD instructions. Add X86_FEATURE_KEYLOCKER to the disabled features mask. It will be enabled under a new config option. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Peter Zijlstra Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Do not publish the feature flag to userspace. * Update the changelog. Changes from RFC v1: * Updated the changelog. --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/cpu/cpuid-deps.c | 1 + 4 files changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index d9c190cdefa9..878d25e38d7c 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -390,6 +390,7 @@ #define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */ #define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */ #define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */ +#define X86_FEATURE_KEYLOCKER (16*32+23) /* "" Key Locker */ #define X86_FEATURE_BUS_LOCK_DETECT (16*32+24) /* Bus Lock detect */ #define X86_FEATURE_CLDEMOTE (16*32+25) /* CLDEMOTE instruction */ #define X86_FEATURE_MOVDIRI (16*32+27) /* MOVDIRI instruction */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 702d93fdd10e..40e8ad2f837d 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -38,6 +38,12 @@ # define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31)) #endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ +#ifdef CONFIG_X86_KEYLOCKER +# define DISABLE_KEYLOCKER 0 +#else +# define DISABLE_KEYLOCKER (1<<(X86_FEATURE_KEYLOCKER & 31)) +#endif /* CONFIG_X86_KEYLOCKER */ + #ifdef CONFIG_X86_5LEVEL # define DISABLE_LA57 0 #else @@ -138,7 +144,7 @@ #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_KEYLOCKER) #define DISABLED_MASK17 0 #define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK19 0 diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index d898432947ff..262348aeaad1 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -128,6 +128,8 @@ #define X86_CR4_PCIDE _BITUL(X86_CR4_PCIDE_BIT) #define X86_CR4_OSXSAVE_BIT 18 /* enable xsave and xrestore */ #define X86_CR4_OSXSAVE _BITUL(X86_CR4_OSXSAVE_BIT) +#define X86_CR4_KEYLOCKER_BIT 19 /* enable Key Locker */ +#define X86_CR4_KEYLOCKER _BITUL(X86_CR4_KEYLOCKER_BIT) #define X86_CR4_SMEP_BIT 20 /* enable SMEP support */ #define X86_CR4_SMEP _BITUL(X86_CR4_SMEP_BIT) #define X86_CR4_SMAP_BIT 21 /* enable SMAP support */ diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index e462c1d3800a..24f54f58dbf8 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -82,6 +82,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, + { X86_FEATURE_KEYLOCKER, X86_FEATURE_XMM2 }, {} }; From patchwork Mon Apr 10 22:59:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206822 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6591C76196 for ; Mon, 10 Apr 2023 23:12:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229899AbjDJXMF (ORCPT ); Mon, 10 Apr 2023 19:12:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229507AbjDJXLn (ORCPT ); Mon, 10 Apr 2023 19:11:43 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA653212F; Mon, 10 Apr 2023 16:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168301; x=1712704301; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=1nnA3+UMtwhKXHQJhoeke18gc8xz62Yzi4vhC4igcco=; b=fVnt/qm4yLQ85OxedmG0vaSltMnLh+2DmHnNgZxmDENXUfKHf/17OAHI 4xYD+JUnLL0BdFAEGJDPX6GKoc/M5JvX7fPd/yQcTVkZtlgiuhtJ96JJm 590OJtTXuR1wYN6BAem5G+l7CzNy4DWSjy29YMDD0jaGXab1doHttpCuG h1NmlLcaTTHOS4vLuQSuxb5O4H+nXPqPeuz2dnA6imcw5WOd3J+RhRilt nstoBWUqSn+CcJS8VE/ITJh3YW2isGxgCfqMCc7dAN+xWKbqP0FhBHDnE KTe+VySHN0XReTqjPWfWKfeDUMmhkThjuSf3izMZs/xfQuCzpyNjNEjl7 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962518" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962518" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757607996" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757607996" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:39 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" Subject: [PATCH v6 03/12] x86/insn: Add Key Locker instructions to the opcode map Date: Mon, 10 Apr 2023 15:59:27 -0700 Message-Id: <20230410225936.8940-4-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add the following Key Locker instructions to the opcode map: LOADIWKEY: Load an CPU-internal wrapping key. ENCODEKEY128: Wrap a 128-bit AES key to a key handle. ENCODEKEY256: Wrap a 256-bit AES key to a key handle. AESENC128KL: Encrypt a 128-bit block of data using a 128-bit AES key indicated by a key handle. AESENC256KL: Encrypt a 128-bit block of data using a 256-bit AES key indicated by a key handle. AESDEC128KL: Decrypt a 128-bit block of data using a 128-bit AES key indicated by a key handle. AESDEC256KL: Decrypt a 128-bit block of data using a 256-bit AES key indicated by a key handle. AESENCWIDE128KL: Encrypt 8 128-bit blocks of data using a 128-bit AES key indicated by a key handle. AESENCWIDE256KL: Encrypt 8 128-bit blocks of data using a 256-bit AES key indicated by a key handle. AESDECWIDE128KL: Decrypt 8 128-bit blocks of data using a 128-bit AES key indicated by a key handle. AESDECWIDE256KL: Decrypt 8 128-bit blocks of data using a 256-bit AES key indicated by a key handle. Details of these instructions can be found in Intel Software Developer's Manual. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Separated out the LOADIWKEY addition in a new patch. * Included AES instructions to avoid warning messages when the AES Key Locker module is built. --- arch/x86/lib/x86-opcode-map.txt | 11 +++++++---- tools/arch/x86/lib/x86-opcode-map.txt | 11 +++++++---- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index 5168ee0360b2..87e9696c47d2 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -800,11 +800,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev) cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev) cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev) cf: vgf2p8mulb Vx,Wx (66) +d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B) db: VAESIMC Vdq,Wdq (66),(v1) -dc: vaesenc Vx,Hx,Wx (66) -dd: vaesenclast Vx,Hx,Wx (66) -de: vaesdec Vx,Hx,Wx (66) -df: vaesdeclast Vx,Hx,Wx (66) +dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3) +dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3) +de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3) +df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3) f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) @@ -814,6 +815,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My, f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3) f9: MOVDIRI My,Gy +fa: ENCODEKEY128 Ew,Ew (F3) +fb: ENCODEKEY256 Ew,Ew (F3) EndTable Table: 3-byte opcode 2 (0x0f 0x3a) diff --git a/tools/arch/x86/lib/x86-opcode-map.txt b/tools/arch/x86/lib/x86-opcode-map.txt index 5168ee0360b2..87e9696c47d2 100644 --- a/tools/arch/x86/lib/x86-opcode-map.txt +++ b/tools/arch/x86/lib/x86-opcode-map.txt @@ -800,11 +800,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev) cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev) cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev) cf: vgf2p8mulb Vx,Wx (66) +d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B) db: VAESIMC Vdq,Wdq (66),(v1) -dc: vaesenc Vx,Hx,Wx (66) -dd: vaesenclast Vx,Hx,Wx (66) -de: vaesdec Vx,Hx,Wx (66) -df: vaesdeclast Vx,Hx,Wx (66) +dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3) +dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3) +de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3) +df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3) f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) @@ -814,6 +815,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My, f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3) f9: MOVDIRI My,Gy +fa: ENCODEKEY128 Ew,Ew (F3) +fb: ENCODEKEY256 Ew,Ew (F3) EndTable Table: 3-byte opcode 2 (0x0f 0x3a) From patchwork Mon Apr 10 22:59:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206824 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81929C77B74 for ; Mon, 10 Apr 2023 23:12:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229914AbjDJXMG (ORCPT ); Mon, 10 Apr 2023 19:12:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229790AbjDJXLn (ORCPT ); Mon, 10 Apr 2023 19:11:43 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC30A2135; Mon, 10 Apr 2023 16:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168302; x=1712704302; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=e8bCb4INlqBkf0K+wkKTyG3ginAA5mKmpmasRm00usQ=; b=kciAi+OyVrPTwC+YTe2Hg54uD7HTaP1j7wFBu0d8ITm2GiBpzPMJPIJ/ y1Z5bf+wZEUWMIgtS5IMMfQJvguk3IcTBcp6q75SjhtWZo/HhqSluqUsT ua9cZ5eAytCVhh1rj8MsVwInejbHG+ZKcPTbMt6WcQe0LYV62IOcNrB4/ t7qu7aUiEe1IcQ+h6tBDQUjhBgmp/o/Erxl2L6LQJMmLCFPEedpUPWghm luufPwJwlpQmHBI9YmYiMm1wovn/GzArJDcxOHg+7H7aX9Qy7oLrkRNZN 2wpkE/3IY/+rSYtSoKhmy/+VoUy6aTPb/RXAeGX+AaKVZT2OCUXAiTFBj A==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962521" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962521" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757607999" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757607999" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:39 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , "Rafael J. Wysocki" , Peter Zijlstra Subject: [PATCH v6 04/12] x86/asm: Add a wrapper function for the LOADIWKEY instruction Date: Mon, 10 Apr 2023 15:59:28 -0700 Message-Id: <20230410225936.8940-5-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker introduces a CPU-internal wrapping key to encode a user key to a key handle. Then a key handle is referenced instead of the plain text key. The new instruction loads an internal wrapping key in the software-inaccessible CPU state. It operates only in kernel mode. Define struct iwkey to pass the key value. The kernel will use this function to load a new key at boot time when the feature is enabled. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: "Rafael J. Wysocki" Cc: Peter Zijlstra Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Separate out the code as a new patch. * Improve the usability with the new struct as an argument. (Dan Williams) Note, Dan wondered if: WARN_ON(!irq_fpu_usable()); would be appropriate in the load_xmm_iwkey() function. --- arch/x86/include/asm/keylocker.h | 25 ++++++++++++++++++++++ arch/x86/include/asm/special_insns.h | 32 ++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 arch/x86/include/asm/keylocker.h diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h new file mode 100644 index 000000000000..df84c83228a1 --- /dev/null +++ b/arch/x86/include/asm/keylocker.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _ASM_KEYLOCKER_H +#define _ASM_KEYLOCKER_H + +#ifndef __ASSEMBLY__ + +#include + +/** + * struct iwkey - A temporary internal wrapping key storage. + * @integrity_key: A 128-bit key to check that key handles have not + * been tampered with. + * @encryption_key: A 256-bit encryption key used in + * wrapping/unwrapping a clear text key. + * + * This storage should be flushed immediately after loaded. + */ +struct iwkey { + struct reg_128_bit integrity_key; + struct reg_128_bit encryption_key[2]; +}; + +#endif /*__ASSEMBLY__ */ +#endif /* _ASM_KEYLOCKER_H */ diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index d6cd9344f6c7..90b23f55970a 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -9,6 +9,7 @@ #include #include #include +#include /* * The compiler should not reorder volatile asm statements with respect to each @@ -296,6 +297,37 @@ static __always_inline void tile_release(void) asm volatile(".byte 0xc4, 0xe2, 0x78, 0x49, 0xc0"); } +/** + * load_xmm_iwkey - Load a CPU-internal wrapping key + * @key: A struct iwkey pointer. + * + * Load @key to XMMs then do LOADIWKEY. After this, flush XMM + * registers. Caller is responsible for kernel_cpu_begin(). + */ +static inline void load_xmm_iwkey(struct iwkey *key) +{ + struct reg_128_bit zeros = { 0 }; + + asm volatile ("movdqu %0, %%xmm0; movdqu %1, %%xmm1; movdqu %2, %%xmm2;" + :: "m"(key->integrity_key), "m"(key->encryption_key[0]), + "m"(key->encryption_key[1])); + + /* + * LOADIWKEY %xmm1,%xmm2 + * + * EAX and XMM0 are implicit operands. Load a key value + * from XMM0-2 to a software-invisible CPU state. With zero + * in EAX, CPU does not do hardware randomization and the key + * backup is allowed. + * + * This instruction is supported by binutils >= 2.36. + */ + asm volatile (".byte 0xf3,0x0f,0x38,0xdc,0xd1" :: "a"(0)); + + asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;" + :: "m"(zeros)); +} + #endif /* __KERNEL__ */ #endif /* _ASM_X86_SPECIAL_INSNS_H */ From patchwork Mon Apr 10 22:59:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206821 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B46D9C77B70 for ; Mon, 10 Apr 2023 23:11:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229879AbjDJXLw (ORCPT ); Mon, 10 Apr 2023 19:11:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229761AbjDJXLm (ORCPT ); Mon, 10 Apr 2023 19:11:42 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E65B82118; Mon, 10 Apr 2023 16:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168301; x=1712704301; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ICOUeWJ4MvP0vL80dft48PfCDjzRn/jTo0cQSkMOY5o=; b=BJVqyUW6fr1++i17en4FHgZjoOT7E6mRvVuu4LVkLkX94Ve67nWn5fFk ho/OoxJQJliW/t1PSfYw1OKyJ2AnPn9LtQurXhS3yFSXvRDII6WTAzyLy PQUWnveojym3rAZcUavXzLJ/wEB/QZBIZZW7wWCgkVoKx1UcuHb6+2hhr 2u6LMp9RGNOVD/MM5uJ+tvCAL8kET4w9lLJC5z8L/1etyuR6nSM7/uDB9 0EZrcmIgYpsPPBO6ZtEqZMDADx1rGJU/faaKmsN0yHElBxXmZwC5fTagb Mql1WALFP/rlrLXuRfAeX83c8bBc/SnlFJz5a7FsIYKuFH4P9Wfn40E1k A==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962538" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962538" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608002" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608002" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:40 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Peter Zijlstra Subject: [PATCH v6 05/12] x86/msr-index: Add MSRs for Key Locker internal wrapping key Date: Mon, 10 Apr 2023 15:59:29 -0700 Message-Id: <20230410225936.8940-6-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The CPU state that contains the internal wrapping key is in the same power domain as the cache. So any sleep state that would invalidate the cache (like S3) also invalidates the state of the wrapping key. But, since the state is inaccessible to software, it needs a special mechanism to save and restore the key during deep sleep. A set of new MSRs are provided as an abstract interface to save and restore the wrapping key, and to check the key status. The wrapping key is saved in a platform-scoped state of non-volatile media. The backup itself and its path from the CPU are encrypted and integrity protected. But this storage's non-volatility is not architecturally guaranteed across off states, such as S5 and G3. The MSRs will be used to back up the key for S3/4 sleep states. Then the kernel code writes one of them to restore the key in each CPU state. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Peter Zijlstra Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Update the changelog. (Dan Williams) * Rename the MSRs. (Dan Williams) --- arch/x86/include/asm/msr-index.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index ad35355ee43e..4fa3b9e26162 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -1115,4 +1115,10 @@ * a #GP */ +/* MSRs for managing an internal wrapping key for Key Locker. */ +#define MSR_IA32_IWKEY_COPY_STATUS 0x00000990 +#define MSR_IA32_IWKEY_BACKUP_STATUS 0x00000991 +#define MSR_IA32_BACKUP_IWKEY_TO_PLATFORM 0x00000d91 +#define MSR_IA32_COPY_IWKEY_TO_LOCAL 0x00000d92 + #endif /* _ASM_X86_MSR_INDEX_H */ From patchwork Mon Apr 10 22:59:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206820 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF527C77B61 for ; Mon, 10 Apr 2023 23:11:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229814AbjDJXLp (ORCPT ); Mon, 10 Apr 2023 19:11:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229743AbjDJXLm (ORCPT ); Mon, 10 Apr 2023 19:11:42 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA528210D; Mon, 10 Apr 2023 16:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168301; x=1712704301; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=JnGN/QzStCZJLTyrQM3ya368mjdTMP3eVWxv2wk0pgc=; b=HnwJ5GPCbD48vmZL4KHROKap7Qjx8FWF9mTTLFLXbMkDjHEYptk/xSgv WupexBecF/v/S4I6cBTIXfmsjDMOwXbqENbVHYcfuXKnxN618/D31FDWD Yb5UFLKfY5LaJA8uq5NOFRyuIJfyYy8VV+g7jQXEeuKrXi5+AIQBFe/xs sBh9+PidNRxVk6MoPfxFCTSNerAZrlbo78qip8L2+J4MJvCIjtENpTDwC 7aBGzYNjgCxKinrcdyw/BlWQAWhgKwahtFf9sBjNL4inmzYBuKC5C4VcS SDpdIA+boDiVVx99SA+/3iv1LnOia1TiuvFU+nITLh1VdSJMrzbYdOMHW A==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962548" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962548" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608005" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608005" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:40 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" Subject: [PATCH v6 06/12] x86/keylocker: Define Key Locker CPUID leaf Date: Mon, 10 Apr 2023 15:59:30 -0700 Message-Id: <20230410225936.8940-7-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Define the feature-specific CPUID leaf and bits. Both Key Locker enabling code in the x86 core and AES Key Locker code in the crypto library will reference them. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Separate out the code as a new patch. --- arch/x86/include/asm/keylocker.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h index df84c83228a1..e85dfb6c1524 100644 --- a/arch/x86/include/asm/keylocker.h +++ b/arch/x86/include/asm/keylocker.h @@ -5,6 +5,7 @@ #ifndef __ASSEMBLY__ +#include #include /** @@ -21,5 +22,11 @@ struct iwkey { struct reg_128_bit encryption_key[2]; }; +#define KEYLOCKER_CPUID 0x019 +#define KEYLOCKER_CPUID_EAX_SUPERVISOR BIT(0) +#define KEYLOCKER_CPUID_EBX_AESKLE BIT(0) +#define KEYLOCKER_CPUID_EBX_WIDE BIT(2) +#define KEYLOCKER_CPUID_EBX_BACKUP BIT(4) + #endif /*__ASSEMBLY__ */ #endif /* _ASM_KEYLOCKER_H */ From patchwork Mon Apr 10 22:59:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206823 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FA8FC77B61 for ; Mon, 10 Apr 2023 23:12:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229906AbjDJXMF (ORCPT ); Mon, 10 Apr 2023 19:12:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229791AbjDJXLo (ORCPT ); Mon, 10 Apr 2023 19:11:44 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC3DB213A; Mon, 10 Apr 2023 16:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168302; x=1712704302; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=8tkH3Bjh5/XGpnX2lOKu5HgHTZmmicSiTnYUuWTuPqQ=; b=lHRqCK6LzO9G+NbONBgNxIYSUHxoX+l5wwvnH3BnfAMBxI1NUpjJ2FrB hISoghDU/y8gYWu7PvuVPaa4ulA+GQqzE6dF3UPKqzBug72jqTRQKxwMx DVNaneJyQcxTeZHMZi96Frl8wCph7sEyKZOburBQtIFpqo2FaFVEQ2E6S jXV+ZryAoKhQHPyQGyUM12VWU3KlHHjut5dQlAfpBArb19JFg2uZvtD+L URYD+/d6TvwG7qzc9kzR6TbsYtht0pXtjqHtJbo45+7WMnf3XFMtKqd1j /8pkSe1vxiYLt8W1SCqkO8jtkR2AuZI4PFf1uIcv13qtfMLa7IaX9tMCs A==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962554" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962554" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608008" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608008" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:40 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" Subject: [PATCH v6 07/12] x86/cpu/keylocker: Load an internal wrapping key at boot-time Date: Mon, 10 Apr 2023 15:59:31 -0700 Message-Id: <20230410225936.8940-8-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The Internal Wrapping Key (IWKey) is an entity of Key Locker to encode a clear text key into a key handle. This key is a pivot in protecting user keys. So the value has to be randomized before being loaded in the software-invisible CPU state. IWKey needs to be established before the first user. Given that the only proposed Linux use case for Key Locker is dm-crypt, the feature could be lazily enabled when the first dm-crypt user arrives, but there is no precedent for late enabling of CPU features and it adds maintenance burden without demonstrative benefit outside of minimizing the visibility of Key Locker to userspace. The kernel generates random bytes and load them at boot time. These bytes are flushed out immediately. Setting the CR4.KL bit does not always enable the feature so ensure the dynamic CPU bit (CPUID.AESKLE) is set before loading the key. Given that the Linux Key Locker support is only intended for bare metal dm-crypt consumption, and that switching IWKey per VM is untenable, explicitly skip Key Locker setup in the X86_FEATURE_HYPERVISOR case. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from v5: * Call out the disabling when the feature is available on a virtual machine. Then, it will turn off the feature flag Changes from RFC v2: * Make bare metal only. * Clean up the code (e.g. dynamically allocate the key cache). (Dan Williams) * Massage the changelog. * Move out the LOADIWKEY wrapper and the Key Locker CPUID defines. Note, Dan wonders that given that the only proposed Linux use case for Key Locker is dm-crypt, the feature could be lazily enabled when the first dm-crypt user arrives, but as Dave notes there is no precedent for late enabling of CPU features and it adds maintenance burden without demonstrative benefit outside of minimizing the visibility of Key Locker to userspace. --- arch/x86/include/asm/keylocker.h | 9 ++++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/cpu/common.c | 5 +- arch/x86/kernel/keylocker.c | 82 ++++++++++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 2 + 5 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kernel/keylocker.c diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h index e85dfb6c1524..820ac29c06d9 100644 --- a/arch/x86/include/asm/keylocker.h +++ b/arch/x86/include/asm/keylocker.h @@ -5,6 +5,7 @@ #ifndef __ASSEMBLY__ +#include #include #include @@ -28,5 +29,13 @@ struct iwkey { #define KEYLOCKER_CPUID_EBX_WIDE BIT(2) #define KEYLOCKER_CPUID_EBX_BACKUP BIT(4) +#ifdef CONFIG_X86_KEYLOCKER +void setup_keylocker(struct cpuinfo_x86 *c); +void destroy_keylocker_data(void); +#else +#define setup_keylocker(c) do { } while (0) +#define destroy_keylocker_data() do { } while (0) +#endif + #endif /*__ASSEMBLY__ */ #endif /* _ASM_KEYLOCKER_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index b366641703e3..20babbaf8c0c 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -133,6 +133,7 @@ obj-$(CONFIG_PERF_EVENTS) += perf_regs.o obj-$(CONFIG_TRACING) += tracepoint.o obj-$(CONFIG_SCHED_MC_PRIO) += itmt.o obj-$(CONFIG_X86_UMIP) += umip.o +obj-$(CONFIG_X86_KEYLOCKER) += keylocker.o obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 3ea06b0b4570..a1edd5997e0a 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -58,6 +58,8 @@ #include #include #include +#include + #include #include #include @@ -1874,10 +1876,11 @@ static void identify_cpu(struct cpuinfo_x86 *c) /* Disable the PN if appropriate */ squash_the_stupid_serial_number(c); - /* Set up SMEP/SMAP/UMIP */ + /* Setup various Intel-specific CPU security features */ setup_smep(c); setup_smap(c); setup_umip(c); + setup_keylocker(c); /* Enable FSGSBASE instructions if available. */ if (cpu_has(c, X86_FEATURE_FSGSBASE)) { diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c new file mode 100644 index 000000000000..2519102f72f1 --- /dev/null +++ b/arch/x86/kernel/keylocker.c @@ -0,0 +1,82 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * Setup Key Locker feature and support internal wrapping key + * management. + */ + +#include +#include + +#include +#include +#include + +static __initdata struct keylocker_setup_data { + struct iwkey key; +} kl_setup; + +static void __init generate_keylocker_data(void) +{ + get_random_bytes(&kl_setup.key.integrity_key, sizeof(kl_setup.key.integrity_key)); + get_random_bytes(&kl_setup.key.encryption_key, sizeof(kl_setup.key.encryption_key)); +} + +void __init destroy_keylocker_data(void) +{ + memset(&kl_setup.key, KEY_DESTROY, sizeof(kl_setup.key)); +} + +static void __init load_keylocker(void) +{ + kernel_fpu_begin(); + load_xmm_iwkey(&kl_setup.key); + kernel_fpu_end(); +} + +/** + * setup_keylocker - Enable the feature. + * @c: A pointer to struct cpuinfo_x86 + */ +void __ref setup_keylocker(struct cpuinfo_x86 *c) +{ + if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER)) + goto out; + + if (cpu_feature_enabled(X86_FEATURE_HYPERVISOR)) { + pr_debug("x86/keylocker: Not compatible with a hypervisor.\n"); + goto disable; + } + + cr4_set_bits(X86_CR4_KEYLOCKER); + + if (c == &boot_cpu_data) { + u32 eax, ebx, ecx, edx; + + cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); + /* + * Check the feature readiness via CPUID. Note that the + * CPUID AESKLE bit is conditionally set only when CR4.KL + * is set. + */ + if (!(ebx & KEYLOCKER_CPUID_EBX_AESKLE) || + !(eax & KEYLOCKER_CPUID_EAX_SUPERVISOR)) { + pr_debug("x86/keylocker: Not fully supported.\n"); + goto disable; + } + + generate_keylocker_data(); + } + + load_keylocker(); + + pr_info_once("x86/keylocker: Enabled.\n"); + return; + +disable: + setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER); + pr_info_once("x86/keylocker: Disabled.\n"); +out: + /* Make sure the feature disabled for kexec-reboot. */ + cr4_clear_bits(X86_CR4_KEYLOCKER); +} diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 851477f7d728..880ed9cc8c53 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -84,6 +84,7 @@ #include #include #include +#include /* representing HT siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); @@ -1500,6 +1501,7 @@ void __init native_smp_cpus_done(unsigned int max_cpus) nmi_selftest(); impress_friends(); cache_aps_init(); + destroy_keylocker_data(); } static int __initdata setup_possible_cpus = -1; From patchwork Mon Apr 10 22:59:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206826 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8318BC77B61 for ; Mon, 10 Apr 2023 23:12:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229952AbjDJXMR (ORCPT ); Mon, 10 Apr 2023 19:12:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229872AbjDJXLv (ORCPT ); Mon, 10 Apr 2023 19:11:51 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55A352694; Mon, 10 Apr 2023 16:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168303; x=1712704303; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=OVcWEfHIwxJSYOH3PtUNQyn0mkISh7WF54jUMQ9NGf0=; b=hR9QD36cyQELP1j2wAI4/EAVDcDcUF++1lSJuzaeYWKpHwmdAkPfNi+5 IFuBCPCt806EbcfhH0dG8jB0LcTP6n5010A7fxAr0JxdcmJmBPN+FLuBq 85SwBX025khVGcX2dLgTMPAyfCw3/v+EY2BJMyAH+6nvrZj8EO56LptbN WJeVr3DX07zf9Nb0I62xIfBhGurcmddyReuxQgZNahVywgHeC129vy2bY qiy/6a5PultrulO+0pMTmvgrw28b9lO0TePxaKmjlNSlAG/fyTLHyhSJ0 AUPOXur3fJA2ONsL+d5PCKmENVB3p0AMjdePxrxane6bMN12HnzP+I50S g==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962566" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962566" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608012" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608012" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:40 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , "Rafael J. Wysocki" , linux-pm@vger.kernel.org Subject: [PATCH v6 08/12] x86/PM/keylocker: Restore internal wrapping key on resume from ACPI S3/4 Date: Mon, 10 Apr 2023 15:59:32 -0700 Message-Id: <20230410225936.8940-9-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When the system enters the ACPI S3 or S4 sleep states, the internal wrapping key is discarded. The primary use case for the feature is bare metal dm-crypt. The key needs to be restored properly on wakeup, as dm-crypt does not prompt for the key on resume from suspend. Even the prompt it does perform for unlocking the volume where the hibernation image is stored, it still expects to reuse the key handles within the hibernation image once it is loaded. So it is motivated to meet dm-crypt's expectation that the key handles in the suspend-image remain valid after resume from an S-state. Key Locker provides a mechanism to back up the internal wrapping key in non-volatile storage. The kernel requests a backup right after the key is loaded at boot time. It is copied back to each CPU upon wakeup. While the backup may be maintained in NVM across S5 and G3 "off" states it is not architecturally guaranteed, nor is it expected by dm-crypt which expects to prompt for the key each time the volume is started. The entirety of Key Locker needs to be disabled if the backup mechanism is not available unless CONFIG_SUSPEND=n, otherwise dm-crypt requires the backup to be available. In the event of a key restore failure the kernel proceeds with an initialized IWKey state. This has the effect of invalidating any key handles that might be present in a suspend-image. When this happens dm-crypt will see I/O errors resulting from error returns from crypto_skcipher_{en,de}crypt(). While this will disrupt operations in the current boot, data is not at risk and access is restored at the next reboot to create new handles relative to the current IWKey. Manage a feature-specific flag to communicate with the crypto implementation. This ensures to stop using the AES instructions upon the key restore failure while not turning off the feature. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Acked-by: Rafael J. Wysocki Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: "Rafael J. Wysocki" Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-pm@vger.kernel.org --- Changes from v5: * Fix the 'valid_kl' flag not to be set when the feature is disabled. (Reported by Marvin Hsu marvin.hsu@intel.com) Add the function comment about this. * Improve the error handling in setup_keylocker(). All the error cases fall through the end that disables the feature. Otherwise, all the successful cases return immediately. Changes from v4: * Update the changelog and title. (Rafael Wysocki) Changes from v3: * Fix the build issue with !X86_KEYLOCKER. (Eric Biggers) Changes from RFC v2: * Change the backup key failure handling. (Dan Williams) Changes from RFC v1: * Folded the warning message into the if condition check. (Rafael Wysocki) * Rebased on the changes of the previous patches. * Added error code for key restoration failures. * Moved the restore helper. * Added function descriptions. --- arch/x86/include/asm/keylocker.h | 4 + arch/x86/kernel/keylocker.c | 138 ++++++++++++++++++++++++++++++- arch/x86/power/cpu.c | 2 + 3 files changed, 140 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h index 820ac29c06d9..c1d27fb5a1c3 100644 --- a/arch/x86/include/asm/keylocker.h +++ b/arch/x86/include/asm/keylocker.h @@ -32,9 +32,13 @@ struct iwkey { #ifdef CONFIG_X86_KEYLOCKER void setup_keylocker(struct cpuinfo_x86 *c); void destroy_keylocker_data(void); +void restore_keylocker(void); +extern bool valid_keylocker(void); #else #define setup_keylocker(c) do { } while (0) #define destroy_keylocker_data() do { } while (0) +#define restore_keylocker() do { } while (0) +static inline bool valid_keylocker(void) { return false; } #endif #endif /*__ASSEMBLY__ */ diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c index 2519102f72f1..72d075499067 100644 --- a/arch/x86/kernel/keylocker.c +++ b/arch/x86/kernel/keylocker.c @@ -11,20 +11,45 @@ #include #include #include +#include static __initdata struct keylocker_setup_data { + bool initialized; struct iwkey key; } kl_setup; +/* + * This flag is set with IWKey load. When the key restore fails, it is + * reset. This restore state is exported to the crypto library, then AES-KL + * will not be used there. So, the feature is soft-disabled with this flag. + */ +static bool valid_kl; + +bool valid_keylocker(void) +{ + return valid_kl; +} +EXPORT_SYMBOL_GPL(valid_keylocker); + static void __init generate_keylocker_data(void) { get_random_bytes(&kl_setup.key.integrity_key, sizeof(kl_setup.key.integrity_key)); get_random_bytes(&kl_setup.key.encryption_key, sizeof(kl_setup.key.encryption_key)); } +/* + * This is invoked when the bootup is finished, which means IWKey is + * loaded. Then, the 'valid_kl' flag is set here when the feature is + * enabled. + */ void __init destroy_keylocker_data(void) { + if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER)) + return; + memset(&kl_setup.key, KEY_DESTROY, sizeof(kl_setup.key)); + kl_setup.initialized = true; + valid_kl = true; } static void __init load_keylocker(void) @@ -34,6 +59,27 @@ static void __init load_keylocker(void) kernel_fpu_end(); } +/** + * copy_keylocker - Copy the internal wrapping key from the backup. + * + * Request hardware to copy the key in non-volatile storage to the CPU + * state. + * + * Returns: -EBUSY if the copy fails, 0 if successful. + */ +static int copy_keylocker(void) +{ + u64 status; + + wrmsrl(MSR_IA32_COPY_IWKEY_TO_LOCAL, 1); + + rdmsrl(MSR_IA32_IWKEY_COPY_STATUS, status); + if (status & BIT(0)) + return 0; + else + return -EBUSY; +} + /** * setup_keylocker - Enable the feature. * @c: A pointer to struct cpuinfo_x86 @@ -52,6 +98,7 @@ void __ref setup_keylocker(struct cpuinfo_x86 *c) if (c == &boot_cpu_data) { u32 eax, ebx, ecx, edx; + bool backup_available; cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); /* @@ -65,13 +112,54 @@ void __ref setup_keylocker(struct cpuinfo_x86 *c) goto disable; } + backup_available = !!(ebx & KEYLOCKER_CPUID_EBX_BACKUP); + /* + * The internal wrapping key in CPU state is volatile in + * S3/4 states. So ensure the backup capability along with + * S-states. + */ + if (!backup_available && IS_ENABLED(CONFIG_SUSPEND)) { + pr_debug("x86/keylocker: No key backup support with possible S3/4.\n"); + goto disable; + } + generate_keylocker_data(); - } + load_keylocker(); - load_keylocker(); + /* Backup an internal wrapping key in non-volatile media. */ + if (backup_available) + wrmsrl(MSR_IA32_BACKUP_IWKEY_TO_PLATFORM, 1); - pr_info_once("x86/keylocker: Enabled.\n"); - return; + pr_info("x86/keylocker: Enabled.\n"); + return; + } else { + int rc; + + /* + * Load the internal wrapping key directly when available + * in memory, which is only possible at boot-time. + * + * NB: When system wakes up, this path also recovers the + * internal wrapping key. + */ + if (!kl_setup.initialized) { + load_keylocker(); + return; + } else if (valid_kl) { + rc = copy_keylocker(); + if (!rc) + return; + + /* + * The boot CPU was successful but the key copy + * fails here. Then, the subsequent feature use + * will have inconsistent keys and failures. So, + * invalidate the feature via the flag. + */ + valid_kl = false; + pr_err_once("x86/keylocker: Invalid copy status (rc: %d).\n", rc); + } + } disable: setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER); @@ -80,3 +168,45 @@ void __ref setup_keylocker(struct cpuinfo_x86 *c) /* Make sure the feature disabled for kexec-reboot. */ cr4_clear_bits(X86_CR4_KEYLOCKER); } + +/** + * restore_keylocker - Restore the internal wrapping key. + * + * The boot CPU executes this while other CPUs restore it through the setup + * function. + */ +void restore_keylocker(void) +{ + u64 backup_status; + int rc; + + if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER) || !valid_kl) + return; + + /* + * The IA32_IWKEYBACKUP_STATUS MSR contains a bitmap that indicates + * an invalid backup if bit 0 is set and a read (or write) error if + * bit 2 is set. + */ + rdmsrl(MSR_IA32_IWKEY_BACKUP_STATUS, backup_status); + if (backup_status & BIT(0)) { + rc = copy_keylocker(); + if (rc) + pr_err("x86/keylocker: Invalid copy state (rc: %d).\n", rc); + else + return; + } else { + pr_err("x86/keylocker: The key backup access failed with %s.\n", + (backup_status & BIT(2)) ? "read error" : "invalid status"); + } + + /* + * Now the backup key is not available. Invalidate the feature via + * the flag to avoid any subsequent use. But keep the feature with + * zero IWKeys instead of disabling it. The current users will see + * key handle integrity failure but that's because of the internal + * key change. + */ + pr_err("x86/keylocker: Failed to restore internal wrapping key.\n"); + valid_kl = false; +} diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c index 236447ee9beb..34d2cf946b13 100644 --- a/arch/x86/power/cpu.c +++ b/arch/x86/power/cpu.c @@ -27,6 +27,7 @@ #include #include #include +#include #ifdef CONFIG_X86_32 __visible unsigned long saved_context_ebx; @@ -264,6 +265,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt) x86_platform.restore_sched_clock_state(); cache_bp_restore(); perf_restore_debug_store(); + restore_keylocker(); c = &cpu_data(smp_processor_id()); if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL)) From patchwork Mon Apr 10 22:59:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206825 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45FC6C77B73 for ; Mon, 10 Apr 2023 23:12:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229917AbjDJXMH (ORCPT ); Mon, 10 Apr 2023 19:12:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229802AbjDJXLo (ORCPT ); Mon, 10 Apr 2023 19:11:44 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B823F2699; Mon, 10 Apr 2023 16:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168303; x=1712704303; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=gGdT5ei86w9Hizyjm1aYXsjM9Us57NAQM61P97l7pFk=; b=mezJxstoAG2m6k/+UNrGcnTtbun9WK3YGF54kfGaeAOaCkE/yf+o2IA3 1wejJBeqqW1XZg72EEU5xjFcOp4Zacn37KgHWbCD4oToBh4kCGkzmnWwP AJhC6uRX0lhV08NdSFfUKVuj1fM+5IJoYT5J2WGgWumBjiu22iY8WeA1M HY/PoqZ6G9OlSRqwYGqHHxB+ia4cJOoLfMgWx3CJgLLz8W8/RWj7TuswY TpEYb9vSQtVZJPA6Za6vTQmpV+SpQucbjJ28kIYcRsmdR8xifcBzvsIxa ksq7VTASCNZonqcZMQp//y+QwkM00Bsh0YvsuXuD5S9d5joL0RmRczpV4 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962575" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962575" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608015" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608015" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:41 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, Jonathan Corbet , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , linux-doc@vger.kernel.org Subject: [PATCH v6 09/12] x86/cpu: Add a configuration and command line option for Key Locker Date: Mon, 10 Apr 2023 15:59:33 -0700 Message-Id: <20230410225936.8940-10-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add CONFIG_X86_KEYLOCKER to gate whether Key Locker is initialized at boot. The option is selected by the Key Locker cipher module CRYPTO_AES_KL (to be added in a later patch). Add a new command line option "nokeylocker" to optionally override the default CONFIG_X86_KEYLOCKER=y behavior. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Jonathan Corbet Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Make the option selected by CRYPTO_AES_KL. (Dan Williams) * Massage the changelog and the config option description. --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/x86/Kconfig | 3 +++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 3 files changed, 21 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 5f2ec4b0f927..6534e6217e56 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3655,6 +3655,8 @@ nohugevmalloc [KNL,X86,PPC,ARM64] Disable kernel huge vmalloc mappings. + nokeylocker [X86] Disable Key Locker hardware feature. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index c94297369448..91f2063ab283 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1894,6 +1894,9 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_KEYLOCKER + bool + choice prompt "TSX enable mode" depends on CPU_SUP_INTEL diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index a1edd5997e0a..c5550b8f030d 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -419,6 +419,22 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c) static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | X86_CR4_FSGSBASE | X86_CR4_CET; + +static __init int x86_nokeylocker_setup(char *arg) +{ + /* Expect an exact match without trailing characters. */ + if (strlen(arg)) + return 0; + + if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER); + pr_info("x86/keylocker: Disabled by kernel command line.\n"); + return 1; +} +__setup("nokeylocker", x86_nokeylocker_setup); + static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning); static unsigned long cr4_pinned_bits __ro_after_init; From patchwork Mon Apr 10 22:59:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206829 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75B21C77B61 for ; Mon, 10 Apr 2023 23:12:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229985AbjDJXMi (ORCPT ); Mon, 10 Apr 2023 19:12:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbjDJXME (ORCPT ); Mon, 10 Apr 2023 19:12:04 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7687026A6; Mon, 10 Apr 2023 16:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168304; x=1712704304; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QEwQL4DpThhIDfU++MHf+y7EfevZqyiC5hzpV4vl/iM=; b=XxS+aMyextG+6H/jYjKsDNIGhYv39BsAzR3l1LwL/R0iWsZTYpskNJfM BK9iCciAY/VfbPyMCxbxNY985FG3JRtPiMqRO9/pOOJvSAcdaOr5dcWsK ekp5T5N1mjhQG5J0gThIdQ4qfHXk7fr8vFqYIgBTwTesA6QbroIJOy4VS 2JDRLRjaDLYMoRURS1PyXYrCojJCuMfg+V8Yaf7fPjUlm0BTXAUzcxGJ1 1rm6CnIf2pplAd10l1oeDcNYKJcXH0IKzhYv5nuty9nzSlzXBiK+mo4Mr m2Ny13wYyV4IIALuoWZ04EAfoPoNa+W2G2qzjkeXLSx/1JMq30COiLWbf w==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962585" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962585" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608018" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608018" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:41 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, "David S. Miller" , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" Subject: [PATCH v6 10/12] crypto: x86/aes - Prepare for a new AES implementation Date: Mon, 10 Apr 2023 15:59:34 -0700 Message-Id: <20230410225936.8940-11-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker's AES instruction set ('AES-KL') has a similar programming interface to AES-NI. The internal ABI in the assembly code will have the same prototype as AES-NI. Then, the glue code will be the same as AES-NI's. Refactor the common C code to avoid code duplication. The AES-NI code uses it with a function pointer argument to call back the AES-NI assembly code. So will the AES-KL code. Introduce wrappers for data transformation functions to return an error code. AES-KL may populate an error during data transformation. Export those refactored functions and new wrappers that the AES-KL code will reference. Also move some constant values to a common assembly code. No functional change intended. Signed-off-by: Chang S. Bae Acked-by: Dan Williams Cc: Herbert Xu Cc: "David S. Miller" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Ard Biesheuvel Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from v5: * Clean up the staled function definition -- cbc_crypt_common(). * Ensure kernel_fpu_end() for the possible error return from xts_crypt_common()->crypt1_fn(). Changes from v4: * Drop CBC mode changes. (Eric Biggers) Changes from v3: * Drop ECB and CTR mode changes. (Eric Biggers) * Export symbols. (Eric Biggers) Changes from RFC v2: * Massage the changelog. (Dan Williams) Changes from RFC v1: * Added as a new patch. (Ard Biesheuvel) Link: https://lore.kernel.org/lkml/CAMj1kXGa4f21eH0mdxd1pQsZMUjUr1Btq+Dgw-gC=O-yYft7xw@mail.gmail.com/ --- arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/aes-intel_asm.S | 26 ++++ arch/x86/crypto/aes-intel_glue.c | 127 ++++++++++++++++ arch/x86/crypto/aes-intel_glue.h | 44 ++++++ arch/x86/crypto/aesni-intel_asm.S | 58 +++---- arch/x86/crypto/aesni-intel_glue.c | 235 +++++++++-------------------- arch/x86/crypto/aesni-intel_glue.h | 17 +++ 7 files changed, 305 insertions(+), 204 deletions(-) create mode 100644 arch/x86/crypto/aes-intel_asm.S create mode 100644 arch/x86/crypto/aes-intel_glue.c create mode 100644 arch/x86/crypto/aes-intel_glue.h create mode 100644 arch/x86/crypto/aesni-intel_glue.h diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 9aa46093c91b..8469b0a09cb5 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -47,7 +47,7 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o -aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o +aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o diff --git a/arch/x86/crypto/aes-intel_asm.S b/arch/x86/crypto/aes-intel_asm.S new file mode 100644 index 000000000000..98abf875af79 --- /dev/null +++ b/arch/x86/crypto/aes-intel_asm.S @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +/* + * Constant values shared between AES implementations: + */ + +.pushsection .rodata +.align 16 +.Lcts_permute_table: + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 +#ifdef __x86_64__ +.Lbswap_mask: + .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +#endif +.popsection + +.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 +.align 16 +.Lgf128mul_x_ble_mask: + .octa 0x00000000000000010000000000000087 +.previous diff --git a/arch/x86/crypto/aes-intel_glue.c b/arch/x86/crypto/aes-intel_glue.c new file mode 100644 index 000000000000..d485577e121b --- /dev/null +++ b/arch/x86/crypto/aes-intel_glue.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include +#include +#include +#include +#include +#include +#include +#include "aes-intel_glue.h" + +int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen, + int (*fn)(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len)) +{ + struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int err; + + err = xts_verify_key(tfm, key, keylen); + if (err) + return err; + + keylen /= 2; + + /* first half of xts-key is for crypt */ + err = fn(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, key, keylen); + if (err) + return err; + + /* second half of xts-key is for tweak */ + return fn(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, key + keylen, keylen); +} +EXPORT_SYMBOL_GPL(xts_setkey_common); + +int xts_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct skcipher_request subreq; + struct skcipher_walk walk; + int err; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + + err = skcipher_walk_virt(&walk, req, false); + if (!walk.nbytes) + return err; + + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + blocks * AES_BLOCK_SIZE, req->iv); + req = &subreq; + + err = skcipher_walk_virt(&walk, req, false); + if (!walk.nbytes) + return err; + } else { + tail = 0; + } + + kernel_fpu_begin(); + + /* calculate first value of T */ + err = crypt1_fn(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); + if (err) { + kernel_fpu_end(); + return err; + } + + while (walk.nbytes > 0) { + int nbytes = walk.nbytes; + + if (nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + + err = crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, + nbytes, walk.iv); + kernel_fpu_end(); + if (err) + return err; + + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); + + if (walk.nbytes > 0) + kernel_fpu_begin(); + } + + if (unlikely(tail > 0 && !err)) { + struct scatterlist sg_src[2], sg_dst[2]; + struct scatterlist *src, *dst; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + kernel_fpu_begin(); + err = crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, + walk.nbytes, walk.iv); + kernel_fpu_end(); + if (err) + return err; + + err = skcipher_walk_done(&walk, 0); + } + return err; +} +EXPORT_SYMBOL_GPL(xts_crypt_common); diff --git a/arch/x86/crypto/aes-intel_glue.h b/arch/x86/crypto/aes-intel_glue.h new file mode 100644 index 000000000000..a5bd528ac072 --- /dev/null +++ b/arch/x86/crypto/aes-intel_glue.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Shared glue code between AES implementations, refactored from the AES-NI's. + */ + +#ifndef _AES_INTEL_GLUE_H +#define _AES_INTEL_GLUE_H + +#include + +#define AES_ALIGN 16 +#define AES_ALIGN_ATTR __attribute__((__aligned__(AES_ALIGN))) +#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) +#define AES_ALIGN_EXTRA ((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) +#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AES_ALIGN_EXTRA) +#define XTS_AES_CTX_SIZE (sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA) + +struct aes_xts_ctx { + u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR; + u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR; +}; + +static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) +{ + unsigned long addr = (unsigned long)raw_ctx; + unsigned long align = AES_ALIGN; + + if (align <= crypto_tfm_ctx_alignment()) + align = 1; + + return (struct crypto_aes_ctx *)ALIGN(addr, align); +} + +int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen, + int (*fn)(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len)); + +int xts_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)); + +#endif /* _AES_INTEL_GLUE_H */ diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S index 837c1e0aa021..9ea4f13464d5 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -28,6 +28,7 @@ #include #include #include +#include "aes-intel_asm.S" /* * The following macros are used to move an (un)aligned 16 byte value to/from @@ -1820,10 +1821,10 @@ SYM_FUNC_START_LOCAL(_key_expansion_256b) SYM_FUNC_END(_key_expansion_256b) /* - * int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - * unsigned int key_len) + * int _aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + * unsigned int key_len) */ -SYM_FUNC_START(aesni_set_key) +SYM_FUNC_START(_aesni_set_key) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -1932,12 +1933,12 @@ SYM_FUNC_START(aesni_set_key) #endif FRAME_END RET -SYM_FUNC_END(aesni_set_key) +SYM_FUNC_END(_aesni_set_key) /* - * void aesni_enc(const void *ctx, u8 *dst, const u8 *src) + * void _aesni_enc(const void *ctx, u8 *dst, const u8 *src) */ -SYM_FUNC_START(aesni_enc) +SYM_FUNC_START(_aesni_enc) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -1956,7 +1957,7 @@ SYM_FUNC_START(aesni_enc) #endif FRAME_END RET -SYM_FUNC_END(aesni_enc) +SYM_FUNC_END(_aesni_enc) /* * _aesni_enc1: internal ABI @@ -2124,9 +2125,9 @@ SYM_FUNC_START_LOCAL(_aesni_enc4) SYM_FUNC_END(_aesni_enc4) /* - * void aesni_dec (const void *ctx, u8 *dst, const u8 *src) + * void _aesni_dec (const void *ctx, u8 *dst, const u8 *src) */ -SYM_FUNC_START(aesni_dec) +SYM_FUNC_START(_aesni_dec) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -2146,7 +2147,7 @@ SYM_FUNC_START(aesni_dec) #endif FRAME_END RET -SYM_FUNC_END(aesni_dec) +SYM_FUNC_END(_aesni_dec) /* * _aesni_dec1: internal ABI @@ -2689,21 +2690,6 @@ SYM_FUNC_START(aesni_cts_cbc_dec) RET SYM_FUNC_END(aesni_cts_cbc_dec) -.pushsection .rodata -.align 16 -.Lcts_permute_table: - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 - .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 -#ifdef __x86_64__ -.Lbswap_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -#endif -.popsection - #ifdef __x86_64__ /* * _aesni_inc_init: internal ABI @@ -2819,12 +2805,6 @@ SYM_FUNC_END(aesni_ctr_enc) #endif -.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 -.align 16 -.Lgf128mul_x_ble_mask: - .octa 0x00000000000000010000000000000087 -.previous - /* * _aesni_gf128mul_x_ble: internal ABI * Multiply in GF(2^128) for XTS IVs @@ -2844,10 +2824,10 @@ SYM_FUNC_END(aesni_ctr_enc) pxor KEY, IV; /* - * void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) + * void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_encrypt) +SYM_FUNC_START(_aesni_xts_encrypt) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2996,13 +2976,13 @@ SYM_FUNC_START(aesni_xts_encrypt) movups STATE, (OUTP) jmp .Lxts_enc_ret -SYM_FUNC_END(aesni_xts_encrypt) +SYM_FUNC_END(_aesni_xts_encrypt) /* - * void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) + * void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_decrypt) +SYM_FUNC_START(_aesni_xts_decrypt) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -3158,4 +3138,4 @@ SYM_FUNC_START(aesni_xts_decrypt) movups STATE, (OUTP) jmp .Lxts_dec_ret -SYM_FUNC_END(aesni_xts_decrypt) +SYM_FUNC_END(_aesni_xts_decrypt) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index a5b0cb3efeba..e7a9e8e01680 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -36,33 +36,24 @@ #include #include +#include "aes-intel_glue.h" +#include "aesni-intel_glue.h" -#define AESNI_ALIGN 16 -#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN))) -#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) #define RFC4106_HASH_SUBKEY_SIZE 16 -#define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) -#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AESNI_ALIGN_EXTRA) -#define XTS_AES_CTX_SIZE (sizeof(struct aesni_xts_ctx) + AESNI_ALIGN_EXTRA) /* This data is stored at the end of the crypto_tfm struct. * It's a type of per "session" data storage location. * This needs to be 16 byte aligned. */ struct aesni_rfc4106_gcm_ctx { - u8 hash_subkey[16] AESNI_ALIGN_ATTR; - struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR; + u8 hash_subkey[16] AES_ALIGN_ATTR; + struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR; u8 nonce[4]; }; struct generic_gcmaes_ctx { - u8 hash_subkey[16] AESNI_ALIGN_ATTR; - struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR; -}; - -struct aesni_xts_ctx { - u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; - u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; + u8 hash_subkey[16] AES_ALIGN_ATTR; + struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR; }; #define GCM_BLOCK_LEN 16 @@ -80,10 +71,10 @@ struct gcm_context_data { u8 hash_keys[GCM_BLOCK_LEN * 16]; }; -asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len); -asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); -asmlinkage void aesni_dec(const void *ctx, u8 *out, const u8 *in); +asmlinkage int _aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len); +asmlinkage void _aesni_enc(const void *ctx, u8 *out, const u8 *in); +asmlinkage void _aesni_dec(const void *ctx, u8 *out, const u8 *in); asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); asmlinkage void aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, @@ -97,14 +88,51 @@ asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); +int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len) +{ + return _aesni_set_key(ctx, in_key, key_len); +} +EXPORT_SYMBOL_GPL(aesni_set_key); + +int aesni_enc(const void *ctx, u8 *out, const u8 *in) +{ + _aesni_enc(ctx, out, in); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_enc); + +int aesni_dec(const void *ctx, u8 *out, const u8 *in) +{ + _aesni_dec(ctx, out, in); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_dec); + #define AVX_GEN2_OPTSIZE 640 #define AVX_GEN4_OPTSIZE 4096 -asmlinkage void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); +asmlinkage void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); +asmlinkage void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv); + +int aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_xts_encrypt(ctx, out, in, len, iv); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_xts_encrypt); + +int aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_xts_decrypt(ctx, out, in, len, iv); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_xts_decrypt); #ifdef CONFIG_X86_64 @@ -201,7 +229,7 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(gcm_use_avx2); static inline struct aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) { - unsigned long align = AESNI_ALIGN; + unsigned long align = AES_ALIGN; if (align <= crypto_tfm_ctx_alignment()) align = 1; @@ -211,7 +239,7 @@ aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) static inline struct generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) { - unsigned long align = AESNI_ALIGN; + unsigned long align = AES_ALIGN; if (align <= crypto_tfm_ctx_alignment()) align = 1; @@ -219,16 +247,6 @@ generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) } #endif -static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) -{ - unsigned long addr = (unsigned long)raw_ctx; - unsigned long align = AESNI_ALIGN; - - if (align <= crypto_tfm_ctx_alignment()) - align = 1; - return (struct crypto_aes_ctx *)ALIGN(addr, align); -} - static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key, unsigned int key_len) { @@ -243,7 +261,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, err = aes_expandkey(ctx, in_key, key_len); else { kernel_fpu_begin(); - err = aesni_set_key(ctx, in_key, key_len); + err = _aesni_set_key(ctx, in_key, key_len); kernel_fpu_end(); } @@ -264,7 +282,7 @@ static void aesni_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) aes_encrypt(ctx, dst, src); } else { kernel_fpu_begin(); - aesni_enc(ctx, dst, src); + _aesni_enc(ctx, dst, src); kernel_fpu_end(); } } @@ -277,7 +295,7 @@ static void aesni_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) aes_decrypt(ctx, dst, src); } else { kernel_fpu_begin(); - aesni_dec(ctx, dst, src); + _aesni_dec(ctx, dst, src); kernel_fpu_end(); } } @@ -491,7 +509,7 @@ static int cts_cbc_decrypt(struct skcipher_request *req) #ifdef CONFIG_X86_64 static void aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv) + const u8 *in, unsigned int len, u8 *iv) { /* * based on key length, override with the by8 version @@ -528,7 +546,7 @@ static int ctr_crypt(struct skcipher_request *req) nbytes &= ~AES_BLOCK_MASK; if (walk.nbytes == walk.total && nbytes > 0) { - aesni_enc(ctx, keystream, walk.iv); + _aesni_enc(ctx, keystream, walk.iv); crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes, walk.src.virt.addr + walk.nbytes - nbytes, keystream, nbytes); @@ -673,8 +691,8 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req, u8 *iv, void *aes_ctx, u8 *auth_tag, unsigned long auth_tag_len) { - u8 databuf[sizeof(struct gcm_context_data) + (AESNI_ALIGN - 8)] __aligned(8); - struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AESNI_ALIGN); + u8 databuf[sizeof(struct gcm_context_data) + (AES_ALIGN - 8)] __aligned(8); + struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AES_ALIGN); unsigned long left = req->cryptlen; struct scatter_walk assoc_sg_walk; struct skcipher_walk walk; @@ -829,8 +847,8 @@ static int helper_rfc4106_encrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); unsigned int i; __be32 counter = cpu_to_be32(1); @@ -857,8 +875,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); unsigned int i; if (unlikely(req->assoclen != 16 && req->assoclen != 20)) @@ -883,128 +901,17 @@ static int helper_rfc4106_decrypt(struct aead_request *req) static int xts_aesni_setkey(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen) { - struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); - int err; - - err = xts_verify_key(tfm, key, keylen); - if (err) - return err; - - keylen /= 2; - - /* first half of xts-key is for crypt */ - err = aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, - key, keylen); - if (err) - return err; - - /* second half of xts-key is for tweak */ - return aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, - key + keylen, keylen); -} - -static int xts_crypt(struct skcipher_request *req, bool encrypt) -{ - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); - int tail = req->cryptlen % AES_BLOCK_SIZE; - struct skcipher_request subreq; - struct skcipher_walk walk; - int err; - - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - - err = skcipher_walk_virt(&walk, req, false); - if (!walk.nbytes) - return err; - - if (unlikely(tail > 0 && walk.nbytes < walk.total)) { - int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; - - skcipher_walk_abort(&walk); - - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, - skcipher_request_flags(req), - NULL, NULL); - skcipher_request_set_crypt(&subreq, req->src, req->dst, - blocks * AES_BLOCK_SIZE, req->iv); - req = &subreq; - - err = skcipher_walk_virt(&walk, req, false); - if (!walk.nbytes) - return err; - } else { - tail = 0; - } - - kernel_fpu_begin(); - - /* calculate first value of T */ - aesni_enc(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); - - while (walk.nbytes > 0) { - int nbytes = walk.nbytes; - - if (nbytes < walk.total) - nbytes &= ~(AES_BLOCK_SIZE - 1); - - if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - nbytes, walk.iv); - else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - nbytes, walk.iv); - kernel_fpu_end(); - - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); - - if (walk.nbytes > 0) - kernel_fpu_begin(); - } - - if (unlikely(tail > 0 && !err)) { - struct scatterlist sg_src[2], sg_dst[2]; - struct scatterlist *src, *dst; - - dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); - if (req->dst != req->src) - dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); - - skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, - req->iv); - - err = skcipher_walk_virt(&walk, &subreq, false); - if (err) - return err; - - kernel_fpu_begin(); - if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - kernel_fpu_end(); - - err = skcipher_walk_done(&walk, 0); - } - return err; + return xts_setkey_common(tfm, key, keylen, aes_set_key_common); } static int xts_encrypt(struct skcipher_request *req) { - return xts_crypt(req, true); + return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc); } static int xts_decrypt(struct skcipher_request *req) { - return xts_crypt(req, false); + return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc); } static struct crypto_alg aesni_cipher_alg = { @@ -1160,8 +1067,8 @@ static int generic_gcmaes_encrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); __be32 counter = cpu_to_be32(1); memcpy(iv, req->iv, 12); @@ -1177,8 +1084,8 @@ static int generic_gcmaes_decrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); memcpy(iv, req->iv, 12); *((__be32 *)(iv+12)) = counter; diff --git a/arch/x86/crypto/aesni-intel_glue.h b/arch/x86/crypto/aesni-intel_glue.h new file mode 100644 index 000000000000..81ecacb4e54c --- /dev/null +++ b/arch/x86/crypto/aesni-intel_glue.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Support for Intel AES-NI instructions. This file contains function + * prototypes to be referenced for other AES implementations + */ + +int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); + +int aesni_enc(const void *ctx, u8 *out, const u8 *in); +int aesni_dec(const void *ctx, u8 *out, const u8 *in); + +int aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +int aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); + From patchwork Mon Apr 10 22:59:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206827 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 964ACC77B61 for ; Mon, 10 Apr 2023 23:12:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229970AbjDJXM0 (ORCPT ); Mon, 10 Apr 2023 19:12:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229888AbjDJXME (ORCPT ); Mon, 10 Apr 2023 19:12:04 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3068426A0; Mon, 10 Apr 2023 16:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168304; x=1712704304; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=oHbg3AIw6EWjGuPusXhPaguYziSB624KfaS9uRwcK7A=; b=QOSepriM0sBRHpQU/S0aJFUuTodmrVCHiwGcl67dnvk7CjT7N6IuCSrS QisXFEn5oNx72b/+j9lwKniUPb1m6yOB2oV4DSh50/+JDEzuQs5ufxZ/T d8gYaEPqvqEanp+GiKFjEAdLvcCHMGokIvu5X0wXyQbM0MWTYuj6P4Obf pTEnU5kM6GdFsZrvNB8Ken04IdC0dVlEAqy5gu6K0P4YCr96oLsIRDFQh Kagoxuxsjx9qgfaE5wwHjJtvmAS/lA6RGWcy+vCP7GSfy61l4iASaDbXu jKG6WRXn0YNiJCdbeiiqvXov2FyfXjD18pckbSOv43vCMy6wQ7lZF/Rzz Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962596" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962596" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608021" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608021" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:41 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, "David S. Miller" , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Nathan Chancellor , Nick Desaulniers , Tom Rix Subject: [PATCH v6 11/12] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Date: Mon, 10 Apr 2023 15:59:35 -0700 Message-Id: <20230410225936.8940-12-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker is a CPU feature to reduce key exfiltration opportunities while maintaining a programming interface similar to AES-NI. It converts the AES key into an encoded form, called the 'key handle'. The key handle is a wrapped version of the clear-text key where the wrapping key has limited exposure. Once converted via setkey(), all subsequent data encryption using new AES instructions ('AES-KL') uses this key handle, reducing the exposure of private key material in memory. AES-KL is analogous to that of AES-NI. Most assembly code is translated from the AES-NI code. They are operational in both 32-bit and 64-bit modes like AES-NI. However, users need to be aware of the following differences: == Key Handle == AES-KL may fail with an invalid key handle. It could be corrupted or fail with handle restriction. A key handle may be encoded with some restrictions. The implementation restricts every handle only available in kernel mode via setkey(). === AES Compliance === Key Locker is not AES compliant in that it lacks support for 192-bit keys. However, per the expectations of Linux crypto-cipher implementations the software cipher implementation must support all the AES compliant key-sizes. The AES-KL cipher implementation achieves this constraint by logging a warning and falling back to AES-NI. In other words the 192-bit key-size limitation for what can be converted into a Key Locker key-handle is only documented, not enforced. This along with the below performance and failure mode implications is an end-user consideration for selecting AES-KL vs AES-NI. == API Limitation == The setkey() function transforms an AES key to a handle. An extended key is a usual outcome of setkey() in other AES cipher implementations. For this reason, a setkey() failure does not fall back to the other. So, AES-KL will be exposed via synchronous interfaces only. == Wrapping-key Restore Failure == The setkey() failure is also possible with the wrapping-key restore failure. In an event of hardware failure, the wrapping key is lost from deep sleep states. Then, setkey() will fail with the ENODEV error. And the suspended data transforming task may resume but will fail due to the wrapping key change. == Performance == This feature comes with some performance penalties vs AES-NI. The cryptsetup benchmark indicates Key Locker raw throughput can be ~5x slower than AES-NI. For disk encryption, storage bandwidth may be the bottleneck before encryption bandwidth, but the potential performance difference is why AES-KL is advertised as a distinct cipher in /proc/crypto rather than the kernel transparently replacing AES-NI usage with AES-KL. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: Herbert Xu Cc: "David S. Miller" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Nathan Chancellor Cc: Nick Desaulniers Cc: Tom Rix Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from v5: * Replace the ret instruction with RET as rebased on the upstream -- commit f94909ceb1ed ("x86: Prepare asm files for straight-line-speculation"). Changes from v3: * Exclude non-AES-KL objects. (Eric Biggers) * Simplify the assembler dependency check. (Peter Zijlstra) * Trim the Kconfig help text. (Dan Williams) * Fix a defined-but-not-used warning. Changes from RFC v2: * Move out each mode support in new patches. * Update the changelog to describe the limitation and the tradeoff clearly. (Andy Lutomirski) Changes from RFC v1: * Rebased on the refactored code. (Ard Biesheuvel) * Dropped exporting the single block interface. (Ard Biesheuvel) * Fixed the fallback and error handling paths. (Ard Biesheuvel) * Revised the module description. (Dave Hansen and Peter Zijlsta) * Made the build depend on the binutils version to support new instructions. (Borislav Petkov and Peter Zijlstra) * Updated the changelog accordingly. Link: https://lore.kernel.org/lkml/CAMj1kXGa4f21eH0mdxd1pQsZMUjUr1Btq+Dgw-gC=O-yYft7xw@mail.gmail.com/ --- arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aeskl-intel_asm.S | 184 +++++++++++++++++++++++++++++ arch/x86/crypto/aeskl-intel_glue.c | 115 ++++++++++++++++++ crypto/Kconfig | 36 ++++++ 4 files changed, 338 insertions(+) create mode 100644 arch/x86/crypto/aeskl-intel_asm.S create mode 100644 arch/x86/crypto/aeskl-intel_glue.c diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 8469b0a09cb5..ab9bd2b102dd 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o +obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o +aeskl-intel-y := aeskl-intel_asm.o aeskl-intel_glue.o + obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o diff --git a/arch/x86/crypto/aeskl-intel_asm.S b/arch/x86/crypto/aeskl-intel_asm.S new file mode 100644 index 000000000000..17c7b306a766 --- /dev/null +++ b/arch/x86/crypto/aeskl-intel_asm.S @@ -0,0 +1,184 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Implement AES algorithm using Intel AES Key Locker instructions. + * + * Most code is based from the AES-NI implementation, aesni-intel_asm.S + * + */ + +#include +#include +#include +#include "aes-intel_asm.S" + +.text + +#define STATE1 %xmm0 +#define STATE2 %xmm1 +#define STATE3 %xmm2 +#define STATE4 %xmm3 +#define STATE5 %xmm4 +#define STATE6 %xmm5 +#define STATE7 %xmm6 +#define STATE8 %xmm7 +#define STATE STATE1 + +#define IV %xmm9 +#define KEY %xmm10 +#define BSWAP_MASK %xmm11 +#define CTR %xmm12 +#define INC %xmm13 + +#ifdef __x86_64__ +#define IN1 %xmm8 +#define IN2 %xmm9 +#define IN3 %xmm10 +#define IN4 %xmm11 +#define IN5 %xmm12 +#define IN6 %xmm13 +#define IN7 %xmm14 +#define IN8 %xmm15 +#define IN IN1 +#define TCTR_LOW %r11 +#else +#define IN %xmm1 +#endif + +#ifdef __x86_64__ +#define AREG %rax +#define HANDLEP %rdi +#define OUTP %rsi +#define KLEN %r9d +#define INP %rdx +#define T1 %r10 +#define LEN %rcx +#define IVP %r8 +#else +#define AREG %eax +#define HANDLEP %edi +#define OUTP AREG +#define KLEN %ebx +#define INP %edx +#define T1 %ecx +#define LEN %esi +#define IVP %ebp +#endif + +#define UKEYP OUTP +#define GF128MUL_MASK %xmm11 + +/* + * int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len) + */ +SYM_FUNC_START(aeskl_setkey) + FRAME_BEGIN +#ifndef __x86_64__ + push HANDLEP + movl (FRAME_OFFSET+8)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+12)(%esp), UKEYP # in_key + movl (FRAME_OFFSET+16)(%esp), %edx # key_len +#endif + movl %edx, 480(HANDLEP) + movdqu (UKEYP), STATE1 + mov $1, %eax + cmp $16, %dl + je .Lsetkey_128 + + movdqu 0x10(UKEYP), STATE2 + encodekey256 %eax, %eax + movdqu STATE4, 0x30(HANDLEP) + jmp .Lsetkey_end +.Lsetkey_128: + encodekey128 %eax, %eax + +.Lsetkey_end: + movdqu STATE1, (HANDLEP) + movdqu STATE2, 0x10(HANDLEP) + movdqu STATE3, 0x20(HANDLEP) + + xor AREG, AREG +#ifndef __x86_64__ + popl HANDLEP +#endif + FRAME_END + RET +SYM_FUNC_END(aeskl_setkey) + +/* + * int _aeskl_enc(const void *ctx, u8 *dst, const u8 *src) + */ +SYM_FUNC_START(_aeskl_enc) + FRAME_BEGIN +#ifndef __x86_64__ + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+12)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+16)(%esp), OUTP # dst + movl (FRAME_OFFSET+20)(%esp), INP # src +#endif + movdqu (INP), STATE + movl 480(HANDLEP), KLEN + + cmp $16, KLEN + je .Lenc_128 + aesenc256kl (HANDLEP), STATE + jz .Lenc_err + jmp .Lenc_noerr +.Lenc_128: + aesenc128kl (HANDLEP), STATE + jz .Lenc_err + +.Lenc_noerr: + xor AREG, AREG + jmp .Lenc_end +.Lenc_err: + mov $1, AREG +.Lenc_end: + movdqu STATE, (OUTP) +#ifndef __x86_64__ + popl KLEN + popl HANDLEP +#endif + FRAME_END + RET +SYM_FUNC_END(_aeskl_enc) + +/* + * int _aeskl_dec(const void *ctx, u8 *dst, const u8 *src) + */ +SYM_FUNC_START(_aeskl_dec) + FRAME_BEGIN +#ifndef __x86_64__ + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+12)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+16)(%esp), OUTP # dst + movl (FRAME_OFFSET+20)(%esp), INP # src +#endif + movdqu (INP), STATE + mov 480(HANDLEP), KLEN + + cmp $16, KLEN + je .Ldec_128 + aesdec256kl (HANDLEP), STATE + jz .Ldec_err + jmp .Ldec_noerr +.Ldec_128: + aesdec128kl (HANDLEP), STATE + jz .Ldec_err + +.Ldec_noerr: + xor AREG, AREG + jmp .Ldec_end +.Ldec_err: + mov $1, AREG +.Ldec_end: + movdqu STATE, (OUTP) +#ifndef __x86_64__ + popl KLEN + popl HANDLEP +#endif + FRAME_END + RET +SYM_FUNC_END(_aeskl_dec) + diff --git a/arch/x86/crypto/aeskl-intel_glue.c b/arch/x86/crypto/aeskl-intel_glue.c new file mode 100644 index 000000000000..0062baaaf7b2 --- /dev/null +++ b/arch/x86/crypto/aeskl-intel_glue.c @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Support for AES Key Locker instructions. This file contains glue + * code and the real AES implementation is in aeskl-intel_asm.S. + * + * Most code is based on AES-NI glue code, aesni-intel_glue.c + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "aes-intel_glue.h" +#include "aesni-intel_glue.h" + +asmlinkage int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); + +asmlinkage int _aeskl_enc(const void *ctx, u8 *out, const u8 *in); +asmlinkage int _aeskl_dec(const void *ctx, u8 *out, const u8 *in); + +static int __maybe_unused aeskl_setkey_common(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len) +{ + struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx); + int err; + + if (!crypto_simd_usable()) + return -EBUSY; + + if (key_len != AES_KEYSIZE_128 && key_len != AES_KEYSIZE_192 && + key_len != AES_KEYSIZE_256) + return -EINVAL; + + kernel_fpu_begin(); + if (unlikely(key_len == AES_KEYSIZE_192)) { + pr_warn_once("AES-KL does not support 192-bit key. Use AES-NI.\n"); + err = aesni_set_key(ctx, in_key, key_len); + } else { + if (!valid_keylocker()) + err = -ENODEV; + else + err = aeskl_setkey(ctx, in_key, key_len); + } + kernel_fpu_end(); + + return err; +} + +static inline u32 keylength(const void *raw_ctx) +{ + struct crypto_aes_ctx *ctx = aes_ctx((void *)raw_ctx); + + return ctx->key_length; +} + +static inline int aeskl_enc(const void *ctx, u8 *out, const u8 *in) +{ + if (unlikely(keylength(ctx) == AES_KEYSIZE_192)) + return -EINVAL; + else if (!valid_keylocker()) + return -ENODEV; + else if (_aeskl_enc(ctx, out, in)) + return -EINVAL; + else + return 0; +} + +static inline int aeskl_dec(const void *ctx, u8 *out, const u8 *in) +{ + if (unlikely(keylength(ctx) == AES_KEYSIZE_192)) + return -EINVAL; + else if (!valid_keylocker()) + return -ENODEV; + else if (_aeskl_dec(ctx, out, in)) + return -EINVAL; + else + return 0; +} + +static int __init aeskl_init(void) +{ + if (!valid_keylocker()) + return -ENODEV; + + /* + * AES-KL itself does not depend on AES-NI. But AES-KL does not + * support 192-bit keys. To make itself AES-compliant, it falls + * back to AES-NI. + */ + if (!boot_cpu_has(X86_FEATURE_AES)) + return -ENODEV; + + return 0; +} + +static void __exit aeskl_exit(void) +{ + return; +} + +late_initcall(aeskl_init); +module_exit(aeskl_exit); + +MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm, AES Key Locker implementation"); +MODULE_LICENSE("GPL"); +MODULE_ALIAS_CRYPTO("aes"); diff --git a/crypto/Kconfig b/crypto/Kconfig index 9c86f7045157..e432d1ded391 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -573,6 +573,42 @@ config CRYPTO_TWOFISH See https://www.schneier.com/twofish.html for further information. +config AS_HAS_KEYLOCKER + def_bool $(as-instr,encodekey256 %eax$(comma)%eax) + help + Supported by binutils >= 2.36 and LLVM integrated assembler >= V12 + +config CRYPTO_AES_KL + tristate "AES cipher algorithms (AES-KL)" + depends on AS_HAS_KEYLOCKER + depends on CRYPTO_AES_NI_INTEL + select X86_KEYLOCKER + + help + Key Locker provides AES SIMD instructions (AES-KL) for secure + data encryption and decryption. While this new instruction + set is analogous to AES-NI, AES-KL supports to encode an AES + key to an encoded form ('key handle') and uses it to transform + data instead of accessing the AES key. + + The setkey() transforms an AES key to a key handle, then the AES + key is no longer needed for data transformation. A user may + displace their keys from possible exposition. + + This key encryption is done by the CPU-internal wrapping key. + This wrapping key support is provided with X86_KEYLOCKER. + + AES-KL supports 128-/256-bit keys only. While giving a 192-bit + key does not return an error, as the AES-NI driver is chosen to + process it, the claimed security property is not available with + that. + + Bare metal disk encryption is the preferred use case. Key Locker + usage requires explicit opt-in at setup time. So select it if + unsure. + + See Documentation/x86/keylocker.rst for more details. + config CRYPTO_TWOFISH_COMMON tristate help From patchwork Mon Apr 10 22:59:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 13206828 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07711C76196 for ; Mon, 10 Apr 2023 23:12:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229980AbjDJXMh (ORCPT ); Mon, 10 Apr 2023 19:12:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbjDJXME (ORCPT ); Mon, 10 Apr 2023 19:12:04 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2ACE26AC; Mon, 10 Apr 2023 16:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681168304; x=1712704304; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=8JvxfkPIvKe4GbMc6HW6G0HpLSMAyF4KJSCjKVejYqE=; b=jjR3M5XmtoBqtJpoxKdBMfubQU5AgLbbk1SYU5ew9Wm6BRTYgXiP8Ma4 UseJI822rjwAfgIcjoEHuxKEHlp1o1h7tB8LPTpylachP61gZ7kfe/6+5 gPU+PgvdJH7AeQedLHcd5MaAwYoavgL2AO4sR/U7wbu9VOg790lWpHUSU Lvjne8mf0gi68HIXyCo+XbvEvJn6nGE798MJ7EmszD0MxKwATxtjibhkw 5XA4zFiWjV9JQbtpWs4T5kU1XtZYYheDr+ckiXqWK8YAaL+ypYd522Dcq 6WVsqu1ajSsZ5Chb8M2+zT0I18JDcFdxonDx75YGgfeQ0z6mpzYCkzsqR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="340962609" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="340962609" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2023 16:11:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10676"; a="757608024" X-IronPort-AV: E=Sophos;i="5.98,335,1673942400"; d="scan'208";a="757608024" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by fmsmga004.fm.intel.com with ESMTP; 10 Apr 2023 16:11:42 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dm-devel@redhat.com Cc: ebiggers@kernel.org, gmazyland@gmail.com, luto@kernel.org, dave.hansen@linux.intel.com, tglx@linutronix.de, bp@suse.de, mingo@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au, ardb@kernel.org, dan.j.williams@intel.com, bernie.keany@intel.com, charishma1.gairuboyina@intel.com, lalithambika.krishnakumar@intel.com, chang.seok.bae@intel.com, "David S. Miller" , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" Subject: [PATCH v6 12/12] crypto: x86/aes-kl - Support XTS mode Date: Mon, 10 Apr 2023 15:59:36 -0700 Message-Id: <20230410225936.8940-13-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230410225936.8940-1-chang.seok.bae@intel.com> References: <20220112211258.21115-1-chang.seok.bae@intel.com> <20230410225936.8940-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Implement XTS mode using AES-KL. Export the methods with a lower priority than AES-NI to avoid from selected by default. The assembly code clobbers more than eight 128-bit registers to make use of performant wide instructions that process eight 128-bit blocks at once. But that many 128-bit registers are not available in 32-bit mode, so support 64-bit mode only. Signed-off-by: Chang S. Bae Acked-by: Dan Williams Cc: Herbert Xu Cc: "David S. Miller" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from v5: * Replace the ret instruction with RET as rebased on the upstream -- commit f94909ceb1ed ("x86: Prepare asm files for straight-line-speculation"). Changes from RFC v2: * Separate out the code as a new patch. --- arch/x86/crypto/aeskl-intel_asm.S | 449 +++++++++++++++++++++++++++++ arch/x86/crypto/aeskl-intel_glue.c | 107 ++++++- 2 files changed, 553 insertions(+), 3 deletions(-) diff --git a/arch/x86/crypto/aeskl-intel_asm.S b/arch/x86/crypto/aeskl-intel_asm.S index 17c7b306a766..05134327bcb2 100644 --- a/arch/x86/crypto/aeskl-intel_asm.S +++ b/arch/x86/crypto/aeskl-intel_asm.S @@ -182,3 +182,452 @@ SYM_FUNC_START(_aeskl_dec) RET SYM_FUNC_END(_aeskl_dec) +#ifdef __x86_64__ + +/* + * XTS implementation + */ + +/* + * _aeskl_gf128mul_x_ble: internal ABI + * Multiply in GF(2^128) for XTS IVs + * input: + * IV: current IV + * GF128MUL_MASK == mask with 0x87 and 0x01 + * output: + * IV: next IV + * changed: + * CTR: == temporary value + */ +#define _aeskl_gf128mul_x_ble() \ + pshufd $0x13, IV, KEY; \ + paddq IV, IV; \ + psrad $31, KEY; \ + pand GF128MUL_MASK, KEY; \ + pxor KEY, IV; + +/* + * int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) + */ +SYM_FUNC_START(_aeskl_xts_encrypt) + FRAME_BEGIN + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK + movups (IVP), IV + + mov 480(HANDLEP), KLEN + +.Lxts_enc8: + sub $128, LEN + jl .Lxts_enc1_pre + + movdqa IV, STATE1 + movdqu (INP), INC + pxor INC, STATE1 + movdqu IV, (OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE5 + movdqu 0x40(INP), INC + pxor INC, STATE5 + movdqu IV, 0x40(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE6 + movdqu 0x50(INP), INC + pxor INC, STATE6 + movdqu IV, 0x50(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE7 + movdqu 0x60(INP), INC + pxor INC, STATE7 + movdqu IV, 0x60(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE8 + movdqu 0x70(INP), INC + pxor INC, STATE8 + movdqu IV, 0x70(OUTP) + + cmp $16, KLEN + je .Lxts_enc8_128 + aesencwide256kl (%rdi) + jz .Lxts_enc_ret_err + jmp .Lxts_enc8_end +.Lxts_enc8_128: + aesencwide128kl (%rdi) + jz .Lxts_enc_ret_err + +.Lxts_enc8_end: + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC + pxor INC, STATE2 + movdqu STATE2, 0x10(OUTP) + + movdqu 0x20(OUTP), INC + pxor INC, STATE3 + movdqu STATE3, 0x20(OUTP) + + movdqu 0x30(OUTP), INC + pxor INC, STATE4 + movdqu STATE4, 0x30(OUTP) + + movdqu 0x40(OUTP), INC + pxor INC, STATE5 + movdqu STATE5, 0x40(OUTP) + + movdqu 0x50(OUTP), INC + pxor INC, STATE6 + movdqu STATE6, 0x50(OUTP) + + movdqu 0x60(OUTP), INC + pxor INC, STATE7 + movdqu STATE7, 0x60(OUTP) + + movdqu 0x70(OUTP), INC + pxor INC, STATE8 + movdqu STATE8, 0x70(OUTP) + + _aeskl_gf128mul_x_ble() + + add $128, INP + add $128, OUTP + test LEN, LEN + jnz .Lxts_enc8 + +.Lxts_enc_ret_iv: + movups IV, (IVP) +.Lxts_enc_ret_noerr: + xor AREG, AREG + jmp .Lxts_enc_ret +.Lxts_enc_ret_err: + mov $1, AREG +.Lxts_enc_ret: + FRAME_END + RET + +.Lxts_enc1_pre: + add $128, LEN + jz .Lxts_enc_ret_iv + sub $16, LEN + jl .Lxts_enc_cts4 + +.Lxts_enc1: + movdqu (INP), STATE1 + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_enc1_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + jmp .Lxts_enc1_end +.Lxts_enc1_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + +.Lxts_enc1_end: + pxor IV, STATE1 + _aeskl_gf128mul_x_ble() + + test LEN, LEN + jz .Lxts_enc1_out + + add $16, INP + sub $16, LEN + jl .Lxts_enc_cts1 + + movdqu STATE1, (OUTP) + add $16, OUTP + jmp .Lxts_enc1 + +.Lxts_enc1_out: + movdqu STATE1, (OUTP) + jmp .Lxts_enc_ret_iv + +.Lxts_enc_cts4: + movdqu STATE8, STATE1 + sub $16, OUTP + +.Lxts_enc_cts1: + lea .Lcts_permute_table(%rip), T1 + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), STATE2 + movaps STATE1, STATE3 + pshufb STATE2, STATE1 + movups STATE1, (LEN) + + movups (IVP), STATE1 + pshufb STATE1, IN1 + pblendvb STATE3, IN1 + movaps IN1, STATE1 + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_enc1_cts_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + jmp .Lxts_enc1_cts_end +.Lxts_enc1_cts_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + +.Lxts_enc1_cts_end: + pxor IV, STATE1 + movups STATE1, (OUTP) + jmp .Lxts_enc_ret_noerr +SYM_FUNC_END(_aeskl_xts_encrypt) + +/* + * int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) + */ +SYM_FUNC_START(_aeskl_xts_decrypt) + FRAME_BEGIN + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK + movups (IVP), IV + + mov 480(HANDLEP), KLEN + + test $15, LEN + jz .Lxts_dec8 + sub $16, LEN + +.Lxts_dec8: + sub $128, LEN + jl .Lxts_dec1_pre + + movdqa IV, STATE1 + movdqu (INP), INC + pxor INC, STATE1 + movdqu IV, (OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE5 + movdqu 0x40(INP), INC + pxor INC, STATE5 + movdqu IV, 0x40(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE6 + movdqu 0x50(INP), INC + pxor INC, STATE6 + movdqu IV, 0x50(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE7 + movdqu 0x60(INP), INC + pxor INC, STATE7 + movdqu IV, 0x60(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE8 + movdqu 0x70(INP), INC + pxor INC, STATE8 + movdqu IV, 0x70(OUTP) + + cmp $16, KLEN + je .Lxts_dec8_128 + aesdecwide256kl (%rdi) + jz .Lxts_dec_ret_err + jmp .Lxts_dec8_end +.Lxts_dec8_128: + aesdecwide128kl (%rdi) + jz .Lxts_dec_ret_err + +.Lxts_dec8_end: + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC + pxor INC, STATE2 + movdqu STATE2, 0x10(OUTP) + + movdqu 0x20(OUTP), INC + pxor INC, STATE3 + movdqu STATE3, 0x20(OUTP) + + movdqu 0x30(OUTP), INC + pxor INC, STATE4 + movdqu STATE4, 0x30(OUTP) + + movdqu 0x40(OUTP), INC + pxor INC, STATE5 + movdqu STATE5, 0x40(OUTP) + + movdqu 0x50(OUTP), INC + pxor INC, STATE6 + movdqu STATE6, 0x50(OUTP) + + movdqu 0x60(OUTP), INC + pxor INC, STATE7 + movdqu STATE7, 0x60(OUTP) + + movdqu 0x70(OUTP), INC + pxor INC, STATE8 + movdqu STATE8, 0x70(OUTP) + + _aeskl_gf128mul_x_ble() + + add $128, INP + add $128, OUTP + test LEN, LEN + jnz .Lxts_dec8 + +.Lxts_dec_ret_iv: + movups IV, (IVP) +.Lxts_dec_ret_noerr: + xor AREG, AREG + jmp .Lxts_dec_ret +.Lxts_dec_ret_err: + mov $1, AREG +.Lxts_dec_ret: + FRAME_END + RET + +.Lxts_dec1_pre: + add $128, LEN + jz .Lxts_dec_ret_iv + +.Lxts_dec1: + movdqu (INP), STATE1 + + add $16, INP + sub $16, LEN + jl .Lxts_dec_cts1 + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_end +.Lxts_dec1_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_end: + pxor IV, STATE1 + _aeskl_gf128mul_x_ble() + + test LEN, LEN + jz .Lxts_dec1_out + + movdqu STATE1, (OUTP) + add $16, OUTP + jmp .Lxts_dec1 + +.Lxts_dec1_out: + movdqu STATE1, (OUTP) + jmp .Lxts_dec_ret_iv + +.Lxts_dec_cts1: + movdqa IV, STATE5 + _aeskl_gf128mul_x_ble() + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_cts_pre_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_cts_pre_end +.Lxts_dec1_cts_pre_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_cts_pre_end: + pxor IV, STATE1 + + lea .Lcts_permute_table(%rip), T1 + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), STATE2 + movaps STATE1, STATE3 + pshufb STATE2, STATE1 + movups STATE1, (LEN) + + movups (IVP), STATE1 + pshufb STATE1, IN1 + pblendvb STATE3, IN1 + movaps IN1, STATE1 + + pxor STATE5, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_cts_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_cts_end +.Lxts_dec1_cts_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_cts_end: + pxor STATE5, STATE1 + + movups STATE1, (OUTP) + jmp .Lxts_dec_ret_noerr + +SYM_FUNC_END(_aeskl_xts_decrypt) + +#endif diff --git a/arch/x86/crypto/aeskl-intel_glue.c b/arch/x86/crypto/aeskl-intel_glue.c index 0062baaaf7b2..6616a2fc6f04 100644 --- a/arch/x86/crypto/aeskl-intel_glue.c +++ b/arch/x86/crypto/aeskl-intel_glue.c @@ -27,8 +27,15 @@ asmlinkage int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsign asmlinkage int _aeskl_enc(const void *ctx, u8 *out, const u8 *in); asmlinkage int _aeskl_dec(const void *ctx, u8 *out, const u8 *in); -static int __maybe_unused aeskl_setkey_common(struct crypto_tfm *tfm, void *raw_ctx, - const u8 *in_key, unsigned int key_len) +#ifdef CONFIG_X86_64 +asmlinkage int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +asmlinkage int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +#endif + +static int aeskl_setkey_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key, + unsigned int key_len) { struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx); int err; @@ -86,11 +93,99 @@ static inline int aeskl_dec(const void *ctx, u8 *out, const u8 *in) return 0; } +#ifdef CONFIG_X86_64 + +static inline int aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + else if (!valid_keylocker()) + return -ENODEV; + else if (_aeskl_xts_encrypt(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +static inline int aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + else if (!valid_keylocker()) + return -ENODEV; + else if (_aeskl_xts_decrypt(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +static int aeskl_xts_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keylen) +{ + return xts_setkey_common(tfm, key, keylen, aeskl_setkey_common); +} + +static int xts_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return xts_crypt_common(req, aeskl_xts_encrypt, aeskl_enc); + else + return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc); +} + +static int xts_decrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return xts_crypt_common(req, aeskl_xts_decrypt, aeskl_enc); + else + return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc); +} + +#endif /* CONFIG_X86_64 */ + +static struct skcipher_alg aeskl_skciphers[] = { + { +#ifdef CONFIG_X86_64 + .base = { + .cra_name = "__xts(aes)", + .cra_driver_name = "__xts-aes-aeskl", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = XTS_AES_CTX_SIZE, + .cra_module = THIS_MODULE, + }, + .min_keysize = 2 * AES_MIN_KEY_SIZE, + .max_keysize = 2 * AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, + .setkey = aeskl_xts_setkey, + .encrypt = xts_encrypt, + .decrypt = xts_decrypt, +#endif + } +}; + +static struct simd_skcipher_alg *aeskl_simd_skciphers[ARRAY_SIZE(aeskl_skciphers)]; + static int __init aeskl_init(void) { + u32 eax, ebx, ecx, edx; + int err; + if (!valid_keylocker()) return -ENODEV; + cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); + if (!(ebx & KEYLOCKER_CPUID_EBX_WIDE)) + return -ENODEV; + /* * AES-KL itself does not depend on AES-NI. But AES-KL does not * support 192-bit keys. To make itself AES-compliant, it falls @@ -99,12 +194,18 @@ static int __init aeskl_init(void) if (!boot_cpu_has(X86_FEATURE_AES)) return -ENODEV; + err = simd_register_skciphers_compat(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers), + aeskl_simd_skciphers); + if (err) + return err; + return 0; } static void __exit aeskl_exit(void) { - return; + simd_unregister_skciphers(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers), + aeskl_simd_skciphers); } late_initcall(aeskl_init);