From patchwork Fri May 14 20:14:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258891 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A371C433B4 for ; Fri, 14 May 2021 20:20:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B54161480 for ; Fri, 14 May 2021 20:20:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232870AbhENUVv (ORCPT ); Fri, 14 May 2021 16:21:51 -0400 Received: from mga14.intel.com ([192.55.52.115]:16070 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233253AbhENUVq (ORCPT ); Fri, 14 May 2021 16:21:46 -0400 IronPort-SDR: ozx4UuUzySV8X7Hi3fS3xV0MrOepyt1CV6U9ozjQCq1KVlntSJJ4hphygIJaiO148velbXySms n9omXINk4zCw== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921578" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921578" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:33 -0700 IronPort-SDR: R5zDJJEHoqtoSHdw7toPBPOEISIs+NVl/kEYiyfdxsdRKXH1v05oxWYPGjnENsSvTvzFPUZUWJ cKIeWMWY1jgg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147153" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:33 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [RFC PATCH v2 01/11] x86/cpufeature: Enumerate Key Locker feature Date: Fri, 14 May 2021 13:14:58 -0700 Message-Id: <20210514201508.27967-2-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Intel's Key Locker is a new security feature providing a mechanism to protect data encryption keys when processing the Advanced Encryption Standard (AES) algorithm. The feature accompanies new AES instructions as a successor to Intel's AES New Instructions (AES-NI). It also follows AES-NI's other feature dependency. Here add it to enumerate the hardware capability, shown as 'keylocker' in /proc/cpuinfo. Define the feature-specific CPUID leaf and bits for the feature enablement. Add X86_FEATURE_KEYLOCKER to the disabled features mask for the use of compile-time configuration. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Updated the changelog. --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/include/asm/keylocker.h | 18 ++++++++++++++++++ arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/cpu/cpuid-deps.c | 1 + 5 files changed, 29 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/keylocker.h diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index ac37830ae941..578cf3fe7182 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -359,6 +359,7 @@ #define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */ #define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */ #define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */ +#define X86_FEATURE_KEYLOCKER (16*32+23) /* Key Locker */ #define X86_FEATURE_BUS_LOCK_DETECT (16*32+24) /* Bus Lock detect */ #define X86_FEATURE_CLDEMOTE (16*32+25) /* CLDEMOTE instruction */ #define X86_FEATURE_MOVDIRI (16*32+27) /* MOVDIRI instruction */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index b7dd944dc867..96acda147a50 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -44,6 +44,12 @@ # define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31)) #endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ +#ifdef CONFIG_X86_KEYLOCKER +# define DISABLE_KEYLOCKER 0 +#else +# define DISABLE_KEYLOCKER (1<<(X86_FEATURE_KEYLOCKER & 31)) +#endif /* CONFIG_X86_KEYLOCKER */ + #ifdef CONFIG_X86_5LEVEL # define DISABLE_LA57 0 #else @@ -88,7 +94,7 @@ #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_KEYLOCKER) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK19 0 diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h new file mode 100644 index 000000000000..0597d6b1cd05 --- /dev/null +++ b/arch/x86/include/asm/keylocker.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _ASM_KEYLOCKER_H +#define _ASM_KEYLOCKER_H + +#ifndef __ASSEMBLY__ + +#include + +#define KEYLOCKER_CPUID 0x019 +#define KEYLOCKER_CPUID_EAX_SUPERVISOR BIT(0) +#define KEYLOCKER_CPUID_EBX_AESKLE BIT(0) +#define KEYLOCKER_CPUID_EBX_WIDE BIT(2) +#define KEYLOCKER_CPUID_EBX_BACKUP BIT(4) +#define KEYLOCKER_CPUID_ECX_RAND BIT(1) + +#endif /*__ASSEMBLY__ */ +#endif /* _ASM_KEYLOCKER_H */ diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..b958a95a0908 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -124,6 +124,8 @@ #define X86_CR4_PCIDE _BITUL(X86_CR4_PCIDE_BIT) #define X86_CR4_OSXSAVE_BIT 18 /* enable xsave and xrestore */ #define X86_CR4_OSXSAVE _BITUL(X86_CR4_OSXSAVE_BIT) +#define X86_CR4_KEYLOCKER_BIT 19 /* enable Key Locker */ +#define X86_CR4_KEYLOCKER _BITUL(X86_CR4_KEYLOCKER_BIT) #define X86_CR4_SMEP_BIT 20 /* enable SMEP support */ #define X86_CR4_SMEP _BITUL(X86_CR4_SMEP_BIT) #define X86_CR4_SMAP_BIT 21 /* enable SMAP support */ diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index defda61f372d..ce2020b9ff4d 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -75,6 +75,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_SGX_LC, X86_FEATURE_SGX }, { X86_FEATURE_SGX1, X86_FEATURE_SGX }, { X86_FEATURE_SGX2, X86_FEATURE_SGX1 }, + { X86_FEATURE_KEYLOCKER, X86_FEATURE_XMM2 }, {} }; From patchwork Fri May 14 20:14:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258893 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEE4CC43460 for ; Fri, 14 May 2021 20:20:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BD49F61460 for ; Fri, 14 May 2021 20:20:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232975AbhENUVw (ORCPT ); Fri, 14 May 2021 16:21:52 -0400 Received: from mga14.intel.com ([192.55.52.115]:16076 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232930AbhENUVt (ORCPT ); Fri, 14 May 2021 16:21:49 -0400 IronPort-SDR: GkQ7ql11/hW13GCqnD0s1pUEs5EgyFpJU3EG4bEydd5DI+NHjFJenCDKQ7lmQMR7iGELw0jGlK oxIcBK64R/Dg== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921579" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921579" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:34 -0700 IronPort-SDR: M2gl9988+3rPvzmwlFWhI0P2l+8QUngvrP3jErN5kWk9xrFfGFQ7Tp2P355thZqCzyMGdv6ZMI xV5RWy36AkKw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147159" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:33 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [RFC PATCH v2 02/11] x86/insn: Add Key Locker instructions to the opcode map Date: Fri, 14 May 2021 13:14:59 -0700 Message-Id: <20210514201508.27967-3-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add the following Key Locker instructions to the opcode map: LOADIWKEY: Load an CPU-internal wrapping key. ENCODEKEY128: Wrap a 128-bit AES key to a key handle. ENCODEKEY256: Wrap a 256-bit AES key to a key handle. AESENC128KL: Encrypt a 128-bit block of data using a 128-bit AES key indicated by a key handle. AESENC256KL: Encrypt a 128-bit block of data using a 256-bit AES key indicated by a key handle. AESDEC128KL: Decrypt a 128-bit block of data using a 128-bit AES key indicated by a key handle. AESDEC256KL: Decrypt a 128-bit block of data using a 256-bit AES key indicated by a key handle. AESENCWIDE128KL: Encrypt 8 128-bit blocks of data using a 128-bit AES key indicated by a key handle. AESENCWIDE256KL: Encrypt 8 128-bit blocks of data using a 256-bit AES key indicated by a key handle. AESDECWIDE128KL: Decrypt 8 128-bit blocks of data using a 128-bit AES key indicated by a key handle. AESDECWIDE256KL: Decrypt 8 128-bit blocks of data using a 256-bit AES key indicated by a key handle. Details of these instructions can be found in Intel Software Developer's Manual. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Separated out the LOADIWKEY addition in a new patch. * Included AES instructions to avoid warning messages when the AES Key Locker module is built. --- arch/x86/lib/x86-opcode-map.txt | 11 +++++++---- tools/arch/x86/lib/x86-opcode-map.txt | 11 +++++++---- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index ec31f5b60323..eb702fc6572e 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -794,11 +794,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev) cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev) cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev) cf: vgf2p8mulb Vx,Wx (66) +d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B) db: VAESIMC Vdq,Wdq (66),(v1) -dc: vaesenc Vx,Hx,Wx (66) -dd: vaesenclast Vx,Hx,Wx (66) -de: vaesdec Vx,Hx,Wx (66) -df: vaesdeclast Vx,Hx,Wx (66) +dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3) +dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3) +de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3) +df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3) f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) @@ -808,6 +809,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My, f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3) f9: MOVDIRI My,Gy +fa: ENCODEKEY128 Ew,Ew (F3) +fb: ENCODEKEY256 Ew,Ew (F3) EndTable Table: 3-byte opcode 2 (0x0f 0x3a) diff --git a/tools/arch/x86/lib/x86-opcode-map.txt b/tools/arch/x86/lib/x86-opcode-map.txt index ec31f5b60323..eb702fc6572e 100644 --- a/tools/arch/x86/lib/x86-opcode-map.txt +++ b/tools/arch/x86/lib/x86-opcode-map.txt @@ -794,11 +794,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev) cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev) cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev) cf: vgf2p8mulb Vx,Wx (66) +d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B) db: VAESIMC Vdq,Wdq (66),(v1) -dc: vaesenc Vx,Hx,Wx (66) -dd: vaesenclast Vx,Hx,Wx (66) -de: vaesdec Vx,Hx,Wx (66) -df: vaesdeclast Vx,Hx,Wx (66) +dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3) +dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3) +de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3) +df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3) f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) @@ -808,6 +809,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My, f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3) f9: MOVDIRI My,Gy +fa: ENCODEKEY128 Ew,Ew (F3) +fb: ENCODEKEY256 Ew,Ew (F3) EndTable Table: 3-byte opcode 2 (0x0f 0x3a) From patchwork Fri May 14 20:15:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258899 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CCF8C433ED for ; Fri, 14 May 2021 20:20:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 827836145D for ; Fri, 14 May 2021 20:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233327AbhENUVy (ORCPT ); Fri, 14 May 2021 16:21:54 -0400 Received: from mga14.intel.com ([192.55.52.115]:16070 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbhENUVu (ORCPT ); Fri, 14 May 2021 16:21:50 -0400 IronPort-SDR: ZiXSCo9IxpOaeQk2YHoSI+e7bDjjDbyePm9AUBTOaIFUs46MOmEw0YvWrApe634cffn5gsr8Mw 9jrCxi9oxr+g== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921587" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921587" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:36 -0700 IronPort-SDR: WkWppAfQtuJhmG2mbvn8VBYSCiDVNLoAVIrCEgrTcFdHwLOLftwmn4yZUeVcwRREMMG1yp8Da6 MoVyoQVt1Y/Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147162" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:34 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [RFC PATCH v2 03/11] x86/cpu: Load Key Locker internal key at boot-time Date: Fri, 14 May 2021 13:15:00 -0700 Message-Id: <20210514201508.27967-4-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Internal (Wrapping) Key is a new entity of Intel Key Locker feature. This internal key is loaded in a software-inaccessible CPU state and used to encode a data encryption key. The kernel makes random data and loads it as the internal key in each CPU. The data need to be flushed as soon as the load is done. System firmware may disable the feature. Check the dynamic CPUID bit (KEYLOCKER_CPUID_EBX_AESKLE) at first. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Added to flush cache line when removing an internal key value. Also, renamed the helper. * Reorganized the helper functions. * Changed to reference boot CPU's feature flag consistently, and to update it if the feature is disabled. * Refactored LOADIWKEY instruction in a new helper. * Added function descriptions. --- arch/x86/include/asm/keylocker.h | 9 ++ arch/x86/include/asm/special_insns.h | 36 ++++++++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/cpu/common.c | 5 +- arch/x86/kernel/keylocker.c | 125 +++++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 2 + 6 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kernel/keylocker.c diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h index 0597d6b1cd05..870832f007ec 100644 --- a/arch/x86/include/asm/keylocker.h +++ b/arch/x86/include/asm/keylocker.h @@ -5,6 +5,7 @@ #ifndef __ASSEMBLY__ +#include #include #define KEYLOCKER_CPUID 0x019 @@ -14,5 +15,13 @@ #define KEYLOCKER_CPUID_EBX_BACKUP BIT(4) #define KEYLOCKER_CPUID_ECX_RAND BIT(1) +#ifdef CONFIG_X86_KEYLOCKER +void setup_keylocker(struct cpuinfo_x86 *c); +void flush_keylocker_data(void); +#else +#define setup_keylocker(c) do { } while (0) +#define flush_keylocker_data() do { } while (0) +#endif + #endif /*__ASSEMBLY__ */ #endif /* _ASM_KEYLOCKER_H */ diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 2acd6cb62328..0511b2d72ab8 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -306,6 +306,42 @@ static inline int enqcmds(void __iomem *dst, const void *src) return 0; } +/** + * loadiwkey() - Load CPU-internal wrapping key + * @keyconf: Indicate the key source and the key backup allowance. + * + * The key value is presumed in XMM0-2 registers. So make sure the states + * are available before calling this. + * + * If bit 0 of @keyconf is set, key backup is not permitted. If bit 1 is + * set, a hardware-randomized key is loaded. + * + * Returns: -EAGAIN if hardware randomization fails; otherwise, 0. + * + */ +static inline int __must_check loadiwkey(u32 keyconf) +{ + int zf; + + /* + * LOADIWKEY %xmm1,%xmm2 + * + * EAX and XMM0 are implicit operands. Load key value from XMM0-2 + * to software-invisible CPU state. + * + * This instruction is supported by binutils >= 2.36. + */ + asm volatile (".byte 0xf3,0x0f,0x38,0xdc,0xd1" + CC_SET(z) + : CC_OUT(z) (zf) + : "a"(keyconf)); + + /* Hardware randomization failure is indicated via EFLAGS.ZF=1. */ + if (zf) + return -EAGAIN; + return 0; +} + #endif /* __KERNEL__ */ #endif /* _ASM_X86_SPECIAL_INSNS_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 0704c2a94272..5662876e175b 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -143,6 +143,7 @@ obj-$(CONFIG_PERF_EVENTS) += perf_regs.o obj-$(CONFIG_TRACING) += tracepoint.o obj-$(CONFIG_SCHED_MC_PRIO) += itmt.o obj-$(CONFIG_X86_UMIP) += umip.o +obj-$(CONFIG_X86_KEYLOCKER) += keylocker.o obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index a1b756c49a93..2e64371acb81 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -57,6 +57,8 @@ #include #include #include +#include + #include #include "cpu.h" @@ -1555,10 +1557,11 @@ static void identify_cpu(struct cpuinfo_x86 *c) /* Disable the PN if appropriate */ squash_the_stupid_serial_number(c); - /* Set up SMEP/SMAP/UMIP */ + /* Setup various Intel-specific CPU security features */ setup_smep(c); setup_smap(c); setup_umip(c); + setup_keylocker(c); /* Enable FSGSBASE instructions if available. */ if (cpu_has(c, X86_FEATURE_FSGSBASE)) { diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c new file mode 100644 index 000000000000..d590815de508 --- /dev/null +++ b/arch/x86/kernel/keylocker.c @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * Setup Key Locker feature and support the internal key management. + */ + +#include + +#include +#include +#include +#include +#include + +/* Internal (Wrapping) Key size fits in three 128-bit registers. */ +#define KEYSIZE_128BIT 3 + +static struct _keydata { + struct reg_128_bit value[KEYSIZE_128BIT]; +} keydata; + +/** + * make_keylocker_data() - Generate the internal key. + * + * Return: Nothing + */ +static void make_keylocker_data(void) +{ + int i; + + for (i = 0; i < KEYSIZE_128BIT; i++) + get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit)); +} + +/** + * flush_keylocker_data() - Flush the internal key in memory. + * + * Returns: Nothing + */ +void flush_keylocker_data(void) +{ + unsigned int size = sizeof(struct reg_128_bit) * KEYSIZE_128BIT; + void *keyaddr = &keydata.value; + + if (!boot_cpu_has(X86_FEATURE_KEYLOCKER)) + return; + + memset(keyaddr, 0, size); + clflush_cache_range(keyaddr, size); +} + +#define KEYSRC_SWRAND 0 + +/** + * load_keylocker() - Load the internal key. + * + * Returns: -EAGAIN if hardware randomization fails; otherwise, 0. + */ +static int load_keylocker(void) +{ + struct reg_128_bit zeros = { 0 }; + u32 keysrc = KEYSRC_SWRAND; + int err; + + kernel_fpu_begin(); + + asm volatile ("movdqu %0, %%xmm0; movdqu %1, %%xmm1; movdqu %2, %%xmm2;" + :: "m"(keydata.value[0]), + "m"(keydata.value[1]), + "m"(keydata.value[2])); + + err = loadiwkey(keysrc); + + asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;" + :: "m"(zeros)); + + kernel_fpu_end(); + + return err; +} + +/** + * setup_keylocker() - Enable the feature if supported. + * @c: A pointer to struct cpuinfo_x86 + * + * Returns: Nothing + */ +void setup_keylocker(struct cpuinfo_x86 *c) +{ + int err; + + if (!boot_cpu_has(X86_FEATURE_KEYLOCKER)) + goto out; + + cr4_set_bits(X86_CR4_KEYLOCKER); + + if (c == &boot_cpu_data) { + u32 eax, ebx, ecx, edx; + + cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); + /* BIOS may not enable it on some systems. */ + if (!(ebx & KEYLOCKER_CPUID_EBX_AESKLE)) { + pr_debug("x86/keylocker: not fully supported.\n"); + goto disable; + } + + make_keylocker_data(); + } + + err = load_keylocker(); + if (err) { + pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err); + goto disable; + } + + pr_info_once("x86/keylocker: Enabled.\n"); + return; + +disable: + setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER); + pr_info_once("x86/keylocker: Disabled.\n"); +out: + /* Make sure the feature disabled for kexec-reboot. */ + cr4_clear_bits(X86_CR4_KEYLOCKER); +} diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 0ad5214f598a..0266d70a44be 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -82,6 +82,7 @@ #include #include #include +#include #ifdef CONFIG_ACPI_CPPC_LIB #include @@ -1429,6 +1430,7 @@ void __init native_smp_cpus_done(unsigned int max_cpus) nmi_selftest(); impress_friends(); mtrr_aps_init(); + flush_keylocker_data(); } static int __initdata setup_possible_cpus = -1; From patchwork Fri May 14 20:15:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258897 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44FC5C43600 for ; Fri, 14 May 2021 20:20:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 282AA61221 for ; Fri, 14 May 2021 20:20:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233405AbhENUVy (ORCPT ); Fri, 14 May 2021 16:21:54 -0400 Received: from mga14.intel.com ([192.55.52.115]:16073 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232976AbhENUVw (ORCPT ); Fri, 14 May 2021 16:21:52 -0400 IronPort-SDR: tIh8XyzJccQjgVYnxFdfnSB+zUZGYJNLr3/kyQphGi7pO7LEEdNhj0QhgGGCd54oYa4E0MKLoI mNUrlsFA9Jcg== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921589" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921589" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:36 -0700 IronPort-SDR: fxmkGVWNesbentQ4qBTmI2CcIAe5+kvl6c0wjeqbQjI/mu+qd9YBV5CZ3FDIzHm/1S76LxkQ/+ CZKX+Z4XIyZg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147169" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:36 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [RFC PATCH v2 04/11] x86/msr-index: Add MSRs for Key Locker internal key Date: Fri, 14 May 2021 13:15:01 -0700 Message-Id: <20210514201508.27967-5-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker internal key in a CPU state can be backed up in a platform register. The backup can be also copied back to a CPU state. This mechanism is useful to restore the key (after system sleep). Add MSRs for the internal key backup, copy, and status check. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- arch/x86/include/asm/msr-index.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 742d89a00721..f8e78782c8f4 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -935,4 +935,10 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* MSRs for Key Locker Internal (Wrapping) Key management */ +#define MSR_IA32_COPY_LOCAL_TO_PLATFORM 0x00000d91 +#define MSR_IA32_COPY_PLATFORM_TO_LOCAL 0x00000d92 +#define MSR_IA32_COPY_STATUS 0x00000990 +#define MSR_IA32_IWKEYBACKUP_STATUS 0x00000991 + #endif /* _ASM_X86_MSR_INDEX_H */ From patchwork Fri May 14 20:15:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258903 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C279C43603 for ; Fri, 14 May 2021 20:20:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 319066145B for ; Fri, 14 May 2021 20:20:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233446AbhENUV4 (ORCPT ); Fri, 14 May 2021 16:21:56 -0400 Received: from mga14.intel.com ([192.55.52.115]:16076 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232996AbhENUVw (ORCPT ); Fri, 14 May 2021 16:21:52 -0400 IronPort-SDR: hXC4Znb3M/8x9nWoAtPK7hgxLjizphYTCkyDRdL30lgavj1qmTa+40+JNf6eDJg7JNEDLCHUB4 y33zs9F13alA== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921590" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921590" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:37 -0700 IronPort-SDR: 6rYghBLjw96bsgqcY8Lr2iCL9IAz5zvRh6BG48L0q9BNVq7ay38CyLqgg9CfXhobLAcvzI8zWE BnE9EaB87Ing== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147173" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:36 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, linux-pm@vger.kernel.org Subject: [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states Date: Fri, 14 May 2021 13:15:02 -0700 Message-Id: <20210514201508.27967-6-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When the system state switches to these sleep states, the internal key gets reset. Since this system transition is transparent to userspace, the internal key needs to be restored properly. Key Locker provides a mechanism to back up the internal key in non-volatile memory. The kernel requests a backup right after the key loaded at boot-time and copies it later when the system wakes up. The backup during the S5 sleep state is not trusted. It is overwritten by a new key at the next boot. On a system with the S3/4 states, enable the feature only when the backup mechanism is supported. Disable the feature when the copy fails (or the backup corrupts). The shutdown is considered too noisy. A new key is considerable only when threads can be synchronously suspended. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-pm@vger.kernel.org Acked-by: Rafael J. Wysocki --- Changes from RFC v1: * Folded the warning message into the if condition check. (Rafael Wysocki) * Rebased on the changes of the previous patches. * Added error code for key restoration failures. * Moved the restore helper. * Added function descriptions. --- arch/x86/include/asm/keylocker.h | 3 + arch/x86/kernel/keylocker.c | 122 +++++++++++++++++++++++++++++-- arch/x86/power/cpu.c | 2 + 3 files changed, 122 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h index 870832f007ec..74b806346bee 100644 --- a/arch/x86/include/asm/keylocker.h +++ b/arch/x86/include/asm/keylocker.h @@ -7,6 +7,7 @@ #include #include +#include #define KEYLOCKER_CPUID 0x019 #define KEYLOCKER_CPUID_EAX_SUPERVISOR BIT(0) @@ -18,9 +19,11 @@ #ifdef CONFIG_X86_KEYLOCKER void setup_keylocker(struct cpuinfo_x86 *c); void flush_keylocker_data(void); +void restore_keylocker(void); #else #define setup_keylocker(c) do { } while (0) #define flush_keylocker_data() do { } while (0) +#define restore_keylocker() do { } while (0) #endif #endif /*__ASSEMBLY__ */ diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c index d590815de508..0f60350944fa 100644 --- a/arch/x86/kernel/keylocker.c +++ b/arch/x86/kernel/keylocker.c @@ -5,6 +5,8 @@ */ #include +#include +#include #include #include @@ -12,10 +14,13 @@ #include #include +static bool keybackup_available; + /* Internal (Wrapping) Key size fits in three 128-bit registers. */ #define KEYSIZE_128BIT 3 static struct _keydata { + bool valid; struct reg_128_bit value[KEYSIZE_128BIT]; } keydata; @@ -30,6 +35,8 @@ static void make_keylocker_data(void) for (i = 0; i < KEYSIZE_128BIT; i++) get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit)); + + keydata.valid = true; } /** @@ -47,6 +54,8 @@ void flush_keylocker_data(void) memset(keyaddr, 0, size); clflush_cache_range(keyaddr, size); + + keydata.valid = false; } #define KEYSRC_SWRAND 0 @@ -79,6 +88,40 @@ static int load_keylocker(void) return err; } +#define KEYRESTORE_RETRY 1 + +/** + * copy_keylocker() - Copy the internal key from the backup. + * + * Request hardware to copy the key in non-volatile memory to the CPU state. Do this + * again if the copy fails. The key may not be ready when the precedent backup is + * still in progress. + * + * Returns: -EBUSY if the copy fails, -ENODEV if no backup is available, or 0 if + * successful. + */ +static int copy_keylocker(void) +{ + int i; + + if (!keybackup_available) + return -ENODEV; + + wrmsrl(MSR_IA32_COPY_PLATFORM_TO_LOCAL, 1); + + for (i = 0; i <= KEYRESTORE_RETRY; i++) { + u64 status; + + if (i) + udelay(1); + + rdmsrl(MSR_IA32_COPY_STATUS, status); + if (status & BIT(0)) + return 0; + } + return -EBUSY; +} + /** * setup_keylocker() - Enable the feature if supported. * @c: A pointer to struct cpuinfo_x86 @@ -104,13 +147,43 @@ void setup_keylocker(struct cpuinfo_x86 *c) goto disable; } + keybackup_available = (ebx & KEYLOCKER_CPUID_EBX_BACKUP); + /* Internal key backup is essential with S3/4 states. */ + if (!keybackup_available && + (acpi_sleep_state_supported(ACPI_STATE_S3) || + acpi_sleep_state_supported(ACPI_STATE_S4))) { + pr_debug("x86/keylocker: no key backup support with possible S3/4.\n"); + goto disable; + } + make_keylocker_data(); - } - err = load_keylocker(); - if (err) { - pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err); - goto disable; + err = load_keylocker(); + if (err) { + pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err); + goto disable; + } + + /* Back up the internal key in non-volatile memory if supported. */ + if (keybackup_available) + wrmsrl(MSR_IA32_COPY_LOCAL_TO_PLATFORM, 1); + } else { + + /* + * Load the internal key directly when available in memory, which is only + * possible at boot-time. + * + * NB: When system wakes up, this path also recovers the internal key. + */ + if (keydata.valid) + err = load_keylocker(); + else + err = copy_keylocker(); + if (err) { + pr_err_once("x86/keylocker: Fail to %s internal key (rc: %d).\n", + keydata.valid ? "load" : "copy", err); + goto disable; + } } pr_info_once("x86/keylocker: Enabled.\n"); @@ -123,3 +196,42 @@ void setup_keylocker(struct cpuinfo_x86 *c) /* Make sure the feature disabled for kexec-reboot. */ cr4_clear_bits(X86_CR4_KEYLOCKER); } + +/** + * restore_keylocker() - Restore the internal key. + * + * The boot CPU executes this while other CPUs restore it through the setup function. + * + * Returns: Nothing + */ +void restore_keylocker(void) +{ + u64 backup_status; + int err; + + if (!boot_cpu_has(X86_FEATURE_KEYLOCKER)) + return; + + /* + * IA32_IWKEYBACKUP_STATUS MSR contains a bitmap that indicates an invalid backup if bit 0 + * is set and a read (or write) error if bit 2 is set. + */ + rdmsrl(MSR_IA32_IWKEYBACKUP_STATUS, backup_status); + if (WARN(!(backup_status & BIT(0)), + "x86/keylocker: Internal key backup access failed with %s.\n", + (backup_status & BIT(2)) ? "read error" : "invalid status")) + goto disable_out; + + err = copy_keylocker(); + if (err) { + pr_err("x86/keylocker: Internal key restoration failed (rc: %d).\n", err); + goto disable_out; + } + + return; + +disable_out: + pr_info("x86/keylocker: Disabled with internal key restoration failure.\n"); + setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER); + cr4_clear_bits(X86_CR4_KEYLOCKER); +} diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c index 3a070e7cdb8b..ace94f07701a 100644 --- a/arch/x86/power/cpu.c +++ b/arch/x86/power/cpu.c @@ -25,6 +25,7 @@ #include #include #include +#include #ifdef CONFIG_X86_32 __visible unsigned long saved_context_ebx; @@ -261,6 +262,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt) mtrr_bp_restore(); perf_restore_debug_store(); msr_restore_context(ctxt); + restore_keylocker(); c = &cpu_data(smp_processor_id()); if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL)) From patchwork Fri May 14 20:15:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258901 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB42AC4360C for ; Fri, 14 May 2021 20:20:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8DB6361460 for ; Fri, 14 May 2021 20:20:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233459AbhENUV4 (ORCPT ); Fri, 14 May 2021 16:21:56 -0400 Received: from mga14.intel.com ([192.55.52.115]:16070 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233008AbhENUVw (ORCPT ); Fri, 14 May 2021 16:21:52 -0400 IronPort-SDR: bWO3tqoUDaz1abDlLVzJ4Na2+9UCw0eKHuNElZj3diDTMlfs8MfLFduT5kB8dYjUqSquPjtAjh 2W4f//8Cse8w== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921592" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921592" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:37 -0700 IronPort-SDR: PENpovH2k+3B1gl75FEYU68QMWaA7N8r8mm1bAC8cweXNONJq1/xuiTfwhrloOeEexext0cqWC E6q6+7iTKxyg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147177" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:36 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, linux-doc@vger.kernel.org Subject: [RFC PATCH v2 06/11] x86/cpu: Add a config option and a chicken bit for Key Locker Date: Fri, 14 May 2021 13:15:03 -0700 Message-Id: <20210514201508.27967-7-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add a kernel config option to enable the feature (disabled by default) at compile-time. Also, add a new command-line parameter -- 'nokeylocker' to disable the feature at boot-time. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/x86/Kconfig | 14 ++++++++++++++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 3 files changed, 32 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index cb89dbdedc46..d0afe36c1802 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3284,6 +3284,8 @@ nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nokeylocker [X86] Disables Key Locker hardware feature. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 0045e1b44190..de8eeb705ed8 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1885,6 +1885,20 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_KEYLOCKER + prompt "Key Locker" + def_bool n + depends on CPU_SUP_INTEL + help + Key Locker is a new security feature to protect data encryption + keys for the Advanced Encryption Standard (AES) algorithm. + + When enabled, every CPU has a unique internal key to wrap AES + keys in an encoded format. The internal key is not accessible + to software once loaded. + + If unsure, say y. + choice prompt "TSX enable mode" depends on CPU_SUP_INTEL diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 2e64371acb81..c655cce30c2b 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -353,6 +353,22 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c) /* These bits should not change their value after CPU init is finished. */ static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | X86_CR4_FSGSBASE; + +static __init int x86_nokeylocker_setup(char *arg) +{ + /* Expect an exact match without trailing characters. */ + if (strlen(arg)) + return 0; + + if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER); + pr_info("x86/keylocker: Disabled by kernel command line.\n"); + return 1; +} +__setup("nokeylocker", x86_nokeylocker_setup); + static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning); static unsigned long cr4_pinned_bits __ro_after_init; From patchwork Fri May 14 20:15:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258907 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39866C433B4 for ; Fri, 14 May 2021 20:20:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 151EB6145B for ; Fri, 14 May 2021 20:20:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233480AbhENUV5 (ORCPT ); Fri, 14 May 2021 16:21:57 -0400 Received: from mga14.intel.com ([192.55.52.115]:16073 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233028AbhENUVw (ORCPT ); Fri, 14 May 2021 16:21:52 -0400 IronPort-SDR: Qu3mheomfwnO2BGcGmhC9gEs6SCutxOf36Y+rLelUDponityOvyTj5BaFn8HwH18TTrJfGYyUt fl2NZBrGPMeA== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921594" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921594" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:37 -0700 IronPort-SDR: MjA9L+QCKBblRHPeAED79usTmz3wz+ESaOPB52gdmKCkObI/tOqCBP5pdO9FGsO7QRSXPK4cpC b/ILTC1ODd+g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147183" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:37 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, linux-kselftest@vger.kernel.org Subject: [RFC PATCH v2 07/11] selftests/x86: Test Key Locker internal key maintenance Date: Fri, 14 May 2021 13:15:04 -0700 Message-Id: <20210514201508.27967-8-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The test validates the internal key to be the same in all CPUs. It performs the validation again with the Suspend-To-RAM (ACPI S3) state. Signed-off-by: Chang S. Bae Cc: linux-kernel@vger.kernel.org Cc: linux-kselftest@vger.kernel.org --- Changes from RFC v1: * Commented the binutils version number for ENCODEKEY128 (Peter Zijlstra) --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/keylocker.c | 177 ++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/keylocker.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 333980375bc7..09237cc84108 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -13,7 +13,7 @@ CAN_BUILD_WITH_NOPIE := $(shell ./check_cc.sh $(CC) trivial_program.c -no-pie) TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \ check_initial_reg_state sigreturn iopl ioperm \ test_vsyscall mov_ss_trap \ - syscall_arg_fault fsgsbase_restore + syscall_arg_fault fsgsbase_restore keylocker TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer diff --git a/tools/testing/selftests/x86/keylocker.c b/tools/testing/selftests/x86/keylocker.c new file mode 100644 index 000000000000..78bbb7939b1a --- /dev/null +++ b/tools/testing/selftests/x86/keylocker.c @@ -0,0 +1,177 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * keylocker.c, validate the internal key management. + */ +#undef _GNU_SOURCE +#define _GNU_SOURCE 1 + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define HANDLE_SIZE 48 + +static bool keylocker_disabled; + +/* Encode a 128-bit key to a 384-bit handle */ +static inline void __encode_key(char *handle) +{ + static const unsigned char aeskey[] = { 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, + 0x71, 0x77, 0x74, 0x69, 0x6f, 0x6b, 0x6c, 0x78 }; + + asm volatile ("movdqu %0, %%xmm0" : : "m" (*aeskey) :); + + /* Set no restriction to the handle */ + asm volatile ("mov $0, %%eax" :); + + /* ENCODEKEY128 %EAX (supported by binutils >= 2.36) */ + asm volatile (".byte 0xf3, 0xf, 0x38, 0xfa, 0xc0"); + + asm volatile ("movdqu %%xmm0, %0; movdqu %%xmm1, %1; movdqu %%xmm2, %2;" + : "=m" (handle[0]), "=m" (handle[0x10]), "=m" (handle[0x20])); +} + +static jmp_buf jmpbuf; + +static void handle_sigill(int sig, siginfo_t *si, void *ctx_void) +{ + keylocker_disabled = true; + siglongjmp(jmpbuf, 1); +} + +static bool encode_key(char *handle) +{ + bool success = true; + struct sigaction sa; + int ret; + + memset(&sa, 0, sizeof(sa)); + + /* Set signal handler */ + sa.sa_flags = SA_SIGINFO; + sa.sa_sigaction = handle_sigill; + sigemptyset(&sa.sa_mask); + ret = sigaction(SIGILL, &sa, 0); + if (ret) + err(1, "sigaction"); + + if (sigsetjmp(jmpbuf, 1)) + success = false; + else + __encode_key(handle); + + /* Clear signal handler */ + sa.sa_flags = 0; + sa.sa_sigaction = NULL; + sa.sa_handler = SIG_DFL; + sigemptyset(&sa.sa_mask); + ret = sigaction(SIGILL, &sa, 0); + if (ret) + err(1, "sigaction"); + + return success; +} + +/* + * Test if the internal key is the same in all the CPUs: + * + * Since the value is not readable, compare the encoded output of a AES key + * between CPUs. + */ + +static int nerrs; + +static unsigned char cpu0_handle[HANDLE_SIZE] = { 0 }; + +static void test_internal_key(bool slept, long cpus) +{ + int cpu, errs; + + printf("Test the internal key consistency between CPUs\n"); + + for (cpu = 0, errs = 0; cpu < cpus; cpu++) { + char handle[HANDLE_SIZE] = { 0 }; + cpu_set_t mask; + bool success; + + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + sched_setaffinity(0, sizeof(cpu_set_t), &mask); + + success = encode_key(handle); + if (!success) { + /* The encode should success after the S3 sleep */ + if (slept) + errs++; + printf("[%s]\tKey Locker disabled at CPU%d\n", + slept ? "FAIL" : "NOTE", cpu); + continue; + } + + if (cpu == 0 && !slept) { + /* Record the first handle value as reference */ + memcpy(cpu0_handle, handle, HANDLE_SIZE); + } else if (memcmp(cpu0_handle, handle, HANDLE_SIZE)) { + printf("[FAIL]\tMismatched internal key at CPU%d\n", + cpu); + errs++; + } + } + + if (errs == 0 && !keylocker_disabled) + printf("[OK]\tAll the internal keys are the same\n"); + else + nerrs += errs; +} + +static void switch_to_sleep(bool *slept) +{ + ssize_t bytes; + int fd; + + printf("Transition to Suspend-To-RAM state\n"); + + fd = open("/sys/power/mem_sleep", O_RDWR); + if (fd < 0) + err(1, "Open /sys/power/mem_sleep"); + + bytes = write(fd, "deep", strlen("deep")); + if (bytes != strlen("deep")) + err(1, "Write /sys/power/mem_sleep"); + close(fd); + + fd = open("/sys/power/state", O_RDWR); + if (fd < 0) + err(1, "Open /sys/power/state"); + + bytes = write(fd, "mem", strlen("mem")); + if (bytes != strlen("mem")) + err(1, "Write /sys/power/state"); + close(fd); + + printf("Wake up from Suspend-To-RAM state\n"); + *slept = true; +} + +int main(void) +{ + bool slept = false; + long cpus; + + cpus = sysconf(_SC_NPROCESSORS_ONLN); + printf("%ld CPUs in the system\n", cpus); + + test_internal_key(slept, cpus); + if (keylocker_disabled) + return nerrs ? 1 : 0; + + switch_to_sleep(&slept); + test_internal_key(slept, cpus); + return nerrs ? 1 : 0; +} From patchwork Fri May 14 20:15:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258905 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23DB8C43616 for ; Fri, 14 May 2021 20:20:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D64C86121E for ; Fri, 14 May 2021 20:20:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233522AbhENUV6 (ORCPT ); Fri, 14 May 2021 16:21:58 -0400 Received: from mga14.intel.com ([192.55.52.115]:16076 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233202AbhENUVy (ORCPT ); Fri, 14 May 2021 16:21:54 -0400 IronPort-SDR: wBpT6zhHoDEOgu30pD3ewKHWN32t0mL4ma7a+jqQOykJk008ex+CNwcg3iBviB4jAcCpHwPF5u fWSEXKG3gapA== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921597" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921597" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:37 -0700 IronPort-SDR: IeFs2124po+FNEVQbH2f/r8tF+6C4qmz9uoynzTwnulps3dckFJ22vyb3097JFFy5WPgkBo4dQ BHI/EDiijW0Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147186" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:37 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [RFC PATCH v2 08/11] crypto: x86/aes-ni - Improve error handling Date: Fri, 14 May 2021 13:15:05 -0700 Message-Id: <20210514201508.27967-9-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Some error case in the glue code is possibly ignored and thus not handled correctly. Make sure each error code is not overwritten. Signed-off-by: Chang S. Bae Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Added as a new patch. This change prepares to address Ard's feedback. --- arch/x86/crypto/aesni-intel_glue.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 2144e54a6c89..685943f0e5a3 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -291,7 +291,7 @@ static int ecb_encrypt(struct skcipher_request *req) nbytes & AES_BLOCK_MASK); kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; - err = skcipher_walk_done(&walk, nbytes); + err |= skcipher_walk_done(&walk, nbytes); } return err; @@ -313,7 +313,7 @@ static int ecb_decrypt(struct skcipher_request *req) nbytes & AES_BLOCK_MASK); kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; - err = skcipher_walk_done(&walk, nbytes); + err |= skcipher_walk_done(&walk, nbytes); } return err; @@ -335,7 +335,7 @@ static int cbc_encrypt(struct skcipher_request *req) nbytes & AES_BLOCK_MASK, walk.iv); kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; - err = skcipher_walk_done(&walk, nbytes); + err |= skcipher_walk_done(&walk, nbytes); } return err; @@ -357,7 +357,7 @@ static int cbc_decrypt(struct skcipher_request *req) nbytes & AES_BLOCK_MASK, walk.iv); kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; - err = skcipher_walk_done(&walk, nbytes); + err |= skcipher_walk_done(&walk, nbytes); } return err; @@ -522,7 +522,7 @@ static int ctr_crypt(struct skcipher_request *req) nbytes = 0; } kernel_fpu_end(); - err = skcipher_walk_done(&walk, nbytes); + err |= skcipher_walk_done(&walk, nbytes); } return err; } @@ -691,7 +691,7 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req, } kernel_fpu_end(); - err = skcipher_walk_done(&walk, 0); + err |= skcipher_walk_done(&walk, 0); } if (err) @@ -862,7 +862,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt) skcipher_request_set_crypt(&subreq, req->src, req->dst, blocks * AES_BLOCK_SIZE, req->iv); req = &subreq; - err = skcipher_walk_virt(&walk, req, false); + err |= skcipher_walk_virt(&walk, req, false); } else { tail = 0; } @@ -888,7 +888,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt) nbytes, walk.iv); kernel_fpu_end(); - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); + err |= skcipher_walk_done(&walk, walk.nbytes - nbytes); if (walk.nbytes > 0) kernel_fpu_begin(); @@ -905,7 +905,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt) skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, req->iv); - err = skcipher_walk_virt(&walk, &subreq, false); + err |= skcipher_walk_virt(&walk, &subreq, false); if (err) return err; From patchwork Fri May 14 20:15:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258911 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F073C43461 for ; Fri, 14 May 2021 20:20:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0E7BF61221 for ; Fri, 14 May 2021 20:20:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233681AbhENUWD (ORCPT ); Fri, 14 May 2021 16:22:03 -0400 Received: from mga14.intel.com ([192.55.52.115]:16070 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233193AbhENUVy (ORCPT ); Fri, 14 May 2021 16:21:54 -0400 IronPort-SDR: ZaE2Ns66gVT3Ay5X+Z8rwu1sE0P/+gbf/v79/kzy1NOwIMPeMNvEs6QViYcGYNe+a8AAa7J2Dp hSOOKhl9o0JA== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921602" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921602" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:40 -0700 IronPort-SDR: 35iBmBoIogZvBqinjDg+jGF9uda2ZvdbaJswGYfLmLjdw/F/q/vqvBApbwVK3FgkWyyfonLGud zz64f6gOGrqA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147194" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:37 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, Ard Biesheuvel Subject: [RFC PATCH v2 09/11] crypto: x86/aes-ni - Refactor to prepare a new AES implementation Date: Fri, 14 May 2021 13:15:06 -0700 Message-Id: <20210514201508.27967-10-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Refactor some glue code and constant values of assembly code in preparation for a new AES implementation. Also, introduce wrappers for data transformation functions to return an error value. New AES instructions may populate an error condition while AES-NI does not. No functional change. Signed-off-by: Chang S. Bae Cc: Ard Biesheuvel Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Added as a new patch. (Ard Biesheuvel) --- arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/aes-intel_asm.S | 26 +++ arch/x86/crypto/aes-intel_glue.c | 208 +++++++++++++++++++ arch/x86/crypto/aes-intel_glue.h | 61 ++++++ arch/x86/crypto/aesni-intel_asm.S | 90 ++++----- arch/x86/crypto/aesni-intel_glue.c | 308 ++++------------------------- arch/x86/crypto/aesni-intel_glue.h | 88 +++++++++ 7 files changed, 452 insertions(+), 331 deletions(-) create mode 100644 arch/x86/crypto/aes-intel_asm.S create mode 100644 arch/x86/crypto/aes-intel_glue.c create mode 100644 arch/x86/crypto/aes-intel_glue.h create mode 100644 arch/x86/crypto/aesni-intel_glue.h diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index d0959e7b809f..30d558df67cd 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -47,7 +47,7 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o -aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o +aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o diff --git a/arch/x86/crypto/aes-intel_asm.S b/arch/x86/crypto/aes-intel_asm.S new file mode 100644 index 000000000000..98abf875af79 --- /dev/null +++ b/arch/x86/crypto/aes-intel_asm.S @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +/* + * Constant values shared between AES implementations: + */ + +.pushsection .rodata +.align 16 +.Lcts_permute_table: + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 +#ifdef __x86_64__ +.Lbswap_mask: + .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +#endif +.popsection + +.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 +.align 16 +.Lgf128mul_x_ble_mask: + .octa 0x00000000000000010000000000000087 +.previous diff --git a/arch/x86/crypto/aes-intel_glue.c b/arch/x86/crypto/aes-intel_glue.c new file mode 100644 index 000000000000..66c67f1610c9 --- /dev/null +++ b/arch/x86/crypto/aes-intel_glue.c @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include +#include +#include +#include +#include +#include +#include +#include "aes-intel_glue.h" + +int ecb_crypt_common(struct skcipher_request *req, + int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); + struct skcipher_walk walk; + unsigned int nbytes; + int err; + + err = skcipher_walk_virt(&walk, req, false); + + while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); + err = fn(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK); + kernel_fpu_end(); + if (err) + return err; + + nbytes &= AES_BLOCK_SIZE - 1; + err |= skcipher_walk_done(&walk, nbytes); + } + + return err; +} + +int cbc_crypt_common(struct skcipher_request *req, + int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); + struct skcipher_walk walk; + unsigned int nbytes; + int err; + + err = skcipher_walk_virt(&walk, req, false); + + while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); + err = fn(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK, + walk.iv); + kernel_fpu_end(); + if (err) + return err; + + nbytes &= AES_BLOCK_SIZE - 1; + err |= skcipher_walk_done(&walk, nbytes); + } + + return err; +} + +int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen, + int (*fn)(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len)) +{ + struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int err; + + err = xts_verify_key(tfm, key, keylen); + if (err) + return err; + + keylen /= 2; + + /* first half of xts-key is for crypt */ + err = fn(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, key, keylen); + if (err) + return err; + + /* second half of xts-key is for tweak */ + return fn(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, key + keylen, keylen); +} + +int xts_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct skcipher_request subreq; + struct skcipher_walk walk; + int err; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + + err = skcipher_walk_virt(&walk, req, false); + + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + blocks * AES_BLOCK_SIZE, req->iv); + req = &subreq; + err |= skcipher_walk_virt(&walk, req, false); + } else { + tail = 0; + } + + kernel_fpu_begin(); + + /* calculate first value of T */ + err |= crypt1_fn(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); + + while (walk.nbytes > 0) { + int nbytes = walk.nbytes; + + if (nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + + err |= crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, + nbytes, walk.iv); + kernel_fpu_end(); + + err |= skcipher_walk_done(&walk, walk.nbytes - nbytes); + + if (walk.nbytes > 0) + kernel_fpu_begin(); + } + + if (unlikely(tail > 0 && !err)) { + struct scatterlist sg_src[2], sg_dst[2]; + struct scatterlist *src, *dst; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err |= skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + kernel_fpu_begin(); + err = crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, + walk.nbytes, walk.iv); + kernel_fpu_end(); + + err |= skcipher_walk_done(&walk, 0); + } + return err; +} + +#ifdef CONFIG_X86_64 + +int ctr_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); + u8 keystream[AES_BLOCK_SIZE]; + struct skcipher_walk walk; + unsigned int nbytes; + int err; + + err = skcipher_walk_virt(&walk, req, false); + + while ((nbytes = walk.nbytes) > 0) { + kernel_fpu_begin(); + if (nbytes & AES_BLOCK_MASK) { + err = crypt_fn(ctx, walk.dst.virt.addr, walk.src.virt.addr, + nbytes & AES_BLOCK_MASK, walk.iv); + } + nbytes &= ~AES_BLOCK_MASK; + + if (walk.nbytes == walk.total && nbytes > 0) { + err = crypt1_fn(ctx, keystream, walk.iv); + crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes, + walk.src.virt.addr + walk.nbytes - nbytes, + keystream, nbytes); + crypto_inc(walk.iv, AES_BLOCK_SIZE); + nbytes = 0; + } + kernel_fpu_end(); + if (err) + return err; + + err |= skcipher_walk_done(&walk, nbytes); + } + return err; +} + +#endif /* CONFIG_X86_64 */ diff --git a/arch/x86/crypto/aes-intel_glue.h b/arch/x86/crypto/aes-intel_glue.h new file mode 100644 index 000000000000..41e93c0f7932 --- /dev/null +++ b/arch/x86/crypto/aes-intel_glue.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Shared glue code between AES implementations, refactored from the AES-NI's. + */ + +#ifndef _AES_INTEL_GLUE_H +#define _AES_INTEL_GLUE_H + +#include + +#define AES_ALIGN 16 +#define AES_ALIGN_ATTR __attribute__((__aligned__(AES_ALIGN))) +#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) +#define AES_ALIGN_EXTRA ((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) +#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AES_ALIGN_EXTRA) +#define XTS_AES_CTX_SIZE (sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA) + +struct aes_xts_ctx { + u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR; + u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR; +}; + +static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) +{ + unsigned long addr = (unsigned long)raw_ctx; + unsigned long align = AES_ALIGN; + + if (align <= crypto_tfm_ctx_alignment()) + align = 1; + + return (struct crypto_aes_ctx *)ALIGN(addr, align); +} + +int ecb_crypt_common(struct skcipher_request *req, + int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len)); + +int cbc_crypt_common(struct skcipher_request *req, + int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv)); + +int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen, + int (*fn)(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len)); + +int xts_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)); + +#ifdef CONFIG_X86_64 + +int ctr_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)); + +#endif /*CONFIG_X86_64 */ + +#endif /* _AES_INTEL_GLUE_H */ diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S index 4e3972570916..06dfd71ffdf4 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -28,6 +28,7 @@ #include #include #include +#include "aes-intel_asm.S" /* * The following macros are used to move an (un)aligned 16 byte value to/from @@ -1937,9 +1938,9 @@ SYM_FUNC_START(aesni_set_key) SYM_FUNC_END(aesni_set_key) /* - * void aesni_enc(const void *ctx, u8 *dst, const u8 *src) + * void _aesni_enc(const void *ctx, u8 *dst, const u8 *src) */ -SYM_FUNC_START(aesni_enc) +SYM_FUNC_START(_aesni_enc) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -1958,7 +1959,7 @@ SYM_FUNC_START(aesni_enc) #endif FRAME_END ret -SYM_FUNC_END(aesni_enc) +SYM_FUNC_END(_aesni_enc) /* * _aesni_enc1: internal ABI @@ -2126,9 +2127,9 @@ SYM_FUNC_START_LOCAL(_aesni_enc4) SYM_FUNC_END(_aesni_enc4) /* - * void aesni_dec (const void *ctx, u8 *dst, const u8 *src) + * void _aesni_dec (const void *ctx, u8 *dst, const u8 *src) */ -SYM_FUNC_START(aesni_dec) +SYM_FUNC_START(_aesni_dec) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -2148,7 +2149,7 @@ SYM_FUNC_START(aesni_dec) #endif FRAME_END ret -SYM_FUNC_END(aesni_dec) +SYM_FUNC_END(_aesni_dec) /* * _aesni_dec1: internal ABI @@ -2316,10 +2317,10 @@ SYM_FUNC_START_LOCAL(_aesni_dec4) SYM_FUNC_END(_aesni_dec4) /* - * void aesni_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len) + * void _aesni_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len) */ -SYM_FUNC_START(aesni_ecb_enc) +SYM_FUNC_START(_aesni_ecb_enc) FRAME_BEGIN #ifndef __x86_64__ pushl LEN @@ -2373,13 +2374,13 @@ SYM_FUNC_START(aesni_ecb_enc) #endif FRAME_END ret -SYM_FUNC_END(aesni_ecb_enc) +SYM_FUNC_END(_aesni_ecb_enc) /* - * void aesni_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len); + * void _aesni_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len); */ -SYM_FUNC_START(aesni_ecb_dec) +SYM_FUNC_START(_aesni_ecb_dec) FRAME_BEGIN #ifndef __x86_64__ pushl LEN @@ -2434,13 +2435,13 @@ SYM_FUNC_START(aesni_ecb_dec) #endif FRAME_END ret -SYM_FUNC_END(aesni_ecb_dec) +SYM_FUNC_END(_aesni_ecb_dec) /* - * void aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) + * void _aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len, u8 *iv) */ -SYM_FUNC_START(aesni_cbc_enc) +SYM_FUNC_START(_aesni_cbc_enc) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2478,13 +2479,13 @@ SYM_FUNC_START(aesni_cbc_enc) #endif FRAME_END ret -SYM_FUNC_END(aesni_cbc_enc) +SYM_FUNC_END(_aesni_cbc_enc) /* - * void aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) + * void _aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len, u8 *iv) */ -SYM_FUNC_START(aesni_cbc_dec) +SYM_FUNC_START(_aesni_cbc_dec) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2571,7 +2572,7 @@ SYM_FUNC_START(aesni_cbc_dec) #endif FRAME_END ret -SYM_FUNC_END(aesni_cbc_dec) +SYM_FUNC_END(_aesni_cbc_dec) /* * void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, @@ -2691,21 +2692,6 @@ SYM_FUNC_START(aesni_cts_cbc_dec) ret SYM_FUNC_END(aesni_cts_cbc_dec) -.pushsection .rodata -.align 16 -.Lcts_permute_table: - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 - .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 -#ifdef __x86_64__ -.Lbswap_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -#endif -.popsection - #ifdef __x86_64__ /* * _aesni_inc_init: internal ABI @@ -2757,10 +2743,10 @@ SYM_FUNC_START_LOCAL(_aesni_inc) SYM_FUNC_END(_aesni_inc) /* - * void aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) + * void _aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len, u8 *iv) */ -SYM_FUNC_START(aesni_ctr_enc) +SYM_FUNC_START(_aesni_ctr_enc) FRAME_BEGIN cmp $16, LEN jb .Lctr_enc_just_ret @@ -2817,16 +2803,10 @@ SYM_FUNC_START(aesni_ctr_enc) .Lctr_enc_just_ret: FRAME_END ret -SYM_FUNC_END(aesni_ctr_enc) +SYM_FUNC_END(_aesni_ctr_enc) #endif -.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 -.align 16 -.Lgf128mul_x_ble_mask: - .octa 0x00000000000000010000000000000087 -.previous - /* * _aesni_gf128mul_x_ble: internal ABI * Multiply in GF(2^128) for XTS IVs @@ -2846,10 +2826,10 @@ SYM_FUNC_END(aesni_ctr_enc) pxor KEY, IV; /* - * void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) + * void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_encrypt) +SYM_FUNC_START(_aesni_xts_encrypt) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2998,13 +2978,13 @@ SYM_FUNC_START(aesni_xts_encrypt) movups STATE, (OUTP) jmp .Lxts_enc_ret -SYM_FUNC_END(aesni_xts_encrypt) +SYM_FUNC_END(_aesni_xts_encrypt) /* - * void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) + * void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_decrypt) +SYM_FUNC_START(_aesni_xts_decrypt) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -3160,4 +3140,4 @@ SYM_FUNC_START(aesni_xts_decrypt) movups STATE, (OUTP) jmp .Lxts_dec_ret -SYM_FUNC_END(aesni_xts_decrypt) +SYM_FUNC_END(_aesni_xts_decrypt) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 685943f0e5a3..4c6733c40ac4 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -35,34 +35,24 @@ #include #include #include +#include "aes-intel_glue.h" +#include "aesni-intel_glue.h" - -#define AESNI_ALIGN 16 -#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN))) -#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) #define RFC4106_HASH_SUBKEY_SIZE 16 -#define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) -#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AESNI_ALIGN_EXTRA) -#define XTS_AES_CTX_SIZE (sizeof(struct aesni_xts_ctx) + AESNI_ALIGN_EXTRA) /* This data is stored at the end of the crypto_tfm struct. * It's a type of per "session" data storage location. * This needs to be 16 byte aligned. */ struct aesni_rfc4106_gcm_ctx { - u8 hash_subkey[16] AESNI_ALIGN_ATTR; - struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR; + u8 hash_subkey[16] AES_ALIGN_ATTR; + struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR; u8 nonce[4]; }; struct generic_gcmaes_ctx { - u8 hash_subkey[16] AESNI_ALIGN_ATTR; - struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR; -}; - -struct aesni_xts_ctx { - u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; - u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; + u8 hash_subkey[16] AES_ALIGN_ATTR; + struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR; }; #define GCM_BLOCK_LEN 16 @@ -80,18 +70,6 @@ struct gcm_context_data { u8 hash_keys[GCM_BLOCK_LEN * 16]; }; -asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len); -asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); -asmlinkage void aesni_dec(const void *ctx, u8 *out, const u8 *in); -asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len); -asmlinkage void aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len); -asmlinkage void aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, @@ -100,16 +78,8 @@ asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, #define AVX_GEN2_OPTSIZE 640 #define AVX_GEN4_OPTSIZE 4096 -asmlinkage void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); - -asmlinkage void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); - #ifdef CONFIG_X86_64 -asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); DEFINE_STATIC_CALL(aesni_ctr_enc_tfm, aesni_ctr_enc); /* Scatter / Gather routines, with args similar to above */ @@ -187,7 +157,7 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(gcm_use_avx2); static inline struct aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) { - unsigned long align = AESNI_ALIGN; + unsigned long align = AES_ALIGN; if (align <= crypto_tfm_ctx_alignment()) align = 1; @@ -197,7 +167,7 @@ aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) static inline struct generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) { - unsigned long align = AESNI_ALIGN; + unsigned long align = AES_ALIGN; if (align <= crypto_tfm_ctx_alignment()) align = 1; @@ -205,16 +175,6 @@ generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) } #endif -static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) -{ - unsigned long addr = (unsigned long)raw_ctx; - unsigned long align = AESNI_ALIGN; - - if (align <= crypto_tfm_ctx_alignment()) - align = 1; - return (struct crypto_aes_ctx *)ALIGN(addr, align); -} - static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key, unsigned int key_len) { @@ -250,7 +210,7 @@ static void aesni_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) aes_encrypt(ctx, dst, src); } else { kernel_fpu_begin(); - aesni_enc(ctx, dst, src); + _aesni_enc(ctx, dst, src); kernel_fpu_end(); } } @@ -263,7 +223,7 @@ static void aesni_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) aes_decrypt(ctx, dst, src); } else { kernel_fpu_begin(); - aesni_dec(ctx, dst, src); + _aesni_dec(ctx, dst, src); kernel_fpu_end(); } } @@ -277,90 +237,22 @@ static int aesni_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key, static int ecb_encrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes)) { - kernel_fpu_begin(); - aesni_ecb_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK); - kernel_fpu_end(); - nbytes &= AES_BLOCK_SIZE - 1; - err |= skcipher_walk_done(&walk, nbytes); - } - - return err; + return ecb_crypt_common(req, aesni_ecb_enc); } static int ecb_decrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes)) { - kernel_fpu_begin(); - aesni_ecb_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK); - kernel_fpu_end(); - nbytes &= AES_BLOCK_SIZE - 1; - err |= skcipher_walk_done(&walk, nbytes); - } - - return err; + return ecb_crypt_common(req, aesni_ecb_dec); } static int cbc_encrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes)) { - kernel_fpu_begin(); - aesni_cbc_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - kernel_fpu_end(); - nbytes &= AES_BLOCK_SIZE - 1; - err |= skcipher_walk_done(&walk, nbytes); - } - - return err; + return cbc_crypt_common(req, aesni_cbc_enc); } static int cbc_decrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes)) { - kernel_fpu_begin(); - aesni_cbc_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - kernel_fpu_end(); - nbytes &= AES_BLOCK_SIZE - 1; - err |= skcipher_walk_done(&walk, nbytes); - } - - return err; + return cbc_crypt_common(req, aesni_cbc_dec); } static int cts_cbc_encrypt(struct skcipher_request *req) @@ -476,8 +368,8 @@ static int cts_cbc_decrypt(struct skcipher_request *req) } #ifdef CONFIG_X86_64 -static void aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv) +static int aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv) { /* * based on key length, override with the by8 version @@ -491,40 +383,12 @@ static void aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out, aes_ctr_enc_192_avx_by8(in, iv, (void *)ctx, out, len); else aes_ctr_enc_256_avx_by8(in, iv, (void *)ctx, out, len); + return 0; } static int ctr_crypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - u8 keystream[AES_BLOCK_SIZE]; - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes) > 0) { - kernel_fpu_begin(); - if (nbytes & AES_BLOCK_MASK) - static_call(aesni_ctr_enc_tfm)(ctx, walk.dst.virt.addr, - walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, - walk.iv); - nbytes &= ~AES_BLOCK_MASK; - - if (walk.nbytes == walk.total && nbytes > 0) { - aesni_enc(ctx, keystream, walk.iv); - crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes, - walk.src.virt.addr + walk.nbytes - nbytes, - keystream, nbytes); - crypto_inc(walk.iv, AES_BLOCK_SIZE); - nbytes = 0; - } - kernel_fpu_end(); - err |= skcipher_walk_done(&walk, nbytes); - } - return err; + return ctr_crypt_common(req, static_call(aesni_ctr_enc_tfm), aesni_enc); } static int @@ -606,8 +470,8 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req, u8 *iv, void *aes_ctx, u8 *auth_tag, unsigned long auth_tag_len) { - u8 databuf[sizeof(struct gcm_context_data) + (AESNI_ALIGN - 8)] __aligned(8); - struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AESNI_ALIGN); + u8 databuf[sizeof(struct gcm_context_data) + (AES_ALIGN - 8)] __aligned(8); + struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AES_ALIGN); unsigned long left = req->cryptlen; struct scatter_walk assoc_sg_walk; struct skcipher_walk walk; @@ -762,8 +626,8 @@ static int helper_rfc4106_encrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); unsigned int i; __be32 counter = cpu_to_be32(1); @@ -790,8 +654,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); unsigned int i; if (unlikely(req->assoclen != 16 && req->assoclen != 20)) @@ -816,123 +680,17 @@ static int helper_rfc4106_decrypt(struct aead_request *req) static int xts_aesni_setkey(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen) { - struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); - int err; - - err = xts_verify_key(tfm, key, keylen); - if (err) - return err; - - keylen /= 2; - - /* first half of xts-key is for crypt */ - err = aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, - key, keylen); - if (err) - return err; - - /* second half of xts-key is for tweak */ - return aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, - key + keylen, keylen); -} - -static int xts_crypt(struct skcipher_request *req, bool encrypt) -{ - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); - int tail = req->cryptlen % AES_BLOCK_SIZE; - struct skcipher_request subreq; - struct skcipher_walk walk; - int err; - - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - - err = skcipher_walk_virt(&walk, req, false); - - if (unlikely(tail > 0 && walk.nbytes < walk.total)) { - int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; - - skcipher_walk_abort(&walk); - - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, - skcipher_request_flags(req), - NULL, NULL); - skcipher_request_set_crypt(&subreq, req->src, req->dst, - blocks * AES_BLOCK_SIZE, req->iv); - req = &subreq; - err |= skcipher_walk_virt(&walk, req, false); - } else { - tail = 0; - } - - kernel_fpu_begin(); - - /* calculate first value of T */ - aesni_enc(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); - - while (walk.nbytes > 0) { - int nbytes = walk.nbytes; - - if (nbytes < walk.total) - nbytes &= ~(AES_BLOCK_SIZE - 1); - - if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - nbytes, walk.iv); - else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - nbytes, walk.iv); - kernel_fpu_end(); - - err |= skcipher_walk_done(&walk, walk.nbytes - nbytes); - - if (walk.nbytes > 0) - kernel_fpu_begin(); - } - - if (unlikely(tail > 0 && !err)) { - struct scatterlist sg_src[2], sg_dst[2]; - struct scatterlist *src, *dst; - - dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); - if (req->dst != req->src) - dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); - - skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, - req->iv); - - err |= skcipher_walk_virt(&walk, &subreq, false); - if (err) - return err; - - kernel_fpu_begin(); - if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - kernel_fpu_end(); - - err = skcipher_walk_done(&walk, 0); - } - return err; + return xts_setkey_common(tfm, key, keylen, aes_set_key_common); } static int xts_encrypt(struct skcipher_request *req) { - return xts_crypt(req, true); + return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc); } static int xts_decrypt(struct skcipher_request *req) { - return xts_crypt(req, false); + return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc); } static struct crypto_alg aesni_cipher_alg = { @@ -1061,8 +819,8 @@ static int generic_gcmaes_encrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); __be32 counter = cpu_to_be32(1); memcpy(iv, req->iv, 12); @@ -1078,8 +836,8 @@ static int generic_gcmaes_decrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); memcpy(iv, req->iv, 12); *((__be32 *)(iv+12)) = counter; @@ -1102,7 +860,7 @@ static struct aead_alg aesni_aeads[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct aesni_rfc4106_gcm_ctx), - .cra_alignmask = AESNI_ALIGN - 1, + .cra_alignmask = AES_ALIGN - 1, .cra_module = THIS_MODULE, }, }, { @@ -1119,7 +877,7 @@ static struct aead_alg aesni_aeads[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct generic_gcmaes_ctx), - .cra_alignmask = AESNI_ALIGN - 1, + .cra_alignmask = AES_ALIGN - 1, .cra_module = THIS_MODULE, }, } }; diff --git a/arch/x86/crypto/aesni-intel_glue.h b/arch/x86/crypto/aesni-intel_glue.h new file mode 100644 index 000000000000..51049f86d78e --- /dev/null +++ b/arch/x86/crypto/aesni-intel_glue.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Support for Intel AES-NI instructions. This file contains function + * prototypes to be referenced for other AES implementations + */ + +asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len); +asmlinkage void _aesni_enc(const void *ctx, u8 *out, const u8 *in); +asmlinkage void _aesni_dec(const void *ctx, u8 *out, const u8 *in); +asmlinkage void _aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); +asmlinkage void _aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); +asmlinkage void _aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv); +asmlinkage void _aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv); +asmlinkage void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +asmlinkage void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); + +static inline int aesni_enc(const void *ctx, u8 *out, const u8 *in) +{ + _aesni_enc(ctx, out, in); + return 0; +} + +static inline int aesni_dec(const void *ctx, u8 *out, const u8 *in) +{ + _aesni_dec(ctx, out, in); + return 0; +} + +static inline int aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len) +{ + _aesni_ecb_enc(ctx, out, in, len); + return 0; +} + +static inline int aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len) +{ + _aesni_ecb_dec(ctx, out, in, len); + return 0; +} + +static inline int aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv) +{ + _aesni_cbc_enc(ctx, out, in, len, iv); + return 0; +} + +static inline int aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv) +{ + _aesni_cbc_dec(ctx, out, in, len, iv); + return 0; +} + +static inline int aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_xts_encrypt(ctx, out, in, len, iv); + return 0; +} + +static inline int aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_xts_decrypt(ctx, out, in, len, iv); + return 0; +} + +#ifdef CONFIG_X86_64 + +asmlinkage void _aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv); + +static inline int aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv) +{ + _aesni_ctr_enc(ctx, out, in, len, iv); + return 0; +} + +#endif /* CONFIG_X86_64 */ + From patchwork Fri May 14 20:15:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258913 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98954C43462 for ; Fri, 14 May 2021 20:20:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B12361465 for ; Fri, 14 May 2021 20:20:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233688AbhENUWD (ORCPT ); Fri, 14 May 2021 16:22:03 -0400 Received: from mga06.intel.com ([134.134.136.31]:58439 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233159AbhENUVy (ORCPT ); Fri, 14 May 2021 16:21:54 -0400 IronPort-SDR: Gta2bt4XfsiSHVTozbktEjojuvGZ6KWRFwGZwQ/sN+yw/1aEYOE0WMQuLZJUCjNXiXVEbJI+de NsbkS4QqfG8Q== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="261469085" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="261469085" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:40 -0700 IronPort-SDR: /TWRE/pK27TU8ThON76+0YO+Ew27t0d1zYiFo+DjpPCaxfHzn6lreAzm29K3S8g4ziJdFZnerP zBDPZ11J10HQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147202" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:40 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Date: Fri, 14 May 2021 13:15:07 -0700 Message-Id: <20210514201508.27967-11-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker (KL) is Intel's new security feature that protects the AES key at the time of data transformation. New AES SIMD instructions -- as a successor of Intel's AES-NI -- are provided to encode an AES key and reference it for the AES algorithm. New instructions support 128/256-bit keys. While it is not desirable to receive any 192-bit key, AES-NI instructions are taken to serve this size. New instructions are operational in both 32-/64-bit modes. Included are methods for ECB, CBC, CTR, and XTS modes. They are not compatible with other implementations as referencing an encrypted form only. XTS needs a separate 32-bit assembly code as the 64-bit version clobbers more than eight 128-bit registers. Support 64-bit mode only. setkey() encodes an AES key. Users may displace the AES key right after this as encrypt()/decrypt() methods do not need the key. Most glue code reuses the refactored functions from the AES-NI part. When enabled, it will have a bit higher priority than the AES-NI as providing key protection. Signed-off-by: Chang S. Bae Cc: Herbert Xu Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v1: * Rebased on the refactored code. (Ard Biesheuvel) * Dropped exporting the single block interface. (Ard Biesheuvel) * Fixed the fallback and error handling paths. (Ard Biesheuvel) * Revised the module description. (Dave Hansen and Peter Zijlsta) * Made the build depend on the binutils version to support new instructions. (Borislav Petkov and Peter Zijlstra) * Updated the changelog accordingly. --- arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aeskl-intel_asm.S | 1181 ++++++++++++++++++++++++++++ arch/x86/crypto/aeskl-intel_glue.c | 390 +++++++++ crypto/Kconfig | 23 + 4 files changed, 1597 insertions(+) create mode 100644 arch/x86/crypto/aeskl-intel_asm.S create mode 100644 arch/x86/crypto/aeskl-intel_glue.c diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 30d558df67cd..6bb4e2fb2b47 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o +obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o +aeskl-intel-y := aeskl-intel_asm.o aesni-intel_asm.o aeskl-intel_glue.o aes-intel_glue.o + obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o diff --git a/arch/x86/crypto/aeskl-intel_asm.S b/arch/x86/crypto/aeskl-intel_asm.S new file mode 100644 index 000000000000..34426f11219d --- /dev/null +++ b/arch/x86/crypto/aeskl-intel_asm.S @@ -0,0 +1,1181 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Implement AES algorithm using Intel AES Key Locker instructions. + * + * Most code is based from the AES-NI implementation, aesni-intel_asm.S + * + */ + +#include +#include +#include +#include "aes-intel_asm.S" + +.text + +#define STATE1 %xmm0 +#define STATE2 %xmm1 +#define STATE3 %xmm2 +#define STATE4 %xmm3 +#define STATE5 %xmm4 +#define STATE6 %xmm5 +#define STATE7 %xmm6 +#define STATE8 %xmm7 +#define STATE STATE1 + +#define IV %xmm9 +#define KEY %xmm10 +#define BSWAP_MASK %xmm11 +#define CTR %xmm12 +#define INC %xmm13 + +#ifdef __x86_64__ +#define IN1 %xmm8 +#define IN2 %xmm9 +#define IN3 %xmm10 +#define IN4 %xmm11 +#define IN5 %xmm12 +#define IN6 %xmm13 +#define IN7 %xmm14 +#define IN8 %xmm15 +#define IN IN1 +#define TCTR_LOW %r11 +#else +#define IN %xmm1 +#endif + +#ifdef __x86_64__ +#define AREG %rax +#define HANDLEP %rdi +#define OUTP %rsi +#define KLEN %r9d +#define INP %rdx +#define T1 %r10 +#define LEN %rcx +#define IVP %r8 +#else +#define AREG %eax +#define HANDLEP %edi +#define OUTP AREG +#define KLEN %ebx +#define INP %edx +#define T1 %ecx +#define LEN %esi +#define IVP %ebp +#endif + +#define UKEYP OUTP +#define GF128MUL_MASK %xmm11 + +/* + * int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len) + */ +SYM_FUNC_START(aeskl_setkey) + FRAME_BEGIN +#ifndef __x86_64__ + push HANDLEP + movl (FRAME_OFFSET+8)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+12)(%esp), UKEYP # in_key + movl (FRAME_OFFSET+16)(%esp), %edx # key_len +#endif + movl %edx, 480(HANDLEP) + movdqu (UKEYP), STATE1 + mov $1, %eax + cmp $16, %dl + je .Lsetkey_128 + + movdqu 0x10(UKEYP), STATE2 + encodekey256 %eax, %eax + movdqu STATE4, 0x30(HANDLEP) + jmp .Lsetkey_end +.Lsetkey_128: + encodekey128 %eax, %eax + +.Lsetkey_end: + movdqu STATE1, (HANDLEP) + movdqu STATE2, 0x10(HANDLEP) + movdqu STATE3, 0x20(HANDLEP) + + xor AREG, AREG +#ifndef __x86_64__ + popl HANDLEP +#endif + FRAME_END + ret +SYM_FUNC_END(aeskl_setkey) + +/* + * int _aeskl_enc(const void *ctx, u8 *dst, const u8 *src) + */ +SYM_FUNC_START(_aeskl_enc) + FRAME_BEGIN +#ifndef __x86_64__ + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+12)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+16)(%esp), OUTP # dst + movl (FRAME_OFFSET+20)(%esp), INP # src +#endif + movdqu (INP), STATE + movl 480(HANDLEP), KLEN + + cmp $16, KLEN + je .Lenc_128 + aesenc256kl (HANDLEP), STATE + jz .Lenc_err + jmp .Lenc_noerr +.Lenc_128: + aesenc128kl (HANDLEP), STATE + jz .Lenc_err + +.Lenc_noerr: + xor AREG, AREG + jmp .Lenc_end +.Lenc_err: + mov $1, AREG +.Lenc_end: + movdqu STATE, (OUTP) +#ifndef __x86_64__ + popl KLEN + popl HANDLEP +#endif + FRAME_END + ret +SYM_FUNC_END(_aeskl_enc) + +/* + * int _aeskl_dec(const void *ctx, u8 *dst, const u8 *src) + */ +SYM_FUNC_START(_aeskl_dec) + FRAME_BEGIN +#ifndef __x86_64__ + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+12)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+16)(%esp), OUTP # dst + movl (FRAME_OFFSET+20)(%esp), INP # src +#endif + movdqu (INP), STATE + mov 480(HANDLEP), KLEN + + cmp $16, KLEN + je .Ldec_128 + aesdec256kl (HANDLEP), STATE + jz .Ldec_err + jmp .Ldec_noerr +.Ldec_128: + aesdec128kl (HANDLEP), STATE + jz .Ldec_err + +.Ldec_noerr: + xor AREG, AREG + jmp .Ldec_end +.Ldec_err: + mov $1, AREG +.Ldec_end: + movdqu STATE, (OUTP) +#ifndef __x86_64__ + popl KLEN + popl HANDLEP +#endif + FRAME_END + ret +SYM_FUNC_END(_aeskl_dec) + +/* + * int _aeskl_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len) + */ +SYM_FUNC_START(_aeskl_ecb_enc) + FRAME_BEGIN +#ifndef __x86_64__ + pushl LEN + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+16)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+20)(%esp), OUTP # dst + movl (FRAME_OFFSET+24)(%esp), INP # src + movl (FRAME_OFFSET+28)(%esp), LEN # len +#endif + test LEN, LEN + jz .Lecb_enc_noerr + mov 480(HANDLEP), KLEN + cmp $16, LEN + jb .Lecb_enc_noerr + cmp $128, LEN + jb .Lecb_enc1 + +.align 4 +.Lecb_enc8: + movdqu (INP), STATE1 + movdqu 0x10(INP), STATE2 + movdqu 0x20(INP), STATE3 + movdqu 0x30(INP), STATE4 + movdqu 0x40(INP), STATE5 + movdqu 0x50(INP), STATE6 + movdqu 0x60(INP), STATE7 + movdqu 0x70(INP), STATE8 + + cmp $16, KLEN + je .Lecb_enc8_128 + aesencwide256kl (HANDLEP) + jz .Lecb_enc_err + jmp .Lecb_enc8_end +.Lecb_enc8_128: + aesencwide128kl (HANDLEP) + jz .Lecb_enc_err + +.Lecb_enc8_end: + movdqu STATE1, (OUTP) + movdqu STATE2, 0x10(OUTP) + movdqu STATE3, 0x20(OUTP) + movdqu STATE4, 0x30(OUTP) + movdqu STATE5, 0x40(OUTP) + movdqu STATE6, 0x50(OUTP) + movdqu STATE7, 0x60(OUTP) + movdqu STATE8, 0x70(OUTP) + + sub $128, LEN + add $128, INP + add $128, OUTP + cmp $128, LEN + jge .Lecb_enc8 + cmp $16, LEN + jb .Lecb_enc_noerr + +.align 4 +.Lecb_enc1: + movdqu (INP), STATE1 + cmp $16, KLEN + je .Lecb_enc1_128 + aesenc256kl (HANDLEP), STATE + jz .Lecb_enc_err + jmp .Lecb_enc1_end +.Lecb_enc1_128: + aesenc128kl (HANDLEP), STATE + jz .Lecb_enc_err + +.Lecb_enc1_end: + movdqu STATE1, (OUTP) + sub $16, LEN + add $16, INP + add $16, OUTP + cmp $16, LEN + jge .Lecb_enc1 + +.Lecb_enc_noerr: + xor AREG, AREG + jmp .Lecb_enc_end +.Lecb_enc_err: + mov $1, AREG +.Lecb_enc_end: +#ifndef __x86_64__ + popl KLEN + popl HANDLEP + popl LEN +#endif + FRAME_END + ret +SYM_FUNC_END(_aeskl_ecb_enc) + +/* + * int _aeskl_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len); + */ +SYM_FUNC_START(_aeskl_ecb_dec) + FRAME_BEGIN +#ifndef __x86_64__ + pushl LEN + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+16)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+20)(%esp), OUTP # dst + movl (FRAME_OFFSET+24)(%esp), INP # src + movl (FRAME_OFFSET+28)(%esp), LEN # len +#endif + + test LEN, LEN + jz .Lecb_dec_noerr + mov 480(HANDLEP), KLEN + cmp $16, LEN + jb .Lecb_dec_noerr + cmp $128, LEN + jb .Lecb_dec1 + +.align 4 +.Lecb_dec8: + movdqu (INP), STATE1 + movdqu 0x10(INP), STATE2 + movdqu 0x20(INP), STATE3 + movdqu 0x30(INP), STATE4 + movdqu 0x40(INP), STATE5 + movdqu 0x50(INP), STATE6 + movdqu 0x60(INP), STATE7 + movdqu 0x70(INP), STATE8 + + cmp $16, KLEN + je .Lecb_dec8_128 + aesdecwide256kl (HANDLEP) + jz .Lecb_dec_err + jmp .Lecb_dec8_end +.Lecb_dec8_128: + aesdecwide128kl (HANDLEP) + jz .Lecb_dec_err + +.Lecb_dec8_end: + movdqu STATE1, (OUTP) + movdqu STATE2, 0x10(OUTP) + movdqu STATE3, 0x20(OUTP) + movdqu STATE4, 0x30(OUTP) + movdqu STATE5, 0x40(OUTP) + movdqu STATE6, 0x50(OUTP) + movdqu STATE7, 0x60(OUTP) + movdqu STATE8, 0x70(OUTP) + + sub $128, LEN + add $128, INP + add $128, OUTP + cmp $128, LEN + jge .Lecb_dec8 + cmp $16, LEN + jb .Lecb_dec_noerr + +.align 4 +.Lecb_dec1: + movdqu (INP), STATE1 + cmp $16, KLEN + je .Lecb_dec1_128 + aesdec256kl (HANDLEP), STATE + jz .Lecb_dec_err + jmp .Lecb_dec1_end +.Lecb_dec1_128: + aesdec128kl (HANDLEP), STATE + jz .Lecb_dec_err + +.Lecb_dec1_end: + movdqu STATE1, (OUTP) + sub $16, LEN + add $16, INP + add $16, OUTP + cmp $16, LEN + jge .Lecb_dec1 + +.Lecb_dec_noerr: + xor AREG, AREG + jmp .Lecb_dec_end +.Lecb_dec_err: + mov $1, AREG +.Lecb_dec_end: +#ifndef __x86_64__ + popl KLEN + popl HANDLEP + popl LEN +#endif + FRAME_END + ret +SYM_FUNC_END(_aeskl_ecb_dec) + +/* + * int _aeskl_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len, u8 *iv) + */ +SYM_FUNC_START(_aeskl_cbc_enc) + FRAME_BEGIN +#ifndef __x86_64__ + pushl IVP + pushl LEN + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+20)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+24)(%esp), OUTP # dst + movl (FRAME_OFFSET+28)(%esp), INP # src + movl (FRAME_OFFSET+32)(%esp), LEN # len + movl (FRAME_OFFSET+36)(%esp), IVP # iv +#endif + + cmp $16, LEN + jb .Lcbc_enc_noerr + mov 480(HANDLEP), KLEN + movdqu (IVP), STATE + +.align 4 +.Lcbc_enc1: + movdqu (INP), IN + pxor IN, STATE + + cmp $16, KLEN + je .Lcbc_enc1_128 + aesenc256kl (HANDLEP), STATE + jz .Lcbc_enc_err + jmp .Lcbc_enc1_end +.Lcbc_enc1_128: + aesenc128kl (HANDLEP), STATE + jz .Lcbc_enc_err + +.Lcbc_enc1_end: + movdqu STATE, (OUTP) + sub $16, LEN + add $16, INP + add $16, OUTP + cmp $16, LEN + jge .Lcbc_enc1 + movdqu STATE, (IVP) + +.Lcbc_enc_noerr: + xor AREG, AREG + jmp .Lcbc_enc_end +.Lcbc_enc_err: + mov $1, AREG +.Lcbc_enc_end: +#ifndef __x86_64__ + popl KLEN + popl HANDLEP + popl LEN + popl IVP +#endif + FRAME_END + ret +SYM_FUNC_END(_aeskl_cbc_enc) + +/* + * int _aeskl_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len, u8 *iv) + */ +SYM_FUNC_START(_aeskl_cbc_dec) + FRAME_BEGIN +#ifndef __x86_64__ + pushl IVP + pushl LEN + pushl HANDLEP + pushl KLEN + movl (FRAME_OFFSET+20)(%esp), HANDLEP # ctx + movl (FRAME_OFFSET+24)(%esp), OUTP # dst + movl (FRAME_OFFSET+28)(%esp), INP # src + movl (FRAME_OFFSET+32)(%esp), LEN # len + movl (FRAME_OFFSET+36)(%esp), IVP # iv +#endif + + cmp $16, LEN + jb .Lcbc_dec_noerr + mov 480(HANDLEP), KLEN +#ifdef __x86_64__ + cmp $128, LEN + jb .Lcbc_dec1_pre + +.align 4 +.Lcbc_dec8: + movdqu 0x0(INP), STATE1 + movdqu 0x10(INP), STATE2 + movdqu 0x20(INP), STATE3 + movdqu 0x30(INP), STATE4 + movdqu 0x40(INP), STATE5 + movdqu 0x50(INP), STATE6 + movdqu 0x60(INP), STATE7 + movdqu 0x70(INP), STATE8 + + movdqu (IVP), IN1 + movdqa STATE1, IN2 + movdqa STATE2, IN3 + movdqa STATE3, IN4 + movdqa STATE4, IN5 + movdqa STATE5, IN6 + movdqa STATE6, IN7 + movdqa STATE7, IN8 + movdqu STATE8, (IVP) + + cmp $16, KLEN + je .Lcbc_dec8_128 + aesdecwide256kl (HANDLEP) + jz .Lcbc_dec_err + jmp .Lcbc_dec8_end +.Lcbc_dec8_128: + aesdecwide128kl (HANDLEP) + jz .Lcbc_dec_err + +.Lcbc_dec8_end: + pxor IN1, STATE1 + pxor IN2, STATE2 + pxor IN3, STATE3 + pxor IN4, STATE4 + pxor IN5, STATE5 + pxor IN6, STATE6 + pxor IN7, STATE7 + pxor IN8, STATE8 + + movdqu STATE1, 0x0(OUTP) + movdqu STATE2, 0x10(OUTP) + movdqu STATE3, 0x20(OUTP) + movdqu STATE4, 0x30(OUTP) + movdqu STATE5, 0x40(OUTP) + movdqu STATE6, 0x50(OUTP) + movdqu STATE7, 0x60(OUTP) + movdqu STATE8, 0x70(OUTP) + + sub $128, LEN + add $128, INP + add $128, OUTP + cmp $128, LEN + jge .Lcbc_dec8 + cmp $16, LEN + jb .Lcbc_dec_noerr +#endif + +.align 4 +.Lcbc_dec1_pre: + movdqu (IVP), STATE3 +.Lcbc_dec1: + movdqu (INP), STATE2 + movdqa STATE2, STATE1 + + cmp $16, KLEN + je .Lcbc_dec1_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lcbc_dec_err + jmp .Lcbc_dec1_end +.Lcbc_dec1_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lcbc_dec_err + +.Lcbc_dec1_end: + pxor STATE3, STATE1 + movdqu STATE1, (OUTP) + movdqa STATE2, STATE3 + sub $16, LEN + add $16, INP + add $16, OUTP + cmp $16, LEN + jge .Lcbc_dec1 + movdqu STATE3, (IVP) + +.Lcbc_dec_noerr: + xor AREG, AREG + jmp .Lcbc_dec_end +.Lcbc_dec_err: + mov $1, AREG +.Lcbc_dec_end: +#ifndef __x86_64__ + popl KLEN + popl HANDLEP + popl LEN + popl IVP +#endif + FRAME_END + ret +SYM_FUNC_END(_aeskl_cbc_dec) + +#ifdef __x86_64__ + +/* + * _aeskl_ctr_inc_init: internal ABI + * setup registers used by _aesni_inc + * input: + * IV + * output: + * CTR: == IV, in little endian + * TCTR_LOW: == lower qword of CTR + * INC: == 1, in little endian + * BSWAP_MASK == endian swapping mask + */ +SYM_FUNC_START_LOCAL(_aeskl_ctr_inc_init) + movaps .Lbswap_mask, BSWAP_MASK + movaps IV, CTR + pshufb BSWAP_MASK, CTR + mov $1, TCTR_LOW + movq TCTR_LOW, INC + movq CTR, TCTR_LOW + ret +SYM_FUNC_END(_aeskl_ctr_inc_init) + +/* + * _aeskl_ctr_inc: internal ABI + * Increase IV by 1, IV is in big endian + * input: + * IV + * CTR: == IV, in little endian + * TCTR_LOW: == lower qword of CTR + * INC: == 1, in little endian + * BSWAP_MASK == endian swapping mask + * output: + * IV: Increase by 1 + * changed: + * CTR: == output IV, in little endian + * TCTR_LOW: == lower qword of CTR + */ +SYM_FUNC_START_LOCAL(_aeskl_ctr_inc) + paddq INC, CTR + add $1, TCTR_LOW + jnc .Linc_low + pslldq $8, INC + paddq INC, CTR + psrldq $8, INC +.Linc_low: + movaps CTR, IV + pshufb BSWAP_MASK, IV + ret +SYM_FUNC_END(_aeskl_ctr_inc) + +/* + * CTR implementations + */ + +/* + * int _aeskl_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len, u8 *iv) + */ +SYM_FUNC_START(_aeskl_ctr_enc) + FRAME_BEGIN + cmp $16, LEN + jb .Lctr_enc_noerr + mov 480(HANDLEP), KLEN + movdqu (IVP), IV + call _aeskl_ctr_inc_init + cmp $128, LEN + jb .Lctr_enc1 + +.align 4 +.Lctr_enc8: + movaps IV, STATE1 + call _aeskl_ctr_inc + movaps IV, STATE2 + call _aeskl_ctr_inc + movaps IV, STATE3 + call _aeskl_ctr_inc + movaps IV, STATE4 + call _aeskl_ctr_inc + movaps IV, STATE5 + call _aeskl_ctr_inc + movaps IV, STATE6 + call _aeskl_ctr_inc + movaps IV, STATE7 + call _aeskl_ctr_inc + movaps IV, STATE8 + call _aeskl_ctr_inc + + cmp $16, KLEN + je .Lctr_enc8_128 + aesencwide256kl (%rdi) + jz .Lctr_enc_err + jmp .Lctr_enc8_end +.Lctr_enc8_128: + aesencwide128kl (%rdi) + jz .Lctr_enc_err +.Lctr_enc8_end: + + movups (INP), IN1 + pxor IN1, STATE1 + movups STATE1, (OUTP) + + movups 0x10(INP), IN1 + pxor IN1, STATE2 + movups STATE2, 0x10(OUTP) + + movups 0x20(INP), IN1 + pxor IN1, STATE3 + movups STATE3, 0x20(OUTP) + + movups 0x30(INP), IN1 + pxor IN1, STATE4 + movups STATE4, 0x30(OUTP) + + movups 0x40(INP), IN1 + pxor IN1, STATE5 + movups STATE5, 0x40(OUTP) + + movups 0x50(INP), IN1 + pxor IN1, STATE6 + movups STATE6, 0x50(OUTP) + + movups 0x60(INP), IN1 + pxor IN1, STATE7 + movups STATE7, 0x60(OUTP) + + movups 0x70(INP), IN1 + pxor IN1, STATE8 + movups STATE8, 0x70(OUTP) + + sub $128, LEN + add $128, INP + add $128, OUTP + cmp $128, LEN + jge .Lctr_enc8 + cmp $16, LEN + jb .Lctr_enc_end + +.align 4 +.Lctr_enc1: + movaps IV, STATE1 + call _aeskl_ctr_inc + + cmp $16, KLEN + je .Lctr_enc1_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lctr_enc_err + jmp .Lctr_enc1_end +.Lctr_enc1_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lctr_enc_err + +.Lctr_enc1_end: + movups (INP), IN1 + pxor IN1, STATE1 + movups STATE1, (OUTP) + sub $16, LEN + add $16, INP + add $16, OUTP + cmp $16, LEN + jge .Lctr_enc1 + +.Lctr_enc_end: + movdqu IV, (IVP) +.Lctr_enc_noerr: + xor AREG, AREG + jmp .Lctr_enc_ret +.Lctr_enc_err: + mov $1, AREG +.Lctr_enc_ret: + FRAME_END + ret +SYM_FUNC_END(_aeskl_ctr_enc) + +/* + * XTS implementation + */ + +/* + * _aeskl_gf128mul_x_ble: internal ABI + * Multiply in GF(2^128) for XTS IVs + * input: + * IV: current IV + * GF128MUL_MASK == mask with 0x87 and 0x01 + * output: + * IV: next IV + * changed: + * CTR: == temporary value + */ +#define _aeskl_gf128mul_x_ble() \ + pshufd $0x13, IV, KEY; \ + paddq IV, IV; \ + psrad $31, KEY; \ + pand GF128MUL_MASK, KEY; \ + pxor KEY, IV; + +/* + * int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src, + * unsigned int len, le128 *iv) + */ +SYM_FUNC_START(_aeskl_xts_encrypt) + FRAME_BEGIN + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK + movups (IVP), IV + + mov 480(HANDLEP), KLEN + +.Lxts_enc8: + sub $128, LEN + jl .Lxts_enc1_pre + + movdqa IV, STATE1 + movdqu (INP), INC + pxor INC, STATE1 + movdqu IV, (OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE5 + movdqu 0x40(INP), INC + pxor INC, STATE5 + movdqu IV, 0x40(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE6 + movdqu 0x50(INP), INC + pxor INC, STATE6 + movdqu IV, 0x50(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE7 + movdqu 0x60(INP), INC + pxor INC, STATE7 + movdqu IV, 0x60(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE8 + movdqu 0x70(INP), INC + pxor INC, STATE8 + movdqu IV, 0x70(OUTP) + + cmp $16, KLEN + je .Lxts_enc8_128 + aesencwide256kl (%rdi) + jz .Lxts_enc_ret_err + jmp .Lxts_enc8_end +.Lxts_enc8_128: + aesencwide128kl (%rdi) + jz .Lxts_enc_ret_err + +.Lxts_enc8_end: + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC + pxor INC, STATE2 + movdqu STATE2, 0x10(OUTP) + + movdqu 0x20(OUTP), INC + pxor INC, STATE3 + movdqu STATE3, 0x20(OUTP) + + movdqu 0x30(OUTP), INC + pxor INC, STATE4 + movdqu STATE4, 0x30(OUTP) + + movdqu 0x40(OUTP), INC + pxor INC, STATE5 + movdqu STATE5, 0x40(OUTP) + + movdqu 0x50(OUTP), INC + pxor INC, STATE6 + movdqu STATE6, 0x50(OUTP) + + movdqu 0x60(OUTP), INC + pxor INC, STATE7 + movdqu STATE7, 0x60(OUTP) + + movdqu 0x70(OUTP), INC + pxor INC, STATE8 + movdqu STATE8, 0x70(OUTP) + + _aeskl_gf128mul_x_ble() + + add $128, INP + add $128, OUTP + test LEN, LEN + jnz .Lxts_enc8 + +.Lxts_enc_ret_iv: + movups IV, (IVP) +.Lxts_enc_ret_noerr: + xor AREG, AREG + jmp .Lxts_enc_ret +.Lxts_enc_ret_err: + mov $1, AREG +.Lxts_enc_ret: + FRAME_END + ret + +.Lxts_enc1_pre: + add $128, LEN + jz .Lxts_enc_ret_iv + sub $16, LEN + jl .Lxts_enc_cts4 + +.Lxts_enc1: + movdqu (INP), STATE1 + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_enc1_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + jmp .Lxts_enc1_end +.Lxts_enc1_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + +.Lxts_enc1_end: + pxor IV, STATE1 + _aeskl_gf128mul_x_ble() + + test LEN, LEN + jz .Lxts_enc1_out + + add $16, INP + sub $16, LEN + jl .Lxts_enc_cts1 + + movdqu STATE1, (OUTP) + add $16, OUTP + jmp .Lxts_enc1 + +.Lxts_enc1_out: + movdqu STATE1, (OUTP) + jmp .Lxts_enc_ret_iv + +.Lxts_enc_cts4: + movdqu STATE8, STATE1 + sub $16, OUTP + +.Lxts_enc_cts1: + lea .Lcts_permute_table(%rip), T1 + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), STATE2 + movaps STATE1, STATE3 + pshufb STATE2, STATE1 + movups STATE1, (LEN) + + movups (IVP), STATE1 + pshufb STATE1, IN1 + pblendvb STATE3, IN1 + movaps IN1, STATE1 + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_enc1_cts_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + jmp .Lxts_enc1_cts_end +.Lxts_enc1_cts_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + +.Lxts_enc1_cts_end: + pxor IV, STATE1 + movups STATE1, (OUTP) + jmp .Lxts_enc_ret_noerr +SYM_FUNC_END(_aeskl_xts_encrypt) + +/* + * int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src, + * unsigned int len, le128 *iv) + */ +SYM_FUNC_START(_aeskl_xts_decrypt) + FRAME_BEGIN + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK + movups (IVP), IV + + mov 480(HANDLEP), KLEN + + test $15, LEN + jz .Lxts_dec8 + sub $16, LEN + +.Lxts_dec8: + sub $128, LEN + jl .Lxts_dec1_pre + + movdqa IV, STATE1 + movdqu (INP), INC + pxor INC, STATE1 + movdqu IV, (OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE5 + movdqu 0x40(INP), INC + pxor INC, STATE5 + movdqu IV, 0x40(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE6 + movdqu 0x50(INP), INC + pxor INC, STATE6 + movdqu IV, 0x50(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE7 + movdqu 0x60(INP), INC + pxor INC, STATE7 + movdqu IV, 0x60(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE8 + movdqu 0x70(INP), INC + pxor INC, STATE8 + movdqu IV, 0x70(OUTP) + + cmp $16, KLEN + je .Lxts_dec8_128 + aesdecwide256kl (%rdi) + jz .Lxts_dec_ret_err + jmp .Lxts_dec8_end +.Lxts_dec8_128: + aesdecwide128kl (%rdi) + jz .Lxts_dec_ret_err + +.Lxts_dec8_end: + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC + pxor INC, STATE2 + movdqu STATE2, 0x10(OUTP) + + movdqu 0x20(OUTP), INC + pxor INC, STATE3 + movdqu STATE3, 0x20(OUTP) + + movdqu 0x30(OUTP), INC + pxor INC, STATE4 + movdqu STATE4, 0x30(OUTP) + + movdqu 0x40(OUTP), INC + pxor INC, STATE5 + movdqu STATE5, 0x40(OUTP) + + movdqu 0x50(OUTP), INC + pxor INC, STATE6 + movdqu STATE6, 0x50(OUTP) + + movdqu 0x60(OUTP), INC + pxor INC, STATE7 + movdqu STATE7, 0x60(OUTP) + + movdqu 0x70(OUTP), INC + pxor INC, STATE8 + movdqu STATE8, 0x70(OUTP) + + _aeskl_gf128mul_x_ble() + + add $128, INP + add $128, OUTP + test LEN, LEN + jnz .Lxts_dec8 + +.Lxts_dec_ret_iv: + movups IV, (IVP) +.Lxts_dec_ret_noerr: + xor AREG, AREG + jmp .Lxts_dec_ret +.Lxts_dec_ret_err: + mov $1, AREG +.Lxts_dec_ret: + FRAME_END + ret + +.Lxts_dec1_pre: + add $128, LEN + jz .Lxts_dec_ret_iv + +.Lxts_dec1: + movdqu (INP), STATE1 + + add $16, INP + sub $16, LEN + jl .Lxts_dec_cts1 + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_end +.Lxts_dec1_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_end: + pxor IV, STATE1 + _aeskl_gf128mul_x_ble() + + test LEN, LEN + jz .Lxts_dec1_out + + movdqu STATE1, (OUTP) + add $16, OUTP + jmp .Lxts_dec1 + +.Lxts_dec1_out: + movdqu STATE1, (OUTP) + jmp .Lxts_dec_ret_iv + +.Lxts_dec_cts1: + movdqa IV, STATE5 + _aeskl_gf128mul_x_ble() + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_cts_pre_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_cts_pre_end +.Lxts_dec1_cts_pre_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_cts_pre_end: + pxor IV, STATE1 + + lea .Lcts_permute_table(%rip), T1 + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), STATE2 + movaps STATE1, STATE3 + pshufb STATE2, STATE1 + movups STATE1, (LEN) + + movups (IVP), STATE1 + pshufb STATE1, IN1 + pblendvb STATE3, IN1 + movaps IN1, STATE1 + + pxor STATE5, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_cts_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_cts_end +.Lxts_dec1_cts_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_cts_end: + pxor STATE5, STATE1 + + movups STATE1, (OUTP) + jmp .Lxts_dec_ret_noerr + +SYM_FUNC_END(_aeskl_xts_decrypt) + +#endif diff --git a/arch/x86/crypto/aeskl-intel_glue.c b/arch/x86/crypto/aeskl-intel_glue.c new file mode 100644 index 000000000000..e0a052e89fa0 --- /dev/null +++ b/arch/x86/crypto/aeskl-intel_glue.c @@ -0,0 +1,390 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Support for AES Key Locker instructions. This file contains glue + * code and the real AES implementation is in aeskl-intel_asm.S. + * + * Most code is based on AES-NI glue code, aesni-intel_glue.c + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "aes-intel_glue.h" +#include "aesni-intel_glue.h" + +asmlinkage int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); + +asmlinkage int _aeskl_enc(const void *ctx, u8 *out, const u8 *in); +asmlinkage int _aeskl_dec(const void *ctx, u8 *out, const u8 *in); + +asmlinkage int _aeskl_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); +asmlinkage int _aeskl_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); + +asmlinkage int _aeskl_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv); +asmlinkage int _aeskl_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv); + +#ifdef CONFIG_X86_64 +asmlinkage int _aeskl_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv); + +asmlinkage int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +asmlinkage int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +#endif + +static int aeskl_setkey_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key, + unsigned int key_len) +{ + struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx); + int err; + + if (!crypto_simd_usable()) + return -EBUSY; + + if ((key_len != AES_KEYSIZE_128) && (key_len != AES_KEYSIZE_192) && + (key_len != AES_KEYSIZE_256)) + return -EINVAL; + + kernel_fpu_begin(); + /* + * For 128-bit and 256-bit keys, encode to a handle that is usable only + * in kernel mode. Use AES-NI instructions for 192-bit keys. + */ + if (likely(key_len != AES_KEYSIZE_192)) + err = aeskl_setkey(ctx, in_key, key_len); + else + err = aesni_set_key(ctx, in_key, key_len); + kernel_fpu_end(); + + return err; +} + +static inline u32 keylength(const void *raw_ctx) +{ + struct crypto_aes_ctx *ctx = aes_ctx((void *)raw_ctx); + + return ctx->key_length; +} + +static inline int aeskl_enc(const void *ctx, u8 *out, const u8 *in) +{ + if (unlikely(keylength(ctx) == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_enc(ctx, out, in)) + return -EINVAL; + else + return 0; +} + +static inline int aeskl_dec(const void *ctx, u8 *out, const u8 *in) +{ + if (unlikely(keylength(ctx) == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_dec(ctx, out, in)) + return -EINVAL; + else + return 0; +} + +static int aeskl_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_ecb_enc(ctx, out, in, len)) + return -EINVAL; + else + return 0; +} + +static int aeskl_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_ecb_dec(ctx, out, in, len)) + return -EINVAL; + else + return 0; +} + +static int aeskl_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_cbc_enc(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +static int aeskl_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_cbc_dec(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +#ifdef CONFIG_X86_64 + +static int aeskl_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, + u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_ctr_enc(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +static inline int aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_xts_encrypt(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +static inline int aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + + if (_aeskl_xts_decrypt(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +#endif /* CONFIG_X86_64 */ + +static int aeskl_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int len) +{ + struct crypto_tfm *crypto_tfm = crypto_skcipher_tfm(tfm); + void *raw_ctx = crypto_skcipher_ctx(tfm); + + return aeskl_setkey_common(crypto_tfm, raw_ctx, key, len); +} + +static int ecb_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return ecb_crypt_common(req, aeskl_ecb_enc); + else + return ecb_crypt_common(req, aesni_ecb_enc); +} + +static int ecb_decrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return ecb_crypt_common(req, aeskl_ecb_dec); + else + return ecb_crypt_common(req, aesni_ecb_dec); +} + +static int cbc_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return cbc_crypt_common(req, aeskl_cbc_enc); + else + return cbc_crypt_common(req, aesni_cbc_enc); +} + +static int cbc_decrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return cbc_crypt_common(req, aeskl_cbc_dec); + else + return cbc_crypt_common(req, aesni_cbc_dec); +} + +#ifdef CONFIG_X86_64 + +static int ctr_crypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return ctr_crypt_common(req, aeskl_ctr_enc, aeskl_enc); + else + return ctr_crypt_common(req, aesni_ctr_enc, aesni_enc); +} + +static int aeskl_xts_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keylen) +{ + return xts_setkey_common(tfm, key, keylen, aeskl_setkey_common); +} + +static int xts_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return xts_crypt_common(req, aeskl_xts_encrypt, aeskl_enc); + else + return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc); +} + +static int xts_decrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return xts_crypt_common(req, aeskl_xts_decrypt, aeskl_enc); + else + return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc); +} + +#endif /* CONFIG_X86_64 */ + +static struct skcipher_alg aeskl_skciphers[] = { + { + .base = { + .cra_name = "__ecb(aes)", + .cra_driver_name = "__ecb-aes-aeskl", + .cra_priority = 401, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = CRYPTO_AES_CTX_SIZE, + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .setkey = aeskl_skcipher_setkey, + .encrypt = ecb_encrypt, + .decrypt = ecb_decrypt, + }, { + .base = { + .cra_name = "__cbc(aes)", + .cra_driver_name = "__cbc-aes-aeskl", + .cra_priority = 401, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = CRYPTO_AES_CTX_SIZE, + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .setkey = aeskl_skcipher_setkey, + .encrypt = cbc_encrypt, + .decrypt = cbc_decrypt, +#ifdef CONFIG_X86_64 + }, { + .base = { + .cra_name = "__ctr(aes)", + .cra_driver_name = "__ctr-aes-aeskl", + .cra_priority = 401, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = 1, + .cra_ctxsize = CRYPTO_AES_CTX_SIZE, + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .chunksize = AES_BLOCK_SIZE, + .setkey = aeskl_skcipher_setkey, + .encrypt = ctr_crypt, + .decrypt = ctr_crypt, + }, { + .base = { + .cra_name = "__xts(aes)", + .cra_driver_name = "__xts-aes-aeskl", + .cra_priority = 400, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = XTS_AES_CTX_SIZE, + .cra_module = THIS_MODULE, + }, + .min_keysize = 2 * AES_MIN_KEY_SIZE, + .max_keysize = 2 * AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, + .setkey = aeskl_xts_setkey, + .encrypt = xts_encrypt, + .decrypt = xts_decrypt, +#endif + } +}; + +static struct simd_skcipher_alg *aeskl_simd_skciphers[ARRAY_SIZE(aeskl_skciphers)]; + +static const struct x86_cpu_id aes_keylocker_cpuid[] = { + X86_MATCH_FEATURE(X86_FEATURE_AES, NULL), + X86_MATCH_FEATURE(X86_FEATURE_KEYLOCKER, NULL), + {} +}; + +static int __init aeskl_init(void) +{ + u32 eax, ebx, ecx, edx; + int err; + + if (!x86_match_cpu(aes_keylocker_cpuid)) + return -ENODEV; + + cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); + if (!(ebx & KEYLOCKER_CPUID_EBX_AESKLE) || + !(eax & KEYLOCKER_CPUID_EAX_SUPERVISOR) || + !(ebx & KEYLOCKER_CPUID_EBX_WIDE)) + return -ENODEV; + + err = simd_register_skciphers_compat(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers), + aeskl_simd_skciphers); + if (err) + return err; + + return 0; +} + +static void __exit aeskl_exit(void) +{ + simd_unregister_skciphers(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers), + aeskl_simd_skciphers); +} + +late_initcall(aeskl_init); +module_exit(aeskl_exit); + +MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm, AES Key Locker implementation"); +MODULE_LICENSE("GPL"); +MODULE_ALIAS_CRYPTO("aes"); diff --git a/crypto/Kconfig b/crypto/Kconfig index ca3b02dcbbfa..5f3d4052fa3c 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1113,6 +1113,29 @@ config CRYPTO_AES_NI_INTEL ECB, CBC, LRW, XTS. The 64 bit version has additional acceleration for CTR. +config CRYPTO_AES_KL + tristate "AES cipher algorithms (AES-KL)" + depends on X86_KEYLOCKER + depends on LD_VERSION >= 23600 + select CRYPTO_AES_NI_INTEL + help + Key Locker provides AES SIMD instructions (AES-KL) to protect AES + keys. They are analogous to AES-NI. AES-KL supports to encode AES + keys and use an encoded form to transform data instead of + referencing an AES key all the way. + + The CPU-internal key is used to wrap AES keys. AES-KL relies on + the x86 core code that manages the internal key. + + AES-KL supports 128-/256-bit keys. This module uses the AES-NI + methods for 192-bit keys. It includes such block cipher mode + support as ECB, CBC, CTR, and XTS. + + binutils version 2.36 or later is needed to assemble AES-KL. + + See the CRYPTO_AES_NI_INTEL description for more about the AES + algorithm. + config CRYPTO_AES_SPARC64 tristate "AES cipher algorithms (SPARC64)" depends on SPARC64 From patchwork Fri May 14 20:15:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12258909 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE6ADC43617 for ; Fri, 14 May 2021 20:20:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 794276145D for ; Fri, 14 May 2021 20:20:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233548AbhENUV6 (ORCPT ); Fri, 14 May 2021 16:21:58 -0400 Received: from mga14.intel.com ([192.55.52.115]:16073 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233309AbhENUVy (ORCPT ); Fri, 14 May 2021 16:21:54 -0400 IronPort-SDR: zvnmxi8aJEfgTiTVI1iam1tQnDRGB1WJPS6oBXH/jzjOmPh+eEt9m3CGrAWM9Z/3oddaxRmBty 5obd3qvTsE/A== X-IronPort-AV: E=McAfee;i="6200,9189,9984"; a="199921604" X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="199921604" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 13:20:40 -0700 IronPort-SDR: B6O2zx/sWz/6z03jUjwrUDxo3gUUbtiHGVsZFkRbotPsDE0F3uA3phjEV73PJQxCX/pPBuNrLR LfxMXQYQnQmA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,300,1613462400"; d="scan'208";a="438147206" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 14 May 2021 13:20:40 -0700 From: "Chang S. Bae" To: tglx@linutronix.de, mingo@kernel.org, bp@suse.de, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: dan.j.williams@intel.com, dave.hansen@intel.com, ravi.v.shankar@intel.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, Mark Brown , linux-doc@vger.kernel.org Subject: [RFC PATCH v2 11/11] x86/cpu: Support the hardware randomization option for Key Locker internal key Date: Fri, 14 May 2021 13:15:08 -0700 Message-Id: <20210514201508.27967-12-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514201508.27967-1-chang.seok.bae@intel.com> References: <20210514201508.27967-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Hardware can load the internal key with randomization. The random.trust_cpu parameter determines the use of the CPU's random number generator. Take it to use the CPU's internal key randomization. The feature's backup mechanism is required to distribute an internal key. This is the only way to copy it to other CPUs if using hardware randomization. This option is disabled when hardware does not support the key backup. Signed-off-by: Chang S. Bae Cc: Mark Brown Cc: x86@kernel.org Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- arch/x86/kernel/keylocker.c | 37 ++++++++++++++++++++++++++++++++----- drivers/char/random.c | 6 ++++++ include/linux/random.h | 2 ++ 3 files changed, 40 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c index 0f60350944fa..5a784492195b 100644 --- a/arch/x86/kernel/keylocker.c +++ b/arch/x86/kernel/keylocker.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -15,27 +16,34 @@ #include static bool keybackup_available; +static bool keyhwrand_available; /* Internal (Wrapping) Key size fits in three 128-bit registers. */ #define KEYSIZE_128BIT 3 static struct _keydata { bool valid; + bool hwrand; struct reg_128_bit value[KEYSIZE_128BIT]; } keydata; /** * make_keylocker_data() - Generate the internal key. + * @use_hwrand: True if use hardware randomization; otherwise, false. * * Return: Nothing */ -static void make_keylocker_data(void) +static void make_keylocker_data(bool use_hwrand) { int i; for (i = 0; i < KEYSIZE_128BIT; i++) get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit)); + keydata.hwrand = (use_hwrand && keyhwrand_available && keybackup_available); + if (use_hwrand && !keydata.hwrand) + pr_warn("x86/keylocker: hardware random key is not fully supported\n"); + keydata.valid = true; } @@ -59,6 +67,8 @@ void flush_keylocker_data(void) } #define KEYSRC_SWRAND 0 +#define KEYSRC_HWRAND BIT(1) +#define KEYSRC_HWRAND_RETRY 10 /** * load_keylocker() - Load the internal key. @@ -68,8 +78,16 @@ void flush_keylocker_data(void) static int load_keylocker(void) { struct reg_128_bit zeros = { 0 }; - u32 keysrc = KEYSRC_SWRAND; - int err; + int retry, err; + u32 keysrc; + + if (keydata.hwrand) { + keysrc = KEYSRC_HWRAND; + retry = KEYSRC_HWRAND_RETRY; + } else { + keysrc = KEYSRC_SWRAND; + retry = 0; + } kernel_fpu_begin(); @@ -78,13 +96,19 @@ static int load_keylocker(void) "m"(keydata.value[1]), "m"(keydata.value[2])); - err = loadiwkey(keysrc); + do { + err = loadiwkey(keysrc); + retry--; + } while (err && retry >= 0); asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;" :: "m"(zeros)); kernel_fpu_end(); + if (keydata.hwrand) + flush_keylocker_data(); + return err; } @@ -138,6 +162,7 @@ void setup_keylocker(struct cpuinfo_x86 *c) cr4_set_bits(X86_CR4_KEYLOCKER); if (c == &boot_cpu_data) { + bool use_hwrand = check_random_trust_cpu(); u32 eax, ebx, ecx, edx; cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); @@ -156,7 +181,9 @@ void setup_keylocker(struct cpuinfo_x86 *c) goto disable; } - make_keylocker_data(); + keyhwrand_available = (ecx & KEYLOCKER_CPUID_ECX_RAND); + + make_keylocker_data(use_hwrand); err = load_keylocker(); if (err) { diff --git a/drivers/char/random.c b/drivers/char/random.c index 605969ed0f96..a3ece5968497 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -771,6 +771,12 @@ static int __init parse_trust_cpu(char *arg) } early_param("random.trust_cpu", parse_trust_cpu); +bool check_random_trust_cpu(void) +{ + return trust_cpu; +} +EXPORT_SYMBOL(check_random_trust_cpu); + static bool crng_init_try_arch(struct crng_state *crng) { int i; diff --git a/include/linux/random.h b/include/linux/random.h index f45b8be3e3c4..f08f44988b13 100644 --- a/include/linux/random.h +++ b/include/linux/random.h @@ -158,4 +158,6 @@ static inline bool __init arch_get_random_long_early(unsigned long *v) } #endif +extern bool check_random_trust_cpu(void); + #endif /* _LINUX_RANDOM_H */