From patchwork Sat Jul 1 15:17:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 9820913 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E54F360375 for ; Sat, 1 Jul 2017 15:18:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4536284AF for ; Sat, 1 Jul 2017 15:18:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C81B228520; Sat, 1 Jul 2017 15:18:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CB741284AF for ; Sat, 1 Jul 2017 15:18:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=y+/Ho9MvgZTCahm0nVCLpCue6ncpFlfWVTRFbXtr+JU=; b=f1q 4MZPWaNAw+2Za3QCsGHjxANf4c7AA4BELSg0zsxbjKyNTGKeFMUz9Wt9EqQ8yNzv6OIoDr8TyzYk+ +CKXPSzgZxxJtsg95y8AtK4OHpThQrbwXaL6XCILcDkYmEHArtRoAZaYIUPSLUUB2JgwJMhSwF8iN YYLF9Tn45Uhte3EygfZek4GivEtM3oXxt+saRBP1ZaQsDZvCDl5Q+GGB56u4kwtc+K50o/X/L7hm9 HUgkb1OybI7M5ulq3i0RxSXriUQTgGRcBYLdMjXLQBLSOmJzTlplskJcjfcxHFy7CEq9zMJM20vUE hMlKFhsJOeWV9Ri1FKU3BiZocfVIziA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dRKA4-0001nb-1F; Sat, 01 Jul 2017 15:18:16 +0000 Received: from mail-wm0-x22e.google.com ([2a00:1450:400c:c09::22e]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dRK9z-0001mZ-GV for linux-arm-kernel@lists.infradead.org; Sat, 01 Jul 2017 15:18:14 +0000 Received: by mail-wm0-x22e.google.com with SMTP id f67so17396755wmh.1 for ; Sat, 01 Jul 2017 08:17:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=g6r3LPv2pqJcclVQhno8nC5+JF0sHcxq8EA+IHyfYxw=; b=akYA9uGA39TgPKPEy8GWaQe/9IBXlWxd8fFkZ/x4+X00tT5lpBzN2Y6vs8UzYzHMDP pAu/sAdle+Ggei+NEawBYflH9YSNzDL1yzrgOPHHptF3CVsNlV7X0DinyYtOR1Vqpbqb hlOXaGik3GW+Cs0l+t3Ao8mOlbWGrNy8dRvus= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=g6r3LPv2pqJcclVQhno8nC5+JF0sHcxq8EA+IHyfYxw=; b=L5mFzuHvc4f60gqDCAu+QQwrrcgenvZN+cID9V6ZXePnyd5EDr7HjOMWhLWELquKdG 2RpTcRiau08+UhkxjmVKi/zXBn8kAQB58wPbpxnxnrDr4Jt02TvJGXAARpHyszxa5yWs vkwhFxvfbTxn49xsmSgA3NlXF1D7uIeN/SGr6/C5kzhhA5KjNdJ8pNFHkXq1bkMaJ3ot 9JQycTKeZGaQkKl/UDKqAyEH3trEjPqiJimkNx7yj2nJK1JU3GoowomJ9d26t0Y+q8Is K1DNas/wuyXABvP2nGCqnMuFIVc1HZRt/hqpdiSBmtpPOGWlRz4HfYGqw4CHGOaL39my Dvtg== X-Gm-Message-State: AKS2vOziROBelbZqKtC77lI6LuNrHisfvN7q0G0XA/uRf2SICHWSO2Yh W74y0pLQr02baQkoTL+fCg== X-Received: by 10.28.131.129 with SMTP id f123mr17292805wmd.7.1498922268416; Sat, 01 Jul 2017 08:17:48 -0700 (PDT) Received: from localhost.localdomain ([196.90.171.18]) by smtp.gmail.com with ESMTPSA id 63sm7872070wmi.8.2017.07.01.08.17.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 01 Jul 2017 08:17:47 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [RFT PATCH] crypto: arm/ghash - add NEON accelerated fallback for vmull.p64 Date: Sat, 1 Jul 2017 15:17:40 +0000 Message-Id: <20170701151740.9513-1-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.9.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170701_081812_023966_E4166DC3 X-CRM114-Status: GOOD ( 16.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: herbert@gondor.apana.org.au, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Implement a NEON fallback for systems that do support NEON but have no support for the optional 64x64->128 polynomial multiplication instruction that is part of the ARMv8 Crypto Extensions. It is based on the paper "Fast Software Polynomial Multiplication on ARM Processors Using the NEON Engine" by Danilo Camara, Conrado Gouvea, Julio Lopez and Ricardo Dahab (https://hal.inria.fr/hal-01506572) On a 32-bit guest executing under KVM on a Cortex-A57, the new code is not only >3x faster than the generic table based GHASH driver, it is also time invariant. (Note that the existing vmull.p64 code is 16x faster on this core). Signed-off-by: Ard Biesheuvel --- Raw numbers for a 2 GHz AMD Seattle (A57 r1p2) after the patch. arch/arm/crypto/Kconfig | 5 +- arch/arm/crypto/ghash-ce-core.S | 110 +++++++++++++++++--- arch/arm/crypto/ghash-ce-glue.c | 24 ++++- 3 files changed, 119 insertions(+), 20 deletions(-) diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index d8f3336bfc88..0b960ed124ae 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -106,14 +106,15 @@ config CRYPTO_AES_ARM_CE ARMv8 Crypto Extensions config CRYPTO_GHASH_ARM_CE - tristate "PMULL-accelerated GHASH using ARMv8 Crypto Extensions" + tristate "PMULL-accelerated GHASH using NEON/ARMv8 Crypto Extensions" depends on KERNEL_MODE_NEON select CRYPTO_HASH select CRYPTO_CRYPTD help Use an implementation of GHASH (used by the GCM AEAD chaining mode) that uses the 64x64 to 128 bit polynomial multiplication (vmull.p64) - that is part of the ARMv8 Crypto Extensions + that is part of the ARMv8 Crypto Extensions, or a slower variant that + uses the vmull.p8 instruction that is part of the basic NEON ISA. config CRYPTO_CRCT10DIF_ARM_CE tristate "CRCT10DIF digest algorithm using PMULL instructions" diff --git a/arch/arm/crypto/ghash-ce-core.S b/arch/arm/crypto/ghash-ce-core.S index f6ab8bcc9efe..a017a9213f7e 100644 --- a/arch/arm/crypto/ghash-ce-core.S +++ b/arch/arm/crypto/ghash-ce-core.S @@ -1,7 +1,7 @@ /* - * Accelerated GHASH implementation with ARMv8 vmull.p64 instructions. + * Accelerated GHASH implementation with NEON/ARMv8 vmull.p8/64 instructions. * - * Copyright (C) 2015 Linaro Ltd. + * Copyright (C) 2015 - 2017 Linaro Ltd. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License version 2 as published @@ -20,26 +20,90 @@ XM .req q6 XH .req q7 IN1 .req q7 + T3 .req q8 SHASH_L .req d0 SHASH_H .req d1 SHASH2_L .req d2 T1_L .req d4 + T2_L .req d6 MASK_L .req d8 XL_L .req d10 XL_H .req d11 XM_L .req d12 XM_H .req d13 XH_L .req d14 + T3_L .req d16 + + k16 .req d21 + k32 .req d22 + k48 .req d23 + + t0l .req d24 + t0h .req d25 + t1l .req d26 + t1h .req d27 + t2l .req d28 + t2h .req d29 + t3l .req d30 + t3h .req d31 + + t0q .req q12 + t1q .req q13 + t2q .req q14 + t3q .req q15 .text .fpu crypto-neon-fp-armv8 /* - * void pmull_ghash_update(int blocks, u64 dg[], const char *src, - * struct ghash_key const *k, const char *head) + * This implementation of 64x64 -> 128 bit polynomial multiplication + * using vmull.p8 instructions (8x8 -> 16) is taken from the paper + * "Fast Software Polynomial Multiplication on ARM Processors Using + * the NEON Engine" by Danilo Camara, Conrado Gouvea, Julio Lopez and + * Ricardo Dahab (https://hal.inria.fr/hal-01506572) */ -ENTRY(pmull_ghash_update) + .macro vmull_p64, rq, ad, bd + vext.8 t0l, \ad, \ad, #1 @ A1 + vmull.p8 t0q, t0l, \bd @ F = A1*B + vext.8 \rq\()_L, \bd, \bd, #1 @ B1 + vmull.p8 \rq, \ad, \rq\()_L @ E = A*B1 + vext.8 t1l, \ad, \ad, #2 @ A2 + vmull.p8 t1q, t1l, \bd @ H = A2*B + vext.8 t3l, \bd, \bd, #2 @ B2 + vmull.p8 t3q, \ad, t3l @ G = A*B2 + vext.8 t2l, \ad, \ad, #3 @ A3 + vmull.p8 t2q, t2l, \bd @ J = A3*B + veor t0q, t0q, \rq @ L = E + F + vext.8 \rq\()_L, \bd, \bd, #3 @ B3 + vmull.p8 \rq, \ad, \rq\()_L @ I = A*B3 + veor t1q, t1q, t3q @ M = G + H + vext.8 t3l, \bd, \bd, #4 @ B4 + vmull.p8 t3q, \ad, t3l @ K = A*B4 + veor t0l, t0l, t0h @ t0 = (L) (P0 + P1) << 8 + vand t0h, t0h, k48 + veor t1l, t1l, t1h @ t1 = (M) (P2 + P3) << 16 + vand t1h, t1h, k32 + veor t2q, t2q, \rq @ N = I + J + veor t0l, t0l, t0h + veor t1l, t1l, t1h + veor t2l, t2l, t2h @ t2 = (N) (P4 + P5) << 24 + vand t2h, t2h, k16 + veor t3l, t3l, t3h @ t3 = (K) (P6 + P7) << 32 + vmov.i64 t3h, #0 + vext.8 t0q, t0q, t0q, #15 + veor t2l, t2l, t2h + vext.8 t1q, t1q, t1q, #14 + vmull.p8 \rq, \ad, \bd @ D = A*B + vext.8 t2q, t2q, t2q, #13 + vext.8 t3q, t3q, t3q, #12 + veor t0q, t0q, t1q + veor t2q, t2q, t3q + veor \rq, \rq, t0q + veor \rq, \rq, t2q + .endm + + .macro ghash_update, vp64 vld1.64 {SHASH}, [r3] vld1.64 {XL}, [r1] vmov.i8 MASK, #0xe1 @@ -67,28 +131,44 @@ ENTRY(pmull_ghash_update) veor T1, T1, T2 veor XL, XL, IN1 - vmull.p64 XH, SHASH_H, XL_H @ a1 * b1 + \vp64 XH, SHASH_H, XL_H @ a1 * b1 veor T1, T1, XL - vmull.p64 XL, SHASH_L, XL_L @ a0 * b0 - vmull.p64 XM, SHASH2_L, T1_L @ (a1 + a0)(b1 + b0) + \vp64 T3, SHASH_L, XL_L @ a0 * b0 + \vp64 XM, SHASH2_L, T1_L @ (a1 + a0)(b1 + b0) - vext.8 T1, XL, XH, #8 - veor T2, XL, XH + vext.8 T1, T3, XH, #8 + veor T2, T3, XH veor XM, XM, T1 veor XM, XM, T2 - vmull.p64 T2, XL_L, MASK_L + \vp64 T2, T3_L, MASK_L vmov XH_L, XM_H - vmov XM_H, XL_L + vmov XM_H, T3_L veor XL, XM, T2 vext.8 T2, XL, XL, #8 - vmull.p64 XL, XL_L, MASK_L + \vp64 T1, XL_L, MASK_L veor T2, T2, XH - veor XL, XL, T2 + veor XL, T1, T2 bne 0b vst1.64 {XL}, [r1] bx lr -ENDPROC(pmull_ghash_update) + .endm + + /* + * void pmull_ghash_update(int blocks, u64 dg[], const char *src, + * struct ghash_key const *k, const char *head) + */ +ENTRY(pmull_ghash_update_p64) + ghash_update vmull.p64 +ENDPROC(pmull_ghash_update_p64) + +ENTRY(pmull_ghash_update_p8) + vmov.i64 k16, #0xffff + vmov.i64 k32, #0xffffffff + vmov.i64 k48, #0xffffffffffff + + ghash_update vmull_p64 +ENDPROC(pmull_ghash_update_p8) diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c index 6bac8bea9f1e..d9bb52cae2ac 100644 --- a/arch/arm/crypto/ghash-ce-glue.c +++ b/arch/arm/crypto/ghash-ce-glue.c @@ -22,6 +22,7 @@ MODULE_DESCRIPTION("GHASH secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); +MODULE_ALIAS_CRYPTO("ghash"); #define GHASH_BLOCK_SIZE 16 #define GHASH_DIGEST_SIZE 16 @@ -41,8 +42,17 @@ struct ghash_async_ctx { struct cryptd_ahash *cryptd_tfm; }; -asmlinkage void pmull_ghash_update(int blocks, u64 dg[], const char *src, - struct ghash_key const *k, const char *head); +asmlinkage void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src, + struct ghash_key const *k, + const char *head); + +asmlinkage void pmull_ghash_update_p8(int blocks, u64 dg[], const char *src, + struct ghash_key const *k, + const char *head); + +static void (*pmull_ghash_update)(int blocks, u64 dg[], const char *src, + struct ghash_key const *k, + const char *head); static int ghash_init(struct shash_desc *desc) { @@ -312,6 +322,14 @@ static int __init ghash_ce_mod_init(void) { int err; + if (!(elf_hwcap & HWCAP_NEON)) + return -ENODEV; + + if (elf_hwcap2 & HWCAP2_PMULL) + pmull_ghash_update = pmull_ghash_update_p64; + else + pmull_ghash_update = pmull_ghash_update_p8; + err = crypto_register_shash(&ghash_alg); if (err) return err; @@ -332,5 +350,5 @@ static void __exit ghash_ce_mod_exit(void) crypto_unregister_shash(&ghash_alg); } -module_cpu_feature_match(PMULL, ghash_ce_mod_init); +module_init(ghash_ce_mod_init); module_exit(ghash_ce_mod_exit);