From patchwork Wed Mar 13 05:12:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850603 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42A2A1850 for ; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 295B2299E0 for ; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0AFA5299E9; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95060299D2 for ; Wed, 13 Mar 2019 05:15:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726184AbfCMFPf (ORCPT ); Wed, 13 Mar 2019 01:15:35 -0400 Received: from mail.kernel.org ([198.145.29.99]:50104 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726043AbfCMFPf (ORCPT ); Wed, 13 Mar 2019 01:15:35 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 464F82183E; Wed, 13 Mar 2019 05:15:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454134; bh=vHRu0W9KwUGaloXhGjaFpSkQdr99+L7HhU0sp5EPj+s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qU1LRDFUW97TFZ1BeNbXj7hLZ9eyZr7HZRg6/2fXtXeYvSmqalfxeUwvQchaDg8CP QE0cxQGVVLQUvjAzptDVmZwrW9dpBZj6Xt+rYEu5CIhleXDDBoKZCwKF0f6IOQmms+ V6Sd/9yMLMJ1ByU/RfNLGZs1HfSDN9e2/5jvLomc= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback Date: Tue, 12 Mar 2019 22:12:45 -0700 Message-Id: <20190313051252.2917-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers The arm64 implementations of ChaCha and XChaCha are failing the extra crypto self-tests following my patches to test the !may_use_simd() code paths, which previously were untested. The problem is as follows: When !may_use_simd(), the arm64 NEON implementations fall back to the generic implementation, which uses the skcipher_walk API to iterate through the src/dst scatterlists. Due to how the skcipher_walk API works, walk.stride is set from the skcipher_alg actually being used, which in this case is the arm64 NEON algorithm. Thus walk.stride is 5*CHACHA_BLOCK_SIZE, not CHACHA_BLOCK_SIZE. This unnecessarily large stride shouldn't cause an actual problem. However, the generic implementation computes round_down(nbytes, walk.stride). round_down() assumes the round amount is a power of 2, which 5*CHACHA_BLOCK_SIZE is not, so it gives the wrong result. This causes the following case in skcipher_walk_done() to be hit, causing a WARN() and failing the encryption operation: if (WARN_ON(err)) { /* unexpected case; didn't process all bytes */ err = -EINVAL; goto finish; } Fix it by rounding down to CHACHA_BLOCK_SIZE instead of walk.stride. (Or we could replace round_down() with rounddown(), but that would add a slow division operation every time, which I think we should avoid.) Fixes: 2fe55987b262 ("crypto: arm64/chacha - use combined SIMD/ALU routine for more speed") Cc: # v5.0+ Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- crypto/chacha_generic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crypto/chacha_generic.c b/crypto/chacha_generic.c index 35b583101f4f..90ec0ec1b4f7 100644 --- a/crypto/chacha_generic.c +++ b/crypto/chacha_generic.c @@ -52,7 +52,7 @@ static int chacha_stream_xor(struct skcipher_request *req, unsigned int nbytes = walk.nbytes; if (nbytes < walk.total) - nbytes = round_down(nbytes, walk.stride); + nbytes = round_down(nbytes, CHACHA_BLOCK_SIZE); chacha_docrypt(state, walk.dst.virt.addr, walk.src.virt.addr, nbytes, ctx->nrounds); From patchwork Wed Mar 13 05:12:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850601 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B5EF17DF for ; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0146299E1 for ; Wed, 13 Mar 2019 05:15:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9BEB299E0; Wed, 13 Mar 2019 05:15:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74933299CC for ; Wed, 13 Mar 2019 05:15:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726043AbfCMFPf (ORCPT ); Wed, 13 Mar 2019 01:15:35 -0400 Received: from mail.kernel.org ([198.145.29.99]:50136 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726097AbfCMFPf (ORCPT ); Wed, 13 Mar 2019 01:15:35 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 952B921841; Wed, 13 Mar 2019 05:15:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454134; bh=dADKUnSVG3rfqdhSW0LWNdj60ivQO/IRlbfO5L21RY8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jFX/QrIy5NcnQ2YGiJB4Zn3EQ92zGzBjvghYw4FuTvUMK0CULT59Hv0HoimUW0N2f +qHBN/Hvtmu67MWrBBdDQtergqC8ZB5F6xTOcv9ch8EwP5hxHFAMt6GAlA/7pFXcCA ZFIkbKbRAgMIt3Xaq/GrvaxmWASpDu/RFyjGZle4= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code Date: Tue, 12 Mar 2019 22:12:46 -0700 Message-Id: <20190313051252.2917-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers The arm64 gcm-aes-ce algorithm is failing the extra crypto self-tests following my patches to test the !may_use_simd() code paths, which previously were untested. The problem is that in the !may_use_simd() case, an odd number of AES blocks can be processed within each step of the skcipher_walk. However, the skcipher_walk is being done with a "stride" of 2 blocks and is advanced by an even number of blocks after each step. This causes the encryption to produce the wrong ciphertext and authentication tag, and causes the decryption to incorrectly fail. Fix it by only processing an even number of blocks per step. Fixes: c2b24c36e0a3 ("crypto: arm64/aes-gcm-ce - fix scatterwalk API violation") Fixes: 71e52c278c54 ("crypto: arm64/aes-ce-gcm - operate on two input blocks at a time") Cc: # v4.19+ Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- arch/arm64/crypto/ghash-ce-glue.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index 791ad422c427..089b09286da7 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -473,9 +473,11 @@ static int gcm_encrypt(struct aead_request *req) put_unaligned_be32(2, iv + GCM_IV_SIZE); while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) { - int blocks = walk.nbytes / AES_BLOCK_SIZE; + const int blocks = + walk.nbytes / (2 * AES_BLOCK_SIZE) * 2; u8 *dst = walk.dst.virt.addr; u8 *src = walk.src.virt.addr; + int remaining = blocks; do { __aes_arm64_encrypt(ctx->aes_key.key_enc, @@ -485,9 +487,9 @@ static int gcm_encrypt(struct aead_request *req) dst += AES_BLOCK_SIZE; src += AES_BLOCK_SIZE; - } while (--blocks > 0); + } while (--remaining > 0); - ghash_do_update(walk.nbytes / AES_BLOCK_SIZE, dg, + ghash_do_update(blocks, dg, walk.dst.virt.addr, &ctx->ghash_key, NULL, pmull_ghash_update_p64); @@ -609,7 +611,7 @@ static int gcm_decrypt(struct aead_request *req) put_unaligned_be32(2, iv + GCM_IV_SIZE); while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) { - int blocks = walk.nbytes / AES_BLOCK_SIZE; + int blocks = walk.nbytes / (2 * AES_BLOCK_SIZE) * 2; u8 *dst = walk.dst.virt.addr; u8 *src = walk.src.virt.addr; From patchwork Wed Mar 13 05:12:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850611 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0DFA01575 for ; Wed, 13 Mar 2019 05:15:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EEF48299CC for ; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E3B27299D6; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DFBC5299CC for ; Wed, 13 Mar 2019 05:15:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726528AbfCMFPg (ORCPT ); Wed, 13 Mar 2019 01:15:36 -0400 Received: from mail.kernel.org ([198.145.29.99]:50154 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726183AbfCMFPg (ORCPT ); Wed, 13 Mar 2019 01:15:36 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E5B402184D; Wed, 13 Mar 2019 05:15:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454135; bh=lzkvbYwAbu6mfSnM6YOnS1aqXWrT8fyKTWOJhCF764w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VJhFhqKzwsbnjcwlw6WhjSo+O9KOE/k6i3AOMjOimKnZyt2MJaUKeV8aiF9n645SQ yaIgugpVpbQtqWHhF6gEoE8jBbF3QDOeFyod7wg547F8yUuvIsieNfu55RItVfdmwh DKMDeK//xTUVHS08VKDzPy930GblsERB9h21PcEk= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 3/8] crypto: simd,testmgr - introduce crypto_simd_usable() Date: Tue, 12 Mar 2019 22:12:47 -0700 Message-Id: <20190313051252.2917-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers So that the no-SIMD fallback code can be tested by the crypto self-tests, add a macro crypto_simd_usable() which wraps may_use_simd(), but also returns false if the crypto self-tests have set a per-CPU bool to disable SIMD in crypto code on the current CPU. Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- crypto/testmgr.c | 26 +++++++++++++++++++++++++- include/crypto/internal/simd.h | 24 ++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/crypto/testmgr.c b/crypto/testmgr.c index 5d56b2990762..52417dde811f 100644 --- a/crypto/testmgr.c +++ b/crypto/testmgr.c @@ -37,6 +37,7 @@ #include #include #include +#include #include "internal.h" @@ -52,6 +53,9 @@ MODULE_PARM_DESC(noextratests, "disable expensive crypto self-tests"); static unsigned int fuzz_iterations = 100; module_param(fuzz_iterations, uint, 0644); MODULE_PARM_DESC(fuzz_iterations, "number of fuzz test iterations"); + +DEFINE_PER_CPU(bool, crypto_simd_disabled_for_test); +EXPORT_PER_CPU_SYMBOL_GPL(crypto_simd_disabled_for_test); #endif #ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS @@ -838,7 +842,27 @@ static void generate_random_testvec_config(struct testvec_config *cfg, WARN_ON_ONCE(!valid_testvec_config(cfg)); } -#endif /* CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ + +static void crypto_disable_simd_for_test(void) +{ + preempt_disable(); + __this_cpu_write(crypto_simd_disabled_for_test, true); +} + +static void crypto_reenable_simd_for_test(void) +{ + __this_cpu_write(crypto_simd_disabled_for_test, false); + preempt_enable(); +} +#else /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ +static void crypto_disable_simd_for_test(void) +{ +} + +static void crypto_reenable_simd_for_test(void) +{ +} +#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ static int check_nonfinal_hash_op(const char *op, int err, u8 *result, unsigned int digestsize, diff --git a/include/crypto/internal/simd.h b/include/crypto/internal/simd.h index a23b18b6ad61..d2316242a988 100644 --- a/include/crypto/internal/simd.h +++ b/include/crypto/internal/simd.h @@ -6,6 +6,9 @@ #ifndef _CRYPTO_INTERNAL_SIMD_H #define _CRYPTO_INTERNAL_SIMD_H +#include +#include + /* skcipher support */ struct simd_skcipher_alg; @@ -42,4 +45,25 @@ int simd_register_aeads_compat(struct aead_alg *algs, int count, void simd_unregister_aeads(struct aead_alg *algs, int count, struct simd_aead_alg **simd_algs); +/* + * crypto_simd_usable() - is it allowed at this time to use SIMD instructions or + * access the SIMD register file? + * + * This delegates to may_use_simd(), except that this also returns false if SIMD + * in crypto code has been temporarily disabled on this CPU by the crypto + * self-tests, in order to test the no-SIMD fallback code. This override is + * currently limited to configurations where the extra self-tests are enabled, + * because it might be a bit too invasive to be part of the regular self-tests. + * + * This is a macro so that , which some architectures don't have, + * doesn't have to be included directly here. + */ +#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS +DECLARE_PER_CPU(bool, crypto_simd_disabled_for_test); +#define crypto_simd_usable() \ + (may_use_simd() && !this_cpu_read(crypto_simd_disabled_for_test)) +#else +#define crypto_simd_usable() may_use_simd() +#endif + #endif /* _CRYPTO_INTERNAL_SIMD_H */ From patchwork Wed Mar 13 05:12:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850609 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 464591575 for ; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D65F299D2 for ; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 21703299D6; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E741299D2 for ; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726544AbfCMFPh (ORCPT ); Wed, 13 Mar 2019 01:15:37 -0400 Received: from mail.kernel.org ([198.145.29.99]:50178 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726097AbfCMFPg (ORCPT ); Wed, 13 Mar 2019 01:15:36 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 42DDC21850; Wed, 13 Mar 2019 05:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454135; bh=YAi4QJd2zHGKbIpLPQRQWJs4ObvfD20mdrtgQquHyRM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xHPBgwARpnCS+OR84p4JB5XH3d9u9GcvEfLwCcljdH241HIbQh6cauYM9ebYCUQlq 2a7DQGKNqgBeXi/Ly4TGLCWGqY9FC3zUo5i6A+FFLuQV3y4WYksiaB0bR9p14PvwCx h58g8qLXDdc089FEPaI4ES2eIuaVIDyQ4OFluEGQ= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable() Date: Tue, 12 Mar 2019 22:12:48 -0700 Message-Id: <20190313051252.2917-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers Replace all calls to irq_fpu_usable() in the x86 crypto code with crypto_simd_usable(), in order to allow testing the no-SIMD code paths. Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- arch/x86/crypto/aesni-intel_glue.c | 8 ++++---- arch/x86/crypto/chacha_glue.c | 6 +++--- arch/x86/crypto/crc32-pclmul_glue.c | 5 +++-- arch/x86/crypto/crc32c-intel_glue.c | 7 ++++--- arch/x86/crypto/crct10dif-pclmul_glue.c | 7 ++++--- arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +++++---- arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +++-- arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +++-- arch/x86/crypto/poly1305_glue.c | 4 ++-- arch/x86/crypto/sha1_ssse3_glue.c | 7 ++++--- arch/x86/crypto/sha256_ssse3_glue.c | 7 ++++--- arch/x86/crypto/sha512_ssse3_glue.c | 10 +++++----- 12 files changed, 44 insertions(+), 36 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 1466d59384ac..21c246799aa5 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -30,8 +30,8 @@ #include #include #include -#include #include +#include #include #include #include @@ -332,7 +332,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, return -EINVAL; } - if (!irq_fpu_usable()) + if (!crypto_simd_usable()) err = crypto_aes_expand_key(ctx, in_key, key_len); else { kernel_fpu_begin(); @@ -353,7 +353,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) { struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm)); - if (!irq_fpu_usable()) + if (!crypto_simd_usable()) crypto_aes_encrypt_x86(ctx, dst, src); else { kernel_fpu_begin(); @@ -366,7 +366,7 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) { struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm)); - if (!irq_fpu_usable()) + if (!crypto_simd_usable()) crypto_aes_decrypt_x86(ctx, dst, src); else { kernel_fpu_begin(); diff --git a/arch/x86/crypto/chacha_glue.c b/arch/x86/crypto/chacha_glue.c index 45c1c4143176..4967ad620775 100644 --- a/arch/x86/crypto/chacha_glue.c +++ b/arch/x86/crypto/chacha_glue.c @@ -12,10 +12,10 @@ #include #include +#include #include #include #include -#include #include #define CHACHA_STATE_ALIGN 16 @@ -170,7 +170,7 @@ static int chacha_simd(struct skcipher_request *req) struct skcipher_walk walk; int err; - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !irq_fpu_usable()) + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) return crypto_chacha_crypt(req); err = skcipher_walk_virt(&walk, req, true); @@ -193,7 +193,7 @@ static int xchacha_simd(struct skcipher_request *req) u8 real_iv[16]; int err; - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !irq_fpu_usable()) + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) return crypto_xchacha_crypt(req); err = skcipher_walk_virt(&walk, req, true); diff --git a/arch/x86/crypto/crc32-pclmul_glue.c b/arch/x86/crypto/crc32-pclmul_glue.c index c8d9cdacbf10..cb4ab6645106 100644 --- a/arch/x86/crypto/crc32-pclmul_glue.c +++ b/arch/x86/crypto/crc32-pclmul_glue.c @@ -32,10 +32,11 @@ #include #include #include +#include #include #include -#include +#include #define CHKSUM_BLOCK_SIZE 1 #define CHKSUM_DIGEST_SIZE 4 @@ -54,7 +55,7 @@ static u32 __attribute__((pure)) unsigned int iremainder; unsigned int prealign; - if (len < PCLMUL_MIN_LEN + SCALE_F_MASK || !irq_fpu_usable()) + if (len < PCLMUL_MIN_LEN + SCALE_F_MASK || !crypto_simd_usable()) return crc32_le(crc, p, len); if ((long)p & SCALE_F_MASK) { diff --git a/arch/x86/crypto/crc32c-intel_glue.c b/arch/x86/crypto/crc32c-intel_glue.c index 5773e1161072..a58fe217c856 100644 --- a/arch/x86/crypto/crc32c-intel_glue.c +++ b/arch/x86/crypto/crc32c-intel_glue.c @@ -29,10 +29,11 @@ #include #include #include +#include #include #include -#include +#include #define CHKSUM_BLOCK_SIZE 1 #define CHKSUM_DIGEST_SIZE 4 @@ -177,7 +178,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data, * use faster PCL version if datasize is large enough to * overcome kernel fpu state save/restore overhead */ - if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) { + if (len >= CRC32C_PCL_BREAKEVEN && crypto_simd_usable()) { kernel_fpu_begin(); *crcp = crc_pcl(data, len, *crcp); kernel_fpu_end(); @@ -189,7 +190,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data, static int __crc32c_pcl_intel_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out) { - if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) { + if (len >= CRC32C_PCL_BREAKEVEN && crypto_simd_usable()) { kernel_fpu_begin(); *(__le32 *)out = ~cpu_to_le32(crc_pcl(data, len, *crcp)); kernel_fpu_end(); diff --git a/arch/x86/crypto/crct10dif-pclmul_glue.c b/arch/x86/crypto/crct10dif-pclmul_glue.c index 0e785c0b2354..64f17d2d8b67 100644 --- a/arch/x86/crypto/crct10dif-pclmul_glue.c +++ b/arch/x86/crypto/crct10dif-pclmul_glue.c @@ -26,12 +26,13 @@ #include #include #include +#include #include #include #include -#include #include #include +#include asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len); @@ -53,7 +54,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data, { struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - if (length >= 16 && irq_fpu_usable()) { + if (length >= 16 && crypto_simd_usable()) { kernel_fpu_begin(); ctx->crc = crc_t10dif_pcl(ctx->crc, data, length); kernel_fpu_end(); @@ -73,7 +74,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out) static int __chksum_finup(__u16 *crcp, const u8 *data, unsigned int len, u8 *out) { - if (len >= 16 && irq_fpu_usable()) { + if (len >= 16 && crypto_simd_usable()) { kernel_fpu_begin(); *(__u16 *)out = crc_t10dif_pcl(*crcp, data, len); kernel_fpu_end(); diff --git a/arch/x86/crypto/ghash-clmulni-intel_glue.c b/arch/x86/crypto/ghash-clmulni-intel_glue.c index 3582ae885ee1..4099a0ae17dd 100644 --- a/arch/x86/crypto/ghash-clmulni-intel_glue.c +++ b/arch/x86/crypto/ghash-clmulni-intel_glue.c @@ -19,8 +19,9 @@ #include #include #include -#include +#include #include +#include #define GHASH_BLOCK_SIZE 16 #define GHASH_DIGEST_SIZE 16 @@ -182,7 +183,7 @@ static int ghash_async_update(struct ahash_request *req) struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm); struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; - if (!irq_fpu_usable() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { memcpy(cryptd_req, req, sizeof(*req)); ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); @@ -200,7 +201,7 @@ static int ghash_async_final(struct ahash_request *req) struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm); struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; - if (!irq_fpu_usable() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { memcpy(cryptd_req, req, sizeof(*req)); ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); @@ -241,7 +242,7 @@ static int ghash_async_digest(struct ahash_request *req) struct ahash_request *cryptd_req = ahash_request_ctx(req); struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; - if (!irq_fpu_usable() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { memcpy(cryptd_req, req, sizeof(*req)); ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); diff --git a/arch/x86/crypto/nhpoly1305-avx2-glue.c b/arch/x86/crypto/nhpoly1305-avx2-glue.c index 20d815ea4b6a..f7567cbd35b6 100644 --- a/arch/x86/crypto/nhpoly1305-avx2-glue.c +++ b/arch/x86/crypto/nhpoly1305-avx2-glue.c @@ -7,9 +7,10 @@ */ #include +#include #include #include -#include +#include asmlinkage void nh_avx2(const u32 *key, const u8 *message, size_t message_len, u8 hash[NH_HASH_BYTES]); @@ -24,7 +25,7 @@ static void _nh_avx2(const u32 *key, const u8 *message, size_t message_len, static int nhpoly1305_avx2_update(struct shash_desc *desc, const u8 *src, unsigned int srclen) { - if (srclen < 64 || !irq_fpu_usable()) + if (srclen < 64 || !crypto_simd_usable()) return crypto_nhpoly1305_update(desc, src, srclen); do { diff --git a/arch/x86/crypto/nhpoly1305-sse2-glue.c b/arch/x86/crypto/nhpoly1305-sse2-glue.c index ed68d164ce14..a661ede3b5cf 100644 --- a/arch/x86/crypto/nhpoly1305-sse2-glue.c +++ b/arch/x86/crypto/nhpoly1305-sse2-glue.c @@ -7,9 +7,10 @@ */ #include +#include #include #include -#include +#include asmlinkage void nh_sse2(const u32 *key, const u8 *message, size_t message_len, u8 hash[NH_HASH_BYTES]); @@ -24,7 +25,7 @@ static void _nh_sse2(const u32 *key, const u8 *message, size_t message_len, static int nhpoly1305_sse2_update(struct shash_desc *desc, const u8 *src, unsigned int srclen) { - if (srclen < 64 || !irq_fpu_usable()) + if (srclen < 64 || !crypto_simd_usable()) return crypto_nhpoly1305_update(desc, src, srclen); do { diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c index 88cc01506c84..6eb65b237b3c 100644 --- a/arch/x86/crypto/poly1305_glue.c +++ b/arch/x86/crypto/poly1305_glue.c @@ -11,11 +11,11 @@ #include #include +#include #include #include #include #include -#include #include struct poly1305_simd_desc_ctx { @@ -126,7 +126,7 @@ static int poly1305_simd_update(struct shash_desc *desc, unsigned int bytes; /* kernel_fpu_begin/end is costly, use fallback for small updates */ - if (srclen <= 288 || !may_use_simd()) + if (srclen <= 288 || !crypto_simd_usable()) return crypto_poly1305_update(desc, src, srclen); kernel_fpu_begin(); diff --git a/arch/x86/crypto/sha1_ssse3_glue.c b/arch/x86/crypto/sha1_ssse3_glue.c index 7391c7de72c7..42f177afc33a 100644 --- a/arch/x86/crypto/sha1_ssse3_glue.c +++ b/arch/x86/crypto/sha1_ssse3_glue.c @@ -22,6 +22,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include #include #include #include @@ -29,7 +30,7 @@ #include #include #include -#include +#include typedef void (sha1_transform_fn)(u32 *digest, const char *data, unsigned int rounds); @@ -39,7 +40,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, { struct sha1_state *sctx = shash_desc_ctx(desc); - if (!irq_fpu_usable() || + if (!crypto_simd_usable() || (sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE) return crypto_sha1_update(desc, data, len); @@ -57,7 +58,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, static int sha1_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out, sha1_transform_fn *sha1_xform) { - if (!irq_fpu_usable()) + if (!crypto_simd_usable()) return crypto_sha1_finup(desc, data, len, out); kernel_fpu_begin(); diff --git a/arch/x86/crypto/sha256_ssse3_glue.c b/arch/x86/crypto/sha256_ssse3_glue.c index 773a873d2b28..73867da3cbee 100644 --- a/arch/x86/crypto/sha256_ssse3_glue.c +++ b/arch/x86/crypto/sha256_ssse3_glue.c @@ -30,6 +30,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include #include #include #include @@ -37,8 +38,8 @@ #include #include #include -#include #include +#include asmlinkage void sha256_transform_ssse3(u32 *digest, const char *data, u64 rounds); @@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, { struct sha256_state *sctx = shash_desc_ctx(desc); - if (!irq_fpu_usable() || + if (!crypto_simd_usable() || (sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE) return crypto_sha256_update(desc, data, len); @@ -67,7 +68,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, static int sha256_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out, sha256_transform_fn *sha256_xform) { - if (!irq_fpu_usable()) + if (!crypto_simd_usable()) return crypto_sha256_finup(desc, data, len, out); kernel_fpu_begin(); diff --git a/arch/x86/crypto/sha512_ssse3_glue.c b/arch/x86/crypto/sha512_ssse3_glue.c index f1b811b60ba6..458356a3f124 100644 --- a/arch/x86/crypto/sha512_ssse3_glue.c +++ b/arch/x86/crypto/sha512_ssse3_glue.c @@ -28,16 +28,16 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include #include #include #include #include +#include #include #include #include -#include - -#include +#include asmlinkage void sha512_transform_ssse3(u64 *digest, const char *data, u64 rounds); @@ -49,7 +49,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data, { struct sha512_state *sctx = shash_desc_ctx(desc); - if (!irq_fpu_usable() || + if (!crypto_simd_usable() || (sctx->count[0] % SHA512_BLOCK_SIZE) + len < SHA512_BLOCK_SIZE) return crypto_sha512_update(desc, data, len); @@ -67,7 +67,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data, static int sha512_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out, sha512_transform_fn *sha512_xform) { - if (!irq_fpu_usable()) + if (!crypto_simd_usable()) return crypto_sha512_finup(desc, data, len, out); kernel_fpu_begin(); From patchwork Wed Mar 13 05:12:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850613 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68B461850 for ; Wed, 13 Mar 2019 05:15:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50FE7299D2 for ; Wed, 13 Mar 2019 05:15:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 45378299D6; Wed, 13 Mar 2019 05:15:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B303299D2 for ; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726183AbfCMFPh (ORCPT ); Wed, 13 Mar 2019 01:15:37 -0400 Received: from mail.kernel.org ([198.145.29.99]:50212 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726326AbfCMFPh (ORCPT ); Wed, 13 Mar 2019 01:15:37 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9DF0621855; Wed, 13 Mar 2019 05:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454135; bh=yr5n0Wxdm1XUCzFgkf3PqRgYoApvSKI9VmTjgjcFNO4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e3INadTDTroCXX099wNTLjHcQqucz95Met5vuINgtdgHwchN0JVBQkSv6oGjvKSxr AnWDvdB5VQxXBC0J3S+teMnuqkdF/x4Yl2y3SpPorzzgmX3qj6ayCRS6Bz5FpZD2ND kjqUeWhflk3uip4rCk/AET8eJoXC4ZMMSGdlmM58= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 5/8] crypto: arm - convert to use crypto_simd_usable() Date: Tue, 12 Mar 2019 22:12:49 -0700 Message-Id: <20190313051252.2917-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers Replace all calls to may_use_simd() in the arm crypto code with crypto_simd_usable(), in order to allow testing the no-SIMD code paths. Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- arch/arm/crypto/chacha-neon-glue.c | 5 +++-- arch/arm/crypto/crc32-ce-glue.c | 5 +++-- arch/arm/crypto/crct10dif-ce-glue.c | 3 ++- arch/arm/crypto/ghash-ce-glue.c | 7 ++++--- arch/arm/crypto/nhpoly1305-neon-glue.c | 3 ++- arch/arm/crypto/sha1-ce-glue.c | 5 +++-- arch/arm/crypto/sha1_neon_glue.c | 5 +++-- arch/arm/crypto/sha2-ce-glue.c | 5 +++-- arch/arm/crypto/sha256_neon_glue.c | 5 +++-- arch/arm/crypto/sha512-neon-glue.c | 5 +++-- 10 files changed, 29 insertions(+), 19 deletions(-) diff --git a/arch/arm/crypto/chacha-neon-glue.c b/arch/arm/crypto/chacha-neon-glue.c index 9d6fda81986d..48a89537b828 100644 --- a/arch/arm/crypto/chacha-neon-glue.c +++ b/arch/arm/crypto/chacha-neon-glue.c @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -93,7 +94,7 @@ static int chacha_neon(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm); - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd()) + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) return crypto_chacha_crypt(req); return chacha_neon_stream_xor(req, ctx, req->iv); @@ -107,7 +108,7 @@ static int xchacha_neon(struct skcipher_request *req) u32 state[16]; u8 real_iv[16]; - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd()) + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) return crypto_xchacha_crypt(req); crypto_chacha_init(state, ctx, req->iv); diff --git a/arch/arm/crypto/crc32-ce-glue.c b/arch/arm/crypto/crc32-ce-glue.c index cd9e93b46c2d..e712c2a7d387 100644 --- a/arch/arm/crypto/crc32-ce-glue.c +++ b/arch/arm/crypto/crc32-ce-glue.c @@ -16,6 +16,7 @@ #include #include +#include #include #include @@ -113,7 +114,7 @@ static int crc32_pmull_update(struct shash_desc *desc, const u8 *data, u32 *crc = shash_desc_ctx(desc); unsigned int l; - if (may_use_simd()) { + if (crypto_simd_usable()) { if ((u32)data % SCALE_F) { l = min_t(u32, length, SCALE_F - ((u32)data % SCALE_F)); @@ -147,7 +148,7 @@ static int crc32c_pmull_update(struct shash_desc *desc, const u8 *data, u32 *crc = shash_desc_ctx(desc); unsigned int l; - if (may_use_simd()) { + if (crypto_simd_usable()) { if ((u32)data % SCALE_F) { l = min_t(u32, length, SCALE_F - ((u32)data % SCALE_F)); diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c index 3d6b800b8396..3b24f2872592 100644 --- a/arch/arm/crypto/crct10dif-ce-glue.c +++ b/arch/arm/crypto/crct10dif-ce-glue.c @@ -15,6 +15,7 @@ #include #include +#include #include #include @@ -36,7 +37,7 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, { u16 *crc = shash_desc_ctx(desc); - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { kernel_neon_begin(); *crc = crc_t10dif_pmull(*crc, data, length); kernel_neon_end(); diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c index b7d30b6cf49c..60123e9ea9d8 100644 --- a/arch/arm/crypto/ghash-ce-glue.c +++ b/arch/arm/crypto/ghash-ce-glue.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -196,7 +197,7 @@ static int ghash_async_update(struct ahash_request *req) struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm); struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { memcpy(cryptd_req, req, sizeof(*req)); ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); @@ -214,7 +215,7 @@ static int ghash_async_final(struct ahash_request *req) struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm); struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { memcpy(cryptd_req, req, sizeof(*req)); ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); @@ -232,7 +233,7 @@ static int ghash_async_digest(struct ahash_request *req) struct ahash_request *cryptd_req = ahash_request_ctx(req); struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { memcpy(cryptd_req, req, sizeof(*req)); ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); diff --git a/arch/arm/crypto/nhpoly1305-neon-glue.c b/arch/arm/crypto/nhpoly1305-neon-glue.c index 49aae87cb2bc..ae5aefc44a4d 100644 --- a/arch/arm/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm/crypto/nhpoly1305-neon-glue.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len, static int nhpoly1305_neon_update(struct shash_desc *desc, const u8 *src, unsigned int srclen) { - if (srclen < 64 || !may_use_simd()) + if (srclen < 64 || !crypto_simd_usable()) return crypto_nhpoly1305_update(desc, src, srclen); do { diff --git a/arch/arm/crypto/sha1-ce-glue.c b/arch/arm/crypto/sha1-ce-glue.c index b732522e20f8..4c6c6900853c 100644 --- a/arch/arm/crypto/sha1-ce-glue.c +++ b/arch/arm/crypto/sha1-ce-glue.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -33,7 +34,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, { struct sha1_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd() || + if (!crypto_simd_usable() || (sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE) return sha1_update_arm(desc, data, len); @@ -47,7 +48,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, static int sha1_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return sha1_finup_arm(desc, data, len, out); kernel_neon_begin(); diff --git a/arch/arm/crypto/sha1_neon_glue.c b/arch/arm/crypto/sha1_neon_glue.c index d15e0ea2c95e..d6c95c213d42 100644 --- a/arch/arm/crypto/sha1_neon_glue.c +++ b/arch/arm/crypto/sha1_neon_glue.c @@ -19,6 +19,7 @@ */ #include +#include #include #include #include @@ -39,7 +40,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data, { struct sha1_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd() || + if (!crypto_simd_usable() || (sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE) return sha1_update_arm(desc, data, len); @@ -54,7 +55,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data, static int sha1_neon_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return sha1_finup_arm(desc, data, len, out); kernel_neon_begin(); diff --git a/arch/arm/crypto/sha2-ce-glue.c b/arch/arm/crypto/sha2-ce-glue.c index 1211a5c129fc..a47a9d4b663e 100644 --- a/arch/arm/crypto/sha2-ce-glue.c +++ b/arch/arm/crypto/sha2-ce-glue.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -34,7 +35,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data, { struct sha256_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd() || + if (!crypto_simd_usable() || (sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE) return crypto_sha256_arm_update(desc, data, len); @@ -49,7 +50,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data, static int sha2_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha256_arm_finup(desc, data, len, out); kernel_neon_begin(); diff --git a/arch/arm/crypto/sha256_neon_glue.c b/arch/arm/crypto/sha256_neon_glue.c index 1d82c6cd31a4..f3f6b1624fc3 100644 --- a/arch/arm/crypto/sha256_neon_glue.c +++ b/arch/arm/crypto/sha256_neon_glue.c @@ -15,6 +15,7 @@ */ #include +#include #include #include #include @@ -34,7 +35,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, { struct sha256_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd() || + if (!crypto_simd_usable() || (sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE) return crypto_sha256_arm_update(desc, data, len); @@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data, static int sha256_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha256_arm_finup(desc, data, len, out); kernel_neon_begin(); diff --git a/arch/arm/crypto/sha512-neon-glue.c b/arch/arm/crypto/sha512-neon-glue.c index 8a5642b41fd6..d33ab59c26c0 100644 --- a/arch/arm/crypto/sha512-neon-glue.c +++ b/arch/arm/crypto/sha512-neon-glue.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -30,7 +31,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data, { struct sha512_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd() || + if (!crypto_simd_usable() || (sctx->count[0] % SHA512_BLOCK_SIZE) + len < SHA512_BLOCK_SIZE) return sha512_arm_update(desc, data, len); @@ -45,7 +46,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data, static int sha512_neon_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return sha512_arm_finup(desc, data, len, out); kernel_neon_begin(); From patchwork Wed Mar 13 05:12:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850617 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A47551850 for ; Wed, 13 Mar 2019 05:15:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8F17B299CC for ; Wed, 13 Mar 2019 05:15:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8372B299D2; Wed, 13 Mar 2019 05:15:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 54EBC299E9 for ; Wed, 13 Mar 2019 05:15:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726542AbfCMFPj (ORCPT ); Wed, 13 Mar 2019 01:15:39 -0400 Received: from mail.kernel.org ([198.145.29.99]:50226 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725886AbfCMFPi (ORCPT ); Wed, 13 Mar 2019 01:15:38 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F3F9321871; Wed, 13 Mar 2019 05:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454136; bh=cyapBt7M4Bzi+09na+L92Osp2bMFOqtEydBtlXm+Vtc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0nF60bKReAZovejhun/eyOWNjJzrXlSRnT1D7yc/Sf4qZmoMdNy62lk7Rjuo3LIQE BmfWn1XME99c2FGxt613p+JpeuYyxFXGWONi887BIagy1exCj9nW/AGyqgWkLTJjdi s8AiBhLtGWh8We9VyyMaV6fJ7MVdDq+St9c10+2E= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 6/8] crypto: arm64 - convert to use crypto_simd_usable() Date: Tue, 12 Mar 2019 22:12:50 -0700 Message-Id: <20190313051252.2917-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers Replace all calls to may_use_simd() in the arm64 crypto code with crypto_simd_usable(), in order to allow testing the no-SIMD code paths. Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 7 ++++--- arch/arm64/crypto/aes-ce-glue.c | 5 +++-- arch/arm64/crypto/aes-glue.c | 4 ++-- arch/arm64/crypto/aes-neonbs-glue.c | 2 +- arch/arm64/crypto/chacha-neon-glue.c | 5 +++-- arch/arm64/crypto/crct10dif-ce-glue.c | 5 +++-- arch/arm64/crypto/ghash-ce-glue.c | 7 ++++--- arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 ++- arch/arm64/crypto/sha1-ce-glue.c | 7 ++++--- arch/arm64/crypto/sha2-ce-glue.c | 7 ++++--- arch/arm64/crypto/sha256-glue.c | 5 +++-- arch/arm64/crypto/sha3-ce-glue.c | 5 +++-- arch/arm64/crypto/sha512-ce-glue.c | 7 ++++--- arch/arm64/crypto/sm3-ce-glue.c | 7 ++++--- arch/arm64/crypto/sm4-ce-glue.c | 5 +++-- 15 files changed, 47 insertions(+), 34 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index 5fc6f51908fd..9dc4110a2e61 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -109,7 +110,7 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen) static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], u32 abytes, u32 *macp) { - if (may_use_simd()) { + if (crypto_simd_usable()) { kernel_neon_begin(); ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc, num_rounds(key)); @@ -255,7 +256,7 @@ static int ccm_encrypt(struct aead_request *req) err = skcipher_walk_aead_encrypt(&walk, req, false); - if (may_use_simd()) { + if (crypto_simd_usable()) { while (walk.nbytes) { u32 tail = walk.nbytes % AES_BLOCK_SIZE; @@ -313,7 +314,7 @@ static int ccm_decrypt(struct aead_request *req) err = skcipher_walk_aead_decrypt(&walk, req, false); - if (may_use_simd()) { + if (crypto_simd_usable()) { while (walk.nbytes) { u32 tail = walk.nbytes % AES_BLOCK_SIZE; diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glue.c index e6b3227bbf57..3213843fcb46 100644 --- a/arch/arm64/crypto/aes-ce-glue.c +++ b/arch/arm64/crypto/aes-ce-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -52,7 +53,7 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); - if (!may_use_simd()) { + if (!crypto_simd_usable()) { __aes_arm64_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); return; } @@ -66,7 +67,7 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) { struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); - if (!may_use_simd()) { + if (!crypto_simd_usable()) { __aes_arm64_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); return; } diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 1e676625ef33..692cb75f2ca2 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -405,7 +405,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); - if (!may_use_simd()) + if (!crypto_simd_usable()) return aes_ctr_encrypt_fallback(ctx, req); return ctr_encrypt(req); @@ -642,7 +642,7 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks, { int rounds = 6 + ctx->key_length / 4; - if (may_use_simd()) { + if (crypto_simd_usable()) { kernel_neon_begin(); aes_mac_update(in, ctx->key_enc, rounds, blocks, dg, enc_before, enc_after); diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index e7a95a566462..4737b6c6c5cf 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -288,7 +288,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); - if (!may_use_simd()) + if (!crypto_simd_usable()) return aes_ctr_encrypt_fallback(&ctx->fallback, req); return ctr_encrypt(req); diff --git a/arch/arm64/crypto/chacha-neon-glue.c b/arch/arm64/crypto/chacha-neon-glue.c index bece1d85bd81..3a26a98a7e17 100644 --- a/arch/arm64/crypto/chacha-neon-glue.c +++ b/arch/arm64/crypto/chacha-neon-glue.c @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -90,7 +91,7 @@ static int chacha_neon(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm); - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd()) + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) return crypto_chacha_crypt(req); return chacha_neon_stream_xor(req, ctx, req->iv); @@ -104,7 +105,7 @@ static int xchacha_neon(struct skcipher_request *req) u32 state[16]; u8 real_iv[16]; - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd()) + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable()) return crypto_xchacha_crypt(req); crypto_chacha_init(state, ctx, req->iv); diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c index dd325829ee44..64e92ab70269 100644 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ b/arch/arm64/crypto/crct10dif-ce-glue.c @@ -16,6 +16,7 @@ #include #include +#include #include #include @@ -38,7 +39,7 @@ static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, { u16 *crc = shash_desc_ctx(desc); - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { kernel_neon_begin(); *crc = crc_t10dif_pmull_p8(*crc, data, length); kernel_neon_end(); @@ -54,7 +55,7 @@ static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, { u16 *crc = shash_desc_ctx(desc); - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) { kernel_neon_begin(); *crc = crc_t10dif_pmull_p64(*crc, data, length); kernel_neon_end(); diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index 089b09286da7..fcd458c83bc1 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -89,7 +90,7 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src, struct ghash_key const *k, const char *head)) { - if (likely(may_use_simd())) { + if (likely(crypto_simd_usable())) { kernel_neon_begin(); simd_update(blocks, dg, src, key, head); kernel_neon_end(); @@ -441,7 +442,7 @@ static int gcm_encrypt(struct aead_request *req) err = skcipher_walk_aead_encrypt(&walk, req, false); - if (likely(may_use_simd() && walk.total >= 2 * AES_BLOCK_SIZE)) { + if (likely(crypto_simd_usable() && walk.total >= 2 * AES_BLOCK_SIZE)) { u32 const *rk = NULL; kernel_neon_begin(); @@ -565,7 +566,7 @@ static int gcm_decrypt(struct aead_request *req) err = skcipher_walk_aead_decrypt(&walk, req, false); - if (likely(may_use_simd() && walk.total >= 2 * AES_BLOCK_SIZE)) { + if (likely(crypto_simd_usable() && walk.total >= 2 * AES_BLOCK_SIZE)) { u32 const *rk = NULL; kernel_neon_begin(); diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/nhpoly1305-neon-glue.c index 22cc32ac9448..d15e872fa3f5 100644 --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len, static int nhpoly1305_neon_update(struct shash_desc *desc, const u8 *src, unsigned int srclen) { - if (srclen < 64 || !may_use_simd()) + if (srclen < 64 || !crypto_simd_usable()) return crypto_nhpoly1305_update(desc, src, srclen); do { diff --git a/arch/arm64/crypto/sha1-ce-glue.c b/arch/arm64/crypto/sha1-ce-glue.c index 17fac2889f56..eaa7a8258f1c 100644 --- a/arch/arm64/crypto/sha1-ce-glue.c +++ b/arch/arm64/crypto/sha1-ce-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -38,7 +39,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data, { struct sha1_ce_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha1_update(desc, data, len); sctx->finalize = 0; @@ -56,7 +57,7 @@ static int sha1_ce_finup(struct shash_desc *desc, const u8 *data, struct sha1_ce_state *sctx = shash_desc_ctx(desc); bool finalize = !sctx->sst.count && !(len % SHA1_BLOCK_SIZE); - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha1_finup(desc, data, len, out); /* @@ -78,7 +79,7 @@ static int sha1_ce_final(struct shash_desc *desc, u8 *out) { struct sha1_ce_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha1_finup(desc, NULL, 0, out); sctx->finalize = 0; diff --git a/arch/arm64/crypto/sha2-ce-glue.c b/arch/arm64/crypto/sha2-ce-glue.c index 261f5195cab7..a725997e55f2 100644 --- a/arch/arm64/crypto/sha2-ce-glue.c +++ b/arch/arm64/crypto/sha2-ce-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -42,7 +43,7 @@ static int sha256_ce_update(struct shash_desc *desc, const u8 *data, { struct sha256_ce_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd()) + if (!crypto_simd_usable()) return sha256_base_do_update(desc, data, len, (sha256_block_fn *)sha256_block_data_order); @@ -61,7 +62,7 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data, struct sha256_ce_state *sctx = shash_desc_ctx(desc); bool finalize = !sctx->sst.count && !(len % SHA256_BLOCK_SIZE); - if (!may_use_simd()) { + if (!crypto_simd_usable()) { if (len) sha256_base_do_update(desc, data, len, (sha256_block_fn *)sha256_block_data_order); @@ -90,7 +91,7 @@ static int sha256_ce_final(struct shash_desc *desc, u8 *out) { struct sha256_ce_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd()) { + if (!crypto_simd_usable()) { sha256_base_do_finalize(desc, (sha256_block_fn *)sha256_block_data_order); return sha256_base_finish(desc, out); diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c index 4aedeaefd61f..54586e0be9fd 100644 --- a/arch/arm64/crypto/sha256-glue.c +++ b/arch/arm64/crypto/sha256-glue.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -89,7 +90,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data, { struct sha256_state *sctx = shash_desc_ctx(desc); - if (!may_use_simd()) + if (!crypto_simd_usable()) return sha256_base_do_update(desc, data, len, (sha256_block_fn *)sha256_block_data_order); @@ -119,7 +120,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data, static int sha256_finup_neon(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) { + if (!crypto_simd_usable()) { if (len) sha256_base_do_update(desc, data, len, (sha256_block_fn *)sha256_block_data_order); diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-glue.c index a336feac0f59..9a4bbfc45f40 100644 --- a/arch/arm64/crypto/sha3-ce-glue.c +++ b/arch/arm64/crypto/sha3-ce-glue.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -32,7 +33,7 @@ static int sha3_update(struct shash_desc *desc, const u8 *data, struct sha3_state *sctx = shash_desc_ctx(desc); unsigned int digest_size = crypto_shash_digestsize(desc->tfm); - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha3_update(desc, data, len); if ((sctx->partial + len) >= sctx->rsiz) { @@ -76,7 +77,7 @@ static int sha3_final(struct shash_desc *desc, u8 *out) __le64 *digest = (__le64 *)out; int i; - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sha3_final(desc, out); sctx->buf[sctx->partial++] = 0x06; diff --git a/arch/arm64/crypto/sha512-ce-glue.c b/arch/arm64/crypto/sha512-ce-glue.c index f2c5f28c622a..2369540040aa 100644 --- a/arch/arm64/crypto/sha512-ce-glue.c +++ b/arch/arm64/crypto/sha512-ce-glue.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -31,7 +32,7 @@ asmlinkage void sha512_block_data_order(u64 *digest, u8 const *src, int blocks); static int sha512_ce_update(struct shash_desc *desc, const u8 *data, unsigned int len) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return sha512_base_do_update(desc, data, len, (sha512_block_fn *)sha512_block_data_order); @@ -46,7 +47,7 @@ static int sha512_ce_update(struct shash_desc *desc, const u8 *data, static int sha512_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) { + if (!crypto_simd_usable()) { if (len) sha512_base_do_update(desc, data, len, (sha512_block_fn *)sha512_block_data_order); @@ -65,7 +66,7 @@ static int sha512_ce_finup(struct shash_desc *desc, const u8 *data, static int sha512_ce_final(struct shash_desc *desc, u8 *out) { - if (!may_use_simd()) { + if (!crypto_simd_usable()) { sha512_base_do_finalize(desc, (sha512_block_fn *)sha512_block_data_order); return sha512_base_finish(desc, out); diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glue.c index 88938a20d9b2..5d15533799a2 100644 --- a/arch/arm64/crypto/sm3-ce-glue.c +++ b/arch/arm64/crypto/sm3-ce-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -28,7 +29,7 @@ asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src, static int sm3_ce_update(struct shash_desc *desc, const u8 *data, unsigned int len) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sm3_update(desc, data, len); kernel_neon_begin(); @@ -40,7 +41,7 @@ static int sm3_ce_update(struct shash_desc *desc, const u8 *data, static int sm3_ce_final(struct shash_desc *desc, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sm3_finup(desc, NULL, 0, out); kernel_neon_begin(); @@ -53,7 +54,7 @@ static int sm3_ce_final(struct shash_desc *desc, u8 *out) static int sm3_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - if (!may_use_simd()) + if (!crypto_simd_usable()) return crypto_sm3_finup(desc, data, len, out); kernel_neon_begin(); diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glue.c index 0c4fc223f225..2754c875d39c 100644 --- a/arch/arm64/crypto/sm4-ce-glue.c +++ b/arch/arm64/crypto/sm4-ce-glue.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -20,7 +21,7 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm); - if (!may_use_simd()) { + if (!crypto_simd_usable()) { crypto_sm4_encrypt(tfm, out, in); } else { kernel_neon_begin(); @@ -33,7 +34,7 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm); - if (!may_use_simd()) { + if (!crypto_simd_usable()) { crypto_sm4_decrypt(tfm, out, in); } else { kernel_neon_begin(); From patchwork Wed Mar 13 05:12:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850607 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DD601850 for ; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B756299D6 for ; Wed, 13 Mar 2019 05:15:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F0D61299E1; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4073A299D4 for ; Wed, 13 Mar 2019 05:15:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726097AbfCMFPh (ORCPT ); Wed, 13 Mar 2019 01:15:37 -0400 Received: from mail.kernel.org ([198.145.29.99]:50246 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726183AbfCMFPh (ORCPT ); Wed, 13 Mar 2019 01:15:37 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 507CA21873; Wed, 13 Mar 2019 05:15:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454136; bh=4bywO5eXatDKtqu/5HOPPSncChimudaqktyjzb5khMw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N0zxmAGwjT8/V73gaaM6KiCK2jKkZT1H2hb4gzX5eR48O5om3ACRG9czPRIBn9HV8 7c+8IWunmG9rxx0HE/Y57qpOjEXgKyN68VTE+S+WNv6+GhpEG8iwrIHKjdAahExyz4 4YbVEVI9xvftGjW6FgaTr4RFs2BJ49XYByBWPtks= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 7/8] crypto: simd - convert to use crypto_simd_usable() Date: Tue, 12 Mar 2019 22:12:51 -0700 Message-Id: <20190313051252.2917-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers Replace all calls to may_use_simd() in the shared SIMD helpers with crypto_simd_usable(), in order to allow testing the no-SIMD code paths. Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel --- crypto/simd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/crypto/simd.c b/crypto/simd.c index 7d62686d3a3f..3e3b1d1a6b1f 100644 --- a/crypto/simd.c +++ b/crypto/simd.c @@ -85,7 +85,7 @@ static int simd_skcipher_encrypt(struct skcipher_request *req) subreq = skcipher_request_ctx(req); *subreq = *req; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_skcipher_queued(ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base; else @@ -106,7 +106,7 @@ static int simd_skcipher_decrypt(struct skcipher_request *req) subreq = skcipher_request_ctx(req); *subreq = *req; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_skcipher_queued(ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base; else @@ -336,7 +336,7 @@ static int simd_aead_encrypt(struct aead_request *req) subreq = aead_request_ctx(req); *subreq = *req; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_aead_queued(ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base; else @@ -357,7 +357,7 @@ static int simd_aead_decrypt(struct aead_request *req) subreq = aead_request_ctx(req); *subreq = *req; - if (!may_use_simd() || + if (!crypto_simd_usable() || (in_atomic() && cryptd_aead_queued(ctx->cryptd_tfm))) child = &ctx->cryptd_tfm->base; else From patchwork Wed Mar 13 05:12:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 10850615 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 77BEB17DF for ; Wed, 13 Mar 2019 05:15:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6298E299CC for ; Wed, 13 Mar 2019 05:15:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 576E4299D4; Wed, 13 Mar 2019 05:15:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FCBF299CC for ; Wed, 13 Mar 2019 05:15:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726326AbfCMFPj (ORCPT ); Wed, 13 Mar 2019 01:15:39 -0400 Received: from mail.kernel.org ([198.145.29.99]:50276 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726542AbfCMFPi (ORCPT ); Wed, 13 Mar 2019 01:15:38 -0400 Received: from sol.localdomain (c-107-3-167-184.hsd1.ca.comcast.net [107.3.167.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9EF82218A2; Wed, 13 Mar 2019 05:15:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1552454136; bh=edsI4P2A+j5OH1u930iSsYk0nQmkkbGDtx8/n5Q4xfQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lkCT0WHg3SA1+iMLbKUH4A4YzaKXCydMwLbJSkOvSO4UXIhBMHcmmCrFa+PEWmP7X S4UYKxHIScvCDxwrr3vdkjuK4/vdgmd5WJzD0JiOz8JZHzk3mNOtWYXIJuR0MNiU3I VQEvxLyLlNqtHYHzwTB0CcPYgtjxx/L8MCB7GJLM= From: Eric Biggers To: linux-crypto@vger.kernel.org, Herbert Xu Cc: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code Date: Tue, 12 Mar 2019 22:12:52 -0700 Message-Id: <20190313051252.2917-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190313051252.2917-1-ebiggers@kernel.org> References: <20190313051252.2917-1-ebiggers@kernel.org> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers All crypto API algorithms are supposed to support the case where they are called in a context where SIMD instructions are unusable, e.g. IRQ context on some architectures. However, this isn't tested for by the self-tests, causing bugs to go undetected. Now that all algorithms have been converted to use crypto_simd_usable(), update the self-tests to test the no-SIMD case. First, a bool testvec_config::nosimd is added. When set, the crypto operation is executed with preemption disabled and with crypto_simd_usable() mocked out to return false on the current CPU. A bool test_sg_division::nosimd is also added. For hash algorithms it's honored by the corresponding ->update(). By setting just a subset of these bools, the case where some ->update()s are done in SIMD context and some are done in no-SIMD context is also tested. These bools are then randomly set by generate_random_testvec_config(). For now, all no-SIMD testing is limited to the extra crypto self-tests, because it might be a bit too invasive for the regular self-tests. But this could be changed later. This has already found bugs in the arm64 AES-GCM and ChaCha algorithms. This would have found some past bugs as well. Signed-off-by: Eric Biggers Acked-by: Ard Biesheuvel --- crypto/testmgr.c | 116 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 92 insertions(+), 24 deletions(-) diff --git a/crypto/testmgr.c b/crypto/testmgr.c index 52417dde811f..2c2ddebb48d3 100644 --- a/crypto/testmgr.c +++ b/crypto/testmgr.c @@ -234,12 +234,14 @@ enum finalization_type { * @offset * @flush_type: for hashes, whether an update() should be done now vs. * continuing to accumulate data + * @nosimd: if doing the pending update(), do it with SIMD disabled? */ struct test_sg_division { unsigned int proportion_of_total; unsigned int offset; bool offset_relative_to_alignmask; enum flush_type flush_type; + bool nosimd; }; /** @@ -259,6 +261,7 @@ struct test_sg_division { * @iv_offset_relative_to_alignmask: if true, add the algorithm's alignmask to * the @iv_offset * @finalization_type: what finalization function to use for hashes + * @nosimd: execute with SIMD disabled? Requires !CRYPTO_TFM_REQ_MAY_SLEEP. */ struct testvec_config { const char *name; @@ -269,6 +272,7 @@ struct testvec_config { unsigned int iv_offset; bool iv_offset_relative_to_alignmask; enum finalization_type finalization_type; + bool nosimd; }; #define TESTVEC_CONFIG_NAMELEN 192 @@ -420,8 +424,11 @@ static unsigned int count_test_sg_divisions(const struct test_sg_division *divs) return ndivs; } +#define SGDIVS_HAVE_FLUSHES BIT(0) +#define SGDIVS_HAVE_NOSIMD BIT(1) + static bool valid_sg_divisions(const struct test_sg_division *divs, - unsigned int count, bool *any_flushes_ret) + unsigned int count, int *flags_ret) { unsigned int total = 0; unsigned int i; @@ -432,7 +439,9 @@ static bool valid_sg_divisions(const struct test_sg_division *divs, return false; total += divs[i].proportion_of_total; if (divs[i].flush_type != FLUSH_TYPE_NONE) - *any_flushes_ret = true; + *flags_ret |= SGDIVS_HAVE_FLUSHES; + if (divs[i].nosimd) + *flags_ret |= SGDIVS_HAVE_NOSIMD; } return total == TEST_SG_TOTAL && memchr_inv(&divs[i], 0, (count - i) * sizeof(divs[0])) == NULL; @@ -445,19 +454,18 @@ static bool valid_sg_divisions(const struct test_sg_division *divs, */ static bool valid_testvec_config(const struct testvec_config *cfg) { - bool any_flushes = false; + int flags = 0; if (cfg->name == NULL) return false; if (!valid_sg_divisions(cfg->src_divs, ARRAY_SIZE(cfg->src_divs), - &any_flushes)) + &flags)) return false; if (cfg->dst_divs[0].proportion_of_total) { if (!valid_sg_divisions(cfg->dst_divs, - ARRAY_SIZE(cfg->dst_divs), - &any_flushes)) + ARRAY_SIZE(cfg->dst_divs), &flags)) return false; } else { if (memchr_inv(cfg->dst_divs, 0, sizeof(cfg->dst_divs))) @@ -470,7 +478,12 @@ static bool valid_testvec_config(const struct testvec_config *cfg) MAX_ALGAPI_ALIGNMASK + 1) return false; - if (any_flushes && cfg->finalization_type == FINALIZATION_TYPE_DIGEST) + if ((flags & (SGDIVS_HAVE_FLUSHES | SGDIVS_HAVE_NOSIMD)) && + cfg->finalization_type == FINALIZATION_TYPE_DIGEST) + return false; + + if ((cfg->nosimd || (flags & SGDIVS_HAVE_NOSIMD)) && + (cfg->req_flags & CRYPTO_TFM_REQ_MAY_SLEEP)) return false; return true; @@ -731,13 +744,14 @@ static int build_cipher_test_sglists(struct cipher_test_sglists *tsgls, #ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS static char *generate_random_sgl_divisions(struct test_sg_division *divs, size_t max_divs, char *p, char *end, - bool gen_flushes) + bool gen_flushes, u32 req_flags) { struct test_sg_division *div = divs; unsigned int remaining = TEST_SG_TOTAL; do { unsigned int this_len; + const char *flushtype_str; if (div == &divs[max_divs - 1] || prandom_u32() % 2 == 0) this_len = remaining; @@ -766,11 +780,31 @@ static char *generate_random_sgl_divisions(struct test_sg_division *divs, } } + if (div->flush_type != FLUSH_TYPE_NONE && + !(req_flags & CRYPTO_TFM_REQ_MAY_SLEEP) && + prandom_u32() % 2 == 0) + div->nosimd = true; + + switch (div->flush_type) { + case FLUSH_TYPE_FLUSH: + if (div->nosimd) + flushtype_str = ""; + else + flushtype_str = ""; + break; + case FLUSH_TYPE_REIMPORT: + if (div->nosimd) + flushtype_str = ""; + else + flushtype_str = ""; + break; + default: + flushtype_str = ""; + break; + } + BUILD_BUG_ON(TEST_SG_TOTAL != 10000); /* for "%u.%u%%" */ - p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", - div->flush_type == FLUSH_TYPE_NONE ? "" : - div->flush_type == FLUSH_TYPE_FLUSH ? - " " : " ", + p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", flushtype_str, this_len / 100, this_len % 100, div->offset_relative_to_alignmask ? "alignmask" : "", @@ -820,18 +854,26 @@ static void generate_random_testvec_config(struct testvec_config *cfg, break; } + if (!(cfg->req_flags & CRYPTO_TFM_REQ_MAY_SLEEP) && + prandom_u32() % 2 == 0) { + cfg->nosimd = true; + p += scnprintf(p, end - p, " nosimd"); + } + p += scnprintf(p, end - p, " src_divs=["); p = generate_random_sgl_divisions(cfg->src_divs, ARRAY_SIZE(cfg->src_divs), p, end, (cfg->finalization_type != - FINALIZATION_TYPE_DIGEST)); + FINALIZATION_TYPE_DIGEST), + cfg->req_flags); p += scnprintf(p, end - p, "]"); if (!cfg->inplace && prandom_u32() % 2 == 0) { p += scnprintf(p, end - p, " dst_divs=["); p = generate_random_sgl_divisions(cfg->dst_divs, ARRAY_SIZE(cfg->dst_divs), - p, end, false); + p, end, false, + cfg->req_flags); p += scnprintf(p, end - p, "]"); } @@ -864,6 +906,23 @@ static void crypto_reenable_simd_for_test(void) } #endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ +static int do_ahash_op(int (*op)(struct ahash_request *req), + struct ahash_request *req, + struct crypto_wait *wait, bool nosimd) +{ + int err; + + if (nosimd) + crypto_disable_simd_for_test(); + + err = op(req); + + if (nosimd) + crypto_reenable_simd_for_test(); + + return crypto_wait_req(err, wait); +} + static int check_nonfinal_hash_op(const char *op, int err, u8 *result, unsigned int digestsize, const char *driver, unsigned int vec_num, @@ -938,7 +997,7 @@ static int test_hash_vec_cfg(const char *driver, ahash_request_set_callback(req, req_flags, crypto_req_done, &wait); ahash_request_set_crypt(req, tsgl->sgl, result, vec->psize); - err = crypto_wait_req(crypto_ahash_digest(req), &wait); + err = do_ahash_op(crypto_ahash_digest, req, &wait, cfg->nosimd); if (err) { pr_err("alg: hash: %s digest() failed with err %d on test vector %u, cfg=\"%s\"\n", driver, err, vec_num, cfg->name); @@ -951,7 +1010,7 @@ static int test_hash_vec_cfg(const char *driver, ahash_request_set_callback(req, req_flags, crypto_req_done, &wait); ahash_request_set_crypt(req, NULL, result, 0); - err = crypto_wait_req(crypto_ahash_init(req), &wait); + err = do_ahash_op(crypto_ahash_init, req, &wait, cfg->nosimd); err = check_nonfinal_hash_op("init", err, result, digestsize, driver, vec_num, cfg); if (err) @@ -967,7 +1026,8 @@ static int test_hash_vec_cfg(const char *driver, crypto_req_done, &wait); ahash_request_set_crypt(req, pending_sgl, result, pending_len); - err = crypto_wait_req(crypto_ahash_update(req), &wait); + err = do_ahash_op(crypto_ahash_update, req, &wait, + divs[i]->nosimd); err = check_nonfinal_hash_op("update", err, result, digestsize, driver, vec_num, cfg); @@ -1010,12 +1070,12 @@ static int test_hash_vec_cfg(const char *driver, ahash_request_set_crypt(req, pending_sgl, result, pending_len); if (cfg->finalization_type == FINALIZATION_TYPE_FINAL) { /* finish with update() and final() */ - err = crypto_wait_req(crypto_ahash_update(req), &wait); + err = do_ahash_op(crypto_ahash_update, req, &wait, cfg->nosimd); err = check_nonfinal_hash_op("update", err, result, digestsize, driver, vec_num, cfg); if (err) return err; - err = crypto_wait_req(crypto_ahash_final(req), &wait); + err = do_ahash_op(crypto_ahash_final, req, &wait, cfg->nosimd); if (err) { pr_err("alg: hash: %s final() failed with err %d on test vector %u, cfg=\"%s\"\n", driver, err, vec_num, cfg->name); @@ -1023,7 +1083,7 @@ static int test_hash_vec_cfg(const char *driver, } } else { /* finish with finup() */ - err = crypto_wait_req(crypto_ahash_finup(req), &wait); + err = do_ahash_op(crypto_ahash_finup, req, &wait, cfg->nosimd); if (err) { pr_err("alg: hash: %s finup() failed with err %d on test vector %u, cfg=\"%s\"\n", driver, err, vec_num, cfg->name); @@ -1259,8 +1319,12 @@ static int test_aead_vec_cfg(const char *driver, int enc, aead_request_set_crypt(req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr, enc ? vec->plen : vec->clen, iv); aead_request_set_ad(req, vec->alen); - err = crypto_wait_req(enc ? crypto_aead_encrypt(req) : - crypto_aead_decrypt(req), &wait); + if (cfg->nosimd) + crypto_disable_simd_for_test(); + err = enc ? crypto_aead_encrypt(req) : crypto_aead_decrypt(req); + if (cfg->nosimd) + crypto_reenable_simd_for_test(); + err = crypto_wait_req(err, &wait); if (err) { if (err == -EBADMSG && vec->novrfy) return 0; @@ -1594,8 +1658,12 @@ static int test_skcipher_vec_cfg(const char *driver, int enc, skcipher_request_set_callback(req, req_flags, crypto_req_done, &wait); skcipher_request_set_crypt(req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr, vec->len, iv); - err = crypto_wait_req(enc ? crypto_skcipher_encrypt(req) : - crypto_skcipher_decrypt(req), &wait); + if (cfg->nosimd) + crypto_disable_simd_for_test(); + err = enc ? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req); + if (cfg->nosimd) + crypto_reenable_simd_for_test(); + err = crypto_wait_req(err, &wait); if (err) { pr_err("alg: skcipher: %s %s failed with err %d on test vector %u, cfg=\"%s\"\n", driver, op, err, vec_num, cfg->name);