From patchwork Sun Apr 13 04:54:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9BF8CC3601E for ; Sun, 13 Apr 2025 04:57:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=O6e/7+xSkqHNapzqaKLz5lcr5fn/gFwGUJi7HyR0IVQ=; b=IiHLqfnoxtD8DBJvF18/uTv2dU 69FSm3B60MdzQ0l8OZlLgdcO8Wc8D0MdI8iFLSgc2O7idSuKagbCnqFnmj43qn5x6vmzCs4FyvME6 hJzx18OGJgHY1tt1d+rgc2D8WbDt6cr15aew42MpG2bbqloCW8VW1CyXBgQL5ycM6wn9qZIY3s5e7 1doAf7M1/A44dNF7nGIaQTXF1CSb6t4P3fd6ABUgaetFIm+mUr3rgGEGYm9JHgdGxhoNEaClfYZLZ ZU1CY1PyvjOcdAY271eAJ42CVACuuTvx9INqy06tGo1byv1I2soWJJqFsuE9easpIzUfhHhJL5GA8 I3A39+MA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pPR-0000000GoEO-2UcO; Sun, 13 Apr 2025 04:57:33 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNd-0000000GnfP-3b5Z for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:43 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 19EBBA407E5; Sun, 13 Apr 2025 04:50:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C7EE6C4CEEC; Sun, 13 Apr 2025 04:55:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520140; bh=0eZGQ4HOV8ila8DyyAF1mDleVNmhJHh2p5GKktNCdP0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=raC1yaz/NDXB5M5reoTn4t/01dfz407KBNfd0K/poDFbBFUEXP/CzzQ/h7BteTKya +l7uKwR0/amZTMR3DiCT4nH7nArCjhZueQexvrtZHd1cqtpaZmRDLh5ctmFa/17nNz YvKl3VfyjfrY+n4+fVBaZFVabVob+HzAYlmZBeuLJed3kN9mqzNFK1Dr+NpPQHmKfv aTwBkkxbcyoedBvl/d93d69Apzs8J+eTE/mQIYs1kMuv509JB0qmRvALse4njknTIM +MqCByN2EQaGo2r6lHoqjSObkDTt2F9oRPuZ1LRLrdmx8gkUI1M9HlEWVfoq+o2o+e OGm+yRM8WpVlw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Danny Tsen Subject: [PATCH 1/9] crypto: powerpc/poly1305 - implement library instead of shash Date: Sat, 12 Apr 2025 21:54:13 -0700 Message-ID: <20250413045421.55100-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215542_033655_766B31ED X-CRM114-Status: GOOD ( 18.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers Currently the Power10 optimized Poly1305 is only wired up to the crypto_shash API, which makes it unavailable to users of the library API. The crypto_shash API for Poly1305 is going to change to be implemented on top of the library API, so the library API needs to be supported. And of course it's needed anyway to serve the library users. Therefore, change the Power10 optimized Poly1305 code to implement the library API instead of the crypto_shash API. Cc: Danny Tsen Signed-off-by: Eric Biggers --- arch/powerpc/crypto/Kconfig | 11 +- arch/powerpc/crypto/poly1305-p10-glue.c | 132 +++++++----------------- 2 files changed, 39 insertions(+), 104 deletions(-) base-commit: 3be3f70ee95da03a87d94c4a714ee679a5c7b34d diff --git a/arch/powerpc/crypto/Kconfig b/arch/powerpc/crypto/Kconfig index 47dccdd49637..cbcf76953d83 100644 --- a/arch/powerpc/crypto/Kconfig +++ b/arch/powerpc/crypto/Kconfig @@ -98,20 +98,15 @@ config CRYPTO_CHACHA20_P10 select CRYPTO_LIB_CHACHA_GENERIC select CRYPTO_ARCH_HAVE_LIB_CHACHA default CRYPTO_LIB_CHACHA_INTERNAL config CRYPTO_POLY1305_P10 - tristate "Hash functions: Poly1305 (P10 or later)" + tristate depends on PPC64 && CPU_LITTLE_ENDIAN && VSX - select CRYPTO_HASH + select CRYPTO_ARCH_HAVE_LIB_POLY1305 select CRYPTO_LIB_POLY1305_GENERIC - help - Poly1305 authenticator algorithm (RFC7539) - - Architecture: PowerPC64 - - Power10 or later - - Little-endian + default CRYPTO_LIB_POLY1305_INTERNAL config CRYPTO_DEV_VMX bool "Support for VMX cryptographic acceleration instructions" depends on PPC64 && VSX help diff --git a/arch/powerpc/crypto/poly1305-p10-glue.c b/arch/powerpc/crypto/poly1305-p10-glue.c index 369686e9370b..87bd5eba4497 100644 --- a/arch/powerpc/crypto/poly1305-p10-glue.c +++ b/arch/powerpc/crypto/poly1305-p10-glue.c @@ -2,28 +2,26 @@ /* * Poly1305 authenticator algorithm, RFC7539. * * Copyright 2023- IBM Corp. All rights reserved. */ - -#include -#include #include #include #include -#include -#include #include +#include #include #include #include #include asmlinkage void poly1305_p10le_4blocks(void *h, const u8 *m, u32 mlen); asmlinkage void poly1305_64s(void *h, const u8 *m, u32 mlen, int highbit); asmlinkage void poly1305_emit_64(void *h, void *s, u8 *dst); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_p10); + static void vsx_begin(void) { preempt_disable(); enable_kernel_vsx(); } @@ -32,82 +30,50 @@ static void vsx_end(void) { disable_kernel_vsx(); preempt_enable(); } -static int crypto_poly1305_p10_init(struct shash_desc *desc) +void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) { - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - poly1305_core_init(&dctx->h); + if (!static_key_enabled(&have_p10)) + return poly1305_init_generic(dctx, key); + + dctx->h = (struct poly1305_state){}; + dctx->core_r.key.r64[0] = get_unaligned_le64(key + 0); + dctx->core_r.key.r64[1] = get_unaligned_le64(key + 8); + dctx->s[0] = get_unaligned_le32(key + 16); + dctx->s[1] = get_unaligned_le32(key + 20); + dctx->s[2] = get_unaligned_le32(key + 24); + dctx->s[3] = get_unaligned_le32(key + 28); dctx->buflen = 0; - dctx->rset = 0; - dctx->sset = false; - - return 0; } +EXPORT_SYMBOL(poly1305_init_arch); -static unsigned int crypto_poly1305_setdctxkey(struct poly1305_desc_ctx *dctx, - const u8 *inp, unsigned int len) +void poly1305_update_arch(struct poly1305_desc_ctx *dctx, + const u8 *src, unsigned int srclen) { - unsigned int acc = 0; - - if (unlikely(!dctx->sset)) { - if (!dctx->rset && len >= POLY1305_BLOCK_SIZE) { - struct poly1305_core_key *key = &dctx->core_r; - - key->key.r64[0] = get_unaligned_le64(&inp[0]); - key->key.r64[1] = get_unaligned_le64(&inp[8]); - inp += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - acc += POLY1305_BLOCK_SIZE; - dctx->rset = 1; - } - if (len >= POLY1305_BLOCK_SIZE) { - dctx->s[0] = get_unaligned_le32(&inp[0]); - dctx->s[1] = get_unaligned_le32(&inp[4]); - dctx->s[2] = get_unaligned_le32(&inp[8]); - dctx->s[3] = get_unaligned_le32(&inp[12]); - acc += POLY1305_BLOCK_SIZE; - dctx->sset = true; - } - } - return acc; -} + unsigned int bytes; -static int crypto_poly1305_p10_update(struct shash_desc *desc, - const u8 *src, unsigned int srclen) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - unsigned int bytes, used; + if (!static_key_enabled(&have_p10)) + return poly1305_update_generic(dctx, src, srclen); if (unlikely(dctx->buflen)) { bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen); memcpy(dctx->buf + dctx->buflen, src, bytes); src += bytes; srclen -= bytes; dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - if (likely(!crypto_poly1305_setdctxkey(dctx, dctx->buf, - POLY1305_BLOCK_SIZE))) { - vsx_begin(); - poly1305_64s(&dctx->h, dctx->buf, - POLY1305_BLOCK_SIZE, 1); - vsx_end(); - } - dctx->buflen = 0; - } + if (dctx->buflen < POLY1305_BLOCK_SIZE) + return; + vsx_begin(); + poly1305_64s(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 1); + vsx_end(); + dctx->buflen = 0; } if (likely(srclen >= POLY1305_BLOCK_SIZE)) { bytes = round_down(srclen, POLY1305_BLOCK_SIZE); - used = crypto_poly1305_setdctxkey(dctx, src, bytes); - if (likely(used)) { - srclen -= used; - src += used; - } if (crypto_simd_usable() && (srclen >= POLY1305_BLOCK_SIZE*4)) { vsx_begin(); poly1305_p10le_4blocks(&dctx->h, src, srclen); vsx_end(); src += srclen - (srclen % (POLY1305_BLOCK_SIZE * 4)); @@ -124,63 +90,37 @@ static int crypto_poly1305_p10_update(struct shash_desc *desc, if (unlikely(srclen)) { dctx->buflen = srclen; memcpy(dctx->buf, src, srclen); } - - return 0; } +EXPORT_SYMBOL(poly1305_update_arch); -static int crypto_poly1305_p10_final(struct shash_desc *desc, u8 *dst) +void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) { - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); + if (!static_key_enabled(&have_p10)) + return poly1305_final_generic(dctx, dst); - if (unlikely(!dctx->sset)) - return -ENOKEY; - - if ((dctx->buflen)) { + if (dctx->buflen) { dctx->buf[dctx->buflen++] = 1; memset(dctx->buf + dctx->buflen, 0, POLY1305_BLOCK_SIZE - dctx->buflen); vsx_begin(); poly1305_64s(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0); vsx_end(); - dctx->buflen = 0; } poly1305_emit_64(&dctx->h, &dctx->s, dst); - return 0; } - -static struct shash_alg poly1305_alg = { - .digestsize = POLY1305_DIGEST_SIZE, - .init = crypto_poly1305_p10_init, - .update = crypto_poly1305_p10_update, - .final = crypto_poly1305_p10_final, - .descsize = sizeof(struct poly1305_desc_ctx), - .base = { - .cra_name = "poly1305", - .cra_driver_name = "poly1305-p10", - .cra_priority = 300, - .cra_blocksize = POLY1305_BLOCK_SIZE, - .cra_module = THIS_MODULE, - }, -}; +EXPORT_SYMBOL(poly1305_final_arch); static int __init poly1305_p10_init(void) { - return crypto_register_shash(&poly1305_alg); -} - -static void __exit poly1305_p10_exit(void) -{ - crypto_unregister_shash(&poly1305_alg); + if (cpu_has_feature(CPU_FTR_ARCH_31)) + static_branch_enable(&have_p10); + return 0; } - -module_cpu_feature_match(PPC_MODULE_FEATURE_P10, poly1305_p10_init); -module_exit(poly1305_p10_exit); +arch_initcall(poly1305_p10_init); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Danny Tsen "); MODULE_DESCRIPTION("Optimized Poly1305 for P10"); -MODULE_ALIAS_CRYPTO("poly1305"); -MODULE_ALIAS_CRYPTO("poly1305-p10"); From patchwork Sun Apr 13 04:54:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68EB3C3601E for ; Sun, 13 Apr 2025 05:06:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mskEvms1UsTwJmKX6VcNSEBWl0uUVmfd7/Jqa50Rpg4=; b=1KOhchoIR4eS5AU1YCsm5riMHN 3xhQfgxg3H3HEKX2rGDXVlpV5c6kYh46VKHwGV7PuFgllmV8gD+CQFagleWdWzxFBeVmIqi84RuUH 8IpLrBootuUsalsipvBy1ZXqRqOlFO+XaGNFHGrKdLXI4gaHyOMx8cFPsE2nTX4K8JKHXayYHLJ9R PB96Pd9cJ/hm7k52LkW4sIN40SiuS3SgmvFp9baP3v2LjIYasNczTwy5ENFxQvT9ti3Co7hVrkDdP yYQuWG5iLUOzxsjEZSiJS0EDfkzXPrOLBtM53GicFToI0CmfmF5SDmLxra62la1TWiJVwJCti5oyJ 42CtdBeQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pYH-0000000Got0-47dQ; Sun, 13 Apr 2025 05:06:41 +0000 Received: from tor.source.kernel.org ([2600:3c04:e001:324:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNg-0000000Gnhs-2eky for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:44 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 7C73661139; Sun, 13 Apr 2025 04:55:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C105C4CEE5; Sun, 13 Apr 2025 04:55:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520140; bh=QuZp4dCBsrytrfUD2yzI6B/K0FCoLtG9KaUBqXXB548=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nHBMw7xiYAkkBYxo0P8tTLJRYnt+0kuvM7E7DyGLRTjn2lov/iQhwse3ldMrMkVKQ 3GRxkC9ZmZYpMHimEjDkRWbVZNhFpibY13Moq+7yXaieGwKqWO4eKvnn0W8kW+Cdfs 8DoLb8WisCutQtZ4o1RQeD16c1oBEmH5Uf7FhOF+2w/8r9cmS7y4/RzaxPueAkfWX1 8h6+CzNo+L2MJBKR2gEGP+9/vC+GrAKI3zzPlzmrDroLx+Npq47KGtr82CzKnXYaxt y7KhiQGn4uucGoKEuBdzrh3mij8snIs99FVKCztwU48BtGGTnK8fPjIvU6OZtdqGXb 6eZ2FGtEDr+xw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 2/9] crypto: poly1305 - centralize the shash wrappers for arch code Date: Sat, 12 Apr 2025 21:54:14 -0700 Message-ID: <20250413045421.55100-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers Following the example of the crc32, crc32c, and chacha code, make the crypto subsystem register both generic and architecture-optimized poly1305 shash algorithms, both implemented on top of the appropriate library functions. This eliminates the need for every architecture to implement the same shash glue code. Note that the poly1305 shash requires that the key be prepended to the data, which differs from the library functions where the key is simply a parameter to poly1305_init(). Previously this was handled at a fairly low level, polluting the library code with shash-specific code. Reorganize things so that the shash code handles this quirk itself. Also, to register the architecture-optimized shashes only when architecture-optimized code is actually being used, add a function poly1305_is_arch_optimized() and make each arch implement it. Change each architecture's Poly1305 module_init function to arch_initcall so that the CPU feature detection is guaranteed to run before poly1305_is_arch_optimized() gets called by crypto/poly1305.c. (In cases where poly1305_is_arch_optimized() just returns true unconditionally, using arch_initcall is not strictly needed, but it's still good to be consistent across architectures.) Signed-off-by: Eric Biggers --- arch/arm/crypto/poly1305-glue.c | 11 +- arch/arm64/crypto/poly1305-glue.c | 9 +- arch/mips/crypto/poly1305-glue.c | 10 +- arch/powerpc/crypto/poly1305-p10-glue.c | 6 + arch/x86/crypto/poly1305_glue.c | 8 +- crypto/Makefile | 3 +- crypto/poly1305.c | 152 ++++++++++++++++++++++++ crypto/poly1305_generic.c | 149 ----------------------- include/crypto/poly1305.h | 9 ++ 9 files changed, 201 insertions(+), 156 deletions(-) create mode 100644 crypto/poly1305.c delete mode 100644 crypto/poly1305_generic.c diff --git a/arch/arm/crypto/poly1305-glue.c b/arch/arm/crypto/poly1305-glue.c index 4464ffbf8fd1..238dbf50d05d 100644 --- a/arch/arm/crypto/poly1305-glue.c +++ b/arch/arm/crypto/poly1305-glue.c @@ -216,11 +216,11 @@ static struct shash_alg arm_poly1305_algs[] = {{ .final = arm_poly1305_final, .digestsize = POLY1305_DIGEST_SIZE, .descsize = sizeof(struct poly1305_desc_ctx), .base.cra_name = "poly1305", - .base.cra_driver_name = "poly1305-arm", + .base.cra_driver_name = "poly1305-arm-old", .base.cra_priority = 150, .base.cra_blocksize = POLY1305_BLOCK_SIZE, .base.cra_module = THIS_MODULE, #ifdef CONFIG_KERNEL_MODE_NEON }, { @@ -236,10 +236,17 @@ static struct shash_alg arm_poly1305_algs[] = {{ .base.cra_blocksize = POLY1305_BLOCK_SIZE, .base.cra_module = THIS_MODULE, #endif }}; +bool poly1305_is_arch_optimized(void) +{ + /* We always can use at least the ARM scalar implementation. */ + return true; +} +EXPORT_SYMBOL(poly1305_is_arch_optimized); + static int __init arm_poly1305_mod_init(void) { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && (elf_hwcap & HWCAP_NEON)) static_branch_enable(&have_neon); @@ -262,11 +269,11 @@ static void __exit arm_poly1305_mod_exit(void) } crypto_unregister_shashes(arm_poly1305_algs, ARRAY_SIZE(arm_poly1305_algs)); } -module_init(arm_poly1305_mod_init); +arch_initcall(arm_poly1305_mod_init); module_exit(arm_poly1305_mod_exit); MODULE_DESCRIPTION("Accelerated Poly1305 transform for ARM"); MODULE_LICENSE("GPL v2"); MODULE_ALIAS_CRYPTO("poly1305"); diff --git a/arch/arm64/crypto/poly1305-glue.c b/arch/arm64/crypto/poly1305-glue.c index 18883ea438f3..89415731d9be 100644 --- a/arch/arm64/crypto/poly1305-glue.c +++ b/arch/arm64/crypto/poly1305-glue.c @@ -204,10 +204,17 @@ static struct shash_alg neon_poly1305_alg = { .base.cra_priority = 200, .base.cra_blocksize = POLY1305_BLOCK_SIZE, .base.cra_module = THIS_MODULE, }; +bool poly1305_is_arch_optimized(void) +{ + /* We always can use at least the ARM64 scalar implementation. */ + return true; +} +EXPORT_SYMBOL(poly1305_is_arch_optimized); + static int __init neon_poly1305_mod_init(void) { if (!cpu_have_named_feature(ASIMD)) return 0; @@ -221,11 +228,11 @@ static void __exit neon_poly1305_mod_exit(void) { if (IS_REACHABLE(CONFIG_CRYPTO_HASH) && cpu_have_named_feature(ASIMD)) crypto_unregister_shash(&neon_poly1305_alg); } -module_init(neon_poly1305_mod_init); +arch_initcall(neon_poly1305_mod_init); module_exit(neon_poly1305_mod_exit); MODULE_DESCRIPTION("Poly1305 transform using NEON instructions"); MODULE_LICENSE("GPL v2"); MODULE_ALIAS_CRYPTO("poly1305"); diff --git a/arch/mips/crypto/poly1305-glue.c b/arch/mips/crypto/poly1305-glue.c index c03ad0bbe69c..9eee71dc0b99 100644 --- a/arch/mips/crypto/poly1305-glue.c +++ b/arch/mips/crypto/poly1305-glue.c @@ -163,16 +163,22 @@ static struct shash_alg mips_poly1305_alg = { .final = mips_poly1305_final, .digestsize = POLY1305_DIGEST_SIZE, .descsize = sizeof(struct poly1305_desc_ctx), .base.cra_name = "poly1305", - .base.cra_driver_name = "poly1305-mips", + .base.cra_driver_name = "poly1305-mips-old", .base.cra_priority = 200, .base.cra_blocksize = POLY1305_BLOCK_SIZE, .base.cra_module = THIS_MODULE, }; +bool poly1305_is_arch_optimized(void) +{ + return true; +} +EXPORT_SYMBOL(poly1305_is_arch_optimized); + static int __init mips_poly1305_mod_init(void) { return IS_REACHABLE(CONFIG_CRYPTO_HASH) ? crypto_register_shash(&mips_poly1305_alg) : 0; } @@ -181,11 +187,11 @@ static void __exit mips_poly1305_mod_exit(void) { if (IS_REACHABLE(CONFIG_CRYPTO_HASH)) crypto_unregister_shash(&mips_poly1305_alg); } -module_init(mips_poly1305_mod_init); +arch_initcall(mips_poly1305_mod_init); module_exit(mips_poly1305_mod_exit); MODULE_DESCRIPTION("Poly1305 transform (MIPS accelerated"); MODULE_LICENSE("GPL v2"); MODULE_ALIAS_CRYPTO("poly1305"); diff --git a/arch/powerpc/crypto/poly1305-p10-glue.c b/arch/powerpc/crypto/poly1305-p10-glue.c index 87bd5eba4497..40d296d52c23 100644 --- a/arch/powerpc/crypto/poly1305-p10-glue.c +++ b/arch/powerpc/crypto/poly1305-p10-glue.c @@ -111,10 +111,16 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) poly1305_emit_64(&dctx->h, &dctx->s, dst); } EXPORT_SYMBOL(poly1305_final_arch); +bool poly1305_is_arch_optimized(void) +{ + return static_key_enabled(&have_p10); +} +EXPORT_SYMBOL(poly1305_is_arch_optimized); + static int __init poly1305_p10_init(void) { if (cpu_has_feature(CPU_FTR_ARCH_31)) static_branch_enable(&have_p10); return 0; diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c index 8b5593c46da7..40d03a9a4fb5 100644 --- a/arch/x86/crypto/poly1305_glue.c +++ b/arch/x86/crypto/poly1305_glue.c @@ -255,10 +255,16 @@ static struct shash_alg alg = { .cra_blocksize = POLY1305_BLOCK_SIZE, .cra_module = THIS_MODULE, }, }; +bool poly1305_is_arch_optimized(void) +{ + return static_key_enabled(&poly1305_use_avx); +} +EXPORT_SYMBOL(poly1305_is_arch_optimized); + static int __init poly1305_simd_mod_init(void) { if (boot_cpu_has(X86_FEATURE_AVX) && cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) static_branch_enable(&poly1305_use_avx); @@ -278,11 +284,11 @@ static void __exit poly1305_simd_mod_exit(void) { if (IS_REACHABLE(CONFIG_CRYPTO_HASH)) crypto_unregister_shash(&alg); } -module_init(poly1305_simd_mod_init); +arch_initcall(poly1305_simd_mod_init); module_exit(poly1305_simd_mod_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Jason A. Donenfeld "); MODULE_DESCRIPTION("Poly1305 authenticator"); diff --git a/crypto/Makefile b/crypto/Makefile index 98510a2aa0b1..53f5db5f1105 100644 --- a/crypto/Makefile +++ b/crypto/Makefile @@ -148,11 +148,12 @@ obj-$(CONFIG_CRYPTO_KHAZAD) += khazad.o obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_SEED) += seed.o obj-$(CONFIG_CRYPTO_ARIA) += aria_generic.o obj-$(CONFIG_CRYPTO_CHACHA20) += chacha.o CFLAGS_chacha.o += -DARCH=$(ARCH) -obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o +obj-$(CONFIG_CRYPTO_POLY1305) += poly1305.o +CFLAGS_poly1305.o += -DARCH=$(ARCH) obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o obj-$(CONFIG_CRYPTO_CRC32C) += crc32c_generic.o obj-$(CONFIG_CRYPTO_CRC32) += crc32_generic.o CFLAGS_crc32c_generic.o += -DARCH=$(ARCH) diff --git a/crypto/poly1305.c b/crypto/poly1305.c new file mode 100644 index 000000000000..e0436bdc462b --- /dev/null +++ b/crypto/poly1305.c @@ -0,0 +1,152 @@ +/* + * Crypto API wrapper for the Poly1305 library functions + * + * Copyright (C) 2015 Martin Willi + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + */ + +#include +#include +#include +#include +#include +#include + +struct crypto_poly1305_desc_ctx { + struct poly1305_desc_ctx base; + u8 key[POLY1305_KEY_SIZE]; + unsigned int keysize; +}; + +static int crypto_poly1305_init(struct shash_desc *desc) +{ + struct crypto_poly1305_desc_ctx *dctx = shash_desc_ctx(desc); + + dctx->keysize = 0; + return 0; +} + +static int crypto_poly1305_update(struct shash_desc *desc, + const u8 *src, unsigned int srclen, bool arch) +{ + struct crypto_poly1305_desc_ctx *dctx = shash_desc_ctx(desc); + unsigned int bytes; + + /* + * The key is passed as the first 32 "data" bytes. The actual + * poly1305_init() can be called only once the full key is available. + */ + if (dctx->keysize < POLY1305_KEY_SIZE) { + bytes = min(srclen, POLY1305_KEY_SIZE - dctx->keysize); + memcpy(&dctx->key[dctx->keysize], src, bytes); + dctx->keysize += bytes; + if (dctx->keysize < POLY1305_KEY_SIZE) + return 0; + if (arch) + poly1305_init(&dctx->base, dctx->key); + else + poly1305_init_generic(&dctx->base, dctx->key); + src += bytes; + srclen -= bytes; + } + + if (arch) + poly1305_update(&dctx->base, src, srclen); + else + poly1305_update_generic(&dctx->base, src, srclen); + + return 0; +} + +static int crypto_poly1305_update_generic(struct shash_desc *desc, + const u8 *src, unsigned int srclen) +{ + return crypto_poly1305_update(desc, src, srclen, false); +} + +static int crypto_poly1305_update_arch(struct shash_desc *desc, + const u8 *src, unsigned int srclen) +{ + return crypto_poly1305_update(desc, src, srclen, true); +} + +static int crypto_poly1305_final(struct shash_desc *desc, u8 *dst, bool arch) +{ + struct crypto_poly1305_desc_ctx *dctx = shash_desc_ctx(desc); + + if (unlikely(dctx->keysize != POLY1305_KEY_SIZE)) + return -ENOKEY; + + if (arch) + poly1305_final(&dctx->base, dst); + else + poly1305_final_generic(&dctx->base, dst); + memzero_explicit(&dctx->key, sizeof(dctx->key)); + return 0; +} + +static int crypto_poly1305_final_generic(struct shash_desc *desc, u8 *dst) +{ + return crypto_poly1305_final(desc, dst, false); +} + +static int crypto_poly1305_final_arch(struct shash_desc *desc, u8 *dst) +{ + return crypto_poly1305_final(desc, dst, true); +} + +static struct shash_alg poly1305_algs[] = { + { + .base.cra_name = "poly1305", + .base.cra_driver_name = "poly1305-generic", + .base.cra_priority = 100, + .base.cra_blocksize = POLY1305_BLOCK_SIZE, + .base.cra_module = THIS_MODULE, + .digestsize = POLY1305_DIGEST_SIZE, + .init = crypto_poly1305_init, + .update = crypto_poly1305_update_generic, + .final = crypto_poly1305_final_generic, + .descsize = sizeof(struct crypto_poly1305_desc_ctx), + }, + { + .base.cra_name = "poly1305", + .base.cra_driver_name = "poly1305-" __stringify(ARCH), + .base.cra_priority = 300, + .base.cra_blocksize = POLY1305_BLOCK_SIZE, + .base.cra_module = THIS_MODULE, + .digestsize = POLY1305_DIGEST_SIZE, + .init = crypto_poly1305_init, + .update = crypto_poly1305_update_arch, + .final = crypto_poly1305_final_arch, + .descsize = sizeof(struct crypto_poly1305_desc_ctx), + }, +}; + +static int num_algs; + +static int __init poly1305_mod_init(void) +{ + /* register the arch flavours only if they differ from generic */ + num_algs = poly1305_is_arch_optimized() ? 2 : 1; + + return crypto_register_shashes(poly1305_algs, num_algs); +} + +static void __exit poly1305_mod_exit(void) +{ + crypto_unregister_shashes(poly1305_algs, num_algs); +} + +subsys_initcall(poly1305_mod_init); +module_exit(poly1305_mod_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Martin Willi "); +MODULE_DESCRIPTION("Crypto API wrapper for the Poly1305 library functions"); +MODULE_ALIAS_CRYPTO("poly1305"); +MODULE_ALIAS_CRYPTO("poly1305-generic"); +MODULE_ALIAS_CRYPTO("poly1305-" __stringify(ARCH)); diff --git a/crypto/poly1305_generic.c b/crypto/poly1305_generic.c deleted file mode 100644 index e6f29a98725a..000000000000 --- a/crypto/poly1305_generic.c +++ /dev/null @@ -1,149 +0,0 @@ -/* - * Poly1305 authenticator algorithm, RFC7539 - * - * Copyright (C) 2015 Martin Willi - * - * Based on public domain code by Andrew Moon and Daniel J. Bernstein. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - */ - -#include -#include -#include -#include -#include -#include -#include - -static int crypto_poly1305_init(struct shash_desc *desc) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - poly1305_core_init(&dctx->h); - dctx->buflen = 0; - dctx->rset = 0; - dctx->sset = false; - - return 0; -} - -static unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx, - const u8 *src, unsigned int srclen) -{ - if (!dctx->sset) { - if (!dctx->rset && srclen >= POLY1305_BLOCK_SIZE) { - poly1305_core_setkey(&dctx->core_r, src); - src += POLY1305_BLOCK_SIZE; - srclen -= POLY1305_BLOCK_SIZE; - dctx->rset = 2; - } - if (srclen >= POLY1305_BLOCK_SIZE) { - dctx->s[0] = get_unaligned_le32(src + 0); - dctx->s[1] = get_unaligned_le32(src + 4); - dctx->s[2] = get_unaligned_le32(src + 8); - dctx->s[3] = get_unaligned_le32(src + 12); - src += POLY1305_BLOCK_SIZE; - srclen -= POLY1305_BLOCK_SIZE; - dctx->sset = true; - } - } - return srclen; -} - -static void poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, - unsigned int srclen) -{ - unsigned int datalen; - - if (unlikely(!dctx->sset)) { - datalen = crypto_poly1305_setdesckey(dctx, src, srclen); - src += srclen - datalen; - srclen = datalen; - } - - poly1305_core_blocks(&dctx->h, &dctx->core_r, src, - srclen / POLY1305_BLOCK_SIZE, 1); -} - -static int crypto_poly1305_update(struct shash_desc *desc, - const u8 *src, unsigned int srclen) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - unsigned int bytes; - - if (unlikely(dctx->buflen)) { - bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen); - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - srclen -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks(dctx, dctx->buf, - POLY1305_BLOCK_SIZE); - dctx->buflen = 0; - } - } - - if (likely(srclen >= POLY1305_BLOCK_SIZE)) { - poly1305_blocks(dctx, src, srclen); - src += srclen - (srclen % POLY1305_BLOCK_SIZE); - srclen %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(srclen)) { - dctx->buflen = srclen; - memcpy(dctx->buf, src, srclen); - } - - return 0; -} - -static int crypto_poly1305_final(struct shash_desc *desc, u8 *dst) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (unlikely(!dctx->sset)) - return -ENOKEY; - - poly1305_final_generic(dctx, dst); - return 0; -} - -static struct shash_alg poly1305_alg = { - .digestsize = POLY1305_DIGEST_SIZE, - .init = crypto_poly1305_init, - .update = crypto_poly1305_update, - .final = crypto_poly1305_final, - .descsize = sizeof(struct poly1305_desc_ctx), - .base = { - .cra_name = "poly1305", - .cra_driver_name = "poly1305-generic", - .cra_priority = 100, - .cra_blocksize = POLY1305_BLOCK_SIZE, - .cra_module = THIS_MODULE, - }, -}; - -static int __init poly1305_mod_init(void) -{ - return crypto_register_shash(&poly1305_alg); -} - -static void __exit poly1305_mod_exit(void) -{ - crypto_unregister_shash(&poly1305_alg); -} - -subsys_initcall(poly1305_mod_init); -module_exit(poly1305_mod_exit); - -MODULE_LICENSE("GPL"); -MODULE_AUTHOR("Martin Willi "); -MODULE_DESCRIPTION("Poly1305 authenticator"); -MODULE_ALIAS_CRYPTO("poly1305"); -MODULE_ALIAS_CRYPTO("poly1305-generic"); diff --git a/include/crypto/poly1305.h b/include/crypto/poly1305.h index 090692ec3bc7..91444965772a 100644 --- a/include/crypto/poly1305.h +++ b/include/crypto/poly1305.h @@ -94,6 +94,15 @@ static inline void poly1305_final(struct poly1305_desc_ctx *desc, u8 *digest) poly1305_final_arch(desc, digest); else poly1305_final_generic(desc, digest); } +#if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305) +bool poly1305_is_arch_optimized(void); +#else +static inline bool poly1305_is_arch_optimized(void) +{ + return false; +} +#endif + #endif From patchwork Sun Apr 13 04:54:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3A4ACC3601E for ; Sun, 13 Apr 2025 05:03:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=6LFKrjXL+ASEJNAYWGzGxdZzCjPuyh8F8fOUR6TZNK0=; b=KG6k1drwFNfIVdSZ6XkWBFEiiX rTepy/rD0pJR9bckx/XmE76TOwfSOQ5InO2eQDelVmWd0n6jYbjxa9KjUTqUlltg2n9wVYbBP0xLd saMm9gfY7DLbVF4PmtxxvSyzw/zDYtSE1QzucHOp1JGo1cpOMnV0LUWjF4rQYEe/Wdana1A/QjGuM Fs8EGpVzeRMifTd/9x26JmESmRuhoo06xOY8m54kfbGGFvswqeKFr8ngtQ/TEMDdlsy/UDH6CBIH5 S5yVb3LQt627KSNLyCu9fD8vesz13b98RBxmPXllBOp+MhUzGg49pYvQGQAzFagD7BNWSsOHnUZnn H2Rtn6kQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pUk-0000000Gof7-2nQ3; Sun, 13 Apr 2025 05:03:02 +0000 Received: from nyc.source.kernel.org ([2604:1380:45d1:ec00::3]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNe-0000000GnfU-1HxS for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:44 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 33772A40DC5; Sun, 13 Apr 2025 04:50:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10ACDC4CEEC; Sun, 13 Apr 2025 04:55:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520141; bh=hZyztrbmssg+PrHVWx2Hv09SPvgkFndX9/sdkxmFyCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aqBiDJNeYaacGQKLSOrtH0dEmHnz8GG7mwtiNLcT21DVDsTUo4a2IaxaEIwwY+a7x dvmdb7NMOO7O4OucwSb/wmk2V6lsZVQTSYVyXTArnKXrs8+w8OvXrsYRPELDOOAozO Z/A4AacddrqD236XPJvlGqNyhS96sc+eDhjDcoCv2uwOCcNh/oW7sXbh+gNdHoBBkW ggSTugWAcoP8HkugkbFl2TPVoL2KfFoQLFclwOoMmJ2EHXE254NfXXBUOJGpjBVCtA cVqQiGeYDG457JRu4L36AywhK9aOWYtt2rOxA/DkCJV0zHKmer/ll2wBuioClhzJ+o x5dfUmi2MXaNg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 3/9] crypto: arm/poly1305 - remove redundant shash algorithm Date: Sat, 12 Apr 2025 21:54:15 -0700 Message-ID: <20250413045421.55100-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215542_475574_47FD0D82 X-CRM114-Status: GOOD ( 16.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers Since crypto/poly1305.c now registers a poly1305-$(ARCH) shash algorithm that uses the architecture's Poly1305 library functions, individual architectures no longer need to do the same. Therefore, remove the redundant shash algorithm from the arch-specific code and leave just the library functions there. Signed-off-by: Eric Biggers --- arch/arm/crypto/Kconfig | 6 -- arch/arm/crypto/poly1305-glue.c | 167 +------------------------------- 2 files changed, 3 insertions(+), 170 deletions(-) diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index 332f2430beec..3530e7c80793 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -46,18 +46,12 @@ config CRYPTO_NHPOLY1305_NEON Architecture: arm using: - NEON (Advanced SIMD) extensions config CRYPTO_POLY1305_ARM tristate - select CRYPTO_HASH select CRYPTO_ARCH_HAVE_LIB_POLY1305 default CRYPTO_LIB_POLY1305_INTERNAL - help - Poly1305 authenticator algorithm (RFC7539) - - Architecture: arm optionally using - - NEON (Advanced SIMD) extensions config CRYPTO_BLAKE2S_ARM bool "Hash functions: BLAKE2s" select CRYPTO_ARCH_HAVE_LIB_BLAKE2S help diff --git a/arch/arm/crypto/poly1305-glue.c b/arch/arm/crypto/poly1305-glue.c index 238dbf50d05d..6d6998b3ec7e 100644 --- a/arch/arm/crypto/poly1305-glue.c +++ b/arch/arm/crypto/poly1305-glue.c @@ -6,19 +6,16 @@ */ #include #include #include -#include -#include -#include -#include +#include #include #include -#include #include #include +#include void poly1305_init_arm(void *state, const u8 *key); void poly1305_blocks_arm(void *state, const u8 *src, u32 len, u32 hibit); void poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit); void poly1305_emit_arm(void *state, u8 *digest, const u32 *nonce); @@ -38,106 +35,10 @@ void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KE dctx->s[3] = get_unaligned_le32(key + 28); dctx->buflen = 0; } EXPORT_SYMBOL(poly1305_init_arch); -static int arm_poly1305_init(struct shash_desc *desc) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - dctx->buflen = 0; - dctx->rset = 0; - dctx->sset = false; - - return 0; -} - -static void arm_poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, - u32 len, u32 hibit, bool do_neon) -{ - if (unlikely(!dctx->sset)) { - if (!dctx->rset) { - poly1305_init_arm(&dctx->h, src); - src += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - dctx->rset = 1; - } - if (len >= POLY1305_BLOCK_SIZE) { - dctx->s[0] = get_unaligned_le32(src + 0); - dctx->s[1] = get_unaligned_le32(src + 4); - dctx->s[2] = get_unaligned_le32(src + 8); - dctx->s[3] = get_unaligned_le32(src + 12); - src += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - dctx->sset = true; - } - if (len < POLY1305_BLOCK_SIZE) - return; - } - - len &= ~(POLY1305_BLOCK_SIZE - 1); - - if (static_branch_likely(&have_neon) && likely(do_neon)) - poly1305_blocks_neon(&dctx->h, src, len, hibit); - else - poly1305_blocks_arm(&dctx->h, src, len, hibit); -} - -static void arm_poly1305_do_update(struct poly1305_desc_ctx *dctx, - const u8 *src, u32 len, bool do_neon) -{ - if (unlikely(dctx->buflen)) { - u32 bytes = min(len, POLY1305_BLOCK_SIZE - dctx->buflen); - - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - len -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - arm_poly1305_blocks(dctx, dctx->buf, - POLY1305_BLOCK_SIZE, 1, false); - dctx->buflen = 0; - } - } - - if (likely(len >= POLY1305_BLOCK_SIZE)) { - arm_poly1305_blocks(dctx, src, len, 1, do_neon); - src += round_down(len, POLY1305_BLOCK_SIZE); - len %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(len)) { - dctx->buflen = len; - memcpy(dctx->buf, src, len); - } -} - -static int arm_poly1305_update(struct shash_desc *desc, - const u8 *src, unsigned int srclen) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - arm_poly1305_do_update(dctx, src, srclen, false); - return 0; -} - -static int __maybe_unused arm_poly1305_update_neon(struct shash_desc *desc, - const u8 *src, - unsigned int srclen) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - bool do_neon = crypto_simd_usable() && srclen > 128; - - if (static_branch_likely(&have_neon) && do_neon) - kernel_neon_begin(); - arm_poly1305_do_update(dctx, src, srclen, do_neon); - if (static_branch_likely(&have_neon) && do_neon) - kernel_neon_end(); - return 0; -} - void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, unsigned int nbytes) { bool do_neon = IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && crypto_simd_usable(); @@ -197,49 +98,10 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) poly1305_emit_arm(&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx){}; } EXPORT_SYMBOL(poly1305_final_arch); -static int arm_poly1305_final(struct shash_desc *desc, u8 *dst) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (unlikely(!dctx->sset)) - return -ENOKEY; - - poly1305_final_arch(dctx, dst); - return 0; -} - -static struct shash_alg arm_poly1305_algs[] = {{ - .init = arm_poly1305_init, - .update = arm_poly1305_update, - .final = arm_poly1305_final, - .digestsize = POLY1305_DIGEST_SIZE, - .descsize = sizeof(struct poly1305_desc_ctx), - - .base.cra_name = "poly1305", - .base.cra_driver_name = "poly1305-arm-old", - .base.cra_priority = 150, - .base.cra_blocksize = POLY1305_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -#ifdef CONFIG_KERNEL_MODE_NEON -}, { - .init = arm_poly1305_init, - .update = arm_poly1305_update_neon, - .final = arm_poly1305_final, - .digestsize = POLY1305_DIGEST_SIZE, - .descsize = sizeof(struct poly1305_desc_ctx), - - .base.cra_name = "poly1305", - .base.cra_driver_name = "poly1305-neon", - .base.cra_priority = 200, - .base.cra_blocksize = POLY1305_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -#endif -}}; - bool poly1305_is_arch_optimized(void) { /* We always can use at least the ARM scalar implementation. */ return true; } @@ -248,34 +110,11 @@ EXPORT_SYMBOL(poly1305_is_arch_optimized); static int __init arm_poly1305_mod_init(void) { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && (elf_hwcap & HWCAP_NEON)) static_branch_enable(&have_neon); - else if (IS_REACHABLE(CONFIG_CRYPTO_HASH)) - /* register only the first entry */ - return crypto_register_shash(&arm_poly1305_algs[0]); - - return IS_REACHABLE(CONFIG_CRYPTO_HASH) ? - crypto_register_shashes(arm_poly1305_algs, - ARRAY_SIZE(arm_poly1305_algs)) : 0; -} - -static void __exit arm_poly1305_mod_exit(void) -{ - if (!IS_REACHABLE(CONFIG_CRYPTO_HASH)) - return; - if (!static_branch_likely(&have_neon)) { - crypto_unregister_shash(&arm_poly1305_algs[0]); - return; - } - crypto_unregister_shashes(arm_poly1305_algs, - ARRAY_SIZE(arm_poly1305_algs)); + return 0; } - arch_initcall(arm_poly1305_mod_init); -module_exit(arm_poly1305_mod_exit); MODULE_DESCRIPTION("Accelerated Poly1305 transform for ARM"); MODULE_LICENSE("GPL v2"); -MODULE_ALIAS_CRYPTO("poly1305"); -MODULE_ALIAS_CRYPTO("poly1305-arm"); -MODULE_ALIAS_CRYPTO("poly1305-neon"); From patchwork Sun Apr 13 04:54:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3DBD9C3601E for ; Sun, 13 Apr 2025 05:01:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ghHI1ua22909A9AO+A0V2TeTXm1nvlLoI7Gwe5xVBGg=; b=YKt4Ck3Ik3dVXQS9aPD2M7H+jI RocpOhb16Zm40mmealUr2V2PlgfSkKxO9VzjQ5h5Xxy29AuRR9pvkjjbh2n2L8t3tv9ZsPROD7UEW CXUZhE0ImvU/sAqGZA7ZDthxnCs4QMEaNP0lgD4ze7rIkgj0SL02QZVx3rSeAdDNM3UyRUeG7gndX MXydmgobDdW4BgaAdrEjEzRFNCOkZFf+8UenBANrzykX3K+bCs0QhhS/CiQrLn3DBTivxRwLyq5lB PuQ5exfI8dQZsmBZxgc7MTdf9ynHXk/6Sb/bng1UIeUQ90kIk0hp6zQfkdVH8JkcfvR/T1smk35Fp eJLj/GOw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pSy-0000000GoZA-3rCj; Sun, 13 Apr 2025 05:01:12 +0000 Received: from nyc.source.kernel.org ([2604:1380:45d1:ec00::3]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNe-0000000Gng1-3ims for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:44 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id C1C63A40E71; Sun, 13 Apr 2025 04:50:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A6BB1C4CEEF; Sun, 13 Apr 2025 04:55:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520142; bh=9FIfB3SWlQ3XpNk3Dx8kombdj2eN1jWYviON2vPLySA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qurcntK5oD7ir9QIYy87uFOTG7DZ5ai9xFFMpiTEhR/j4EdtY12M1OVTdVtw/NJsu rsi1wNoL03woAdSu5nYj1OYQmu0DYRKWlULNk+FTWgaf2Zblpp5kLX7C7phu3fINkJ FaacxnBSITxJUiMkcoVJmZVe9NRVMocxReuTm9Pemzo9v6gGJxHgF6uHuS4P1oa0cy e52Y1ytSrBM2Ff5pvBoMUuPsuevuOXObpmTux9yE2jVJ0ejIcew5LdtJJw2W2JfmFM qFMw+xdBSJRuCKY61p/OVXS05LkzuFyjgccxIl3qFqMfmh5X9gWRb5IiP+vnLiXMN2 d51Zcu6bAdL7w== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 4/9] crypto: arm64/poly1305 - remove redundant shash algorithm Date: Sat, 12 Apr 2025 21:54:16 -0700 Message-ID: <20250413045421.55100-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215543_054579_4E068661 X-CRM114-Status: GOOD ( 16.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers Since crypto/poly1305.c now registers a poly1305-$(ARCH) shash algorithm that uses the architecture's Poly1305 library functions, individual architectures no longer need to do the same. Therefore, remove the redundant shash algorithm from the arch-specific code and leave just the library functions there. Signed-off-by: Eric Biggers --- arch/arm64/crypto/Kconfig | 6 -- arch/arm64/crypto/poly1305-glue.c | 140 ++---------------------------- 2 files changed, 6 insertions(+), 140 deletions(-) diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index ce655da0fbee..c47a7649a241 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -26,18 +26,12 @@ config CRYPTO_NHPOLY1305_NEON - NEON (Advanced SIMD) extensions config CRYPTO_POLY1305_NEON tristate depends on KERNEL_MODE_NEON - select CRYPTO_HASH select CRYPTO_ARCH_HAVE_LIB_POLY1305 default CRYPTO_LIB_POLY1305_INTERNAL - help - Poly1305 authenticator algorithm (RFC7539) - - Architecture: arm64 using: - - NEON (Advanced SIMD) extensions config CRYPTO_SHA1_ARM64_CE tristate "Hash functions: SHA-1 (ARMv8 Crypto Extensions)" depends on KERNEL_MODE_NEON select CRYPTO_HASH diff --git a/arch/arm64/crypto/poly1305-glue.c b/arch/arm64/crypto/poly1305-glue.c index 89415731d9be..cb152ceac14a 100644 --- a/arch/arm64/crypto/poly1305-glue.c +++ b/arch/arm64/crypto/poly1305-glue.c @@ -6,19 +6,16 @@ */ #include #include #include -#include -#include -#include -#include +#include #include #include -#include #include #include +#include asmlinkage void poly1305_init_arm64(void *state, const u8 *key); asmlinkage void poly1305_blocks(void *state, const u8 *src, u32 len, u32 hibit); asmlinkage void poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit); asmlinkage void poly1305_emit(void *state, u8 *digest, const u32 *nonce); @@ -34,96 +31,10 @@ void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KE dctx->s[3] = get_unaligned_le32(key + 28); dctx->buflen = 0; } EXPORT_SYMBOL(poly1305_init_arch); -static int neon_poly1305_init(struct shash_desc *desc) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - dctx->buflen = 0; - dctx->rset = 0; - dctx->sset = false; - - return 0; -} - -static void neon_poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, - u32 len, u32 hibit, bool do_neon) -{ - if (unlikely(!dctx->sset)) { - if (!dctx->rset) { - poly1305_init_arm64(&dctx->h, src); - src += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - dctx->rset = 1; - } - if (len >= POLY1305_BLOCK_SIZE) { - dctx->s[0] = get_unaligned_le32(src + 0); - dctx->s[1] = get_unaligned_le32(src + 4); - dctx->s[2] = get_unaligned_le32(src + 8); - dctx->s[3] = get_unaligned_le32(src + 12); - src += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - dctx->sset = true; - } - if (len < POLY1305_BLOCK_SIZE) - return; - } - - len &= ~(POLY1305_BLOCK_SIZE - 1); - - if (static_branch_likely(&have_neon) && likely(do_neon)) - poly1305_blocks_neon(&dctx->h, src, len, hibit); - else - poly1305_blocks(&dctx->h, src, len, hibit); -} - -static void neon_poly1305_do_update(struct poly1305_desc_ctx *dctx, - const u8 *src, u32 len, bool do_neon) -{ - if (unlikely(dctx->buflen)) { - u32 bytes = min(len, POLY1305_BLOCK_SIZE - dctx->buflen); - - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - len -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - neon_poly1305_blocks(dctx, dctx->buf, - POLY1305_BLOCK_SIZE, 1, false); - dctx->buflen = 0; - } - } - - if (likely(len >= POLY1305_BLOCK_SIZE)) { - neon_poly1305_blocks(dctx, src, len, 1, do_neon); - src += round_down(len, POLY1305_BLOCK_SIZE); - len %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(len)) { - dctx->buflen = len; - memcpy(dctx->buf, src, len); - } -} - -static int neon_poly1305_update(struct shash_desc *desc, - const u8 *src, unsigned int srclen) -{ - bool do_neon = crypto_simd_usable() && srclen > 128; - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (static_branch_likely(&have_neon) && do_neon) - kernel_neon_begin(); - neon_poly1305_do_update(dctx, src, srclen, do_neon); - if (static_branch_likely(&have_neon) && do_neon) - kernel_neon_end(); - return 0; -} - void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, unsigned int nbytes) { if (unlikely(dctx->buflen)) { u32 bytes = min(nbytes, POLY1305_BLOCK_SIZE - dctx->buflen); @@ -179,61 +90,22 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) poly1305_emit(&dctx->h, dst, dctx->s); memzero_explicit(dctx, sizeof(*dctx)); } EXPORT_SYMBOL(poly1305_final_arch); -static int neon_poly1305_final(struct shash_desc *desc, u8 *dst) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (unlikely(!dctx->sset)) - return -ENOKEY; - - poly1305_final_arch(dctx, dst); - return 0; -} - -static struct shash_alg neon_poly1305_alg = { - .init = neon_poly1305_init, - .update = neon_poly1305_update, - .final = neon_poly1305_final, - .digestsize = POLY1305_DIGEST_SIZE, - .descsize = sizeof(struct poly1305_desc_ctx), - - .base.cra_name = "poly1305", - .base.cra_driver_name = "poly1305-neon", - .base.cra_priority = 200, - .base.cra_blocksize = POLY1305_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}; - bool poly1305_is_arch_optimized(void) { /* We always can use at least the ARM64 scalar implementation. */ return true; } EXPORT_SYMBOL(poly1305_is_arch_optimized); static int __init neon_poly1305_mod_init(void) { - if (!cpu_have_named_feature(ASIMD)) - return 0; - - static_branch_enable(&have_neon); - - return IS_REACHABLE(CONFIG_CRYPTO_HASH) ? - crypto_register_shash(&neon_poly1305_alg) : 0; -} - -static void __exit neon_poly1305_mod_exit(void) -{ - if (IS_REACHABLE(CONFIG_CRYPTO_HASH) && cpu_have_named_feature(ASIMD)) - crypto_unregister_shash(&neon_poly1305_alg); + if (cpu_have_named_feature(ASIMD)) + static_branch_enable(&have_neon); + return 0; } - arch_initcall(neon_poly1305_mod_init); -module_exit(neon_poly1305_mod_exit); -MODULE_DESCRIPTION("Poly1305 transform using NEON instructions"); +MODULE_DESCRIPTION("Poly1305 authenticator (ARM64 optimized)"); MODULE_LICENSE("GPL v2"); -MODULE_ALIAS_CRYPTO("poly1305"); -MODULE_ALIAS_CRYPTO("poly1305-neon"); From patchwork Sun Apr 13 04:54:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF44AC3601E for ; Sun, 13 Apr 2025 05:05:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mfq8NUl2WvJrEeqpsBeuYsFmw+cnTBFnk1Up45/MZ2k=; b=Rq4gGzJ8ONUDbLrHyvVUy/Y826 yR4G9VpdiuRbnv/6P3muyw1f/m+dF+2v0vvI1ntQ++jbpiE96+PwSrMpcBKYzdnMrxtBV+WvKLYXT DfTcuxryD1fFNo+5CbLKnRztwug6n7eNAuIDV+BwEnv01zUuvK35sNQT6tDgBXRL0VkmKf3e3ToWa ADc9IwqYcdrQjhUlN88yAtl8Kh1krW0P8I20zjpvwcL1CGGTzf7CeABEzlGcuNoNUQN+a74xieEzc 5VJalPXqjxx1jM7UCsKUPBC+H5Gsn2dztnsXGvJ2+MYli/TnHz75YExTSc18zVDK8yM/C6ns8jl1K dJAPjp2A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pWW-0000000GomO-1PJ4; Sun, 13 Apr 2025 05:04:52 +0000 Received: from tor.source.kernel.org ([2600:3c04:e001:324:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNg-0000000Gnhr-2ekh for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:44 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id C80DA6113B; Sun, 13 Apr 2025 04:55:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 41DD0C4CEE5; Sun, 13 Apr 2025 04:55:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520142; bh=S6EtgyS23+qRGH0eAeuHrJro1hFgR1JlJgbog1ls+WE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ASqL0WN/Mm67yTnOT6LYSFTru/q7hLEducze6iLetJNyVrFU28EBOBehoKHuj4Oxx bjNrhBx1GKB0g3RiBz6f2ge6+haqU+3AGx/seBNICYpwgrSAkVxX7uFXlRysnKMm/g O+De6pWgmW8F/TYbosTH+Sg3Ys6WTqGBK6jlwM2pY+X/gK90/5I10kVKFa28LkyYU6 RglM1wB2KVZKfoRMWa8zL7zk6LfUOR7wBBoj+G0Zm9GzXpR/PH6kNHUuCBkVvRirRB lrHTms4JzO/mMetcYLgGd6h4HxCTgRyvHehtFp2NG6lAZyQce9YRhNenb0nqAWOdxQ sEgZopVA4Pd9Q== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 5/9] crypto: mips/poly1305 - drop redundant dependency on CONFIG_MIPS Date: Sat, 12 Apr 2025 21:54:17 -0700 Message-ID: <20250413045421.55100-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers arch/mips/crypto/Kconfig is sourced only when CONFIG_MIPS is enabled, so there is no need for options defined in that file to depend on it. Signed-off-by: Eric Biggers --- arch/mips/crypto/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/mips/crypto/Kconfig b/arch/mips/crypto/Kconfig index 0189686de3a1..9892391f74ac 100644 --- a/arch/mips/crypto/Kconfig +++ b/arch/mips/crypto/Kconfig @@ -2,11 +2,10 @@ menu "Accelerated Cryptographic Algorithms for CPU (mips)" config CRYPTO_POLY1305_MIPS tristate - depends on MIPS select CRYPTO_HASH select CRYPTO_ARCH_HAVE_LIB_POLY1305 default CRYPTO_LIB_POLY1305_INTERNAL help Poly1305 authenticator algorithm (RFC7539) From patchwork Sun Apr 13 04:54:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 10E22C3601E for ; Sun, 13 Apr 2025 05:08:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9Rtv5yf9SdcNfi/Q6caE6xtWhSbobaxGzn3ctMwJWPU=; b=ub3pzerEIQEINTHiiYBz3vuHQW 3R1AbUVVO8PixMyumhKR8a7A6CPh5Ta4HTPdJ4ciUOnWs3fXvDHrDPqioZ1+LpIeukn9dJk2NV79N R9DfpPbVlOoSKBohUsjnBSydWIqNOp/qIQqmMoqq+Q7ZnJWYDl/CIj+AHJcbhZ8fVAXdYnfSSzi7F vmS6OGRzhuCfz9Ogjl5h+ftKqCenlg1TaX13OhS4IhpOFDuiCDLkF6QEmNeqZp6wpzJrrWhohwKCa utwzq6qAkzqlIuRVuR+7yy188mjce8/95uqoGebLzkHlRlGkSmmA5UWRvcPZC+HGZc/XwfI7QVRF6 LccYOPYA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pa3-0000000Gozh-34J5; Sun, 13 Apr 2025 05:08:31 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNf-0000000Gnh1-2PgR for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:45 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 5C54243F31; Sun, 13 Apr 2025 04:55:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE7ACC4CEEB; Sun, 13 Apr 2025 04:55:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520143; bh=Uv2ivhKGeSVnJttJbFk66Y1ps6xLRxh4yCdmxnmogHY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KY5b4L7n1HqepIWAPKLs1qrotLq2c54a5O7KU0785rvEUYPKN6iDwBm2H9Smjz+s6 Yp9mmMGANuGpo/CTioxu8qeRKTEeTTXAwR1vcwSjvNdNPyCLvrNgbFhEyEhWF7NOkE 96qdSnf50bYKRMWykgRovqVBmMjPND0lspd9/w1Bni+H1UCw/RhJwnvbeZFsQ02x0F k3OIIZ9PGOglVBsOXXM8G3qUCsQ+w3yAwd0V9G0+7/9IuxjMHA698IpLzGy6QH7MGw Ju8HUR/Ph7QsHlMeyL/wpDb+EivNONqlMHtGfyoc0HwNf2SxWk0SP4w3IA54l0olOs LfGZUrCVQQ0pg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 6/9] crypto: mips/poly1305 - remove redundant shash algorithm Date: Sat, 12 Apr 2025 21:54:18 -0700 Message-ID: <20250413045421.55100-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215543_682608_E6089F1D X-CRM114-Status: GOOD ( 13.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers Since crypto/poly1305.c now registers a poly1305-$(ARCH) shash algorithm that uses the architecture's Poly1305 library functions, individual architectures no longer need to do the same. Therefore, remove the redundant shash algorithm from the arch-specific code and leave just the library functions there. Signed-off-by: Eric Biggers --- arch/mips/crypto/Kconfig | 5 -- arch/mips/crypto/poly1305-glue.c | 120 +------------------------------ 2 files changed, 2 insertions(+), 123 deletions(-) diff --git a/arch/mips/crypto/Kconfig b/arch/mips/crypto/Kconfig index 9892391f74ac..8283664a1f24 100644 --- a/arch/mips/crypto/Kconfig +++ b/arch/mips/crypto/Kconfig @@ -2,17 +2,12 @@ menu "Accelerated Cryptographic Algorithms for CPU (mips)" config CRYPTO_POLY1305_MIPS tristate - select CRYPTO_HASH select CRYPTO_ARCH_HAVE_LIB_POLY1305 default CRYPTO_LIB_POLY1305_INTERNAL - help - Poly1305 authenticator algorithm (RFC7539) - - Architecture: mips config CRYPTO_MD5_OCTEON tristate "Digests: MD5 (OCTEON)" depends on CPU_CAVIUM_OCTEON select CRYPTO_MD5 diff --git a/arch/mips/crypto/poly1305-glue.c b/arch/mips/crypto/poly1305-glue.c index 9eee71dc0b99..576e7a58e0b1 100644 --- a/arch/mips/crypto/poly1305-glue.c +++ b/arch/mips/crypto/poly1305-glue.c @@ -3,17 +3,14 @@ * OpenSSL/Cryptogams accelerated Poly1305 transform for MIPS * * Copyright (C) 2019 Linaro Ltd. */ -#include -#include -#include -#include +#include #include -#include #include +#include asmlinkage void poly1305_init_mips(void *state, const u8 *key); asmlinkage void poly1305_blocks_mips(void *state, const u8 *src, u32 len, u32 hibit); asmlinkage void poly1305_emit_mips(void *state, u8 *digest, const u32 *nonce); @@ -26,81 +23,10 @@ void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KE dctx->s[3] = get_unaligned_le32(key + 28); dctx->buflen = 0; } EXPORT_SYMBOL(poly1305_init_arch); -static int mips_poly1305_init(struct shash_desc *desc) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - dctx->buflen = 0; - dctx->rset = 0; - dctx->sset = false; - - return 0; -} - -static void mips_poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, - u32 len, u32 hibit) -{ - if (unlikely(!dctx->sset)) { - if (!dctx->rset) { - poly1305_init_mips(&dctx->h, src); - src += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - dctx->rset = 1; - } - if (len >= POLY1305_BLOCK_SIZE) { - dctx->s[0] = get_unaligned_le32(src + 0); - dctx->s[1] = get_unaligned_le32(src + 4); - dctx->s[2] = get_unaligned_le32(src + 8); - dctx->s[3] = get_unaligned_le32(src + 12); - src += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - dctx->sset = true; - } - if (len < POLY1305_BLOCK_SIZE) - return; - } - - len &= ~(POLY1305_BLOCK_SIZE - 1); - - poly1305_blocks_mips(&dctx->h, src, len, hibit); -} - -static int mips_poly1305_update(struct shash_desc *desc, const u8 *src, - unsigned int len) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (unlikely(dctx->buflen)) { - u32 bytes = min(len, POLY1305_BLOCK_SIZE - dctx->buflen); - - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - len -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - mips_poly1305_blocks(dctx, dctx->buf, POLY1305_BLOCK_SIZE, 1); - dctx->buflen = 0; - } - } - - if (likely(len >= POLY1305_BLOCK_SIZE)) { - mips_poly1305_blocks(dctx, src, len, 1); - src += round_down(len, POLY1305_BLOCK_SIZE); - len %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(len)) { - dctx->buflen = len; - memcpy(dctx->buf, src, len); - } - return 0; -} - void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, unsigned int nbytes) { if (unlikely(dctx->buflen)) { u32 bytes = min(nbytes, POLY1305_BLOCK_SIZE - dctx->buflen); @@ -144,55 +70,13 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) poly1305_emit_mips(&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx){}; } EXPORT_SYMBOL(poly1305_final_arch); -static int mips_poly1305_final(struct shash_desc *desc, u8 *dst) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (unlikely(!dctx->sset)) - return -ENOKEY; - - poly1305_final_arch(dctx, dst); - return 0; -} - -static struct shash_alg mips_poly1305_alg = { - .init = mips_poly1305_init, - .update = mips_poly1305_update, - .final = mips_poly1305_final, - .digestsize = POLY1305_DIGEST_SIZE, - .descsize = sizeof(struct poly1305_desc_ctx), - - .base.cra_name = "poly1305", - .base.cra_driver_name = "poly1305-mips-old", - .base.cra_priority = 200, - .base.cra_blocksize = POLY1305_BLOCK_SIZE, - .base.cra_module = THIS_MODULE, -}; - bool poly1305_is_arch_optimized(void) { return true; } EXPORT_SYMBOL(poly1305_is_arch_optimized); -static int __init mips_poly1305_mod_init(void) -{ - return IS_REACHABLE(CONFIG_CRYPTO_HASH) ? - crypto_register_shash(&mips_poly1305_alg) : 0; -} - -static void __exit mips_poly1305_mod_exit(void) -{ - if (IS_REACHABLE(CONFIG_CRYPTO_HASH)) - crypto_unregister_shash(&mips_poly1305_alg); -} - -arch_initcall(mips_poly1305_mod_init); -module_exit(mips_poly1305_mod_exit); - MODULE_DESCRIPTION("Poly1305 transform (MIPS accelerated"); MODULE_LICENSE("GPL v2"); -MODULE_ALIAS_CRYPTO("poly1305"); -MODULE_ALIAS_CRYPTO("poly1305-mips"); From patchwork Sun Apr 13 04:54:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB4A8C3601E for ; Sun, 13 Apr 2025 05:10:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WlWBjj/lxGNOH0Q4D7F6zVnwaN37HigKAHiZelMg+o8=; b=0klziHrOlBkOqc54/a/e5F3glC nHXbq+/RnFjwg5Q5WOCE+ezNUvsuti7elY/a/Qnd2YbHL0cmyk8NmsZ/BSOZdYajZvJX4ej6nPTaV nZeMxCYD6UHEPDxkYrdtYn4HI9TtW6QGsfpudsVdvefCAV0Yisuw7frru7XEmL2twjHqLwAXyqMBw 5DQRzAzYo/C8kP77u7d9yEZOi+xRm6Sqv0Zq+tBO6UMz/4mncsa2RGWdF3hHriDJcgJpTGHTm4HNU icjnOm822CSHLUsDJ0GVV2vJhZ89kLucgeibrK00zyBvwOQTyMW1XDmF3vlGDJO2/DGeIcxpurS0Z 94WZCZ4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pbp-0000000Gp6c-1XF4; Sun, 13 Apr 2025 05:10:21 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNg-0000000Gnht-0dRa for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:45 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id EDF3E4A374; Sun, 13 Apr 2025 04:55:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63D36C4CEEC; Sun, 13 Apr 2025 04:55:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520143; bh=pU25hoOE+o9gz8inrSmm1Ozslc+pQV84IIV/XOU2PEM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EOmwlI343XmwmzZIUWOVNU7G6Cqc8PiuW2n1l5i5/IAWq1XfjzPCN+LX+VY8dR9UP NehEwfxN4bhQgNu//RpRBAA6146hPT1tzcjQGZ5evFj2BMa9iRgz3DcOKM5y4iOsTY J7AX0FybsD6Ty2fMP1wJO0DyHmh0CDfMKCWQuMg7hIMprIjy+TxOfx7wp4VmYU3UUY vdo1y3P28uc/9ngMZzd6Q/qa9yEHDJZtw+jqLEVcfeg2ntSD5ZNU5q6coLGOS6FMQf 9L66jYsGLo5vT9yNqcqINUchcyHS66J9MnMBjUoo+q/xA50PvKqOAf49rW8g+lbdwP hh93Vi0qDdIDQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 7/9] crypto: x86/poly1305 - remove redundant shash algorithm Date: Sat, 12 Apr 2025 21:54:19 -0700 Message-ID: <20250413045421.55100-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215544_247300_BDBC9E2F X-CRM114-Status: GOOD ( 14.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers Since crypto/poly1305.c now registers a poly1305-$(ARCH) shash algorithm that uses the architecture's Poly1305 library functions, individual architectures no longer need to do the same. Therefore, remove the redundant shash algorithm from the arch-specific code and leave just the library functions there. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 7 --- arch/x86/crypto/poly1305_glue.c | 97 +++------------------------------ 2 files changed, 7 insertions(+), 97 deletions(-) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index 0cec75926380..31804dacf9a4 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -397,20 +397,13 @@ config CRYPTO_POLYVAL_CLMUL_NI - CLMUL-NI (carry-less multiplication new instructions) config CRYPTO_POLY1305_X86_64 tristate depends on X86 && 64BIT - select CRYPTO_HASH select CRYPTO_LIB_POLY1305_GENERIC select CRYPTO_ARCH_HAVE_LIB_POLY1305 default CRYPTO_LIB_POLY1305_INTERNAL - help - Poly1305 authenticator algorithm (RFC7539) - - Architecture: x86_64 using: - - SSE2 (Streaming SIMD Extensions 2) - - AVX2 (Advanced Vector Extensions 2) config CRYPTO_SHA1_SSSE3 tristate "Hash functions: SHA-1 (SSSE3/AVX/AVX2/SHA-NI)" depends on X86 && 64BIT select CRYPTO_SHA1 diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c index 40d03a9a4fb5..a0fc543a0d68 100644 --- a/arch/x86/crypto/poly1305_glue.c +++ b/arch/x86/crypto/poly1305_glue.c @@ -1,19 +1,17 @@ // SPDX-License-Identifier: GPL-2.0 OR MIT /* * Copyright (C) 2015-2019 Jason A. Donenfeld . All Rights Reserved. */ -#include -#include -#include #include -#include +#include #include #include #include #include +#include #include #include asmlinkage void poly1305_init_x86_64(void *ctx, const u8 key[POLY1305_BLOCK_SIZE]); @@ -135,64 +133,36 @@ void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KE dctx->s[0] = get_unaligned_le32(&key[16]); dctx->s[1] = get_unaligned_le32(&key[20]); dctx->s[2] = get_unaligned_le32(&key[24]); dctx->s[3] = get_unaligned_le32(&key[28]); dctx->buflen = 0; - dctx->sset = true; } EXPORT_SYMBOL(poly1305_init_arch); -static unsigned int crypto_poly1305_setdctxkey(struct poly1305_desc_ctx *dctx, - const u8 *inp, unsigned int len) -{ - unsigned int acc = 0; - if (unlikely(!dctx->sset)) { - if (!dctx->rset && len >= POLY1305_BLOCK_SIZE) { - poly1305_simd_init(&dctx->h, inp); - inp += POLY1305_BLOCK_SIZE; - len -= POLY1305_BLOCK_SIZE; - acc += POLY1305_BLOCK_SIZE; - dctx->rset = 1; - } - if (len >= POLY1305_BLOCK_SIZE) { - dctx->s[0] = get_unaligned_le32(&inp[0]); - dctx->s[1] = get_unaligned_le32(&inp[4]); - dctx->s[2] = get_unaligned_le32(&inp[8]); - dctx->s[3] = get_unaligned_le32(&inp[12]); - acc += POLY1305_BLOCK_SIZE; - dctx->sset = true; - } - } - return acc; -} - void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, unsigned int srclen) { - unsigned int bytes, used; + unsigned int bytes; if (unlikely(dctx->buflen)) { bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen); memcpy(dctx->buf + dctx->buflen, src, bytes); src += bytes; srclen -= bytes; dctx->buflen += bytes; if (dctx->buflen == POLY1305_BLOCK_SIZE) { - if (likely(!crypto_poly1305_setdctxkey(dctx, dctx->buf, POLY1305_BLOCK_SIZE))) - poly1305_simd_blocks(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 1); + poly1305_simd_blocks(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 1); dctx->buflen = 0; } } if (likely(srclen >= POLY1305_BLOCK_SIZE)) { bytes = round_down(srclen, POLY1305_BLOCK_SIZE); - srclen -= bytes; - used = crypto_poly1305_setdctxkey(dctx, src, bytes); - if (likely(bytes - used)) - poly1305_simd_blocks(&dctx->h, src + used, bytes - used, 1); + poly1305_simd_blocks(&dctx->h, src, bytes, 1); src += bytes; + srclen -= bytes; } if (unlikely(srclen)) { dctx->buflen = srclen; memcpy(dctx->buf, src, srclen); @@ -212,53 +182,10 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) poly1305_simd_emit(&dctx->h, dst, dctx->s); memzero_explicit(dctx, sizeof(*dctx)); } EXPORT_SYMBOL(poly1305_final_arch); -static int crypto_poly1305_init(struct shash_desc *desc) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - *dctx = (struct poly1305_desc_ctx){}; - return 0; -} - -static int crypto_poly1305_update(struct shash_desc *desc, - const u8 *src, unsigned int srclen) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - poly1305_update_arch(dctx, src, srclen); - return 0; -} - -static int crypto_poly1305_final(struct shash_desc *desc, u8 *dst) -{ - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); - - if (unlikely(!dctx->sset)) - return -ENOKEY; - - poly1305_final_arch(dctx, dst); - return 0; -} - -static struct shash_alg alg = { - .digestsize = POLY1305_DIGEST_SIZE, - .init = crypto_poly1305_init, - .update = crypto_poly1305_update, - .final = crypto_poly1305_final, - .descsize = sizeof(struct poly1305_desc_ctx), - .base = { - .cra_name = "poly1305", - .cra_driver_name = "poly1305-simd", - .cra_priority = 300, - .cra_blocksize = POLY1305_BLOCK_SIZE, - .cra_module = THIS_MODULE, - }, -}; - bool poly1305_is_arch_optimized(void) { return static_key_enabled(&poly1305_use_avx); } EXPORT_SYMBOL(poly1305_is_arch_optimized); @@ -275,22 +202,12 @@ static int __init poly1305_simd_mod_init(void) boot_cpu_has(X86_FEATURE_AVX512F) && cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | XFEATURE_MASK_AVX512, NULL) && /* Skylake downclocks unacceptably much when using zmm, but later generations are fast. */ boot_cpu_data.x86_vfm != INTEL_SKYLAKE_X) static_branch_enable(&poly1305_use_avx512); - return IS_REACHABLE(CONFIG_CRYPTO_HASH) ? crypto_register_shash(&alg) : 0; -} - -static void __exit poly1305_simd_mod_exit(void) -{ - if (IS_REACHABLE(CONFIG_CRYPTO_HASH)) - crypto_unregister_shash(&alg); + return 0; } - arch_initcall(poly1305_simd_mod_init); -module_exit(poly1305_simd_mod_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Jason A. Donenfeld "); MODULE_DESCRIPTION("Poly1305 authenticator"); -MODULE_ALIAS_CRYPTO("poly1305"); -MODULE_ALIAS_CRYPTO("poly1305-simd"); From patchwork Sun Apr 13 04:54:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63CEAC3601E for ; Sun, 13 Apr 2025 05:12:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7dxNG1C6BKTk+JwpAc0vXAIATHvUqCHfYjRvMQl1234=; b=bwa25n14oKEyi9j9ztGkOZpZno xh3S2jnZy0UISg/mGTbPt9r7YJG8vyH6flBaNkMQGXKY3EK5iRbnr0uprwTfQ1FFr1V/AhsN2rU2u 2Sq9dXVoOKw3cGGx9mvX+tOqYBCKFmqoEicZVKJfzm9bXPBg7UbNAQPlutosViNvdi30CVnraN1/4 aBH44i4M9ZaYhoeuuJvNn5Ki9CsdWJIAQlAXllHcRJtdslknRJ3NwlmlaEfM1b+SE4l0hFL+GYC81 7OXM+Z+iMFqPJEr5cwFHNnPTa/hBe/Lw/S30lB8qXL4BOAvS4Lq66TV8AsbknIx9fWvz0VxwfcTlY 1BJR5tCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pdb-0000000GpEB-08GV; Sun, 13 Apr 2025 05:12:11 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNg-0000000Gnio-31qZ for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:46 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 802F84A377; Sun, 13 Apr 2025 04:55:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED062C4CEE5; Sun, 13 Apr 2025 04:55:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520144; bh=Pjh4oxagSvmSLpMze4F+AJtO3zI0zTnD0Z/7n/bgK3Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VPbt2eYHx4ya3q///NGVA+AfQZlakY1oxpGQeb7fGWngp5KXaZC77Br8qfTNiMxOF a0yRZBbFGdP0Ge1p6UvtaE6HpdagU5RcBH8PHsYZznsnc2E+5GSoTGtWjUf5BOZ38d xl2D4cTxo/C6gD3zEbMcY7RGKWgmJvMhxQeyfrpqf8TkMCykFbw0CDhQ/NqWpVSvKr whrKmS6Y3bYwNpgL8SBOl/zDxkVdM8YcH0r540WTf3WAXoTLcnZtbXa6mDy47x6n5t TXob2tBZQ8oVEVJrwPDVxV28rDMGu4vb5MvlT5HCoNUlqboVAlJNnupjCvQN8jVi1+ TFmIPLUmM5/Hg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 8/9] crypto: x86/poly1305 - don't select CRYPTO_LIB_POLY1305_GENERIC Date: Sat, 12 Apr 2025 21:54:20 -0700 Message-ID: <20250413045421.55100-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215544_818072_0F40DFE7 X-CRM114-Status: UNSURE ( 8.75 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers The x86 Poly1305 code never falls back to the generic code, so selecting CRYPTO_LIB_POLY1305_GENERIC is unnecessary. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index 31804dacf9a4..a25765186ba3 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -397,11 +397,10 @@ config CRYPTO_POLYVAL_CLMUL_NI - CLMUL-NI (carry-less multiplication new instructions) config CRYPTO_POLY1305_X86_64 tristate depends on X86 && 64BIT - select CRYPTO_LIB_POLY1305_GENERIC select CRYPTO_ARCH_HAVE_LIB_POLY1305 default CRYPTO_LIB_POLY1305_INTERNAL config CRYPTO_SHA1_SSSE3 tristate "Hash functions: SHA-1 (SSSE3/AVX/AVX2/SHA-NI)" From patchwork Sun Apr 13 04:54:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 14049212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CDA0AC3601E for ; Sun, 13 Apr 2025 05:14:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GrhB+kNPtDwcIO4MusPOmHEI+kY9YhdYT3G1lfPHgZc=; b=xM0v143r0+UECy/vItRliFP1Ho ZVYmp8VaS4YpP34t/JfRDihH4FfhaSBt7s8KRafNvB+IrXikyWV6J6t95Mx0BCE4bIHY+M2tZ2B4K PB09WoUzwTTZd+M6lA60bv2qB824tx9ZW/ILJxJo7Ax8AAUNmNcecl4uE+kCXnyyYRwKA/tsBYgcB 687HptL5aLl20zFZyBgMBxrEBUqEVe4/w+34MT1K26cyDEYecaZpmv9ut1eF4Om4rhJG0ubTROjvF vLi6Hqi3COVx1IJ0rKK2HS9XO2OhjdJmRMhREg4bu13pzP7yYCLdYVfjINoE+4mbKGYTu0lQd+JER YsPKssfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pfM-0000000GpKp-3BDA; Sun, 13 Apr 2025 05:14:00 +0000 Received: from nyc.source.kernel.org ([2604:1380:45d1:ec00::3]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3pNh-0000000GnjW-3hDR for linux-arm-kernel@lists.infradead.org; Sun, 13 Apr 2025 04:55:47 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id BB8DEA40D19; Sun, 13 Apr 2025 04:50:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8414CC4CEEC; Sun, 13 Apr 2025 04:55:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744520144; bh=2gVkbo9MzVz9VvdMrst8TjWJgNjWQ5w/TelaNMUEtGU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dQ3TQJtkQRNwLXjjSV+ZirK4F2x5Ki3Bhv5FTKS0TwurPeEsyoDqm/+11jsc6KW/a sS/UqHVfUu/IOzxUpNwkD1ovfmDwFhEzVnWxOWHFD05K5ZscmwjZU2oMjAfRLzRawn 30yLpt4R8qhT7ec13PW04vyT4DaMPls82NRUZ1euqmDGxs8CSb9p5bWhuXdCviGCUc 1zRw8fPcXC/Zdg2Lmqm7zA1a+dhQrtO9RiXimZ4QHwMHLCsaPR3Bnik+Zq35FnHVcq lQ9gEU05caDHcf1lX0cxLWbGZOh7IM7YG2I8Vl/U07faajdBK8X4wf5HybE8IpkIAd uhuSffzv6mTwg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH 9/9] crypto: poly1305 - remove rset and sset fields of poly1305_desc_ctx Date: Sat, 12 Apr 2025 21:54:21 -0700 Message-ID: <20250413045421.55100-10-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250413045421.55100-1-ebiggers@kernel.org> References: <20250413045421.55100-1-ebiggers@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250412_215545_991576_CC9B6EA8 X-CRM114-Status: GOOD ( 11.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Biggers These fields are no longer needed, so remove them. Signed-off-by: Eric Biggers --- include/crypto/poly1305.h | 4 ---- lib/crypto/poly1305.c | 2 -- 2 files changed, 6 deletions(-) diff --git a/include/crypto/poly1305.h b/include/crypto/poly1305.h index 91444965772a..6e21ec2d1dc2 100644 --- a/include/crypto/poly1305.h +++ b/include/crypto/poly1305.h @@ -41,14 +41,10 @@ struct poly1305_state { struct poly1305_desc_ctx { /* partial buffer */ u8 buf[POLY1305_BLOCK_SIZE]; /* bytes used in partial buffer */ unsigned int buflen; - /* how many keys have been set in r[] */ - unsigned short rset; - /* whether s[] has been set */ - bool sset; /* finalize key */ u32 s[4]; /* accumulator */ struct poly1305_state h; /* key */ diff --git a/lib/crypto/poly1305.c b/lib/crypto/poly1305.c index 6e80214ebad8..b633b043f0f6 100644 --- a/lib/crypto/poly1305.c +++ b/lib/crypto/poly1305.c @@ -20,12 +20,10 @@ void poly1305_init_generic(struct poly1305_desc_ctx *desc, desc->s[1] = get_unaligned_le32(key + 20); desc->s[2] = get_unaligned_le32(key + 24); desc->s[3] = get_unaligned_le32(key + 28); poly1305_core_init(&desc->h); desc->buflen = 0; - desc->sset = true; - desc->rset = 2; } EXPORT_SYMBOL_GPL(poly1305_init_generic); void poly1305_update_generic(struct poly1305_desc_ctx *desc, const u8 *src, unsigned int nbytes)