From patchwork Fri Oct 11 17:08:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 11186019 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D3D03112B for ; Fri, 11 Oct 2019 17:09:41 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B01A7206A1 for ; Fri, 11 Oct 2019 17:09:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="FoAMTQBb"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="apJ0GA9s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B01A7206A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IZ4cGB6+IWeLiQvY3+jRX/ak51Xw2vWl+R6DNV7WbTY=; b=FoAMTQBbB2dQjd gCLQ1X+XKb2JsWwXmgUbjRdVVVfXbRO8h/Vts5f8BR3yNH7BJExfYHxs0GmrkLse920EuwXQxpzXP t57UMlYT/e9yb59I9ITWv82HzlwJTsoX1h0t4jdiD28+hRmm2Z/B0aYONFo4XaJpIwXZx5Kg0I9fq l4E2tPGohw7E2OcvzikP0GU28wTcTStrKckFPZ3wcct8gwF9IkYzUq4oPNBXVVVkzovtSeEh0ewU+ +LZ2R1fov57tisf5V7ZAZDqfJgjtQ0/6YydYHAoB7NkNlPUk5qw+V08roEbE+SY5/InshBnh0k/Rk e9xk+b/wzXiBuVzXbi0A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iIyQ8-0005ZA-Bw; Fri, 11 Oct 2019 17:09:40 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iIyPy-0005Ph-Jn for linux-arm-kernel@lists.infradead.org; Fri, 11 Oct 2019 17:09:32 +0000 Received: by mail-wm1-x342.google.com with SMTP id m18so10926699wmc.1 for ; Fri, 11 Oct 2019 10:09:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3dT5xeg+W/abI4zH3wzED14fHVMhPnQCtSYtaVYmbAk=; b=apJ0GA9s8TIf+XqhnX3fcyr614dCtpruEjBkG0gxLeU9xA3fh7RxI8oqyEzc9kk3i1 6O2SIz7jn2HKKlPCUUh3zZ21EW+wS4Au7E1JQHyArgLyuWvlIu7eZStQDE7cGbidKIGa Ztex0oLLfJQRoiPlYITZKw1wFY2C0FBrSf4NMjhTE61Xu/8M2HnzAMlCWedB521kqpps jOOgoA2ZCUSYef5swrsWfLMGAy0ZnTmPZvhvXIW3PIeFDjHI2HE/Dt0VmU4OzkA7uXet 19aG20hPJUUV/wa1Xht1wrT287YKDowb+BinMIA+SwFCCa6svPDU0nu0gnDfisHgOfqd zJKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3dT5xeg+W/abI4zH3wzED14fHVMhPnQCtSYtaVYmbAk=; b=Rohu3wXTO5iunKOrxn4latF2jIGXMD6PC6YCmO/2JXDSxEWV6/PJDZApT00vm6fFa4 UaWp1+YJu9CR2PzXNdixKXTk08ifjPBChASzlVmrii+Ids/up1x+G5bEKPjXn/eFzHNa POiNJy1vcADer4ehQ8JYFQEYpbRO1Ec/plhywDICF/Xr0RAGsln5uyEqAN+Fb9IbvPAY EdTuoldRtuDnvi+AksrhPQ/VLw4q88pw+Suxv+D186BUULiyTcoMz2mfYY8Gi9O7lOVQ UlxpmHAm08hkZkahIfVCwszCAtHhf2cFWkCdgc1tloiTTqBmXrz0SlIpX8dyrCMdpBwD bqXA== X-Gm-Message-State: APjAAAVxFbjQSmzZGdCnEGIIgp8zuFreJ00gwOTnw52X5Qvvh+AUl7wL MbRzfZbrZaMlx9HQyZTSi2GmxcMUGS7fEQ== X-Google-Smtp-Source: APXvYqxNZuXDIfJfBDc7cNa62DKfqWGbvM/skXG4YSOilCu7Shxfe3VRWO+A4HmwmbyBrHGANp+jFg== X-Received: by 2002:a1c:1d15:: with SMTP id d21mr3998509wmd.5.1570813769234; Fri, 11 Oct 2019 10:09:29 -0700 (PDT) Received: from localhost.localdomain (91-167-84-221.subs.proxad.net. [91.167.84.221]) by smtp.gmail.com with ESMTPSA id f9sm11876875wre.74.2019.10.11.10.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Oct 2019 10:09:28 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH 1/2] crypto: aegis128 - avoid function pointers for parameterization Date: Fri, 11 Oct 2019 19:08:22 +0200 Message-Id: <20191011170823.6713-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191011170823.6713-1-ard.biesheuvel@linaro.org> References: <20191011170823.6713-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191011_100930_652149_B102043C X-CRM114-Status: GOOD ( 17.43 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:342 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Instead of passing around an ops structure with function pointers, which forces indirect calls to be used, refactor the code slightly so we can use ordinary function calls. At the same time, switch to a static key to decide whether or not the SIMD code path may be used. Signed-off-by: Ard Biesheuvel --- crypto/aegis128-core.c | 105 +++++++++----------- 1 file changed, 46 insertions(+), 59 deletions(-) diff --git a/crypto/aegis128-core.c b/crypto/aegis128-core.c index 80e73611bd5c..fe7ab66dd8f9 100644 --- a/crypto/aegis128-core.c +++ b/crypto/aegis128-core.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -35,15 +36,7 @@ struct aegis_ctx { union aegis_block key; }; -struct aegis128_ops { - int (*skcipher_walk_init)(struct skcipher_walk *walk, - struct aead_request *req, bool atomic); - - void (*crypt_chunk)(struct aegis_state *state, u8 *dst, - const u8 *src, unsigned int size); -}; - -static bool have_simd; +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_simd); static const union aegis_block crypto_aegis_const[2] = { { .words64 = { @@ -59,7 +52,7 @@ static const union aegis_block crypto_aegis_const[2] = { static bool aegis128_do_simd(void) { #ifdef CONFIG_CRYPTO_AEGIS128_SIMD - if (have_simd) + if (static_branch_likely(&have_simd)) return crypto_simd_usable(); #endif return false; @@ -323,25 +316,27 @@ static void crypto_aegis128_process_ad(struct aegis_state *state, } } -static void crypto_aegis128_process_crypt(struct aegis_state *state, - struct aead_request *req, - const struct aegis128_ops *ops) +static __always_inline +int crypto_aegis128_process_crypt(struct aegis_state *state, + struct aead_request *req, + struct skcipher_walk *walk, + void (*crypt)(struct aegis_state *state, + u8 *dst, const u8 *src, + unsigned int size)) { - struct skcipher_walk walk; + int err = 0; - ops->skcipher_walk_init(&walk, req, false); + while (walk->nbytes) { + unsigned int nbytes = walk->nbytes; - while (walk.nbytes) { - unsigned int nbytes = walk.nbytes; + if (nbytes < walk->total) + nbytes = round_down(nbytes, walk->stride); - if (nbytes < walk.total) - nbytes = round_down(nbytes, walk.stride); + crypt(state, walk->dst.virt.addr, walk->src.virt.addr, nbytes); - ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); - - skcipher_walk_done(&walk, walk.nbytes - nbytes); + err = skcipher_walk_done(walk, walk->nbytes - nbytes); } + return err; } static void crypto_aegis128_final(struct aegis_state *state, @@ -390,39 +385,27 @@ static int crypto_aegis128_setauthsize(struct crypto_aead *tfm, return 0; } -static void crypto_aegis128_crypt(struct aead_request *req, - union aegis_block *tag_xor, - unsigned int cryptlen, - const struct aegis128_ops *ops) +static int crypto_aegis128_encrypt(struct aead_request *req) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); struct aegis_ctx *ctx = crypto_aead_ctx(tfm); + unsigned int cryptlen = req->cryptlen; + struct skcipher_walk walk; struct aegis_state state; crypto_aegis128_init(&state, &ctx->key, req->iv); crypto_aegis128_process_ad(&state, req->src, req->assoclen); - crypto_aegis128_process_crypt(&state, req, ops); - crypto_aegis128_final(&state, tag_xor, req->assoclen, cryptlen); -} - -static int crypto_aegis128_encrypt(struct aead_request *req) -{ - const struct aegis128_ops *ops = &(struct aegis128_ops){ - .skcipher_walk_init = skcipher_walk_aead_encrypt, - .crypt_chunk = crypto_aegis128_encrypt_chunk, - }; - - struct crypto_aead *tfm = crypto_aead_reqtfm(req); - union aegis_block tag = {}; - unsigned int authsize = crypto_aead_authsize(tfm); - unsigned int cryptlen = req->cryptlen; + skcipher_walk_aead_encrypt(&walk, req, false); if (aegis128_do_simd()) - ops = &(struct aegis128_ops){ - .skcipher_walk_init = skcipher_walk_aead_encrypt, - .crypt_chunk = crypto_aegis128_encrypt_chunk_simd }; - - crypto_aegis128_crypt(req, &tag, cryptlen, ops); + crypto_aegis128_process_crypt(&state, req, &walk, + crypto_aegis128_encrypt_chunk_simd); + else + crypto_aegis128_process_crypt(&state, req, &walk, + crypto_aegis128_encrypt_chunk); + crypto_aegis128_final(&state, &tag, req->assoclen, cryptlen); scatterwalk_map_and_copy(tag.bytes, req->dst, req->assoclen + cryptlen, authsize, 1); @@ -431,26 +414,29 @@ static int crypto_aegis128_encrypt(struct aead_request *req) static int crypto_aegis128_decrypt(struct aead_request *req) { - const struct aegis128_ops *ops = &(struct aegis128_ops){ - .skcipher_walk_init = skcipher_walk_aead_decrypt, - .crypt_chunk = crypto_aegis128_decrypt_chunk, - }; static const u8 zeros[AEGIS128_MAX_AUTH_SIZE] = {}; - struct crypto_aead *tfm = crypto_aead_reqtfm(req); union aegis_block tag; unsigned int authsize = crypto_aead_authsize(tfm); unsigned int cryptlen = req->cryptlen - authsize; + struct aegis_ctx *ctx = crypto_aead_ctx(tfm); + struct skcipher_walk walk; + struct aegis_state state; scatterwalk_map_and_copy(tag.bytes, req->src, req->assoclen + cryptlen, authsize, 0); - if (aegis128_do_simd()) - ops = &(struct aegis128_ops){ - .skcipher_walk_init = skcipher_walk_aead_decrypt, - .crypt_chunk = crypto_aegis128_decrypt_chunk_simd }; + crypto_aegis128_init(&state, &ctx->key, req->iv); + crypto_aegis128_process_ad(&state, req->src, req->assoclen); - crypto_aegis128_crypt(req, &tag, cryptlen, ops); + skcipher_walk_aead_decrypt(&walk, req, false); + if (aegis128_do_simd()) + crypto_aegis128_process_crypt(&state, req, &walk, + crypto_aegis128_decrypt_chunk_simd); + else + crypto_aegis128_process_crypt(&state, req, &walk, + crypto_aegis128_decrypt_chunk); + crypto_aegis128_final(&state, &tag, req->assoclen, cryptlen); return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; } @@ -481,8 +467,9 @@ static struct aead_alg crypto_aegis128_alg = { static int __init crypto_aegis128_module_init(void) { - if (IS_ENABLED(CONFIG_CRYPTO_AEGIS128_SIMD)) - have_simd = crypto_aegis128_have_simd(); + if (IS_ENABLED(CONFIG_CRYPTO_AEGIS128_SIMD) && + crypto_aegis128_have_simd()) + static_branch_enable(&have_simd); return crypto_register_aead(&crypto_aegis128_alg); } From patchwork Fri Oct 11 17:08:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 11186021 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 440C21668 for ; Fri, 11 Oct 2019 17:10:05 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 041ED206A1 for ; Fri, 11 Oct 2019 17:10:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="cyY7bORh"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="AbwEIQB3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 041ED206A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QNg8hJShfsvqiRe372maovYFaTFhm8ClRy7Eoqy42Gg=; b=cyY7bORhfvI4Z1 EpUkidjtBVM0dWJNGFAJVgYbB4S4bIfBaxFiVnS5/geqMQ9tWLIuEwryAAJjnfotT7i5FWmKndLXc Y3dVwm9445cmX/ZeFqVGkkbFrkhXKca3WEDTH8dTgmidL8ilRAYuRoazLQMaOndKYpOJrVzb5zEqR 7dDqCIj8e6694eR77KUnxwwb5QwMN8vwOVdzYvPJfJlwiNrMGCvmp7rLd+/xtKsSoteWV7eC2qV5I X7Hn09r7bvQ2vTFDubYUtnm36T7D2FmlvHHHa+IS0pfBrOVk7YsErI2NPjTVJmEDSCqMEWRbZpvzc bapnXjeedFjT4IhvZAOA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iIyQL-0005nl-3r; Fri, 11 Oct 2019 17:09:53 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iIyPz-0005Qo-Sv for linux-arm-kernel@lists.infradead.org; Fri, 11 Oct 2019 17:09:33 +0000 Received: by mail-wr1-x443.google.com with SMTP id p14so12742748wro.4 for ; Fri, 11 Oct 2019 10:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tKcWa8kiCmuQGLJUVnadD6vpBI/OKGXQya0413QFYWE=; b=AbwEIQB37g1f04wuqsM97MyCuc35DWJL8ZCKfdWIOwtQn3ocjNaf5rPNNLr0dtl/h7 +hJvre0mj01P8zRciGlQ7nxOPjp2nkeN2R5HuY1hV4QrggO8cTOwXC0FXhpeYSQoG/3r CGrmga21odEtNf6IYWiBbXUC5hnQRG9G8PmnsFuIOznnuuhzWhj5t4IIyHqtDvp89RFY TCeWLYLxaN/I++IBymsejrniYKHgPGH2uTM5pmhOzfxFhSYCShrhOpfnk657PpdcUK80 E2kfb4u5elGOZyemRd7rxYIHZIvcbH4Q65XJjIwxYqQ66WHE+mV/pg60ufnF6/GoBiJB bMHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tKcWa8kiCmuQGLJUVnadD6vpBI/OKGXQya0413QFYWE=; b=blmLVTKhSwJ6cvH++y43xHFgDmkdxKn2qAQrNHxXLjbGFEUQaX/3E0H4nHKeL7yB3p HJD2Ln2YDEvGiG1Xgo3o6P6YMNl2F6XjwabKeVox50VfOoK+QhNJ8qh0eTXXkcKx3s9v mbi07GkXKLFye5vgEq3rv6U0/zpiIzAzDz13Vn1YlC1dZNolHo3ur1UXlGNflDy/F7bU K28r8OqmmQ9Vm0oQVW9eTYPqepqnqbTB7byfn1ND4z6k3nxcbgr/TEoDoyKLut57grA9 xkiMhfnsaQ2CecjCdc0/1B2YCaFXjdpbHIdovaFZiiEUiWT+90iV3eWLdLMircDSNseF u82w== X-Gm-Message-State: APjAAAVFjpuSr8wTGaYB8zVDfVAuo+fTvwppLs+9vAE9kHzVqXV/D2wG ft5RHL1/ZMbfMSqRiDuMEzH+Cg== X-Google-Smtp-Source: APXvYqxY3qHWRYIaXSoDLi4qk+VvGdqQk21dysGqo4N7ySiaAiXxVg5jLIDOVni+KGW4XmiMX+bRYw== X-Received: by 2002:a5d:4ec1:: with SMTP id s1mr13562167wrv.42.1570813770511; Fri, 11 Oct 2019 10:09:30 -0700 (PDT) Received: from localhost.localdomain (91-167-84-221.subs.proxad.net. [91.167.84.221]) by smtp.gmail.com with ESMTPSA id f9sm11876875wre.74.2019.10.11.10.09.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Oct 2019 10:09:29 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH 2/2] crypto: aegis128 - duplicate init() and final() hooks in SIMD code Date: Fri, 11 Oct 2019 19:08:23 +0200 Message-Id: <20191011170823.6713-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191011170823.6713-1-ard.biesheuvel@linaro.org> References: <20191011170823.6713-1-ard.biesheuvel@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191011_100931_933534_4467DBC4 X-CRM114-Status: GOOD ( 13.23 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:443 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org In order to speed up aegis128 processing even more, duplicate the init() and final() routines as SIMD versions in their entirety. This results in a 2x speedup on ARM Cortex-A57 for ~1500 byte inputs (using AES instructions). Signed-off-by: Ard Biesheuvel --- crypto/aegis128-core.c | 38 ++++++++++----- crypto/aegis128-neon-inner.c | 49 ++++++++++++++++++++ crypto/aegis128-neon.c | 22 +++++++++ 3 files changed, 97 insertions(+), 12 deletions(-) diff --git a/crypto/aegis128-core.c b/crypto/aegis128-core.c index fe7ab66dd8f9..71c11cb5bad1 100644 --- a/crypto/aegis128-core.c +++ b/crypto/aegis128-core.c @@ -60,10 +60,16 @@ static bool aegis128_do_simd(void) bool crypto_aegis128_have_simd(void); void crypto_aegis128_update_simd(struct aegis_state *state, const void *msg); +void crypto_aegis128_init_simd(struct aegis_state *state, + const union aegis_block *key, + const u8 *iv); void crypto_aegis128_encrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size); void crypto_aegis128_decrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size); +void crypto_aegis128_final_simd(struct aegis_state *state, + union aegis_block *tag_xor, + u64 assoclen, u64 cryptlen); static void crypto_aegis128_update(struct aegis_state *state) { @@ -395,17 +401,21 @@ static int crypto_aegis128_encrypt(struct aead_request *req) struct skcipher_walk walk; struct aegis_state state; - crypto_aegis128_init(&state, &ctx->key, req->iv); - crypto_aegis128_process_ad(&state, req->src, req->assoclen); - skcipher_walk_aead_encrypt(&walk, req, false); - if (aegis128_do_simd()) + if (aegis128_do_simd()) { + crypto_aegis128_init_simd(&state, &ctx->key, req->iv); + crypto_aegis128_process_ad(&state, req->src, req->assoclen); crypto_aegis128_process_crypt(&state, req, &walk, crypto_aegis128_encrypt_chunk_simd); - else + crypto_aegis128_final_simd(&state, &tag, req->assoclen, + cryptlen); + } else { + crypto_aegis128_init(&state, &ctx->key, req->iv); + crypto_aegis128_process_ad(&state, req->src, req->assoclen); crypto_aegis128_process_crypt(&state, req, &walk, crypto_aegis128_encrypt_chunk); - crypto_aegis128_final(&state, &tag, req->assoclen, cryptlen); + crypto_aegis128_final(&state, &tag, req->assoclen, cryptlen); + } scatterwalk_map_and_copy(tag.bytes, req->dst, req->assoclen + cryptlen, authsize, 1); @@ -426,17 +436,21 @@ static int crypto_aegis128_decrypt(struct aead_request *req) scatterwalk_map_and_copy(tag.bytes, req->src, req->assoclen + cryptlen, authsize, 0); - crypto_aegis128_init(&state, &ctx->key, req->iv); - crypto_aegis128_process_ad(&state, req->src, req->assoclen); - skcipher_walk_aead_decrypt(&walk, req, false); - if (aegis128_do_simd()) + if (aegis128_do_simd()) { + crypto_aegis128_init_simd(&state, &ctx->key, req->iv); + crypto_aegis128_process_ad(&state, req->src, req->assoclen); crypto_aegis128_process_crypt(&state, req, &walk, crypto_aegis128_decrypt_chunk_simd); - else + crypto_aegis128_final_simd(&state, &tag, req->assoclen, + cryptlen); + } else { + crypto_aegis128_init(&state, &ctx->key, req->iv); + crypto_aegis128_process_ad(&state, req->src, req->assoclen); crypto_aegis128_process_crypt(&state, req, &walk, crypto_aegis128_decrypt_chunk); - crypto_aegis128_final(&state, &tag, req->assoclen, cryptlen); + crypto_aegis128_final(&state, &tag, req->assoclen, cryptlen); + } return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; } diff --git a/crypto/aegis128-neon-inner.c b/crypto/aegis128-neon-inner.c index f05310ca22aa..16d2e806eb47 100644 --- a/crypto/aegis128-neon-inner.c +++ b/crypto/aegis128-neon-inner.c @@ -132,6 +132,34 @@ void preload_sbox(void) :: "r"(crypto_aes_sbox)); } +void crypto_aegis128_init_neon(void *state, const void *key, const void *iv) +{ + static const uint8_t const0[] = { + 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d, + 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62, + }; + static const uint8_t const1[] = { + 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1, + 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd, + }; + uint8x16_t k = vld1q_u8(key); + uint8x16_t kiv = k ^ vld1q_u8(iv); + struct aegis128_state st = {{ + kiv, + vld1q_u8(const1), + vld1q_u8(const0), + k ^ vld1q_u8(const0), + k ^ vld1q_u8(const1), + }}; + int i; + + for (i = 0; i < 5; i++) { + st = aegis128_update_neon(st, k); + st = aegis128_update_neon(st, kiv); + } + aegis128_save_state_neon(st, state); +} + void crypto_aegis128_update_neon(void *state, const void *msg) { struct aegis128_state st = aegis128_load_state_neon(state); @@ -210,3 +238,24 @@ void crypto_aegis128_decrypt_chunk_neon(void *state, void *dst, const void *src, aegis128_save_state_neon(st, state); } + +void crypto_aegis128_final_neon(void *state, void *tag_xor, + uint64_t assocbits, uint64_t cryptbits) +{ + struct aegis128_state st = aegis128_load_state_neon(state); + union { + uint64_t pair[2]; + uint8x16_t v; + } t = {{ assocbits, cryptbits }}; + uint8x16_t tag; + int i; + + t.v ^= st.v[3]; + + for (i = 0; i < 7; i++) + st = aegis128_update_neon(st, t.v); + + tag = vld1q_u8(tag_xor); + tag ^= st.v[0] ^ st.v[1] ^ st.v[2] ^ st.v[3] ^ st.v[4]; + vst1q_u8(tag_xor, tag); +} diff --git a/crypto/aegis128-neon.c b/crypto/aegis128-neon.c index 751f9c195aa4..5b37b04b8ad4 100644 --- a/crypto/aegis128-neon.c +++ b/crypto/aegis128-neon.c @@ -8,11 +8,14 @@ #include "aegis.h" +void crypto_aegis128_init_neon(void *state, const void *key, const u8 *iv); void crypto_aegis128_update_neon(void *state, const void *msg); void crypto_aegis128_encrypt_chunk_neon(void *state, void *dst, const void *src, unsigned int size); void crypto_aegis128_decrypt_chunk_neon(void *state, void *dst, const void *src, unsigned int size); +void crypto_aegis128_final_neon(void *state, void *tag_xor, + uint64_t assocbits, uint64_t cryptbits); int aegis128_have_aes_insn __ro_after_init; @@ -25,6 +28,15 @@ bool crypto_aegis128_have_simd(void) return IS_ENABLED(CONFIG_ARM64); } +void crypto_aegis128_init_simd(union aegis_block *state, + const union aegis_block *key, + const u8 *iv) +{ + kernel_neon_begin(); + crypto_aegis128_init_neon(state, key, iv); + kernel_neon_end(); +} + void crypto_aegis128_update_simd(union aegis_block *state, const void *msg) { kernel_neon_begin(); @@ -47,3 +59,13 @@ void crypto_aegis128_decrypt_chunk_simd(union aegis_block *state, u8 *dst, crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); kernel_neon_end(); } + +void crypto_aegis128_final_simd(union aegis_block *state, + union aegis_block *tag_xor, + u64 assoclen, u64 cryptlen) +{ + kernel_neon_begin(); + crypto_aegis128_final_neon(state, tag_xor, cpu_to_le64(8 * assoclen), + cpu_to_le64(8 * cryptlen)); + kernel_neon_end(); +}