From patchwork Wed Dec 6 19:43:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10096947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 61F3960327 for ; Wed, 6 Dec 2017 19:47:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53C9C28E2E for ; Wed, 6 Dec 2017 19:47:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4874F28E65; Wed, 6 Dec 2017 19:47:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CB98328E2E for ; Wed, 6 Dec 2017 19:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1ffkudK2MdAObBhjRyfgikreolJ4EBMe+mHpyJ587o4=; b=K6OcJjAtf5HvxdvxUFWOOq2kcH 92x++Z0ma5eZAyTFiMBVzi15EFn5jwuD4nFRvWebfR1NHbB/+99QqVncFNhGO5NuypQhN6qwKQgT2 9USmsvyD93zes8P3JCG08xg+Id/Oi9N1NtQ+CjFVLaTNPQXa275RdbaPk0UpNTY9ln51jwMwaOtwT M4AgXMxNN+aIpTSEh2pvpefgfbecsJH8mzWwiQvxwYdLGXcgMUeCmxExsE7C1VeSdyaLqT20p1ngH rNdAUxwopWgTSYz5oQAG8bmvAOuL6f2lDfCAAX1k8q4nmj98EgdSLwU+AVmfE95YRcyGx2btQ2esX fbKdUsEg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1eMfec-0001Ut-E6; Wed, 06 Dec 2017 19:46:50 +0000 Received: from mail-wr0-x243.google.com ([2a00:1450:400c:c0c::243]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1eMfcR-0006PJ-6r for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2017 19:44:49 +0000 Received: by mail-wr0-x243.google.com with SMTP id s66so5114298wrc.9 for ; Wed, 06 Dec 2017 11:44:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=BfeaCIe3bP4WigQt68E0qH/qrUXmozDFVhJ9mVfSpVq8sWTT3H1/xKTP0S5ULBRNeE i+UVvst52Nr+FIHkQbwrd1cASaskFmYHscpH0XkAXRAKmXEWcmpRgCZY8D9nFRxbQ+JJ KYB59CdIaJwWuiJVhT3r6HljuCYh8nsmJ7Maw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=J/OR8wSLk9EzF4p225Xv8Ie0VnfdBu/Rd7Jid7zDcCauCdCsjnSZaWqyyFqImZKXc6 SrYPpANt6sA+CwrpNFBPDUfa5/qsZgViNOpii71sKjeBKxlvaRxWOjfNYPYEENe5IGqJ 6Ivs8RwOGxPBZN7xyrTJ/2ODEBRS7G8SWcd/5ZTGZ9fZLG2Qs82lUaiXqEZIBIpuhcFX rgwfJ879ZFfxsFOwt0YhCs9xXNBwJ1WqA8CJt1eiVjI5pFeu4219b3CIYA6s+fmBL4OX IKQd3c3izdYQCMqP69/HBpxtSka5q/3xy5JkCKWgP/R4z2iEL0Ud5te/YA91mSw8Hm26 KWTQ== X-Gm-Message-State: AKGB3mKDyrJ+tM437hynyVLjAtImtNblZXrszZRNt8ml7t+LGtyl76wy I0abreWXK6dMl1uUqzEpWTdIeVA0EUA= X-Google-Smtp-Source: AGs4zMb35d/ll8AacUp/zbLt6obE/6kS0cWTg2T1wJwwVCmernjO+gAJijNMedIs3AFZHJplh+gFQg== X-Received: by 10.223.134.184 with SMTP id 53mr509613wrx.17.1512589458265; Wed, 06 Dec 2017 11:44:18 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:17 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH v3 08/20] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC encrypt path Date: Wed, 6 Dec 2017 19:43:34 +0000 Message-Id: <20171206194346.24393-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171206_114435_917191_18F61C03 X-CRM114-Status: GOOD ( 12.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , herbert@gondor.apana.org.au, Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , Sebastian Andrzej Siewior , Will Deacon , Russell King - ARM Linux , Steven Rostedt , Thomas Gleixner , Dave Martin , linux-arm-kernel@lists.infradead.org, linux-rt-users@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP CBC MAC is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC MAC which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 23 ++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index e86535a1329d..a68412e1e3a4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -395,8 +395,28 @@ AES_ENDPROC(aes_xts_decrypt) AES_ENTRY(aes_mac_update) ld1 {v0.16b}, [x4] /* get dg */ enc_prepare w2, x1, x7 - cbnz w5, .Lmacenc + cbz w5, .Lmacloop4x + encrypt_block v0, w2, x1, x7, w8 + +.Lmacloop4x: + subs w3, w3, #4 + bmi .Lmac1x + ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v2.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v3.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v4.16b + cmp w3, wzr + csinv x5, x6, xzr, eq + cbz w5, .Lmacout + encrypt_block v0, w2, x1, x7, w8 + b .Lmacloop4x +.Lmac1x: + add w3, w3, #4 .Lmacloop: cbz w3, .Lmacout ld1 {v1.16b}, [x0], #16 /* get next pt block */ @@ -406,7 +426,6 @@ AES_ENTRY(aes_mac_update) csinv x5, x6, xzr, eq cbz w5, .Lmacout -.Lmacenc: encrypt_block v0, w2, x1, x7, w8 b .Lmacloop