From patchwork Thu Jan 24 18:27:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10779851 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 26FDB13B4 for ; Thu, 24 Jan 2019 18:27:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19ECF306CF for ; Thu, 24 Jan 2019 18:27:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DEE83082B; Thu, 24 Jan 2019 18:27:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A3B6A306CF for ; Thu, 24 Jan 2019 18:27:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EDVGCvmoqbu4jR4t3wW7HeD51mLa5bYsqNPZGNbX2ig=; b=HBZ4CckKUZgU8MU4U1Sqk57Nm1 3W7lyfJaNitn4q31+OGfior+iIjucq0p9BlHVqOlUmkYLd+pbz8x040zW/mfVIXk/1gwiqur9lGf0 zhTvghVbfh0QAxuq4tRnzbMNTsalN51Vq5cg24p1tcreo5kdtL2cMN51f+aOkxHP8ex5EWofrTx3m 3heNV/9RZgCk+3a7TX/TZRbT6Q2OPuguXc6EZcJwdAZWMJ6xKYMynDOLQXJdtEQvMZsq3LHn33eM3 OSe+Qv1rOiX6mDwMT84V2AYvPZdLhGoTmKQSMsRGJrRexY2kd/6pMmJBA+Ew0k1f8xhCWr2pc/75w EEVMPK2g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmjiz-00028c-PU; Thu, 24 Jan 2019 18:27:37 +0000 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmjij-0001vZ-Hd for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 18:27:23 +0000 Received: by mail-wm1-x343.google.com with SMTP id m1so4242774wml.2 for ; Thu, 24 Jan 2019 10:27:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=hl6687nMxC/Lvjr/tM2DjrjI/c8ecYNEFmGYcpWIz77p8QVJs+H+jeO7eVzqdxvccp mTMjUSf5HKVBt5Zic8+nAiGOYWobjoBpdi5d5nsFkkGsb42uTDGMuzhsyH06T4Yk6AE3 jnTYDN0edqWfUgdVxi/XmW+dAlLOU3jED3uRQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=L+fDB8E9AVBipJskD+39SF8Ywlre6pBETVXdjqPhTBxcNMpY9C/JeFe9S3CAjC9qBU jCLWFUiD6ntVbg6pDaUwFu1TlHFS7LRo9uvbCDV6noqKD4k67BOSU9DSgfqPJsBIAa77 gnIB0Pgdz/PUvlx2Tau0ktT287pmTMfkIjXBSayYH9Md6cTs31N4bSYi5oOXtYSmGuyV gniqFhaMp31GjJITcK6F8xxMLld/Kfp78MugAamoOT9RYBTeGsRjt/AofS20sj2eQtyf 9nvl3w4Cp4u3c+IS6I9zw8nak4gp/Z+pSexuXA9WbupQlhZ6W7zEXrKRxC7ms85RjRHl 7Z5A== X-Gm-Message-State: AJcUukeDZc08zbIEs91d1szGynijOOGG6RA6eOlNLFcx9leblSR3l1rR fJpOufj5vFyWrmaRXj+esfuFpA== X-Google-Smtp-Source: ALg8bN6umYxuDwtCZR0kXhWSQn62gKlMp2P3bU8k5ex8ORyT9rov7POdD0GGVR8hsIOR3tcpeqO83Q== X-Received: by 2002:a1c:e044:: with SMTP id x65mr3685792wmg.86.1548354439745; Thu, 24 Jan 2019 10:27:19 -0800 (PST) Received: from dogfood.home ([2a01:cb1d:112:6f00:bcd4:806e:230a:673e]) by smtp.gmail.com with ESMTPSA id 202sm73247138wmt.8.2019.01.24.10.27.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jan 2019 10:27:18 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH 1/2] crypto: arm/crct10dif - revert to C code for short inputs Date: Thu, 24 Jan 2019 19:27:11 +0100 Message-Id: <20190124182712.7142-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190124182712.7142-1-ard.biesheuvel@linaro.org> References: <20190124182712.7142-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190124_102721_579515_F07C3E61 X-CRM114-Status: GOOD ( 16.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ebiggers@kernel.org, herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The SIMD routine ported from x86 used to have a special code path for inputs < 16 bytes, which got lost somewhere along the way. Instead, the current glue code aligns the input pointer to permit the NEON routine to use special versions of the vld1 instructions that assume 16 byte alignment, but this could result in inputs of less than 16 bytes to be passed in. This not only fails the new extended tests that Eric has implemented, it also results in the code reading before the input pointer, which could potentially result in crashes when dealing with less than 16 bytes of input at the start of a page which is preceded by an unmapped page. So update the glue code to only invoke the NEON routine if the input is more than 16 bytes. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/crct10dif-ce-core.S | 20 ++++++++--------- arch/arm/crypto/crct10dif-ce-glue.c | 23 +++++--------------- 2 files changed, 16 insertions(+), 27 deletions(-) diff --git a/arch/arm/crypto/crct10dif-ce-core.S b/arch/arm/crypto/crct10dif-ce-core.S index ce45ba0c0687..3fd13d7c842c 100644 --- a/arch/arm/crypto/crct10dif-ce-core.S +++ b/arch/arm/crypto/crct10dif-ce-core.S @@ -124,10 +124,10 @@ ENTRY(crc_t10dif_pmull) vext.8 q10, qzr, q0, #4 // receive the initial 64B data, xor the initial crc value - vld1.64 {q0-q1}, [arg2, :128]! - vld1.64 {q2-q3}, [arg2, :128]! - vld1.64 {q4-q5}, [arg2, :128]! - vld1.64 {q6-q7}, [arg2, :128]! + vld1.64 {q0-q1}, [arg2]! + vld1.64 {q2-q3}, [arg2]! + vld1.64 {q4-q5}, [arg2]! + vld1.64 {q6-q7}, [arg2]! CPU_LE( vrev64.8 q0, q0 ) CPU_LE( vrev64.8 q1, q1 ) CPU_LE( vrev64.8 q2, q2 ) @@ -150,7 +150,7 @@ CPU_LE( vrev64.8 q7, q7 ) veor.8 q0, q0, q10 adr ip, rk3 - vld1.64 {q10}, [ip, :128] // xmm10 has rk3 and rk4 + vld1.64 {q10}, [ip] // xmm10 has rk3 and rk4 // // we subtract 256 instead of 128 to save one instruction from the loop @@ -167,7 +167,7 @@ CPU_LE( vrev64.8 q7, q7 ) _fold_64_B_loop: .macro fold64, reg1, reg2 - vld1.64 {q11-q12}, [arg2, :128]! + vld1.64 {q11-q12}, [arg2]! vmull.p64 q8, \reg1\()h, d21 vmull.p64 \reg1, \reg1\()l, d20 @@ -203,13 +203,13 @@ CPU_LE( vrev64.8 q12, q12 ) // constants adr ip, rk9 - vld1.64 {q10}, [ip, :128]! + vld1.64 {q10}, [ip]! .macro fold16, reg, rk vmull.p64 q8, \reg\()l, d20 vmull.p64 \reg, \reg\()h, d21 .ifnb \rk - vld1.64 {q10}, [ip, :128]! + vld1.64 {q10}, [ip]! .endif veor.8 q7, q7, q8 veor.8 q7, q7, \reg @@ -238,7 +238,7 @@ _16B_reduction_loop: vmull.p64 q7, d15, d21 veor.8 q7, q7, q8 - vld1.64 {q0}, [arg2, :128]! + vld1.64 {q0}, [arg2]! CPU_LE( vrev64.8 q0, q0 ) vswp d0, d1 veor.8 q7, q7, q0 @@ -335,7 +335,7 @@ _less_than_128: vmov.i8 q0, #0 vmov s3, arg1_low32 // get the initial crc value - vld1.64 {q7}, [arg2, :128]! + vld1.64 {q7}, [arg2]! CPU_LE( vrev64.8 q7, q7 ) vswp d14, d15 veor.8 q7, q7, q0 diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c index d428355cf38d..14c19c70a841 100644 --- a/arch/arm/crypto/crct10dif-ce-glue.c +++ b/arch/arm/crypto/crct10dif-ce-glue.c @@ -35,26 +35,15 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, unsigned int length) { u16 *crc = shash_desc_ctx(desc); - unsigned int l; - if (!may_use_simd()) { - *crc = crc_t10dif_generic(*crc, data, length); + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + kernel_neon_begin(); + *crc = crc_t10dif_pmull(*crc, data, length); + kernel_neon_end(); } else { - if (unlikely((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { - l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE - - ((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)); - - *crc = crc_t10dif_generic(*crc, data, l); - - length -= l; - data += l; - } - if (length > 0) { - kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); - kernel_neon_end(); - } + *crc = crc_t10dif_generic(*crc, data, length); } + return 0; } From patchwork Thu Jan 24 18:27:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10779853 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A8ED13B4 for ; Thu, 24 Jan 2019 18:27:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D4C2306CF for ; Thu, 24 Jan 2019 18:27:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 315F23082B; Thu, 24 Jan 2019 18:27:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B4DCF306CF for ; Thu, 24 Jan 2019 18:27:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=weayzAmtUAQv1wMCXh3Ga5Y6C/URxyMyoHKI7phLF5k=; b=M1OMZ0ItoLPMTeKpPXKN96whBb Ynv7az9zYDjRR6DSQOVZxVmW2DN91WGbU7Ee9hCZ0anKhfzs5a/Z54UN//hzoIQMfhW2ma6jUZKCj 3ze0BUo7x+wJGZTx+eqTy0ubX01ilI18SLrbnoX4R9jSfFNPtujwzQUjEOLP6hjV6uuNl3DF51gfd 9j9LmKHbxksjTX+mqrtw0SzBkcnp0aMYR1OwQQE3j87oWPF/QDG+67Da4TbSexTEjXwbfBPUfCx7W mSggriL0zdLgzXXbM/UWIsblg+2g2m3luW24Tyf4ygOT0kdhzE3uZiLfx7p0KP+taa2bVIQUOuqOZ ZgVFNIDA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmjjF-0002Ma-Et; Thu, 24 Jan 2019 18:27:53 +0000 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmjik-0001vy-VG for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 18:27:24 +0000 Received: by mail-wr1-x444.google.com with SMTP id u4so7598766wrp.3 for ; Thu, 24 Jan 2019 10:27:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3/1JsJbOO9AapMWTRThG5LlPeoGHhP1SPxErwHi+5nY=; b=aIpSZUY3HDECVs6ewDdoEN9Hdyis2ferKU36Vhx9iyGGId0acU/LfAtbxIhx6gcIuE 7XMpikyGfot/vqiMvqbqgqh7pC9Qnp7EPME7IXlJRVzkUbJFdwNoHJfXc82NiVyZWluV 7PGeqVprHgmqPoVodW4vgj54HJ0cZh7Y33O5A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3/1JsJbOO9AapMWTRThG5LlPeoGHhP1SPxErwHi+5nY=; b=ZG9jhhgA8rMvFIcnqVixwfmUQ/mHBK4dq/HM0kiw7YK7TK12I7yVr/1Rcl6Vg/yK5s JT8Rx7hclJh+CsmIZiU11knDGF4Yiey0TgA/1UZM6j22Xh0otMnCitxqbAhNkeg65pMc oMwSBqxJJSWG/PjkqzjHUB9eXn53R6zQpX1b4WxxQAT38a8ie0+DBMKVseOdzNgb6Paq vW+NiPjbehaojDIkdrBUZkKJZwpXR3Xnd7t9pqrUSeIGPMGV7QNO+PYEaU2worwbsEtP Q7s98Bb+GwX9yTebNyJ8eWRpzODRuUi7toGyLKS83WyqVq+GC2R6NZh7fHhKde5iv+fh m/uw== X-Gm-Message-State: AJcUukcVj74EdAO5yQ1Vb79LL3hE56Tw6PUszs9h1So3giDEDOF6aozL cmWHiAlhmzWZJV/s2LFXj8ADFGnQVRdIwQ== X-Google-Smtp-Source: ALg8bN5yh+PmaunQIPmSunyPL7mY0/f+bAONZd8HEZ9jOTYMa5GuN6haDGb2R1JUvqukAQ09htcm7g== X-Received: by 2002:adf:f550:: with SMTP id j16mr8145818wrp.258.1548354441226; Thu, 24 Jan 2019 10:27:21 -0800 (PST) Received: from dogfood.home ([2a01:cb1d:112:6f00:bcd4:806e:230a:673e]) by smtp.gmail.com with ESMTPSA id 202sm73247138wmt.8.2019.01.24.10.27.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jan 2019 10:27:20 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH 2/2] crypto: arm64/crct10dif - revert to C code for short inputs Date: Thu, 24 Jan 2019 19:27:12 +0100 Message-Id: <20190124182712.7142-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190124182712.7142-1-ard.biesheuvel@linaro.org> References: <20190124182712.7142-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190124_102723_000812_9AF71F70 X-CRM114-Status: GOOD ( 15.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ebiggers@kernel.org, herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The SIMD routine ported from x86 used to have a special code path for inputs < 16 bytes, which got lost somewhere along the way. Instead, the current glue code aligns the input pointer to permit the NEON routine to use special versions of the vld1 instructions that assume 16 byte alignment, but this could result in inputs of less than 16 bytes to be passed in. This not only fails the new extended tests that Eric has implemented, it also results in the code reading before the input pointer, which could potentially result in crashes when dealing with less than 16 bytes of input at the start of a page which is preceded by an unmapped page. So update the glue code to only invoke the NEON routine if the input is more than 16 bytes. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/crct10dif-ce-glue.c | 25 +++++--------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c index b461d62023f2..567c24f3d224 100644 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ b/arch/arm64/crypto/crct10dif-ce-glue.c @@ -39,26 +39,13 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, unsigned int length) { u16 *crc = shash_desc_ctx(desc); - unsigned int l; - if (unlikely((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { - l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE - - ((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)); - - *crc = crc_t10dif_generic(*crc, data, l); - - length -= l; - data += l; - } - - if (length > 0) { - if (may_use_simd()) { - kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); - kernel_neon_end(); - } else { - *crc = crc_t10dif_generic(*crc, data, length); - } + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + kernel_neon_begin(); + *crc = crc_t10dif_pmull(*crc, data, length); + kernel_neon_end(); + } else { + *crc = crc_t10dif_generic(*crc, data, length); } return 0;