From patchwork Thu May 1 15:51:24 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 4099491 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 57A46BFF02 for ; Thu, 1 May 2014 15:54:04 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 66B3520357 for ; Thu, 1 May 2014 15:54:03 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 699A22034C for ; Thu, 1 May 2014 15:54:02 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WftHI-0000pe-LY; Thu, 01 May 2014 15:52:04 +0000 Received: from mail-wi0-f174.google.com ([209.85.212.174]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WftHE-0000IU-Hf for linux-arm-kernel@lists.infradead.org; Thu, 01 May 2014 15:52:01 +0000 Received: by mail-wi0-f174.google.com with SMTP id d1so970836wiv.1 for ; Thu, 01 May 2014 08:51:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xhk6U6uhx/dWDxIO6OBgRf0LMsxyjioqot62R6lT1AM=; b=QEl8v4lSAOn4wSAvW9AYE9O8dekrKnTBRiRvrbExDFg7I8Z/vLmip2UZrQp2YdoI07 aiTlXiOGiRmCFU/HJdt5HNZcJi6wqZeRhgBgCW4DDKsLeox+7ZDuyI1CbLlaPemSXWy1 qjFP9FsMkEPsNvmXWyQYWuY729Vu+xGrlf6yKXrcK/sUnz0gbdz8iAknNZ+cFHOzYr9b RMmLt/aEHkF5kP6dc9Kol/yVo8+pCQPPKCGU9y9k026eDnJiQ1Yt6Lo7blJPa4/cKed+ 7u74dwPRauPqhYGN3yKi/YZ/7qE6ZGh3+Xs4V2ywOpemow2gL2AhpbwSJMyT4xlw2cCD vYqg== X-Gm-Message-State: ALoCoQmiis4y7ZELheTpAwSOrYJleWSl6npL2XF7k/HNui/+69JMs9hzNc0M2O5RrUDufFEWx3Er X-Received: by 10.194.189.116 with SMTP id gh20mr9550582wjc.41.1398959497918; Thu, 01 May 2014 08:51:37 -0700 (PDT) Received: from ards-macbook-pro.local (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id kp5sm41727776wjb.30.2014.05.01.08.51.36 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 01 May 2014 08:51:37 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org Subject: [PATCH resend 13/15] arm64/crypto: add voluntary preemption to Crypto Extensions SHA1 Date: Thu, 1 May 2014 17:51:24 +0200 Message-Id: <1398959486-8222-4-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1398959486-8222-1-git-send-email-ard.biesheuvel@linaro.org> References: <1398959486-8222-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140501_085200_889403_9C92E15E X-CRM114-Status: GOOD ( 16.68 ) X-Spam-Score: -0.7 (/) Cc: catalin.marinas@arm.com, will.deacon@arm.com, steve.capper@linaro.org, Ard Biesheuvel X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The Crypto Extensions based SHA1 implementation uses the NEON register file, and hence runs with preemption disabled. This patch adds a TIF_NEED_RESCHED check to its inner loop so we at least give up the CPU voluntarily when we are running in process context and have been tagged for preemption by the scheduler. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha1-ce-core.S | 19 ++++++++-------- arch/arm64/crypto/sha1-ce-glue.c | 49 +++++++++++++++++++++++++++++++--------- 2 files changed, 48 insertions(+), 20 deletions(-) diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S index bd4af29f2722..831e332f9331 100644 --- a/arch/arm64/crypto/sha1-ce-core.S +++ b/arch/arm64/crypto/sha1-ce-core.S @@ -66,8 +66,8 @@ .word 0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6 /* - * void sha1_ce_transform(int blocks, u8 const *src, u32 *state, - * u8 *head, long bytes) + * int sha1_ce_transform(int blocks, u8 const *src, u32 *state, + * u8 *head, long bytes, struct thread_info *ti) */ ENTRY(sha1_ce_transform) /* load round constants */ @@ -127,7 +127,13 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgbv.2s, dgbv.2s, dg1v.2s add dgav.4s, dgav.4s, dg0v.4s - cbnz w0, 0b + cbz w0, 4f + b_if_no_resched x5, x8, 0b + + /* store new state */ +3: str dga, [x2] + str dgb, [x2, #16] + ret /* * Final block: add padding and total bit count. @@ -135,7 +141,7 @@ CPU_LE( rev32 v11.16b, v11.16b ) * size was not a round multiple of the block size, and the padding is * handled by the C code. */ - cbz x4, 3f +4: cbz x4, 3b movi v9.2d, #0 mov x8, #0x80000000 movi v10.2d, #0 @@ -145,9 +151,4 @@ CPU_LE( rev32 v11.16b, v11.16b ) mov v11.d[0], xzr mov v11.d[1], x7 b 2b - - /* store new state */ -3: str dga, [x2] - str dgb, [x2, #16] - ret ENDPROC(sha1_ce_transform) diff --git a/arch/arm64/crypto/sha1-ce-glue.c b/arch/arm64/crypto/sha1-ce-glue.c index 6fe83f37a750..69850a163668 100644 --- a/arch/arm64/crypto/sha1-ce-glue.c +++ b/arch/arm64/crypto/sha1-ce-glue.c @@ -20,8 +20,8 @@ MODULE_DESCRIPTION("SHA1 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -asmlinkage void sha1_ce_transform(int blocks, u8 const *src, u32 *state, - u8 *head, long bytes); +asmlinkage int sha1_ce_transform(int blocks, u8 const *src, u32 *state, + u8 *head, long bytes, struct thread_info *ti); static int sha1_init(struct shash_desc *desc) { @@ -42,6 +42,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, sctx->count += len; if ((partial + len) >= SHA1_BLOCK_SIZE) { + struct thread_info *ti = NULL; int blocks; if (partial) { @@ -52,16 +53,30 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, len -= p; } + /* + * Pass current's thread info pointer to sha1_ce_transform() + * below if we want it to play nice under preemption. + */ + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + blocks = len / SHA1_BLOCK_SIZE; len %= SHA1_BLOCK_SIZE; - kernel_neon_begin_partial(16); - sha1_ce_transform(blocks, data, sctx->state, - partial ? sctx->buffer : NULL, 0); - kernel_neon_end(); + do { + int rem; + + kernel_neon_begin_partial(16); + rem = sha1_ce_transform(blocks, data, sctx->state, + partial ? sctx->buffer : NULL, + 0, ti); + kernel_neon_end(); - data += blocks * SHA1_BLOCK_SIZE; - partial = 0; + data += (blocks - rem) * SHA1_BLOCK_SIZE; + blocks = rem; + partial = 0; + } while (unlikely(ti && blocks > 0)); } if (len) memcpy(sctx->buffer + partial, data, len); @@ -94,6 +109,7 @@ static int sha1_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { struct sha1_state *sctx = shash_desc_ctx(desc); + struct thread_info *ti = NULL; __be32 *dst = (__be32 *)out; int blocks; int i; @@ -111,9 +127,20 @@ static int sha1_finup(struct shash_desc *desc, const u8 *data, */ blocks = len / SHA1_BLOCK_SIZE; - kernel_neon_begin_partial(16); - sha1_ce_transform(blocks, data, sctx->state, NULL, len); - kernel_neon_end(); + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + + do { + int rem; + + kernel_neon_begin_partial(16); + rem = sha1_ce_transform(blocks, data, sctx->state, + NULL, len, ti); + kernel_neon_end(); + data += (blocks - rem) * SHA1_BLOCK_SIZE; + blocks = rem; + } while (unlikely(ti && blocks > 0)); for (i = 0; i < SHA1_DIGEST_SIZE / sizeof(__be32); i++) put_unaligned_be32(sctx->state[i], dst++);