From patchwork Sat Mar 10 15:21:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10273615 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0ADE660594 for ; Sat, 10 Mar 2018 15:23:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE56729563 for ; Sat, 10 Mar 2018 15:23:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E2FCC295BC; Sat, 10 Mar 2018 15:23:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7674529563 for ; Sat, 10 Mar 2018 15:23:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932327AbeCJPW7 (ORCPT ); Sat, 10 Mar 2018 10:22:59 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:37289 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932315AbeCJPW5 (ORCPT ); Sat, 10 Mar 2018 10:22:57 -0500 Received: by mail-wr0-f193.google.com with SMTP id z12so11638518wrg.4 for ; Sat, 10 Mar 2018 07:22:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RYUXyT4yhMIRanK7ihkLKdOKlgN14RDXrNInu0AEK8g=; b=RknFKhhG0D7vyNolv/HJayLrHRepnfuk8E4K5bw7LPuka/mELechLkjTQttpHSUJwJ VkHkVreuHUqi9lVl8N+2jMmTT7NWy2MNDe1TpSQxouT5uHI28Y6WSEkuToVG9+lVXqHO jDxGKUAZUJupEAtqbchw96mD4gOLS8vyUaNH0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RYUXyT4yhMIRanK7ihkLKdOKlgN14RDXrNInu0AEK8g=; b=QZ2u2Rt8VasrSoNt10sMKm923NpekYw89FXMmMITrsFsOFUGxbilOG9bGTNO5j8VGn fmsn7NWOv1fBx4UoQqy5R2DMudAtgiMwvyyn4dCQmk4ovaBasVY73x6D9i/tXF30NjzB C/mDIUsQkdrdfIUrZHIDE+2MkLVQCw4WlC/OxCxeXj32efJduGWwcrnUmzT2K/PCX6+6 9lpXb7n9TEzch+bI+qGJ2iuyo/xGD4Mz6Hk8OVkdrrqg3W26FLXcTCyZx4Em/kj1s/Lo b/9jbddqMxuGzzmXLTycqo8n9QU6tPpkfvt6divhwIu+znAc2zm136kn7EBN2vX9BiXO rjKQ== X-Gm-Message-State: AElRT7E+ZEaYmBJpzu7tDFuaOGLZKLGCnYw+cmNW+BFUVzHM+Zc9PkbY g2MEXYZ32cgJLqXuk6rMdl62mjs9+Kc= X-Google-Smtp-Source: AG47ELvtpsPT2ArscAIOlmIbk7+9x7q0ForCvKymflGRSkryEbDNbU7PTnKzouYF4Wpu3avIzSlrhQ== X-Received: by 10.223.136.15 with SMTP id d15mr1890013wrd.127.1520695375885; Sat, 10 Mar 2018 07:22:55 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:55 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 09/23] crypto: arm64/sha256-neon - play nice with CONFIG_PREEMPT kernels Date: Sat, 10 Mar 2018 15:21:54 +0000 Message-Id: <20180310152208.10369-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Tweak the SHA256 update routines to invoke the SHA256 block transform block by block, to avoid excessive scheduling delays caused by the NEON algorithm running with preemption disabled. Also, remove a stale comment which no longer applies now that kernel mode NEON is actually disallowed in some contexts. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha256-glue.c | 36 +++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c index b064d925fe2a..e8880ccdc71f 100644 --- a/arch/arm64/crypto/sha256-glue.c +++ b/arch/arm64/crypto/sha256-glue.c @@ -89,21 +89,32 @@ static struct shash_alg algs[] = { { static int sha256_update_neon(struct shash_desc *desc, const u8 *data, unsigned int len) { - /* - * Stacking and unstacking a substantial slice of the NEON register - * file may significantly affect performance for small updates when - * executing in interrupt context, so fall back to the scalar code - * in that case. - */ + struct sha256_state *sctx = shash_desc_ctx(desc); + if (!may_use_simd()) return sha256_base_do_update(desc, data, len, (sha256_block_fn *)sha256_block_data_order); - kernel_neon_begin(); - sha256_base_do_update(desc, data, len, - (sha256_block_fn *)sha256_block_neon); - kernel_neon_end(); + while (len > 0) { + unsigned int chunk = len; + + /* + * Don't hog the CPU for the entire time it takes to process all + * input when running on a preemptible kernel, but process the + * data block by block instead. + */ + if (IS_ENABLED(CONFIG_PREEMPT) && + chunk + sctx->count % SHA256_BLOCK_SIZE > SHA256_BLOCK_SIZE) + chunk = SHA256_BLOCK_SIZE - + sctx->count % SHA256_BLOCK_SIZE; + kernel_neon_begin(); + sha256_base_do_update(desc, data, chunk, + (sha256_block_fn *)sha256_block_neon); + kernel_neon_end(); + data += chunk; + len -= chunk; + } return 0; } @@ -117,10 +128,9 @@ static int sha256_finup_neon(struct shash_desc *desc, const u8 *data, sha256_base_do_finalize(desc, (sha256_block_fn *)sha256_block_data_order); } else { - kernel_neon_begin(); if (len) - sha256_base_do_update(desc, data, len, - (sha256_block_fn *)sha256_block_neon); + sha256_update_neon(desc, data, len); + kernel_neon_begin(); sha256_base_do_finalize(desc, (sha256_block_fn *)sha256_block_neon); kernel_neon_end();