From patchwork Sat Mar 10 15:21:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10273665 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1068360594 for ; Sat, 10 Mar 2018 15:29:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F091E299DB for ; Sat, 10 Mar 2018 15:29:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E0ABF299E3; Sat, 10 Mar 2018 15:29:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 68000299DB for ; Sat, 10 Mar 2018 15:29:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=918E9HewUTtp4Js364ASTucHbXN2tYjXabMSMdddf3o=; b=WTpAFk1zwpiuXa8B8J9/afsVdA IujgltnXmnV6aiGVy8+Nr5s6o5RmtDO306VtpeHQvdr45Xyrw5ONDd+oJxhCHqqdKA3npeVOOqSyI o+KSrlabkoDlSY61Q+WdLPvw6luCrWYsNKUGIzCjbUpc+DEVjZxLT9cEsgzMWSmfpAM0nmho1Qi3Z BGyXAzHsWoETInFHMA6tGwxml3C6eqj7D5EXF1pgAYA/CQuNIQrpX2Jm2353UC5KssFrGfwk42UoM lR0VdO9XvJBUiXIYVEKvCjzmSC3Z24MNrS8PtvjxgBG/TDBnpYGa6WCOVdJVmW24n9GBVlF1grxMX kn/l5XPg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eugQy-0004QN-6F; Sat, 10 Mar 2018 15:29:20 +0000 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eugKl-000801-Fu for linux-arm-kernel@lists.infradead.org; Sat, 10 Mar 2018 15:23:05 +0000 Received: by mail-wm0-x242.google.com with SMTP id a20so7233498wmd.1 for ; Sat, 10 Mar 2018 07:22:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=Nc80v7ii0S8D+KJt8wa4vOl+GDzwV1Kka+jcAUJiwWvaMRm3Zb6wyFwN8DcPWLTr87 NaGF1vcrk2pbf9JekkSemNPsthQVkJj87zu5bG3pmpLWJK+bOe5wuboIrXxNJq2lQYcm HK0ereFQisULL6CyFhM4qkZLEkGPHZSoqKvQM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o9DpJHdXibth1JpdGJrmSP+z+3/O3SvHO7sSQoHB+GE=; b=nhRyhK7x6E//2CJ/nuFYxsFVeyXRhcRhI3sVtDVlUYbLvEyya4Q+EAceaQmo6TvwM1 JhoIemBhCB2UaODfNiY399Ju7xLwZH9BDktybCdjj22eP4cWJgZegE1O5uGTsVAkcMqC bZTd+0/cgFtCC/RI4HKXwJBX4FiWhC49uY4vkKlSUrVI84BqzxY1IisMsK7BRumjfpDH fb218oTO1zeOm2ncVWRpFOSYLdWHoPjOxMhQ+NSe5Cixk41YXde6UVG6kzkKDDh1fIFZ yq6SdnJlKxKMjtVMCs3RIeiCjJUK/p7CXMveL7bf/l5WwWPwSQklUZQAqLN++vIzRnHi Em2A== X-Gm-Message-State: AElRT7GTMi7E3unbiAXmgxcoDrdjrQdhJShGRVesisRqS1FLu/CV7ut3 MzA1wEIMeDIPNZbTKAOhtdMcwg== X-Google-Smtp-Source: AG47ELuWqsAu+J8adDvjfDBPNK0lUZLbV/WD0On2OLUqKsXs8/iNozdbb8JoNyJ9ucqCLXWaf4AACQ== X-Received: by 10.28.26.202 with SMTP id a193mr1436610wma.138.1520695363656; Sat, 10 Mar 2018 07:22:43 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:43 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH v5 05/23] crypto: arm64/chacha20 - move kernel mode neon en/disable into loop Date: Sat, 10 Mar 2018 15:21:50 +0000 Message-Id: <20180310152208.10369-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180310_072256_385397_BC81C61F X-CRM114-Status: GOOD ( 18.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , herbert@gondor.apana.org.au, Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , Sebastian Andrzej Siewior , Will Deacon , Russell King - ARM Linux , Steven Rostedt , Thomas Gleixner , Dave Martin , linux-arm-kernel@lists.infradead.org, linux-rt-users@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/chacha20-neon-glue.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/crypto/chacha20-neon-glue.c b/arch/arm64/crypto/chacha20-neon-glue.c index cbdb75d15cd0..727579c93ded 100644 --- a/arch/arm64/crypto/chacha20-neon-glue.c +++ b/arch/arm64/crypto/chacha20-neon-glue.c @@ -37,12 +37,19 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, u8 buf[CHACHA20_BLOCK_SIZE]; while (bytes >= CHACHA20_BLOCK_SIZE * 4) { + kernel_neon_begin(); chacha20_4block_xor_neon(state, dst, src); + kernel_neon_end(); bytes -= CHACHA20_BLOCK_SIZE * 4; src += CHACHA20_BLOCK_SIZE * 4; dst += CHACHA20_BLOCK_SIZE * 4; state[12] += 4; } + + if (!bytes) + return; + + kernel_neon_begin(); while (bytes >= CHACHA20_BLOCK_SIZE) { chacha20_block_xor_neon(state, dst, src); bytes -= CHACHA20_BLOCK_SIZE; @@ -55,6 +62,7 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, chacha20_block_xor_neon(state, buf, buf); memcpy(dst, buf, bytes); } + kernel_neon_end(); } static int chacha20_neon(struct skcipher_request *req) @@ -68,11 +76,10 @@ static int chacha20_neon(struct skcipher_request *req) if (!may_use_simd() || req->cryptlen <= CHACHA20_BLOCK_SIZE) return crypto_chacha20_crypt(req); - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); crypto_chacha20_init(state, ctx, walk.iv); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int nbytes = walk.nbytes; @@ -83,7 +90,6 @@ static int chacha20_neon(struct skcipher_request *req) nbytes); err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - kernel_neon_end(); return err; }