From patchwork Sat Mar 10 15:21:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10273657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9927B60594 for ; Sat, 10 Mar 2018 15:27:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8660729563 for ; Sat, 10 Mar 2018 15:27:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 79E6B29765; Sat, 10 Mar 2018 15:27:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E716829563 for ; Sat, 10 Mar 2018 15:27:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/IF+iyuHWOFa+CZwpEQRvfM5ed6vk20x2+VAqp8eFPQ=; b=A76SddVixUIH9dWr65NLaLlhxs JtIfs+9s1Qct/Ij26wJuXOnCvXNfxOWUexl5c4/YUTkbrrillysvcj8ny0R01oNHQD8I86QvZg2nW 9fRe/PWAum/YCIG+PUd37j9BeIjRypLnij+UpFrOok48Lwq1sNyyS32ymLW5I6JtROUnyhd9Kdqce b7E2ilcCtPKJR7bGWArm/yjxUdWCcly4YsdKkHd3pD09vYeLKMIRqBKCD9+aUM1Ei+GKAP6igCGpD DCmyzz8jL0EgMPQWDKcIPbRjQB09uDsvK5JwOTwx6roZ1XMq8U2IQIaFHql4zA2Wac0xJpPgL+jTb YRqZkOsQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eugOm-0003A4-NJ; Sat, 10 Mar 2018 15:27:04 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eugL9-0008Lv-Dl for linux-arm-kernel@bombadil.infradead.org; Sat, 10 Mar 2018 15:23:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=b5L/U4UbL5kjXRpLckrVX3q+qInCo8IzJ15unk/D7c0=; b=E92qjbqCmuFFetW3zeL0pwBwy ia4NecQ6UT9Rt9mAMK0VpszWP37QJpricYJpfTrQWvV7MetFqZUys8Ayve7q67VTYqj6bYC0C34q0 uDedR0WcBElBOSL2gn4bxecf6wLgtmJO4l5+FfQrVVcgqROkQlcuQ+fsjp3HdpYetwtNfieLeyApN A2lhBGHANKiafFWgsHFK/vZEzt4sPjQfs7AzheGSsKoL5QhQnLNVIMP0Cam257xptJ8/f/5E0dp7j Z2xLUmM4Ze9P4gJHnMSUGjMncnpW1kzG/ks7jt12YFVHqNn0PvSo6PdJ86aBIChNHNfD0kMqb4XPn FLmBUymwQ==; Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]) by casper.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eugL6-0002g9-BA for linux-arm-kernel@lists.infradead.org; Sat, 10 Mar 2018 15:23:18 +0000 Received: by mail-wm0-x242.google.com with SMTP id 139so8786222wmn.2 for ; Sat, 10 Mar 2018 07:23:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=b5L/U4UbL5kjXRpLckrVX3q+qInCo8IzJ15unk/D7c0=; b=BlX59uZrJIhgn55fe1G3KrfItzjy3qR0gn6UBxOds1HAum6BqyJzBJv39VjU/48MC4 Z0YdzpM66a1FOheE4l8rum8zy3o5b9+C6+dN1HOip1E0zWU1w3767SHP3iYmKjg7zXx5 tG2sGTeZmWcRWDJk8Co8aW6z0ROZfDMBfIaZE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=b5L/U4UbL5kjXRpLckrVX3q+qInCo8IzJ15unk/D7c0=; b=P7TJ/Gl7JT8vIG5uwl968JfiXLIq8/LOLMfsADJMPbYVv1H7E2qJlcYrgjlPBGlNfj s9b85ViYoKegDaVpkPdy1JY8o5jk6GSpy9LeNrZ1zMJGMIwq7T8e23qlRt2cuniqt2vp 1I5Z4zyJWTapp/AO4aGqAlED1Y0j/fY/Ir91Yq9I8vXBMc3e14Cn1i+JvuZdeQk/ScdJ ov88Tnc6coXKT3Mdq9/Vph3DKSzOKgqI/BLeBSoKE4GTpPJYHoxuHOd0OrpLOzNXnewA XwWmWEorXkF47zeXTrQ6wYjpGFJ3hMB6zqTlcpg/H2QACJmalHjQ19v4V3kziMdm33hc CASw== X-Gm-Message-State: AElRT7E8ojMb4QnycurQP7Oe4i7xkijFh/ATzH+A8txF8pROvGntIsIo PigGXyLLiJ7chzJY8Ri0FIxFAA== X-Google-Smtp-Source: AG47ELuYZ1n1tKhsRYVvSbDOSoxolgK5m8FLxjlvwpgHU4WMcmUdPW6kVvNiSCVK2UduxkwQScHV5g== X-Received: by 10.28.109.90 with SMTP id i87mr1336898wmc.71.1520695385165; Sat, 10 Mar 2018 07:23:05 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.23.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:23:04 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH v5 12/23] crypto: arm64/sha1-ce - yield NEON after every block of input Date: Sat, 10 Mar 2018 15:21:57 +0000 Message-Id: <20180310152208.10369-13-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180310_152316_380767_26361336 X-CRM114-Status: GOOD ( 15.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , herbert@gondor.apana.org.au, Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , Sebastian Andrzej Siewior , Will Deacon , Russell King - ARM Linux , Steven Rostedt , Thomas Gleixner , Dave Martin , linux-arm-kernel@lists.infradead.org, linux-rt-users@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Avoid excessive scheduling delays under a preemptible kernel by conditionally yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha1-ce-core.S | 42 ++++++++++++++------ 1 file changed, 29 insertions(+), 13 deletions(-) diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S index 46049850727d..78eb35fb5056 100644 --- a/arch/arm64/crypto/sha1-ce-core.S +++ b/arch/arm64/crypto/sha1-ce-core.S @@ -69,30 +69,36 @@ * int blocks) */ ENTRY(sha1_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load round constants */ - loadrc k0.4s, 0x5a827999, w6 +0: loadrc k0.4s, 0x5a827999, w6 loadrc k1.4s, 0x6ed9eba1, w6 loadrc k2.4s, 0x8f1bbcdc, w6 loadrc k3.4s, 0xca62c1d6, w6 /* load state */ - ld1 {dgav.4s}, [x0] - ldr dgb, [x0, #16] + ld1 {dgav.4s}, [x19] + ldr dgb, [x19, #16] /* load sha1_ce_state::finalize */ ldr_l w4, sha1_ce_offsetof_finalize, x4 - ldr w4, [x0, x4] + ldr w4, [x19, x4] /* load input */ -0: ld1 {v8.4s-v11.4s}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v8.4s-v11.4s}, [x20], #64 + sub w21, w21, #1 CPU_LE( rev32 v8.16b, v8.16b ) CPU_LE( rev32 v9.16b, v9.16b ) CPU_LE( rev32 v10.16b, v10.16b ) CPU_LE( rev32 v11.16b, v11.16b ) -1: add t0.4s, v8.4s, k0.4s +2: add t0.4s, v8.4s, k0.4s mov dg0v.16b, dgav.16b add_update c, ev, k0, 8, 9, 10, 11, dgb @@ -123,16 +129,25 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgbv.2s, dgbv.2s, dg1v.2s add dgav.4s, dgav.4s, dg0v.4s - cbnz w2, 0b + cbz w21, 3f + + if_will_cond_yield_neon + st1 {dgav.4s}, [x19] + str dgb, [x19, #16] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* * Final block: add padding and total bit count. * Skip if the input size was not a round multiple of the block size, * the padding is handled by the C code in that case. */ - cbz x4, 3f +3: cbz x4, 4f ldr_l w4, sha1_ce_offsetof_count, x4 - ldr x4, [x0, x4] + ldr x4, [x19, x4] movi v9.2d, #0 mov x8, #0x80000000 movi v10.2d, #0 @@ -141,10 +156,11 @@ CPU_LE( rev32 v11.16b, v11.16b ) mov x4, #0 mov v11.d[0], xzr mov v11.d[1], x7 - b 1b + b 2b /* store new state */ -3: st1 {dgav.4s}, [x0] - str dgb, [x0, #16] +4: st1 {dgav.4s}, [x19] + str dgb, [x19, #16] + frame_pop ret ENDPROC(sha1_ce_transform)