From patchwork Sat Mar 10 15:22:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10273685 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E6AAB601A0 for ; Sat, 10 Mar 2018 15:35:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D482C28EAB for ; Sat, 10 Mar 2018 15:35:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C5A8E29A6C; Sat, 10 Mar 2018 15:35:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 458CF28EAB for ; Sat, 10 Mar 2018 15:35:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1L2+w/HEZyP81PaUmw6Uniu/BedEOS+alQbT90yQVEk=; b=BIMXqF+HMTItMej02TNMNzdn7P XVFvL9Xh6TEEpnGoshKkVkvhepU/GVYigt5ifLepPSXFEm9Ces9Jbj0CYxoV5nHufDchC0yPtHeLH xxtR0vkSa1Rdd9u5CauWaSPdu0e9SE72DDP72ThGX+i4u7HW03sNJXj/yYgBkYaPCLwtIkwJ77FEL SULHE1IuUm4e5qsTZe1DCiam9EnJtmxax9uFWZhMU177xaT0oycCsqvFyP5QR3+8ts7uhQ1MFRhCl fJsZQiblJm86Q8yIQUuAvUxyeKH8U5tAnxigyDkUbbpkhQ/6sxdVLbwZ2ChDI85Nnzs6ml6xQ5QzZ ZEPBIfhw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eugWl-00019g-4A; Sat, 10 Mar 2018 15:35:19 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eugLl-0000FA-P0 for linux-arm-kernel@bombadil.infradead.org; Sat, 10 Mar 2018 15:23:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ONrynQy694N0f18mAFgrnch1E0o/w8Y4ap1M7Kv/2PQ=; b=HwNO8zYBo4t2ituh9/Z8CVimR 3BCcrXQS26MbEiEyjoism+oCKcayEdGNCd3JQHBxdU8K7eHAAKnxXuh/yZOylHgwEoGKQ08qxEj8O 55AFoEohmxTJh0y3PRjOdZQoZA6nGuqS+qUoJzfLAolhs/5RZihhfY67ajibNuM5KWWOVWT4zF6GS arSTHyp2cnYnVpGnoikdcNPdBQD7NGobfuz1RBwQ15JCtA1nY81bzOi1z5nrJYO4Yi9znBCrgecBl LB7H5Bz7cmKVJ+bbILgc6/K+h26vCxmH2QzeT/+lIab/9HEe4/9hyrX01/t2Cm94lldk1XDNCIus4 aOkBIm8vw==; Received: from mail-wr0-x242.google.com ([2a00:1450:400c:c0c::242]) by merlin.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eugLh-0000l9-Gn for linux-arm-kernel@lists.infradead.org; Sat, 10 Mar 2018 15:23:55 +0000 Received: by mail-wr0-x242.google.com with SMTP id n12so503027wra.2 for ; Sat, 10 Mar 2018 07:23:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ONrynQy694N0f18mAFgrnch1E0o/w8Y4ap1M7Kv/2PQ=; b=TkqZNI/SMik+PW8VNPdSpSjqYOXY0AgsToTWPleZMIUoUcr4mbLXuTxlc6ryGal7Ll QbJjSHnu87JjN+dumPSPN04z1wbDIAup0TbJ8/yxiIhh+/LLS/A4KS3LqrzRqn9ZxTRg 7RvJR98iQYfzRnUVkTcR64YFK/BV65z071L9I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ONrynQy694N0f18mAFgrnch1E0o/w8Y4ap1M7Kv/2PQ=; b=M9j+e0uf/nMJbR4BzaGeOiscM/rCY7/+/5zhbTazvF0gY/5/zQPKa+7LReVNZI5Kb+ qM+vJsmeQkRZYoeK/5to0jdh7FzmCo/skb3q2EhtlAZVZ2E4m1ZyQ3ONxiaAT99Tqocm wF/5eMS7RnWjWhsKubARs/RDMphhq1ui5Ij/YSy3+imNwaWfDA1qdSCf6voAHbMsGFcs bMEYVa1ruTb9omFB/RI91hGxeaEe4cyqdhqO5Mrge5cOx4IIpGN2/SLwr3/DPamh8+J6 DvbWHk2wyfeq3Zngoa+bhdcno0FeUt1x53LgLeNOpT6ZPjpwkRyMFEI+InZZgPGMuygy BX+g== X-Gm-Message-State: AElRT7G8QH3OYO8M3G1oMQ47MIFPfsNj8P9eb6l+7qXzJWdC4KlJJmg6 MQqYkdUd1iIT/sPzvetS5THTYQ== X-Google-Smtp-Source: AG47ELtUKfBcdfE278MeGSbUbsVbTBzU7Sc++0gBHPGNrOL3z5pNm2F+SuyhqIHiMgLF1lPQP3qMJA== X-Received: by 10.223.147.36 with SMTP id 33mr1659969wro.133.1520695417246; Sat, 10 Mar 2018 07:23:37 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.23.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:23:36 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH v5 22/23] crypto: arm64/sm3-ce - yield NEON after every block of input Date: Sat, 10 Mar 2018 15:22:07 +0000 Message-Id: <20180310152208.10369-23-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180310_102353_793083_6C7A7E4A X-CRM114-Status: GOOD ( 14.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , herbert@gondor.apana.org.au, Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , Sebastian Andrzej Siewior , Will Deacon , Russell King - ARM Linux , Steven Rostedt , Thomas Gleixner , Dave Martin , linux-arm-kernel@lists.infradead.org, linux-rt-users@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Avoid excessive scheduling delays under a preemptible kernel by conditionally yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm3-ce-core.S | 30 +++++++++++++++----- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/arch/arm64/crypto/sm3-ce-core.S b/arch/arm64/crypto/sm3-ce-core.S index 27169fe07a68..5a116c8d0cee 100644 --- a/arch/arm64/crypto/sm3-ce-core.S +++ b/arch/arm64/crypto/sm3-ce-core.S @@ -77,19 +77,25 @@ */ .text ENTRY(sm3_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load state */ - ld1 {v8.4s-v9.4s}, [x0] + ld1 {v8.4s-v9.4s}, [x19] rev64 v8.4s, v8.4s rev64 v9.4s, v9.4s ext v8.16b, v8.16b, v8.16b, #8 ext v9.16b, v9.16b, v9.16b, #8 - adr_l x8, .Lt +0: adr_l x8, .Lt ldp s13, s14, [x8] /* load input */ -0: ld1 {v0.16b-v3.16b}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v0.16b-v3.16b}, [x20], #64 + sub w21, w21, #1 mov v15.16b, v8.16b mov v16.16b, v9.16b @@ -125,14 +131,24 @@ CPU_LE( rev32 v3.16b, v3.16b ) eor v9.16b, v9.16b, v16.16b /* handled all input blocks? */ - cbnz w2, 0b + cbz w21, 2f + + if_will_cond_yield_neon + st1 {v8.4s-v9.4s}, [x19] + do_cond_yield_neon + ld1 {v8.4s-v9.4s}, [x19] + b 0b + endif_yield_neon + + b 1b /* save state */ - rev64 v8.4s, v8.4s +2: rev64 v8.4s, v8.4s rev64 v9.4s, v9.4s ext v8.16b, v8.16b, v8.16b, #8 ext v9.16b, v9.16b, v9.16b, #8 - st1 {v8.4s-v9.4s}, [x0] + st1 {v8.4s-v9.4s}, [x19] + frame_pop ret ENDPROC(sm3_ce_transform)