From patchwork Mon Aug 27 11:02:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10576961 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87422920 for ; Mon, 27 Aug 2018 11:05:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74C3C29668 for ; Mon, 27 Aug 2018 11:05:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67F5C297F7; Mon, 27 Aug 2018 11:05:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7438929668 for ; Mon, 27 Aug 2018 11:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZFH51NwYpdOz5A+mpkyrigAjgnPNznJYkBG+MTdLYBw=; b=IcB2x7WMN+5bkjJJsqpBgZFEiY aZC6k8wrO1ybQW5SMJtaVt1OdwUjHqoIWKnnxz6Guchv2l52UCfaa6RUsYaAwtzYUqb+aNZKJk5aJ m4ihHYyNBN7sX7vcjodtwkiSMDixAfi6jFnTBZB/n+hZjrL5m5nx+SkBl0nlG3dwzb0RZnmnZL9sX LeMpnwFWOsiTQjVxCWZWuVdhggxx+lLU11i1lKRrQAret06Yo28nkrcMLeVD5HjtWVJVT0DxU49tF gXEBUREGeyhXTXqCsh5sH0745ncYC32+yXJM5h0tbFYsP2maYHeaNiIv3KXNpPIjjRWVmq8yPzweZ PrJQ/acw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fuFKw-00055C-QK; Mon, 27 Aug 2018 11:05:35 +0000 Received: from mail-ed1-x541.google.com ([2a00:1450:4864:20::541]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fuFIg-0002cl-9v for linux-arm-kernel@lists.infradead.org; Mon, 27 Aug 2018 11:03:19 +0000 Received: by mail-ed1-x541.google.com with SMTP id f38-v6so10133829edd.8 for ; Mon, 27 Aug 2018 04:03:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RBWagJrS8izqpWfSj/uAIGFEJFmec204/M/NCMHphvQ=; b=JoYpMQeDGKZ1zPIb0BMTotGaKg0SeZ72J+k3jcjwarBgImQcQVqOO6lafaAInE26Jb K3luZE7h/FLn/GR8AopVaDIhhpcGxZUvQRhxg6UR/95nwjim2/Ll8c9PGqeQz54smymW 2+v2iqer31zdShwwuAKCTtkmD7B45t1uZowzg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RBWagJrS8izqpWfSj/uAIGFEJFmec204/M/NCMHphvQ=; b=NNWlUvrnwneHDEzenxFqhAkRFst4C35wNRupqNPnoFzqpU91XXqOP7/Y0TCNP5HeiM JCZmC6SgksQWoJ9PkvlFnyATJtaWIVmqAW1hudD19N/BK8hVCuCrLtwkYda9gahn5DM/ SoG2gl8sM9KGZsgooQryox5887sfAZkDlO17BtAY1SvH5CHi1YUzZX1yPvB2EITrcPLW WDQsw3cb8Z2hclJNrmidzSvzSMZEWhkL8gs0hE6Hqc5h0b8RdhK7kpF5RZp76+k3R+/R rhKAlw7tJUCTFMtzclOETYo8RZxV9nYcSTFtSBHjYyXjnrRCD6rOu1+1cSQI+COktP6q YC1A== X-Gm-Message-State: APzg51BMad0rC0gYX6boYf1RJNfEmihhyGTwkWK4sWeDdkR72S2wL+GO jNf0VQCcavdvFKxrXsdtr+TYLXfPvQnE7hCO X-Google-Smtp-Source: ANB0VdZ8biaPTWv8M/yLq9HTb+Wu9LRY8RkOyG+W4lu040cGjccHqpmmNhIbm67C2hefIOD82IrTxQ== X-Received: by 2002:a50:8386:: with SMTP id 6-v6mr16195527edi.170.1535367783703; Mon, 27 Aug 2018 04:03:03 -0700 (PDT) Received: from rev02.home ([2a02:a212:9283:9800:24b9:e2d6:9acc:50dd]) by smtp.gmail.com with ESMTPSA id r2-v6sm3114344eda.89.2018.08.27.04.03.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Aug 2018 04:03:02 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org Subject: [PATCH 3/4] arm64/lib: add accelerated crc32 routines Date: Mon, 27 Aug 2018 13:02:44 +0200 Message-Id: <20180827110245.14812-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180827110245.14812-1-ard.biesheuvel@linaro.org> References: <20180827110245.14812-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180827_040314_360983_AED699D8 X-CRM114-Status: GOOD ( 16.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ard Biesheuvel , herbert@gondor.apana.org.au, suzuki.poulose@arm.com, catalin.marinas@arm.com, ebiggers@google.com, will.deacon@arm.com, linux-kernel@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Unlike crc32c(), which is wired up to the crypto API internally so the optimal driver is selected based on the platform's capabilities, crc32_le() is implemented as a library function using a slice-by-8 table based C implementation. Even though few of the call sites may be bottlenecks, calling a time variant implementation with a non-negligible D-cache footprint is a bit of a waste, given that ARMv8.1 and up mandates support for the CRC32 instructions that were optional in ARMv8.0, but are already widely available, even on the Cortex-A53 based Raspberry Pi. So implement routines that use these instructions if available, and fall back to the existing generic routines otherwise. The selection is based on alternatives patching. Note that this unconditionally selects CONFIG_CRC32 as a builtin. Since CRC32 is relied upon by core functionality such as CONFIG_OF_FLATTREE, this just codifies the status quo. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 + arch/arm64/lib/Makefile | 2 + arch/arm64/lib/crc32.S | 60 ++++++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 29e75b47becd..0625355f12fa 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -75,6 +75,7 @@ config ARM64 select CLONE_BACKWARDS select COMMON_CLK select CPU_PM if (SUSPEND || CPU_IDLE) + select CRC32 select DCACHE_WORD_ACCESS select DMA_DIRECT_OPS select EDAC_SUPPORT diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 68755fd70dcf..f28f91fd96a2 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -25,3 +25,5 @@ KCOV_INSTRUMENT_atomic_ll_sc.o := n UBSAN_SANITIZE_atomic_ll_sc.o := n lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o + +obj-$(CONFIG_CRC32) += crc32.o diff --git a/arch/arm64/lib/crc32.S b/arch/arm64/lib/crc32.S new file mode 100644 index 000000000000..5bc1e85b4e1c --- /dev/null +++ b/arch/arm64/lib/crc32.S @@ -0,0 +1,60 @@ +/* + * Accelerated CRC32(C) using AArch64 CRC instructions + * + * Copyright (C) 2016 - 2018 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include + + .cpu generic+crc + + .macro __crc32, c +0: subs x2, x2, #16 + b.mi 8f + ldp x3, x4, [x1], #16 +CPU_BE( rev x3, x3 ) +CPU_BE( rev x4, x4 ) + crc32\c\()x w0, w0, x3 + crc32\c\()x w0, w0, x4 + b.ne 0b + ret + +8: tbz x2, #3, 4f + ldr x3, [x1], #8 +CPU_BE( rev x3, x3 ) + crc32\c\()x w0, w0, x3 +4: tbz x2, #2, 2f + ldr w3, [x1], #4 +CPU_BE( rev w3, w3 ) + crc32\c\()w w0, w0, w3 +2: tbz x2, #1, 1f + ldrh w3, [x1], #2 +CPU_BE( rev16 w3, w3 ) + crc32\c\()h w0, w0, w3 +1: tbz x2, #0, 0f + ldrb w3, [x1] + crc32\c\()b w0, w0, w3 +0: ret + .endm + + .align 5 +ENTRY(crc32_le) +alternative_if_not ARM64_HAS_CRC32 + b crc32_le_base +alternative_else_nop_endif + __crc32 +ENDPROC(crc32_le) + + .align 5 +ENTRY(__crc32c_le) +alternative_if_not ARM64_HAS_CRC32 + b __crc32c_le_base +alternative_else_nop_endif + __crc32 c +ENDPROC(__crc32c_le)