From patchwork Thu Jun 6 15:03:04 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 2681341 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork2.kernel.org Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by patchwork2.kernel.org (Postfix) with ESMTP id 73808DF23A for ; Thu, 6 Jun 2013 15:06:39 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1UkbkD-000352-35; Thu, 06 Jun 2013 15:04:54 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Ukbjl-0003EV-MU; Thu, 06 Jun 2013 15:04:25 +0000 Received: from mail-wg0-x233.google.com ([2a00:1450:400c:c00::233]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1UkbjI-00039u-JE for linux-arm-kernel@lists.infradead.org; Thu, 06 Jun 2013 15:03:58 +0000 Received: by mail-wg0-f51.google.com with SMTP id b13so2208526wgh.30 for ; Thu, 06 Jun 2013 08:03:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references :x-gm-message-state; bh=iBFhMNMPZf/89612P89c2+AK1kTlqA8EsVuLgcxGk2I=; b=nwNsvmruvtqgMf7ouZNbleyUv72gDGlNFqlRd6jXskTicDEFXEsGzaZolp+ohuPpDk E6+p4tum4iRx+f+5BYm/hvLqfK5NhfpPQnk3LPWmFGW2xSXP1t1V+dVcnnUPyEf72yhe vUUCybe1djoBSJZXtFjIF/aAIhMMnUo6YY0rwwzh7IXCrntGrLFj61RP2NoohYBt7bto upqIuf8q+isQNL7zy2EICYEG5sOZvckklOkkfksyHGow1An824BcclymmZ/0bWnoH+Zw SGx82cD8yMbHCR3pqKdmEHv3CiK2dC1GAOWaEgOZR0ztOL59hJRno985Gu5XapGaxKTO BnZw== X-Received: by 10.180.188.141 with SMTP id ga13mr10379710wic.9.1370531014736; Thu, 06 Jun 2013 08:03:34 -0700 (PDT) Received: from ards-mac-mini.local (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id cw8sm16357697wib.7.2013.06.06.08.03.33 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Jun 2013 08:03:34 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 4/5] ARM: crypto: add NEON accelerated XOR implementation Date: Thu, 6 Jun 2013 17:03:04 +0200 Message-Id: <1370530985-20619-5-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.1.2 In-Reply-To: <1370530985-20619-1-git-send-email-ard.biesheuvel@linaro.org> References: <1370530985-20619-1-git-send-email-ard.biesheuvel@linaro.org> X-Gm-Message-State: ALoCoQlWdeAJeInJtYwReAvvSnrIfDjC194LfhthBmauoq1bj5tXnxDog4jDOMAi3Wb5b/Sz2mhx X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20130606_110356_934612_CF23786F X-CRM114-Status: GOOD ( 16.80 ) X-Spam-Score: -1.9 (-) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-1.9 points) pts rule name description ---- ---------------------- -------------------------------------------------- -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: Ard Biesheuvel , patches@linaro.org, linux@arm.linux.org.uk, rob.herring@calxeda.com, nico@linaro.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Add a source file xor-neon.c (which is really just the reference C implementation passed through the GCC vectorizer) and hook it up to the XOR framework. Output captured from a Cortex-A15 @ 1.7 GHz: xor: measuring software checksum speed arm4regs : 2261.600 MB/sec 8regs : 1771.600 MB/sec 32regs : 1441.600 MB/sec neon : 3619.600 MB/sec xor: using function: neon (3619.600 MB/sec) Signed-off-by: Ard Biesheuvel Acked-by: Nicolas Pitre --- arch/arm/include/asm/xor.h | 73 ++++++++++++++++++++++++++++++++++++++++++++++ arch/arm/lib/Makefile | 6 ++++ arch/arm/lib/xor-neon.c | 42 ++++++++++++++++++++++++++ 3 files changed, 121 insertions(+) create mode 100644 arch/arm/lib/xor-neon.c diff --git a/arch/arm/include/asm/xor.h b/arch/arm/include/asm/xor.h index 7604673..4ffb26d 100644 --- a/arch/arm/include/asm/xor.h +++ b/arch/arm/include/asm/xor.h @@ -7,7 +7,10 @@ * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. */ +#include #include +#include +#include #define __XOR(a1, a2) a1 ^= a2 @@ -138,4 +141,74 @@ static struct xor_block_template xor_block_arm4regs = { xor_speed(&xor_block_arm4regs); \ xor_speed(&xor_block_8regs); \ xor_speed(&xor_block_32regs); \ + NEON_TEMPLATES; \ } while (0) + +#ifdef CONFIG_KERNEL_MODE_NEON + +extern struct xor_block_template const xor_block_neon_inner; + +static void +xor_neon_2(unsigned long bytes, unsigned long *p1, unsigned long *p2) +{ + if (in_interrupt()) { + xor_arm4regs_2(bytes, p1, p2); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_2(bytes, p1, p2); + kernel_neon_end(); + } +} + +static void +xor_neon_3(unsigned long bytes, unsigned long *p1, unsigned long *p2, + unsigned long *p3) +{ + if (in_interrupt()) { + xor_arm4regs_3(bytes, p1, p2, p3); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_3(bytes, p1, p2, p3); + kernel_neon_end(); + } +} + +static void +xor_neon_4(unsigned long bytes, unsigned long *p1, unsigned long *p2, + unsigned long *p3, unsigned long *p4) +{ + if (in_interrupt()) { + xor_arm4regs_4(bytes, p1, p2, p3, p4); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4); + kernel_neon_end(); + } +} + +static void +xor_neon_5(unsigned long bytes, unsigned long *p1, unsigned long *p2, + unsigned long *p3, unsigned long *p4, unsigned long *p5) +{ + if (in_interrupt()) { + xor_arm4regs_5(bytes, p1, p2, p3, p4, p5); + } else { + kernel_neon_begin(); + xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5); + kernel_neon_end(); + } +} + +static struct xor_block_template xor_block_neon = { + .name = "neon", + .do_2 = xor_neon_2, + .do_3 = xor_neon_3, + .do_4 = xor_neon_4, + .do_5 = xor_neon_5 +}; + +#define NEON_TEMPLATES \ + do { if (cpu_has_neon()) xor_speed(&xor_block_neon); } while (0) +#else +#define NEON_TEMPLATES +#endif diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile index af72969..aaf3a87 100644 --- a/arch/arm/lib/Makefile +++ b/arch/arm/lib/Makefile @@ -45,3 +45,9 @@ lib-$(CONFIG_ARCH_SHARK) += io-shark.o $(obj)/csumpartialcopy.o: $(obj)/csumpartialcopygeneric.S $(obj)/csumpartialcopyuser.o: $(obj)/csumpartialcopygeneric.S + +ifeq ($(CONFIG_KERNEL_MODE_NEON),y) + NEON_FLAGS := -mfloat-abi=softfp -mfpu=neon + CFLAGS_xor-neon.o += $(NEON_FLAGS) + lib-$(CONFIG_XOR_BLOCKS) += xor-neon.o +endif diff --git a/arch/arm/lib/xor-neon.c b/arch/arm/lib/xor-neon.c new file mode 100644 index 0000000..f485e5a --- /dev/null +++ b/arch/arm/lib/xor-neon.c @@ -0,0 +1,42 @@ +/* + * linux/arch/arm/lib/xor-neon.c + * + * Copyright (C) 2013 Linaro Ltd + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include + +#ifndef __ARM_NEON__ +#error You should compile this file with '-mfloat-abi=softfp -mfpu=neon' +#endif + +/* + * Pull in the reference implementations while instructing GCC (through + * -ftree-vectorize) to attempt to exploit implicit parallelism and emit + * NEON instructions. + */ +#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 6) +#pragma GCC optimize "tree-vectorize" +#else +/* + * While older versions of GCC do not generate incorrect code, they fail to + * recognize the parallel nature of these functions, and emit plain ARM code, + * which is known to be slower than the optimized ARM code in asm-arm/xor.h. + */ +#warning This code requires at least version 4.6 of GCC +#endif + +#pragma GCC diagnostic ignored "-Wunused-variable" +#include + +struct xor_block_template const xor_block_neon_inner = { + .name = "__inner_neon__", + .do_2 = xor_8regs_2, + .do_3 = xor_8regs_3, + .do_4 = xor_8regs_4, + .do_5 = xor_8regs_5, +};