From patchwork Wed Aug 19 00:02:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Augusto von Dentz X-Patchwork-Id: 11722311 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 208D21392 for ; Wed, 19 Aug 2020 00:02:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D28FE207BB for ; Wed, 19 Aug 2020 00:02:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WBTJdZVa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727082AbgHSACQ (ORCPT ); Tue, 18 Aug 2020 20:02:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726676AbgHSACP (ORCPT ); Tue, 18 Aug 2020 20:02:15 -0400 Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2364FC061342 for ; Tue, 18 Aug 2020 17:02:14 -0700 (PDT) Received: by mail-pg1-x535.google.com with SMTP id i10so4961076pgk.1 for ; Tue, 18 Aug 2020 17:02:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=yQ4LWb684mNHZcH5T+SHWEtZWJfEIJln+Io+MEEnEtM=; b=WBTJdZValcXOX61h8fx+y7EXUCnH8IOQfgo/Yl05CnDlYSpGOu2iTlixktq1wdqxjF h+LHNafSUclY9dgjd1iJF4UZGNjOhNC+dyZy3yUqNWFSRTZh153Xi5GpYyCAgWUuK77s G1FKdvqpbGNwExuJVxjnHzYGDaShX/rSlTPwIO0M3syjLJFINwqECEq3P7JYEWtciyJN MSo07VzE+fCA2nptWv07k0FCfArF8tjxWBKI9vwZCQCvemDhRdxuT6LgU08lMeFimezE vfeJ3KmambguXRfSrZuQt5KQ01nq5eZaJAW8tsUzQh6BrjnRbKqiluSFkfrzVbhOKqC0 89LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=yQ4LWb684mNHZcH5T+SHWEtZWJfEIJln+Io+MEEnEtM=; b=a36mcCzwtg4t+Xy89mEFYkNdTfeIlCb9/GyA1NTS0UV7Ldz1MfoThpV/UiEylPpWFK SrMfVMiZqfJ3SmG5VurjnAqDU8lOsPJmsg9A4mjCfmv0vKIrjOjItjUQoY8/VTUksNPf hk+BLZMvbBe13Bh1EWGC7XwJrjn7sZqFwD4Cnk0FtJu7OJXD9u41ptO2v20r4CtzQ9cN LX5afDNQgec/+qQoWskhqYtWRiqYunojF6aHZ/Xxh48Pz3HUpJdrUGf7H8+5AOJpaKUU g7DHDp4t4Jm+OygPZ9So2z/KsDnfH5fhwTPRSr8Zh1OAetDAoK6Oa1ACjE7iMAOig/vO xT7w== X-Gm-Message-State: AOAM530G+zPYgMkoAOx0FSuNfu6689JSku1p1A+WVgVL4UpOdnTy4CjB Bkwi7aQSabpmNRm0gpH1446d7JDvjoo= X-Google-Smtp-Source: ABdhPJygWvso55P70N3AzCgnUJ/1MBN4I7ixIVtnqxal/78ACpbLYMRBTq3IOoN5US3PSF2Ssh+3HA== X-Received: by 2002:a65:4502:: with SMTP id n2mr13862284pgq.132.1597795332966; Tue, 18 Aug 2020 17:02:12 -0700 (PDT) Received: from localhost.localdomain (c-71-56-157-77.hsd1.or.comcast.net. [71.56.157.77]) by smtp.gmail.com with ESMTPSA id d80sm25957673pfd.22.2020.08.18.17.02.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 17:02:12 -0700 (PDT) From: Luiz Augusto von Dentz To: linux-bluetooth@vger.kernel.org Subject: [PATCH v2 1/3] sbc: Add initial code for SSE primitives Date: Tue, 18 Aug 2020 17:02:10 -0700 Message-Id: <20200819000212.211485-1-luiz.dentz@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Sender: linux-bluetooth-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-bluetooth@vger.kernel.org From: Luiz Augusto von Dentz --- Makefile.am | 1 + sbc/sbc_primitives.c | 18 +- sbc/sbc_primitives_mmx.c | 49 +----- sbc/sbc_primitives_sse.c | 365 +++++++++++++++++++++++++++++++++++++++ sbc/sbc_primitives_sse.h | 38 ++++ 5 files changed, 428 insertions(+), 43 deletions(-) create mode 100644 sbc/sbc_primitives_sse.c create mode 100644 sbc/sbc_primitives_sse.h diff --git a/Makefile.am b/Makefile.am index 342043d..7ff0c7d 100644 --- a/Makefile.am +++ b/Makefile.am @@ -14,6 +14,7 @@ sbc_headers = sbc/sbc.h sbc_sources = sbc/sbc.c sbc/sbc_private.h sbc/sbc_math.h sbc/sbc_tables.h \ sbc/sbc_primitives.h sbc/sbc_primitives.c \ + sbc/sbc_primitives_sse.h sbc/sbc_primitives_sse.c \ sbc/sbc_primitives_mmx.h sbc/sbc_primitives_mmx.c \ sbc/sbc_primitives_iwmmxt.h sbc/sbc_primitives_iwmmxt.c \ sbc/sbc_primitives_neon.h sbc/sbc_primitives_neon.c \ diff --git a/sbc/sbc_primitives.c b/sbc/sbc_primitives.c index ff343cf..c9d1045 100644 --- a/sbc/sbc_primitives.c +++ b/sbc/sbc_primitives.c @@ -33,6 +33,7 @@ #include "sbc_tables.h" #include "sbc_primitives.h" +#include "sbc_primitives_sse.h" #include "sbc_primitives_mmx.h" #include "sbc_primitives_iwmmxt.h" #include "sbc_primitives_neon.h" @@ -590,6 +591,19 @@ static int sbc_calc_scalefactors_j( return joint; } +static void sbc_init_primitives_x86(struct sbc_encoder_state *state) +{ + __builtin_cpu_init(); + + if (__builtin_cpu_supports("mmx")) + sbc_init_primitives_mmx(state); + +#ifdef SBC_BUILD_WITH_SSE_SUPPORT + if (__builtin_cpu_supports("sse4.2")) + sbc_init_primitives_sse(state); +#endif +} + /* * Detect CPU features and setup function pointers */ @@ -614,9 +628,7 @@ void sbc_init_primitives(struct sbc_encoder_state *state) state->implementation_info = "Generic C"; /* X86/AMD64 optimizations */ -#ifdef SBC_BUILD_WITH_MMX_SUPPORT - sbc_init_primitives_mmx(state); -#endif + sbc_init_primitives_x86(state); /* ARM optimizations */ #ifdef SBC_BUILD_WITH_ARMV6_SUPPORT diff --git a/sbc/sbc_primitives_mmx.c b/sbc/sbc_primitives_mmx.c index e2c3e56..6984ce5 100644 --- a/sbc/sbc_primitives_mmx.c +++ b/sbc/sbc_primitives_mmx.c @@ -350,50 +350,19 @@ static void sbc_calc_scalefactors_mmx( __asm__ volatile ("emms\n"); } -static int check_mmx_support(void) +void sbc_init_primitives_mmx(struct sbc_encoder_state *state) { -#ifdef __amd64__ - return 1; /* We assume that all 64-bit processors have MMX support */ -#else - int cpuid_feature_information; - __asm__ volatile ( - /* According to Intel manual, CPUID instruction is supported - * if the value of ID bit (bit 21) in EFLAGS can be modified */ - "pushf\n" - "movl (%%esp), %0\n" - "xorl $0x200000, (%%esp)\n" /* try to modify ID bit */ - "popf\n" - "pushf\n" - "xorl (%%esp), %0\n" /* check if ID bit changed */ - "jz 1f\n" - "push %%eax\n" - "push %%ebx\n" - "push %%ecx\n" - "mov $1, %%eax\n" - "cpuid\n" - "pop %%ecx\n" - "pop %%ebx\n" - "pop %%eax\n" - "1:\n" - "popf\n" - : "=d" (cpuid_feature_information) - : - : "cc"); - return cpuid_feature_information & (1 << 23); -#endif + state->sbc_analyze_4s = sbc_analyze_4b_4s_mmx; + if (state->increment == 1) + state->sbc_analyze_8s = sbc_analyze_1b_8s_mmx_odd; + else + state->sbc_analyze_8s = sbc_analyze_4b_8s_mmx; + state->sbc_calc_scalefactors = sbc_calc_scalefactors_mmx; + state->implementation_info = "MMX"; } +#else void sbc_init_primitives_mmx(struct sbc_encoder_state *state) { - if (check_mmx_support()) { - state->sbc_analyze_4s = sbc_analyze_4b_4s_mmx; - if (state->increment == 1) - state->sbc_analyze_8s = sbc_analyze_1b_8s_mmx_odd; - else - state->sbc_analyze_8s = sbc_analyze_4b_8s_mmx; - state->sbc_calc_scalefactors = sbc_calc_scalefactors_mmx; - state->implementation_info = "MMX"; - } } - #endif diff --git a/sbc/sbc_primitives_sse.c b/sbc/sbc_primitives_sse.c new file mode 100644 index 0000000..42cdb03 --- /dev/null +++ b/sbc/sbc_primitives_sse.c @@ -0,0 +1,365 @@ +/* + * + * Bluetooth low-complexity, subband codec (SBC) library + * + * Copyright (C) 2020 Intel Corporation + * + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ + +#include +#include +#include "sbc.h" +#include "sbc_math.h" +#include "sbc_tables.h" + +#include "sbc_primitives_sse.h" + +/* + * SSE optimizations + */ + +#ifdef SBC_BUILD_WITH_SSE_SUPPORT + +static inline void sbc_analyze_four_sse(const int16_t *in, int32_t *out, + const FIXED_T *consts) +{ + static const SBC_ALIGNED int32_t round_c[2] = { + 1 << (SBC_PROTO_FIXED4_SCALE - 1), + 1 << (SBC_PROTO_FIXED4_SCALE - 1), + }; + __asm__ volatile ( + "movq (%0), %%mm0\n" + "movq 8(%0), %%mm1\n" + "pmaddwd (%1), %%mm0\n" + "pmaddwd 8(%1), %%mm1\n" + "paddd (%2), %%mm0\n" + "paddd (%2), %%mm1\n" + "\n" + "movq 16(%0), %%mm2\n" + "movq 24(%0), %%mm3\n" + "pmaddwd 16(%1), %%mm2\n" + "pmaddwd 24(%1), %%mm3\n" + "paddd %%mm2, %%mm0\n" + "paddd %%mm3, %%mm1\n" + "\n" + "movq 32(%0), %%mm2\n" + "movq 40(%0), %%mm3\n" + "pmaddwd 32(%1), %%mm2\n" + "pmaddwd 40(%1), %%mm3\n" + "paddd %%mm2, %%mm0\n" + "paddd %%mm3, %%mm1\n" + "\n" + "movq 48(%0), %%mm2\n" + "movq 56(%0), %%mm3\n" + "pmaddwd 48(%1), %%mm2\n" + "pmaddwd 56(%1), %%mm3\n" + "paddd %%mm2, %%mm0\n" + "paddd %%mm3, %%mm1\n" + "\n" + "movq 64(%0), %%mm2\n" + "movq 72(%0), %%mm3\n" + "pmaddwd 64(%1), %%mm2\n" + "pmaddwd 72(%1), %%mm3\n" + "paddd %%mm2, %%mm0\n" + "paddd %%mm3, %%mm1\n" + "\n" + "psrad %4, %%mm0\n" + "psrad %4, %%mm1\n" + "packssdw %%mm0, %%mm0\n" + "packssdw %%mm1, %%mm1\n" + "\n" + "movq %%mm0, %%mm2\n" + "pmaddwd 80(%1), %%mm0\n" + "pmaddwd 88(%1), %%mm2\n" + "\n" + "movq %%mm1, %%mm3\n" + "pmaddwd 96(%1), %%mm1\n" + "pmaddwd 104(%1), %%mm3\n" + "paddd %%mm1, %%mm0\n" + "paddd %%mm3, %%mm2\n" + "\n" + "movq %%mm0, (%3)\n" + "movq %%mm2, 8(%3)\n" + : + : "r" (in), "r" (consts), "r" (&round_c), "r" (out), + "i" (SBC_PROTO_FIXED4_SCALE) + : "cc", "memory"); +} + +static inline void sbc_analyze_eight_sse(const int16_t *in, int32_t *out, + const FIXED_T *consts) +{ + static const SBC_ALIGNED int32_t round_c[2] = { + 1 << (SBC_PROTO_FIXED8_SCALE - 1), + 1 << (SBC_PROTO_FIXED8_SCALE - 1), + }; + __asm__ volatile ( + "movq (%0), %%mm0\n" + "movq 8(%0), %%mm1\n" + "movq 16(%0), %%mm2\n" + "movq 24(%0), %%mm3\n" + "pmaddwd (%1), %%mm0\n" + "pmaddwd 8(%1), %%mm1\n" + "pmaddwd 16(%1), %%mm2\n" + "pmaddwd 24(%1), %%mm3\n" + "paddd (%2), %%mm0\n" + "paddd (%2), %%mm1\n" + "paddd (%2), %%mm2\n" + "paddd (%2), %%mm3\n" + "\n" + "movq 32(%0), %%mm4\n" + "movq 40(%0), %%mm5\n" + "movq 48(%0), %%mm6\n" + "movq 56(%0), %%mm7\n" + "pmaddwd 32(%1), %%mm4\n" + "pmaddwd 40(%1), %%mm5\n" + "pmaddwd 48(%1), %%mm6\n" + "pmaddwd 56(%1), %%mm7\n" + "paddd %%mm4, %%mm0\n" + "paddd %%mm5, %%mm1\n" + "paddd %%mm6, %%mm2\n" + "paddd %%mm7, %%mm3\n" + "\n" + "movq 64(%0), %%mm4\n" + "movq 72(%0), %%mm5\n" + "movq 80(%0), %%mm6\n" + "movq 88(%0), %%mm7\n" + "pmaddwd 64(%1), %%mm4\n" + "pmaddwd 72(%1), %%mm5\n" + "pmaddwd 80(%1), %%mm6\n" + "pmaddwd 88(%1), %%mm7\n" + "paddd %%mm4, %%mm0\n" + "paddd %%mm5, %%mm1\n" + "paddd %%mm6, %%mm2\n" + "paddd %%mm7, %%mm3\n" + "\n" + "movq 96(%0), %%mm4\n" + "movq 104(%0), %%mm5\n" + "movq 112(%0), %%mm6\n" + "movq 120(%0), %%mm7\n" + "pmaddwd 96(%1), %%mm4\n" + "pmaddwd 104(%1), %%mm5\n" + "pmaddwd 112(%1), %%mm6\n" + "pmaddwd 120(%1), %%mm7\n" + "paddd %%mm4, %%mm0\n" + "paddd %%mm5, %%mm1\n" + "paddd %%mm6, %%mm2\n" + "paddd %%mm7, %%mm3\n" + "\n" + "movq 128(%0), %%mm4\n" + "movq 136(%0), %%mm5\n" + "movq 144(%0), %%mm6\n" + "movq 152(%0), %%mm7\n" + "pmaddwd 128(%1), %%mm4\n" + "pmaddwd 136(%1), %%mm5\n" + "pmaddwd 144(%1), %%mm6\n" + "pmaddwd 152(%1), %%mm7\n" + "paddd %%mm4, %%mm0\n" + "paddd %%mm5, %%mm1\n" + "paddd %%mm6, %%mm2\n" + "paddd %%mm7, %%mm3\n" + "\n" + "psrad %4, %%mm0\n" + "psrad %4, %%mm1\n" + "psrad %4, %%mm2\n" + "psrad %4, %%mm3\n" + "\n" + "packssdw %%mm0, %%mm0\n" + "packssdw %%mm1, %%mm1\n" + "packssdw %%mm2, %%mm2\n" + "packssdw %%mm3, %%mm3\n" + "\n" + "movq %%mm0, %%mm4\n" + "movq %%mm0, %%mm5\n" + "pmaddwd 160(%1), %%mm4\n" + "pmaddwd 168(%1), %%mm5\n" + "\n" + "movq %%mm1, %%mm6\n" + "movq %%mm1, %%mm7\n" + "pmaddwd 192(%1), %%mm6\n" + "pmaddwd 200(%1), %%mm7\n" + "paddd %%mm6, %%mm4\n" + "paddd %%mm7, %%mm5\n" + "\n" + "movq %%mm2, %%mm6\n" + "movq %%mm2, %%mm7\n" + "pmaddwd 224(%1), %%mm6\n" + "pmaddwd 232(%1), %%mm7\n" + "paddd %%mm6, %%mm4\n" + "paddd %%mm7, %%mm5\n" + "\n" + "movq %%mm3, %%mm6\n" + "movq %%mm3, %%mm7\n" + "pmaddwd 256(%1), %%mm6\n" + "pmaddwd 264(%1), %%mm7\n" + "paddd %%mm6, %%mm4\n" + "paddd %%mm7, %%mm5\n" + "\n" + "movq %%mm4, (%3)\n" + "movq %%mm5, 8(%3)\n" + "\n" + "movq %%mm0, %%mm5\n" + "pmaddwd 176(%1), %%mm0\n" + "pmaddwd 184(%1), %%mm5\n" + "\n" + "movq %%mm1, %%mm7\n" + "pmaddwd 208(%1), %%mm1\n" + "pmaddwd 216(%1), %%mm7\n" + "paddd %%mm1, %%mm0\n" + "paddd %%mm7, %%mm5\n" + "\n" + "movq %%mm2, %%mm7\n" + "pmaddwd 240(%1), %%mm2\n" + "pmaddwd 248(%1), %%mm7\n" + "paddd %%mm2, %%mm0\n" + "paddd %%mm7, %%mm5\n" + "\n" + "movq %%mm3, %%mm7\n" + "pmaddwd 272(%1), %%mm3\n" + "pmaddwd 280(%1), %%mm7\n" + "paddd %%mm3, %%mm0\n" + "paddd %%mm7, %%mm5\n" + "\n" + "movq %%mm0, 16(%3)\n" + "movq %%mm5, 24(%3)\n" + : + : "r" (in), "r" (consts), "r" (&round_c), "r" (out), + "i" (SBC_PROTO_FIXED8_SCALE) + : "cc", "memory"); +} + +static inline void sbc_analyze_4b_4s_sse(struct sbc_encoder_state *state, + int16_t *x, int32_t *out, int out_stride) +{ + /* Analyze blocks */ + sbc_analyze_four_sse(x + 12, out, analysis_consts_fixed4_simd_odd); + out += out_stride; + sbc_analyze_four_sse(x + 8, out, analysis_consts_fixed4_simd_even); + out += out_stride; + sbc_analyze_four_sse(x + 4, out, analysis_consts_fixed4_simd_odd); + out += out_stride; + sbc_analyze_four_sse(x + 0, out, analysis_consts_fixed4_simd_even); + + __asm__ volatile ("emms\n"); +} + +static inline void sbc_analyze_4b_8s_sse(struct sbc_encoder_state *state, + int16_t *x, int32_t *out, int out_stride) +{ + /* Analyze blocks */ + sbc_analyze_eight_sse(x + 24, out, analysis_consts_fixed8_simd_odd); + out += out_stride; + sbc_analyze_eight_sse(x + 16, out, analysis_consts_fixed8_simd_even); + out += out_stride; + sbc_analyze_eight_sse(x + 8, out, analysis_consts_fixed8_simd_odd); + out += out_stride; + sbc_analyze_eight_sse(x + 0, out, analysis_consts_fixed8_simd_even); + + __asm__ volatile ("emms\n"); +} + +static inline void sbc_analyze_1b_8s_sse_even(struct sbc_encoder_state *state, + int16_t *x, int32_t *out, int out_stride); + +static inline void sbc_analyze_1b_8s_sse_odd(struct sbc_encoder_state *state, + int16_t *x, int32_t *out, int out_stride) +{ + sbc_analyze_eight_sse(x, out, analysis_consts_fixed8_simd_odd); + state->sbc_analyze_8s = sbc_analyze_1b_8s_sse_even; + + __asm__ volatile ("emms\n"); +} + +static inline void sbc_analyze_1b_8s_sse_even(struct sbc_encoder_state *state, + int16_t *x, int32_t *out, int out_stride) +{ + sbc_analyze_eight_sse(x, out, analysis_consts_fixed8_simd_even); + state->sbc_analyze_8s = sbc_analyze_1b_8s_sse_odd; + + __asm__ volatile ("emms\n"); +} + +static void sbc_calc_scalefactors_sse( + int32_t sb_sample_f[16][2][8], + uint32_t scale_factor[2][8], + int blocks, int channels, int subbands) +{ + static const SBC_ALIGNED int32_t consts[2] = { + 1 << SCALE_OUT_BITS, + 1 << SCALE_OUT_BITS, + }; + int ch, sb; + intptr_t blk; + for (ch = 0; ch < channels; ch++) { + for (sb = 0; sb < subbands; sb += 2) { + blk = (blocks - 1) * (((char *) &sb_sample_f[1][0][0] - + (char *) &sb_sample_f[0][0][0])); + __asm__ volatile ( + "movq (%4), %%mm0\n" + "1:\n" + "movq (%1, %0), %%mm1\n" + "pxor %%mm2, %%mm2\n" + "pcmpgtd %%mm2, %%mm1\n" + "paddd (%1, %0), %%mm1\n" + "pcmpgtd %%mm1, %%mm2\n" + "pxor %%mm2, %%mm1\n" + + "por %%mm1, %%mm0\n" + + "sub %2, %0\n" + "jns 1b\n" + + "movd %%mm0, %k0\n" + "psrlq $32, %%mm0\n" + "bsrl %k0, %k0\n" + "subl %5, %k0\n" + "movl %k0, (%3)\n" + + "movd %%mm0, %k0\n" + "bsrl %k0, %k0\n" + "subl %5, %k0\n" + "movl %k0, 4(%3)\n" + : "+r" (blk) + : "r" (&sb_sample_f[0][ch][sb]), + "i" ((char *) &sb_sample_f[1][0][0] - + (char *) &sb_sample_f[0][0][0]), + "r" (&scale_factor[ch][sb]), + "r" (&consts), + "i" (SCALE_OUT_BITS) + : "cc", "memory"); + } + } + __asm__ volatile ("emms\n"); +} + +void sbc_init_primitives_sse(struct sbc_encoder_state *state) +{ + state->sbc_analyze_4s = sbc_analyze_4b_4s_sse; + if (state->increment == 1) + state->sbc_analyze_8s = sbc_analyze_1b_8s_sse_odd; + else + state->sbc_analyze_8s = sbc_analyze_4b_8s_sse; + state->sbc_calc_scalefactors = sbc_calc_scalefactors_sse; + state->implementation_info = "SSE"; +} + +#else +void sbc_init_primitives_sse(struct sbc_encoder_state *state) +{ +} +#endif diff --git a/sbc/sbc_primitives_sse.h b/sbc/sbc_primitives_sse.h new file mode 100644 index 0000000..8830cfd --- /dev/null +++ b/sbc/sbc_primitives_sse.h @@ -0,0 +1,38 @@ +/* + * + * Bluetooth low-complexity, subband codec (SBC) library + * + * Copyright (C) 2020 Intel Corporation + * + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ + +#ifndef __SBC_PRIMITIVES_SSE_H +#define __SBC_PRIMITIVES_SSE_H + +#include "sbc_primitives.h" + +#if defined(__GNUC__) && (defined(__i386__) || defined(__amd64__)) && \ + !defined(SBC_HIGH_PRECISION) && (SCALE_OUT_BITS == 15) + +#define SBC_BUILD_WITH_SSE_SUPPORT + +void sbc_init_primitives_sse(struct sbc_encoder_state *encoder_state); + +#endif + +#endif From patchwork Wed Aug 19 00:02:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Augusto von Dentz X-Patchwork-Id: 11722313 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 37D14618 for ; Wed, 19 Aug 2020 00:02:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 16C052075E for ; Wed, 19 Aug 2020 00:02:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rgWMc0GF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727776AbgHSACQ (ORCPT ); Tue, 18 Aug 2020 20:02:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727012AbgHSACP (ORCPT ); Tue, 18 Aug 2020 20:02:15 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AFB9C061389 for ; Tue, 18 Aug 2020 17:02:14 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id x25so10765598pff.4 for ; Tue, 18 Aug 2020 17:02:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uSA0Ybo7FyRxzr9ClbGhKtFYud7d1Je4lZohhQzo/UQ=; b=rgWMc0GFFP67vj1Er+uBtgmm8BCRFksV5lkiVAqUZFa0jYwCT3u02NWmnXey/rjiS4 mAD1ALPXcraVBpDRtuXEalMRBgcBz212xosKO1M2JX/siZ6dfY+uHwEUIWwYsdZzlyIg ZYyJyLM1dDkiBVPJEn5DV2ky14NPKs1jrHdfqU+4m7s9ZVqAZhLraUN9WYWto5Pa1YmK UQTU5tyijc6R2Xe4dwxmCuDcgExfUkyN2Qt3sZCDMBB5VaX7DpPxnqMgo+Yj2X1c/eu2 10lOmEkmAZkGvG99aMDPEqKfjSUV35VE/PWAlkQ38Qk0Kt0HvaIu5yj7AhYyoTPzeQQB 6ypQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uSA0Ybo7FyRxzr9ClbGhKtFYud7d1Je4lZohhQzo/UQ=; b=m1F/0h3IFSrky/YUyjpcl+hjliLcXPuitmGW9yb0jWEjCF3mZcsGPBEoTmnaIs/m04 MUeC4P9hhBi1amT19NwibNbHmf6fhs6P/0OocZr6VcyHhabrqc9kqpclXzga9rXqmRfl i+xPuya10+nFEEg1K6O0snAKBE7zWkGlY0JZv84KsXRASM4AK8TfN7dkJSJEHUuXmQ2y UAv9UU/hvUueacmWkpsj9vWD02tAkkyJWG6gvRAsGby6Oi9HWFZ+qMXsYPLE2LimasZv Z4CFte4lLbX/rXscfegyS8TwGcy2b4ypw30hTr+O4qtR+bl1CdA8QJHtAho7+dSDnTRF VcqQ== X-Gm-Message-State: AOAM533xpSx3XOvS1S/f3Zaqq8dEPxNBE99sPbwxOqyU97+TrnpY52Th /Dz1XkLYS3Ou9PdAklR3kwui3kiUABg= X-Google-Smtp-Source: ABdhPJyRwLIfLZqQFQe8RGLC/Vlrqb1fALSCzl9JYxTO/F5zL/YK2OqPuK1RoxWBpxp3ad1p8g7pyw== X-Received: by 2002:a62:1dc3:: with SMTP id d186mr17270021pfd.93.1597795333812; Tue, 18 Aug 2020 17:02:13 -0700 (PDT) Received: from localhost.localdomain (c-71-56-157-77.hsd1.or.comcast.net. [71.56.157.77]) by smtp.gmail.com with ESMTPSA id d80sm25957673pfd.22.2020.08.18.17.02.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 17:02:13 -0700 (PDT) From: Luiz Augusto von Dentz To: linux-bluetooth@vger.kernel.org Subject: [PATCH v2 2/3] sbc/sbc_primitives_sse: Optimize sbc_analyze_4s Date: Tue, 18 Aug 2020 17:02:11 -0700 Message-Id: <20200819000212.211485-2-luiz.dentz@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200819000212.211485-1-luiz.dentz@gmail.com> References: <20200819000212.211485-1-luiz.dentz@gmail.com> MIME-Version: 1.0 Sender: linux-bluetooth-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-bluetooth@vger.kernel.org From: Luiz Augusto von Dentz This makes use 128 bit XMM registers whenever possible. $ time src/sbcenc_mmx -s 4 sin_4m.au > /dev/null real 0m1.073s user 0m1.039s sys 0m0.030s === After ==== $ time src/sbcenc -s 4 sin_4m.au > /dev/null real 0m1.049s user 0m1.000s sys 0m0.047s --- sbc/sbc_primitives_sse.c | 58 +++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 33 deletions(-) diff --git a/sbc/sbc_primitives_sse.c b/sbc/sbc_primitives_sse.c index 42cdb03..2a903e1 100644 --- a/sbc/sbc_primitives_sse.c +++ b/sbc/sbc_primitives_sse.c @@ -38,48 +38,40 @@ static inline void sbc_analyze_four_sse(const int16_t *in, int32_t *out, const FIXED_T *consts) { - static const SBC_ALIGNED int32_t round_c[2] = { + static const SBC_ALIGNED int32_t round_c[4] = { + 1 << (SBC_PROTO_FIXED4_SCALE - 1), + 1 << (SBC_PROTO_FIXED4_SCALE - 1), 1 << (SBC_PROTO_FIXED4_SCALE - 1), 1 << (SBC_PROTO_FIXED4_SCALE - 1), }; __asm__ volatile ( - "movq (%0), %%mm0\n" - "movq 8(%0), %%mm1\n" - "pmaddwd (%1), %%mm0\n" - "pmaddwd 8(%1), %%mm1\n" - "paddd (%2), %%mm0\n" - "paddd (%2), %%mm1\n" + "movdqu (%0), %%xmm0\n" + "pmaddwd (%1), %%xmm0\n" + "paddd (%2), %%xmm0\n" "\n" - "movq 16(%0), %%mm2\n" - "movq 24(%0), %%mm3\n" - "pmaddwd 16(%1), %%mm2\n" - "pmaddwd 24(%1), %%mm3\n" - "paddd %%mm2, %%mm0\n" - "paddd %%mm3, %%mm1\n" + "movdqu 16(%0), %%xmm1\n" + "pmaddwd 16(%1), %%xmm1\n" + "paddd %%xmm1, %%xmm0\n" "\n" - "movq 32(%0), %%mm2\n" - "movq 40(%0), %%mm3\n" - "pmaddwd 32(%1), %%mm2\n" - "pmaddwd 40(%1), %%mm3\n" - "paddd %%mm2, %%mm0\n" - "paddd %%mm3, %%mm1\n" + "movdqu 32(%0), %%xmm1\n" + "pmaddwd 32(%1), %%xmm1\n" + "paddd %%xmm1, %%xmm0\n" "\n" - "movq 48(%0), %%mm2\n" - "movq 56(%0), %%mm3\n" - "pmaddwd 48(%1), %%mm2\n" - "pmaddwd 56(%1), %%mm3\n" - "paddd %%mm2, %%mm0\n" - "paddd %%mm3, %%mm1\n" + "movdqu 48(%0), %%xmm1\n" + "pmaddwd 48(%1), %%xmm1\n" + "paddd %%xmm1, %%xmm0\n" "\n" - "movq 64(%0), %%mm2\n" - "movq 72(%0), %%mm3\n" - "pmaddwd 64(%1), %%mm2\n" - "pmaddwd 72(%1), %%mm3\n" - "paddd %%mm2, %%mm0\n" - "paddd %%mm3, %%mm1\n" + "movdqu 64(%0), %%xmm1\n" + "pmaddwd 64(%1), %%xmm1\n" + "paddd %%xmm1, %%xmm0\n" + "\n" + "psrad %4, %%xmm0\n" + "\n" + "movdqa %%xmm0, %%xmm1\n" + "punpckhqdq %%xmm1, %%xmm1\n" + "movdq2q %%xmm0, %%mm0\n" + "movdq2q %%xmm1, %%mm1\n" "\n" - "psrad %4, %%mm0\n" - "psrad %4, %%mm1\n" "packssdw %%mm0, %%mm0\n" "packssdw %%mm1, %%mm1\n" "\n" From patchwork Wed Aug 19 00:02:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Augusto von Dentz X-Patchwork-Id: 11722315 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D039618 for ; Wed, 19 Aug 2020 00:02:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0376520855 for ; Wed, 19 Aug 2020 00:02:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aBV6XUGv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727780AbgHSACS (ORCPT ); Tue, 18 Aug 2020 20:02:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727012AbgHSACR (ORCPT ); Tue, 18 Aug 2020 20:02:17 -0400 Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2DD5C061389 for ; Tue, 18 Aug 2020 17:02:16 -0700 (PDT) Received: by mail-pg1-x530.google.com with SMTP id x6so10488986pgx.12 for ; Tue, 18 Aug 2020 17:02:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=z0dKBvUf9vGWL9gen29+YT/xiceR0mKKGHWkGGIRMKc=; b=aBV6XUGvXUFLEk0bOeNbgB+C6V9BCQjb1wytJmtfiOZlzs93puXKjR8BK5a2LeWJ0v Z/qu8EzSmqu2HBqozRlGMR0sj10W0YeFTYe+QLfIrAsJZ2uE3rEkSnFE7HQA7uhKkve8 7TbloMMyemuFbNti8UXx1CRIEvLBZF9LQXsgCtdiqKBCVbfR/7UWKNnyb0JefwsZWqbQ a197DbMBJwRdGI8ex3nqolddUGz+rBO6c3irliqfNklOrkqcXORExpCit+KGTRheslnB gfy655KHOv63BX7zOxp5rvYVKNKtl64MmDbetpx2FOsa3mLnUBIIVLi+x1/7frgOE576 0FgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z0dKBvUf9vGWL9gen29+YT/xiceR0mKKGHWkGGIRMKc=; b=XSwKXbl5zV/K7G7Flp4QyL3TKa7KiY+Ocg9VLGBLWwa+h+PynwjZ8zmCvr/poAUvld fY3zQ+mM0Z8zgOdfK183jIiR758pQiiUeMPhNOrILUncmepga5MQ1tuJ0Sagajvnluix s4rBo+On/WWP3lOh4oPiXz6PctxzpGvzY7B+O/7V4VYrp0wUl0q9uK3/lgX/06SYglR5 gU69r+cLg5Qe0q8SesxlFf+UJjVxPr4QbQ0K2/yTR/Uw22eBbzmdZXcQqWwR3yYurB4L pAJF0XQpAfC6sglLjK063lzMeAXZK/mzvJ0T0mHi91xfROFXRMXKn2zSMfqdXPQm3Z5k ZRRA== X-Gm-Message-State: AOAM531ALCrLso5ifcaG1nmSKAOPXTi9TQCdIJ3CGD5d/btQQAr4hPYi 7W2xVqSxQNbgZxk1JKOfB7J6zvWu/lA= X-Google-Smtp-Source: ABdhPJwFna7JKUeYZqTtXCi9ZNN3Tm//jGDR6lnX5jzqZGPuFwbs1JBwBVa/GMX6sdod8VjiGpu/BA== X-Received: by 2002:a05:6a00:78e:: with SMTP id g14mr16805555pfu.171.1597795334713; Tue, 18 Aug 2020 17:02:14 -0700 (PDT) Received: from localhost.localdomain (c-71-56-157-77.hsd1.or.comcast.net. [71.56.157.77]) by smtp.gmail.com with ESMTPSA id d80sm25957673pfd.22.2020.08.18.17.02.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 17:02:14 -0700 (PDT) From: Luiz Augusto von Dentz To: linux-bluetooth@vger.kernel.org Subject: [PATCH v2 3/3] sbc/sbc_primitives_sse: Optimize sbc_analyze_8s Date: Tue, 18 Aug 2020 17:02:12 -0700 Message-Id: <20200819000212.211485-3-luiz.dentz@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200819000212.211485-1-luiz.dentz@gmail.com> References: <20200819000212.211485-1-luiz.dentz@gmail.com> MIME-Version: 1.0 Sender: linux-bluetooth-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-bluetooth@vger.kernel.org From: Luiz Augusto von Dentz This makes use 128 bit XMM registers whenever possible. $ time src/sbcenc_mmx -s 8 sin_64m.au > /dev/null real 0m1.064s user 0m1.012s sys 0m0.049s === After ==== $ time src/sbcenc -s 8 sin_64m.au > /dev/null real 0m1.032s user 0m0.996s sys 0m0.033s --- sbc/sbc_primitives_sse.c | 109 ++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 65 deletions(-) diff --git a/sbc/sbc_primitives_sse.c b/sbc/sbc_primitives_sse.c index 2a903e1..9bff6cf 100644 --- a/sbc/sbc_primitives_sse.c +++ b/sbc/sbc_primitives_sse.c @@ -96,80 +96,59 @@ static inline void sbc_analyze_four_sse(const int16_t *in, int32_t *out, static inline void sbc_analyze_eight_sse(const int16_t *in, int32_t *out, const FIXED_T *consts) { - static const SBC_ALIGNED int32_t round_c[2] = { + static const SBC_ALIGNED int32_t round_c[4] = { + 1 << (SBC_PROTO_FIXED8_SCALE - 1), + 1 << (SBC_PROTO_FIXED8_SCALE - 1), 1 << (SBC_PROTO_FIXED8_SCALE - 1), 1 << (SBC_PROTO_FIXED8_SCALE - 1), }; __asm__ volatile ( - "movq (%0), %%mm0\n" - "movq 8(%0), %%mm1\n" - "movq 16(%0), %%mm2\n" - "movq 24(%0), %%mm3\n" - "pmaddwd (%1), %%mm0\n" - "pmaddwd 8(%1), %%mm1\n" - "pmaddwd 16(%1), %%mm2\n" - "pmaddwd 24(%1), %%mm3\n" - "paddd (%2), %%mm0\n" - "paddd (%2), %%mm1\n" - "paddd (%2), %%mm2\n" - "paddd (%2), %%mm3\n" + "movdqu (%0), %%xmm0\n" + "movdqu 16(%0), %%xmm1\n" + "pmaddwd (%1), %%xmm0\n" + "pmaddwd 16(%1), %%xmm1\n" + "paddd (%2), %%xmm0\n" + "paddd (%2), %%xmm1\n" "\n" - "movq 32(%0), %%mm4\n" - "movq 40(%0), %%mm5\n" - "movq 48(%0), %%mm6\n" - "movq 56(%0), %%mm7\n" - "pmaddwd 32(%1), %%mm4\n" - "pmaddwd 40(%1), %%mm5\n" - "pmaddwd 48(%1), %%mm6\n" - "pmaddwd 56(%1), %%mm7\n" - "paddd %%mm4, %%mm0\n" - "paddd %%mm5, %%mm1\n" - "paddd %%mm6, %%mm2\n" - "paddd %%mm7, %%mm3\n" + "movdqu 32(%0), %%xmm2\n" + "movdqu 48(%0), %%xmm3\n" + "pmaddwd 32(%1), %%xmm2\n" + "pmaddwd 48(%1), %%xmm3\n" + "paddd %%xmm2, %%xmm0\n" + "paddd %%xmm3, %%xmm1\n" "\n" - "movq 64(%0), %%mm4\n" - "movq 72(%0), %%mm5\n" - "movq 80(%0), %%mm6\n" - "movq 88(%0), %%mm7\n" - "pmaddwd 64(%1), %%mm4\n" - "pmaddwd 72(%1), %%mm5\n" - "pmaddwd 80(%1), %%mm6\n" - "pmaddwd 88(%1), %%mm7\n" - "paddd %%mm4, %%mm0\n" - "paddd %%mm5, %%mm1\n" - "paddd %%mm6, %%mm2\n" - "paddd %%mm7, %%mm3\n" + "movdqu 64(%0), %%xmm2\n" + "movdqu 80(%0), %%xmm3\n" + "pmaddwd 64(%1), %%xmm2\n" + "pmaddwd 80(%1), %%xmm3\n" + "paddd %%xmm2, %%xmm0\n" + "paddd %%xmm3, %%xmm1\n" "\n" - "movq 96(%0), %%mm4\n" - "movq 104(%0), %%mm5\n" - "movq 112(%0), %%mm6\n" - "movq 120(%0), %%mm7\n" - "pmaddwd 96(%1), %%mm4\n" - "pmaddwd 104(%1), %%mm5\n" - "pmaddwd 112(%1), %%mm6\n" - "pmaddwd 120(%1), %%mm7\n" - "paddd %%mm4, %%mm0\n" - "paddd %%mm5, %%mm1\n" - "paddd %%mm6, %%mm2\n" - "paddd %%mm7, %%mm3\n" + "movdqu 96(%0), %%xmm2\n" + "movdqu 112(%0), %%xmm3\n" + "pmaddwd 96(%1), %%xmm2\n" + "pmaddwd 112(%1), %%xmm3\n" + "paddd %%xmm2, %%xmm0\n" + "paddd %%xmm3, %%xmm1\n" "\n" - "movq 128(%0), %%mm4\n" - "movq 136(%0), %%mm5\n" - "movq 144(%0), %%mm6\n" - "movq 152(%0), %%mm7\n" - "pmaddwd 128(%1), %%mm4\n" - "pmaddwd 136(%1), %%mm5\n" - "pmaddwd 144(%1), %%mm6\n" - "pmaddwd 152(%1), %%mm7\n" - "paddd %%mm4, %%mm0\n" - "paddd %%mm5, %%mm1\n" - "paddd %%mm6, %%mm2\n" - "paddd %%mm7, %%mm3\n" + "movdqu 128(%0), %%xmm2\n" + "movdqu 144(%0), %%xmm3\n" + "pmaddwd 128(%1), %%xmm2\n" + "pmaddwd 144(%1), %%xmm3\n" + "paddd %%xmm2, %%xmm0\n" + "paddd %%xmm3, %%xmm1\n" + "\n" + "psrad %4, %%xmm0\n" + "psrad %4, %%xmm1\n" "\n" - "psrad %4, %%mm0\n" - "psrad %4, %%mm1\n" - "psrad %4, %%mm2\n" - "psrad %4, %%mm3\n" + "movdqa %%xmm0, %%xmm2\n" + "movdqa %%xmm1, %%xmm3\n" + "punpckhqdq %%xmm2, %%xmm2\n" + "punpckhqdq %%xmm3, %%xmm3\n" + "movdq2q %%xmm0, %%mm0\n" + "movdq2q %%xmm2, %%mm1\n" + "movdq2q %%xmm1, %%mm2\n" + "movdq2q %%xmm3, %%mm3\n" "\n" "packssdw %%mm0, %%mm0\n" "packssdw %%mm1, %%mm1\n"