From patchwork Thu Jun 6 18:32:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesse Taube X-Patchwork-Id: 13688885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 065F5C27C54 for ; Thu, 6 Jun 2024 18:32:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4iWkphv7Vjrvo0Kx5uMxEqp8q1BQ8h/q9MCd605LcoQ=; b=bOp9yNzdkUoNoF N4RFJTCbcWzYlE51fEBpf1DP+R3jocSUSqH817D6Br4wyTneM5WT9VllO7145zuYxJiy/i36Vh+QX gTz/3FMhVI01GQJXybEwtVxJ/b/gLbREGLJwJTbVWEwTRcU5N4WMpmSibslCaQCSE5MPW9aWgDNv/ M/B35VZkfD1svOcY0TjzTs8NehUHihS5kHbpUvyS+JPSnhwEWNiofG5F2Ad2c5GYvaT3deyTvQDEh khnhz0RTak9+oa8tD+8HohqEjt3N6wHUxw9U7h3OTOghQdUNmgd9tDU7VFLWG7JIPsVDH/2qDSSWJ b/qGa7VpOEAKHJhCF55Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sFHup-0000000Axcc-2AmT; Thu, 06 Jun 2024 18:32:47 +0000 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sFHun-0000000Axby-0efp for linux-riscv@lists.infradead.org; Thu, 06 Jun 2024 18:32:46 +0000 Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-70260814b2dso1061773b3a.1 for ; Thu, 06 Jun 2024 11:32:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1717698764; x=1718303564; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VvuTCHkit+Xi04/Y6YOOrb/oKRPfOiVcL1hFmBt4hbc=; b=JiO9eNiqrs36QyTCRjKj/j5aiBh4r3JzSiF1ZQu7xXDioFpyalqFgN5Y95EnNs7CbI dbzS+wbMUhXyYXmthNQT+Yu7MiySTWsNh/Y+fV5VVs1qUtDx+YbaVfndmO0XXIpq1WXz 9SX6Enyqm75nDw06iW+8l7cY5vmP79eP3v0lt8gc+0S2t/MwsglAVup1nqtYBRSth1ty 1Xel7iFTufTG2TFAzwIPjbeWfxjtE60aH2L6cC7gMOIfE6AN0ns7wmlxr6uC4FfH+kOd kxKdt0EvxbupY8iZzNWuSIG+6hcwVT2VviBp5GvLrFHHanBeXcHVPirG1ObRYazJ5ai1 9+9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717698764; x=1718303564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VvuTCHkit+Xi04/Y6YOOrb/oKRPfOiVcL1hFmBt4hbc=; b=b+IBiMatBmiUUlGxySxC46sK53IhqW72K9p33siwjtytUhxF3Yh0DaBkxxELwv/Y/Y hW2Ay1fdSZtgJzVNoAcLJwbeCv9MJmOXbdzAkdCAUdswWUQ6Y7mLKVMl9jh2C3x4EkB5 y0mlhoiqoKbVc/4IP44x/I3yYz+hiUrXY1sAozKT6wkquJdKwuo349bCEzTwIsnVehLp KQzaVdGRheYKWwrdeHh2po619wR0fDqHo3u4n0SEmEy7Lvm3NddJvvVwFmBI8y9chEo5 Km/pWjtrM9T5eKtjH2EGIbDJ4uNcVT1SHnB6ZWa8oX43zxRyjpCcltBk5OC7Z7mUnUEA w/8w== X-Gm-Message-State: AOJu0YzzzC9Ay6y3Xh+8Jud9e8uiCnzYtQ50WL5A7pG9C9xOClv/QrC1 wCmJKP/di9PuTs5eo9SEXWVbFnwfG2f0LAiMqFpe6ThR6Rd+QaDtqRyh1OvFwf0aDvZ900PwQK6 yoc4= X-Google-Smtp-Source: AGHT+IGT/DydJIpJEHegcVhedbVkQodQB11b8IfORR0j4zD4Zrth0xks/SxQQYZsS327tQ3nbYhbaA== X-Received: by 2002:a05:6a00:1403:b0:6ec:ef1c:4dcd with SMTP id d2e1a72fcca58-7040a285b96mr672483b3a.6.1717698763700; Thu, 06 Jun 2024 11:32:43 -0700 (PDT) Received: from jesse-desktop.. (pool-108-26-179-17.bstnma.fios.verizon.net. [108.26.179.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-703fd50f0c6sm1378400b3a.191.2024.06.06.11.32.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jun 2024 11:32:43 -0700 (PDT) From: Jesse Taube To: linux-riscv@lists.infradead.org Cc: Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Evan Green , Charlie Jenkins , Andrew Jones , Jesse Taube , Xiao Wang , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Andy Chiu , Greentime Hu , Heiko Stuebner , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Costa Shulyupin , Andrew Morton , Baoquan He , Sami Tolvanen , Zong Li , Ben Dooks , Erick Archer , Vincent Chen , Joel Granados , linux-kernel@vger.kernel.org Subject: [PATCH 2/3] RISC-V: Detect unaligned vector accesses supported. Date: Thu, 6 Jun 2024 14:32:14 -0400 Message-ID: <20240606183215.416829-2-jesse@rivosinc.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240606183215.416829-1-jesse@rivosinc.com> References: <20240606183215.416829-1-jesse@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240606_113245_220769_CBA19408 X-CRM114-Status: GOOD ( 19.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Run a unaligned vector access to test if the system supports vector unaligned access. Add the result to a new key in hwprobe. This is useful for usermode to know if vector misaligned accesses are supported and if they are faster or slower than equivalent byte accesses. Signed-off-by: Jesse Taube --- arch/riscv/include/asm/cpufeature.h | 2 + arch/riscv/include/asm/hwprobe.h | 2 +- arch/riscv/include/asm/vector.h | 1 + arch/riscv/include/uapi/asm/hwprobe.h | 6 ++ arch/riscv/kernel/sys_hwprobe.c | 34 +++++++++ arch/riscv/kernel/traps_misaligned.c | 84 ++++++++++++++++++++-- arch/riscv/kernel/unaligned_access_speed.c | 4 ++ arch/riscv/kernel/vector.c | 2 +- 8 files changed, 129 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 347805446151..a012c8490a27 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -35,9 +35,11 @@ void riscv_user_isa_enable(void); #if defined(CONFIG_RISCV_MISALIGNED) bool check_unaligned_access_emulated_all_cpus(void); +bool check_vector_unaligned_access_all_cpus(void); void unaligned_emulation_finish(void); bool unaligned_ctl_available(void); DECLARE_PER_CPU(long, misaligned_access_speed); +DECLARE_PER_CPU(long, vector_misaligned_access); #else static inline bool unaligned_ctl_available(void) { diff --git a/arch/riscv/include/asm/hwprobe.h b/arch/riscv/include/asm/hwprobe.h index 630507dff5ea..150a9877b0af 100644 --- a/arch/riscv/include/asm/hwprobe.h +++ b/arch/riscv/include/asm/hwprobe.h @@ -8,7 +8,7 @@ #include -#define RISCV_HWPROBE_MAX_KEY 6 +#define RISCV_HWPROBE_MAX_KEY 7 static inline bool riscv_hwprobe_key_is_valid(__s64 key) { diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 731dcd0ed4de..776af9b37e23 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -21,6 +21,7 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); +bool insn_is_vector(u32 insn_buf); bool riscv_v_first_use_handler(struct pt_regs *regs); void kernel_vector_begin(void); void kernel_vector_end(void); diff --git a/arch/riscv/include/uapi/asm/hwprobe.h b/arch/riscv/include/uapi/asm/hwprobe.h index 060212331a03..ebacff86f134 100644 --- a/arch/riscv/include/uapi/asm/hwprobe.h +++ b/arch/riscv/include/uapi/asm/hwprobe.h @@ -68,6 +68,12 @@ struct riscv_hwprobe { #define RISCV_HWPROBE_MISALIGNED_UNSUPPORTED (4 << 0) #define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0) #define RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE 6 +#define RISCV_HWPROBE_VEC_KEY_MISALIGNED_PERF 7 +#define RISCV_HWPROBE_VEC_MISALIGNED_UNKNOWN 0 +#define RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED 1 +#define RISCV_HWPROBE_VEC_MISALIGNED_SLOW 2 +#define RISCV_HWPROBE_VEC_MISALIGNED_FAST 3 +#define RISCV_HWPROBE_VEC_MISALIGNED_UNSUPPORTED 4 /* Increase RISCV_HWPROBE_MAX_KEY when adding items. */ /* Flags */ diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprobe.c index b286b73e763e..ce641cc6e47a 100644 --- a/arch/riscv/kernel/sys_hwprobe.c +++ b/arch/riscv/kernel/sys_hwprobe.c @@ -184,6 +184,36 @@ static u64 hwprobe_misaligned(const struct cpumask *cpus) } #endif +#if defined(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) +static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) +{ + int cpu; + u64 perf = -1ULL; + + for_each_cpu(cpu, cpus) { + int this_perf = per_cpu(vector_misaligned_access, cpu); + + if (perf == -1ULL) + perf = this_perf; + + if (perf != this_perf) { + perf = RISCV_HWPROBE_VEC_MISALIGNED_UNKNOWN; + break; + } + } + + if (perf == -1ULL) + return RISCV_HWPROBE_VEC_MISALIGNED_UNKNOWN; + + return perf; +} +#else +static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) +{ + return RISCV_HWPROBE_VEC_MISALIGNED_UNKNOWN; +} +#endif + static void hwprobe_one_pair(struct riscv_hwprobe *pair, const struct cpumask *cpus) { @@ -211,6 +241,10 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pair, pair->value = hwprobe_misaligned(cpus); break; + case RISCV_HWPROBE_VEC_KEY_MISALIGNED_PERF: + pair->value = hwprobe_vec_misaligned(cpus); + break; + case RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE: pair->value = 0; if (hwprobe_ext0_has(cpus, RISCV_HWPROBE_EXT_ZICBOZ)) diff --git a/arch/riscv/kernel/traps_misaligned.c b/arch/riscv/kernel/traps_misaligned.c index 2adb7c3e4dd5..8f26c3d92230 100644 --- a/arch/riscv/kernel/traps_misaligned.c +++ b/arch/riscv/kernel/traps_misaligned.c @@ -16,6 +16,7 @@ #include #include #include +#include #define INSN_MATCH_LB 0x3 #define INSN_MASK_LB 0x707f @@ -413,10 +414,6 @@ int handle_misaligned_load(struct pt_regs *regs) perf_sw_event(PERF_COUNT_SW_ALIGNMENT_FAULTS, 1, regs, addr); -#ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS - *this_cpu_ptr(&misaligned_access_speed) = RISCV_HWPROBE_MISALIGNED_EMULATED; -#endif - if (!unaligned_enabled) return -1; @@ -426,6 +423,17 @@ int handle_misaligned_load(struct pt_regs *regs) if (get_insn(regs, epc, &insn)) return -1; +#ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS + if (insn_is_vector(insn) && + *this_cpu_ptr(&vector_misaligned_access) == RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED) { + *this_cpu_ptr(&vector_misaligned_access) = RISCV_HWPROBE_VEC_MISALIGNED_UNSUPPORTED; + regs->epc = epc + INSN_LEN(insn); + return 0; + } + + *this_cpu_ptr(&misaligned_access_speed) = RISCV_HWPROBE_MISALIGNED_EMULATED; +#endif + regs->epc = 0; if ((insn & INSN_MASK_LW) == INSN_MATCH_LW) { @@ -625,6 +633,74 @@ static bool check_unaligned_access_emulated(int cpu) return misaligned_emu_detected; } +#ifdef CONFIG_RISCV_ISA_V +static void check_vector_unaligned_access(struct work_struct *unused) +{ + int cpu = smp_processor_id(); + long *mas_ptr = this_cpu_ptr(&vector_misaligned_access); + unsigned long tmp_var; + + if (!riscv_isa_extension_available(hart_isa[cpu].isa, v)) + return; + + *mas_ptr = RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED; + + local_irq_enable(); + kernel_vector_begin(); + __asm__ __volatile__ ( + ".balign 4\n\t" + ".option push\n\t" + ".option arch, +v\n\t" + " vsetivli zero, 1, e16, m1, ta, ma\n\t" // Vectors of 16b + " vle16.v v0, (%[ptr])\n\t" // Load bytes + ".option pop\n\t" + : : [ptr] "r" ((u8 *)&tmp_var + 1) : "v0", "memory"); + kernel_vector_end(); + + if (*mas_ptr == RISCV_HWPROBE_VEC_MISALIGNED_UNKNOWN) + *mas_ptr = RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED; +} + +bool check_vector_unaligned_access_all_cpus(void) +{ + int cpu; + bool ret = true; + + for_each_online_cpu(cpu) + if (riscv_isa_extension_available(hart_isa[cpu].isa, ZICCLSM)) + per_cpu(vector_misaligned_access, cpu) = RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED; + else + ret = false; + + + if (ret) + return true; + ret = true; + + schedule_on_each_cpu(check_vector_unaligned_access); + + for_each_online_cpu(cpu) + if (per_cpu(vector_misaligned_access, cpu) + != RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED) + return false; + + return ret; +} +#else +bool check_vector_unaligned_access_all_cpus(void) +{ + int cpu; + + for_each_online_cpu(cpu) + if (riscv_isa_extension_available(hart_isa[cpu].isa, ZICCLSM)) + per_cpu(vector_misaligned_access, cpu) = RISCV_HWPROBE_VEC_MISALIGNED_SUPPORTED; + else + per_cpu(vector_misaligned_access, cpu) = RISCV_HWPROBE_VEC_MISALIGNED_UNSUPPORTED; + + return false; +} +#endif + bool check_unaligned_access_emulated_all_cpus(void) { int cpu; diff --git a/arch/riscv/kernel/unaligned_access_speed.c b/arch/riscv/kernel/unaligned_access_speed.c index a9a6bcb02acf..92a84239beaa 100644 --- a/arch/riscv/kernel/unaligned_access_speed.c +++ b/arch/riscv/kernel/unaligned_access_speed.c @@ -20,6 +20,7 @@ #define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) DEFINE_PER_CPU(long, misaligned_access_speed); +DEFINE_PER_CPU(long, vector_misaligned_access) = RISCV_HWPROBE_VEC_MISALIGNED_UNKNOWN; #ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS static cpumask_t fast_misaligned_access; @@ -264,6 +265,8 @@ static int check_unaligned_access_all_cpus(void) { bool all_cpus_emulated = check_unaligned_access_emulated_all_cpus(); + check_vector_unaligned_access_all_cpus(); + if (!all_cpus_emulated) return check_unaligned_access_speed_all_cpus(); @@ -273,6 +276,7 @@ static int check_unaligned_access_all_cpus(void) static int check_unaligned_access_all_cpus(void) { check_unaligned_access_emulated_all_cpus(); + check_vector_unaligned_access_all_cpus(); return 0; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 6727d1d3b8f2..2cceab739b2c 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -66,7 +66,7 @@ void __init riscv_v_setup_ctx_cache(void) #endif } -static bool insn_is_vector(u32 insn_buf) +bool insn_is_vector(u32 insn_buf) { u32 opcode = insn_buf & __INSN_OPCODE_MASK; u32 width, csr;