From patchwork Fri Dec 29 14:36:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 08C11C4706F for ; Fri, 29 Dec 2023 14:37:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0JgZAoHl+45DT3ptZMX8CFBT0tTKFBHgEKRqOb/cmfo=; b=MzHuvjgnA8SyBU 5DBf51RXzgYYfLdaR3rTJAue5J5R+qOzfkXhbv1pkzlykMReBfq8cvO6Qqm4dnddaHIKteyS5ot3q F5Un/GIWHM2UGn9PwQodR6Z3M4kqM2EMvOz9bqYRneZYMIhNsJc6amcauGVeWCgqcM/wOXUULgElv vqbg7JuLL3Dy2rcQJIxbvc99lKSkUL4qja54PqOVBZLVxj+xCMf2869PA6pdEKaU1vutypU0iUUwO j+COUK78G1wuOCFPqPds1mzN4KrBC9qlOScNqTPjMtf3qDFzfYcSvhpH++SIhpwiaMCd9tWTuSmJ6 ykzgERFEq998Q1HsofPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDyy-0012m1-1L; Fri, 29 Dec 2023 14:37:04 +0000 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDyv-0012lD-0h for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:37:03 +0000 Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6d9af1f12d5so3584537b3a.3 for ; Fri, 29 Dec 2023 06:36:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860618; x=1704465418; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M6xmMRjAnbCh5FSE4pZP+lYQaviMGn3QncTRImXF4Tk=; b=JV2O2AeEhir4IE8rakpjeIsvTSM5DzBX3Q0q5InmynbJWZP+mIAsPJB8CV+JNgpkPx cXdQ4wD6cMRCNZkMLCvOzCsps4ulWsJz6QOHPBLpOvmSfjWkyHnLgPlqp5qTdLZ/CvGU TohCz5WN85O2jTzLsZxn0WPIhi3ggGBVPuNNgmJYDFgz42sR4364gXhdGmPmR7Vm5lrF /6VPbBUMZoSkg4mNZRfOdYvctqEY6T6TJ6gNIt8IvFAA2dIMPeXHZjVeB5RCz78hEEU9 HZ5z2J3/hSn63vQ4IOOoHGMVoWtuT9uAkddPaxuCy64wFYItLTuz784YJ1tXPgm0iKHI CupQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860618; x=1704465418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M6xmMRjAnbCh5FSE4pZP+lYQaviMGn3QncTRImXF4Tk=; b=tiqDHF6S0RmFMuN/0VhnADjbekmSfg+cLxS2qu8V4H4O/ehTRDgZX/9rj/IbdFEPyy Lr8A9cNBqqGULjadRYzMcpn5SF8WJnsodZb1Ngdwxll136FJaEwOE8Pgy+T1UgbBIw4p fJHMAS+APkrQfPWLl+1NJmVJ6RFgI610b5gqLlCT/igcrFQKNzEOlLABhd06n7dulnEy YFI5UOToRvXD8CjC0ToRvh2Gn7hhIZ0G2w/2sg3xd6bjH3KYZsnQV9SBvh3snqPbRJKC bFrUGT8gZzVa+PVDGrXsmG6PDO8/USuUjUwuyRSe/dlJ1dvY8K9EFcEJWc5yOBz0vPrN FJWg== X-Gm-Message-State: AOJu0YzGHwHLl72W9zPzsEo44LJAijTSSK/upWasf9NPgHx8VBgK/QJd /zCMoNmBp/Vl/wz8Oqlds+pyLGqU3SjitIGzu9Hxs8gZRfKu1lUrQn+y5iVf9sG6g6sYMG9MrVV zfclc+fc0zR7vVfHt3+SOGJbCxpWrGN9xXGT8cFhLKzlDxKKq6qJQmVD5GndAYZbvD5zmxQrneV jAKxDyre4qA2nYRyKAEY9m X-Google-Smtp-Source: AGHT+IHHmBkOJtGk/J4xDReIiPuFswcH0xMJStm5Ph4SxJPaTkcp535scn5zb2BcyegzdNkRmoxR1Q== X-Received: by 2002:a05:6a00:2e1a:b0:6da:5505:1329 with SMTP id fc26-20020a056a002e1a00b006da55051329mr392720pfb.34.1703860618236; Fri, 29 Dec 2023 06:36:58 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.36.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:36:57 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Vincent Chen , Andy Chiu , Albert Ou , Heiko Stuebner , Baoquan He , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Guo Ren , Xiao Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Alexandre Ghiti , Evan Green , Sami Tolvanen , Sia Jee Heng , Jisheng Zhang Subject: [v9, 01/10] riscv: Add support for kernel mode vector Date: Fri, 29 Dec 2023 14:36:18 +0000 Message-Id: <20231229143627.22898-2-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063701_254614_8D466D3A X-CRM114-Status: GOOD ( 25.30 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu Reviewed-by: Eric Biggers --- Changelog v9: - use bitwise to mask on/off the use of Vector (Eric, Charlie) - BUG_ON when reentrant enablement of Vector happens (Charlie) - Move compiler barrier to the premept_v patch (Eric) Changelog v8: - Refactor unnecessary whitespace change (Eric) Changelog v7: - fix build fail for allmodconfig Changelog v6: - Use 8 bits to track non-preemptible vector context to provide better WARN coverage. Changelog v4: - Use kernel_v_flags and helpers to track vector context. Changelog v3: - Reorder patch 1 to patch 3 to make use of {get,put}_cpu_vector_context later. - Export {get,put}_cpu_vector_context. - Save V context after disabling preemption. (Guo) - Fix a build fail. (Conor) - Remove irqs_disabled() check as it is not needed, fix styling. (Björn) Changelog v2: - 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor) - export may_use_simd to include/asm/simd.h --- arch/riscv/include/asm/processor.h | 13 ++- arch/riscv/include/asm/simd.h | 44 ++++++++++ arch/riscv/include/asm/vector.h | 9 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 116 +++++++++++++++++++++++++ arch/riscv/kernel/process.c | 1 + 6 files changed, 183 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index f19f861cda54..28d19aea24b1 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -73,6 +73,16 @@ struct task_struct; struct pt_regs; +/* + * We use a flag to track in-kernel Vector context. Currently the flag has the + * following meaning: + * + * - bit 0: indicates whether the in-kernel Vector context is active. The + * activation of this state disables the preemption. Currently only 0 and 1 + * are valid value for this field. Other values are reserved for future uses. + */ +#define RISCV_KERNEL_MODE_V 0x1 + /* CPU-specific state of a task */ struct thread_struct { /* Callee-saved registers */ @@ -81,7 +91,8 @@ struct thread_struct { unsigned long s[12]; /* s[0]: frame pointer */ struct __riscv_d_ext_state fstate; unsigned long bad_cause; - unsigned long vstate_ctrl; + u32 riscv_v_flags; + u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; }; diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h new file mode 100644 index 000000000000..ef8af413a9fc --- /dev/null +++ b/arch/riscv/include/asm/simd.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2023 SiFive + */ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include +#include +#include +#include +#include + +#include + +#ifdef CONFIG_RISCV_ISA_V +/* + * may_use_simd - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_simd(void) +{ + /* + * RISCV_KERNEL_MODE_V is only set while preemption is disabled, + * and is clear whenever preemption is enabled. + */ + return !in_hardirq() && !in_nmi() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd(void) +{ + return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 87aaef656257..71af3404fda1 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,15 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +void kernel_vector_begin(void); +void kernel_vector_end(void); +void get_cpu_vector_context(void); +void put_cpu_vector_context(void); + +static inline u32 riscv_v_flags(void) +{ + return current->thread.riscv_v_flags; +} static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index fee22a3d1b53..8c58595696b3 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..114cf4f0a0eb --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include +#include + +static inline void riscv_v_flags_set(u32 flags) +{ + current->thread.riscv_v_flags = flags; +} + +static inline void riscv_v_start(u32 flags) +{ + int orig; + + orig = riscv_v_flags(); + BUG_ON((orig & flags) != 0); + riscv_v_flags_set(orig | flags); +} + +static inline void riscv_v_stop(u32 flags) +{ + int orig; + + orig = riscv_v_flags(); + BUG_ON((orig & flags) == 0); + riscv_v_flags_set(orig & ~flags); +} + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +void get_cpu_vector_context(void) +{ + preempt_disable(); + + riscv_v_start(RISCV_KERNEL_MODE_V); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +void put_cpu_vector_context(void) +{ + riscv_v_stop(RISCV_KERNEL_MODE_V); + + preempt_enable(); +} + +/* + * kernel_vector_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_simd() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_vector_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_vector_end() is + * called. + */ +void kernel_vector_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + BUG_ON(!may_use_simd()); + + get_cpu_vector_context(); + + riscv_v_vstate_save(current, task_pt_regs(current)); + + riscv_v_enable(); +} +EXPORT_SYMBOL_GPL(kernel_vector_begin); + +/* + * kernel_vector_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_vector_begin() was previously + * called, with no call to kernel_vector_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_vector_begin() is called again in the meantime. + */ +void kernel_vector_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + riscv_v_vstate_restore(current, task_pt_regs(current)); + + riscv_v_disable(); + + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4f21d970a129..4a1275db1146 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -221,6 +221,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + p->thread.riscv_v_flags = 0; p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; From patchwork Fri Dec 29 14:36:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0AC3EC4706F for ; Fri, 29 Dec 2023 14:37:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=z4fJUCUW69AVXo+XFpl8Fcr4QFrtNNkNoEtxdEaR0Bg=; b=SNcJp3/5CvVbWp pcS1k4GDnotJPcRpmS/jQibbhVbVg/38lxe7BAbzcOaksgni6wxLU9m6AnU02its95TSiRh2dNxsm hAjmM/m9BG5bvjlS74ZLuwuehK4Bj8ltleg8G5bTgvFLi1ic7FJ2mWz9aE8FnyCgaAT+nGmmE5ozb c44ptKPqpZxL1NQqGcFdkXuKA6UXlZnqZOty5Okt6PxMtONvBEr4xzpe/xILLI5ujK1+/J1bdIETK XAWI+yZHnRfXCErBEv6R991k//wLjH6qELPIaeGSxdb5j24/rQ1zePWOQjDyqWz/U1KBs3gEnwZQV zrOsSCnnFu6zzXZk48Sw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDz5-0012nG-0j; Fri, 29 Dec 2023 14:37:11 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDz2-0012mT-0O for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:37:09 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-6d9b267007fso1504766b3a.3 for ; Fri, 29 Dec 2023 06:37:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860626; x=1704465426; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=VdUeptsKHX9LIW7qCWweJJ78pP9QWx453hTzE07H6mI=; b=WZVBqi6Z5IW4FUuWaRhslRYYomjXCXW1AzWPV6U/ZMzzDottb5cS/C29QVH8QJ+Ce1 2Qg7LulsMHtuTUuit203kRV350toCLaraE5OcRzuyj+hqDdp0+8qfxIGYJGUN4ABvkJi nV58Rvp2cbpcXmItFzIhShY2+zdTcNQc+aZWM815ErmpIKFjinmm1S8se+8/6mpyOuuW FNJFeIevT7Y+89YRORiPfybTm7B9Z+0pW9zu0Tn4rnr60emYeIBgV4CnwKkSUjdgT0XX WtkMIyw802KvQZMePmhXgGpGXnPY5LP9jr+G3Ll2gsWMd044eDfGFASfm9mrZp1tsN8m Rugw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860626; x=1704465426; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VdUeptsKHX9LIW7qCWweJJ78pP9QWx453hTzE07H6mI=; b=FZb2963LnttbgSsWaUegwRzcV1x9g2VevgYLeS20Zti24guLQUtv9rm0z7aQ8uniiz HOifU3z78VRWl4uLAD2kgh/ucE0GYAqdz2TsDp8W1ZHUx6sHZqvDwZXdiHe/ftqFC5zX MHBOwSL8ZHyS93w0FhOVtKroe6hZrDjPO1wme05nJJ10fvtJ+Ijl6Yv+E+bBlM/H2wlv tYJWIDkfX0wvM//dhrWyWGSTws/bmYXLT6Fq5qD8XTyP3V3I0tpABwHsward/I3lPLhV A+lOdLOJmkXTN+xBBSM2ULE067bRfhpdfGixfChL/y0daDStiqggQd7N/Tv5hZFlE8iB 0Zag== X-Gm-Message-State: AOJu0YzzcvJuE8sW/6x4ewOusu3RlaQ9ZGkPDWmptfiNDpKmViAd7/E2 UACPkfXGE/Au8fuPD8zCbwj39YsYh6lJNSIPgtIAvzALEK0l0tJyCoWeHm+OkDScBVwQpCAhyZI 7mmi4rfAY9kIyvcOEcB61Df38NhKICv750jBICJLWD6x/wluadKcS/7pp+7WytsIs8A51WBLsfi ubiiwJKL+Wsysmx8ykhqlL X-Google-Smtp-Source: AGHT+IEyP75mXlR3bkE8TwVV1XWwWz9iEgUZtuWs0gXGzDtoC8us2l0/3X7592CnsqyY+j5g+UL3dQ== X-Received: by 2002:a62:fb0f:0:b0:6d0:89be:e477 with SMTP id x15-20020a62fb0f000000b006d089bee477mr4584760pfm.51.1703860626418; Fri, 29 Dec 2023 06:37:06 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:05 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Conor Dooley , Baoquan He , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= Subject: [v9, 02/10] riscv: vector: make Vector always available for softirq context Date: Fri, 29 Dec 2023 14:36:19 +0000 Message-Id: <20231229143627.22898-3-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063708_159861_A21BBA1F X-CRM114-Status: GOOD ( 17.64 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The goal of this patch is to provide full support of Vector in kernel softirq context. So that some of the crypto alogrithms won't need scalar fallbacks. By disabling bottom halves in active kernel-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. So, softirq context is able to use Vector whenever it runs. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu Reviewed-by: Eric Biggers --- Changelog v8: - refine comments, fix typos (Eric) Changelog v4: - new patch since v4 --- arch/riscv/include/asm/processor.h | 5 +++-- arch/riscv/include/asm/simd.h | 6 +++++- arch/riscv/kernel/kernel_mode_vector.c | 14 ++++++++++++-- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 28d19aea24b1..e76839789067 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -78,8 +78,9 @@ struct pt_regs; * following meaning: * * - bit 0: indicates whether the in-kernel Vector context is active. The - * activation of this state disables the preemption. Currently only 0 and 1 - * are valid value for this field. Other values are reserved for future uses. + * activation of this state disables the preemption. On a non-RT kernel, it + * also disable bh. Currently only 0 and 1 are valid value for this field. + * Other values are reserved for future uses. */ #define RISCV_KERNEL_MODE_V 0x1 diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index ef8af413a9fc..4d699e16c9a9 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -28,8 +28,12 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. + * + * Kernel-mode Vector temporarily disables bh. So we must not return + * true on irq_disabled(). Otherwise we would fail the lockdep check + * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 114cf4f0a0eb..2fc145edae3d 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -46,7 +46,14 @@ static inline void riscv_v_stop(u32 flags) */ void get_cpu_vector_context(void) { - preempt_disable(); + /* + * disable softirqs so it is impossible for softirqs to nest + * get_cpu_vector_context() when kernel is actively using Vector. + */ + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_disable(); + else + preempt_disable(); riscv_v_start(RISCV_KERNEL_MODE_V); } @@ -62,7 +69,10 @@ void put_cpu_vector_context(void) { riscv_v_stop(RISCV_KERNEL_MODE_V); - preempt_enable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_enable(); + else + preempt_enable(); } /* From patchwork Fri Dec 29 14:36:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A77FEC4706F for ; Fri, 29 Dec 2023 14:37:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xaq2TRk3wXbQZSmYt3X3wTn1lW/bK0rQJZnzB7U/D/M=; b=gmTHwrH5qvSL5+ o6fS+8hpSumacN0jSBljOlClM7jUdHpwE6csaeuxdYkjzD0dpxJ8GIn+2Ncwtmv50yIuYufdyBKJf YRD/ExPF9qyIfCf/Jwolk/HrcQwghOrO0KxX8XISwxFkKQp/gLAa6DVQSs7mk+K0i+8eY+9Vo8q45 dQslWuJ6o5GRsCmYVKkUQVWxNVXz1U5/CS68bmyQeGdnPmH3j3sJIiNmRXD+eLocq3PSMOCxsq/uA LDAFhiw4x8O9Uq2doKHwv3myAI7YjV/IpjR7dKx187zvXbfKaNMnEHLeVwCI9++W4y6iA77YV0aCW QcdamaZ9nVlcAu51w3lg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzG-0012qf-0c; Fri, 29 Dec 2023 14:37:22 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzC-0012p1-2j for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:37:20 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-6d9cb95ddd1so1229913b3a.1 for ; Fri, 29 Dec 2023 06:37:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860636; x=1704465436; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=NFnYzFltuLxaA5kymfUv4YQXh/w1s0UKs0KJQeLv+3U=; b=OCqOA7y8a02ELDqoVF0Ef3+vJofML1k+x+gchMsXpvGdddIJUE+cuarVOj+GMPgBH9 ujoQOGSAYhTYC3/RPLnZm5rJ0t0SGjg5z8XW6xVpzTnOIus4Yw52y855XcuQCND3jmkD OwqbLm/b2pKW2XhfKJit3dEn4UD86Ucn6ZgPBV+lKnKRSY0MwUm1oPIpv3jdcVF8GrdS vBQ4fs9C0PlvMJpiHeLPtKb2c/BDMNbImtb9HT6l6EV9H1vyu5Bl/cwbRYPOb0eu+gl2 KJDZt7wkzO8vzJkSlXj6JGnyKpTodzyg9xHdOW7+7HdjQ0MnMLDxJyY9O37RP2UFgeWo 27+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860637; x=1704465437; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NFnYzFltuLxaA5kymfUv4YQXh/w1s0UKs0KJQeLv+3U=; b=AyrzbXV6PayXRc9s8Zkrf7wcAIprauJd7ufJcyXPZodqSK9NEA5p9/t2zwDw8aeTlZ lXdHYtt8oD/s9tyrwbpYfzTnyXV46C44apaOVbUxVbrQbbWdO+9yzOtWrDfauZ32N8YB dzis3Q2MMOxzJuxZpZPNhtsM/FrTotYrWNQ15DuL5OyF7OKMNu95fVNDDkTy3QdAaKGC D1x0bkHyPOtRLbaSDTP6Df0K0kkzQn89I7tzJQQ0UbvjSL7rH5y2Qd06cGpSEeMXmP4z da9EcMrmeg0pwcQKDLHFmysFg/xtkcgmcvTZzjgk6stX9w34ynXBrY10aCs8W/su+iM1 FdDQ== X-Gm-Message-State: AOJu0YxX8n7Araj/pYG9wBV9yOt+Olkfa1ij2bOe2r6S+041LNthEYFz SUhyzo9XpzQ0yYclwD5u0CBdNLOw8NuwP6MGZ8WG86kMajsonQlt8dduuIzcdHQtpgrT7s3VEZZ zPOcYrNk88fOjgHsk5TxMjRisQZWtkMR9s/scoYN/KMIfiROCQsN8pevYWs3C2Tonc3GMHHy6I1 ujUg3zLMshXJ2QZ97Nv+h+ X-Google-Smtp-Source: AGHT+IFxXhswMGGlIx0WTiuegaPKq2mPKZAZjD9NDCIXrayRJEOBfLPAh5IMxm4SbJVderTPFf5FZg== X-Received: by 2002:a62:84d7:0:b0:6d9:abdb:9018 with SMTP id k206-20020a6284d7000000b006d9abdb9018mr3162884pfd.15.1703860636392; Fri, 29 Dec 2023 06:37:16 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:15 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Han-Kuan Chen , Andy Chiu , Albert Ou , Guo Ren , Sami Tolvanen , Deepak Gupta , Andrew Jones , Conor Dooley , Heiko Stuebner Subject: [v9, 03/10] riscv: Add vector extension XOR implementation Date: Fri, 29 Dec 2023 14:36:20 +0000 Message-Id: <20231229143627.22898-4-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063718_887237_C2D34A6C X-CRM114-Status: GOOD ( 15.62 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- Changelog v8: - wrap xor function prototypes with CONFIG_RISCV_ISA_V Changelog v7: - fix build warning message and use proper entry/exit macro for assembly. Drop Conor's A-b Changelog v2: - 's/rvv/vector/' (Conor) --- arch/riscv/include/asm/asm-prototypes.h | 18 ++++++ arch/riscv/include/asm/xor.h | 68 +++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++ 4 files changed, 168 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 36b955c762ba..6db1a9bbff4c 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -9,6 +9,24 @@ long long __lshrti3(long long a, int b); long long __ashrti3(long long a, int b); long long __ashlti3(long long a, int b); +#ifdef CONFIG_RISCV_ISA_V + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +#endif /* CONFIG_RISCV_ISA_V */ #define DECLARE_DO_ERROR_INFO(name) asmlinkage void name(struct pt_regs *regs) diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..96011861e46b --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include +#include + +static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_vector_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_vector_end(); +} + +static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_vector_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_vector_end(); +} + +static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_vector_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_vector_end(); +} + +static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_vector_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_vector_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..b28f2430e52f --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +SYM_FUNC_START(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +SYM_FUNC_END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +SYM_FUNC_START(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +SYM_FUNC_END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +SYM_FUNC_START(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +SYM_FUNC_END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +SYM_FUNC_START(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +SYM_FUNC_END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Fri Dec 29 14:36:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E7E0C46CD3 for ; Fri, 29 Dec 2023 14:37:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OXxORX3SzIyy3luvK5xBJ2UEh+qCquso1noI4RdJ8RY=; b=bq2RdrqCl1hwXp V/ISkOyC7NDEeUU+s7UoGWGjFCz1E2XqvNzXfaBA8qsNONWSnevRSNFdhOBxLWIApf76mJjsMzp8W YiKORiAVK2BIOEJ7wkuI5AM15Iwy6+y5RFOdLZt3j/Hqqo/SfElUOg4A3ZF6hZfXhSrEOQcaa3qY0 vrgU2By762BlY+XD5Gwp6BJjfqxSOxOMXcI5XUlbUOjZ87ri8Ku+WeuGzkeUQ4VaZJq4gC7xZqkIY kMst1mW5DvOspZZmj5dV71DIW3DruAS72SUgW3zIfV/zs2W8UEDP25x3jRqLocYdlmXU8c95a8ezy 1UODLzZ3Zo20gnlqR+lw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzT-0012th-0U; Fri, 29 Dec 2023 14:37:35 +0000 Received: from mail-oa1-x2a.google.com ([2001:4860:4864:20::2a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzQ-0012sb-1d for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:37:34 +0000 Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-2046b2cd2d3so4025143fac.0 for ; Fri, 29 Dec 2023 06:37:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860650; x=1704465450; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0RHjwWZdLq7YyQR5jZZoQUPYbpELlC3XDHPQclpwmMI=; b=nOh81/XFCKYPLXpsQJRztmZDQiCZFHm2UwURY+FiqB298M5MKU3WwuNCWlVC/op3V+ DOiBjZVuoK1BP81X9gkxNyOlsWHTBelm+0pdsKERxMqrZ0qM4/lwbNv7UnR5sBp2zpji KY/g9G/Ha92KShZknN4UJCHRhgRxbSl0EhqMwlXB0YP2X+yZ48e2CoqQqxu1DEZ/JzBD YSjEa0nAkkXdVdo3kBKaylLLvDLbY9kz7hFJ/l5n2PSWzzTyaZQJj0SA2NVJw07qC7OY JJrqTx4tmn3DzA2TVEz+TEWKqu/tW4FzHhKCawwE01GFoYL6103a5b7XOSuPMsJw/VyH wWCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860650; x=1704465450; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0RHjwWZdLq7YyQR5jZZoQUPYbpELlC3XDHPQclpwmMI=; b=xGmSNKFnZhlxU3qx/9GRDIKp0mkjWxBGs2rd99nm52kTUXWvd7Vz1y/a18I5wlboht v2FVvZyIsrdFbjefIW8PDuGH78vC5l751LzC4A++qpIOQDQEyAKCgyN/AXugrBQkFw4g GXbIpt88EUN2qgf5pM9CdbaPX3J83g85rbThJzz2J5mQb6qhJgKdVYlwpW1eq2YUF/dI zE9lARR08KSuhlhTCI166zF+Bl5Gd09b7HNLE3I2uNecWZu2rLxJ7OgDmvzFmoB3VZZg sy8KmpUUBRb/FohvIxBxdWQZGy+5jPEJ5+hrTOamHkLgil6eVLFd39vg7ihe0zAi/oLA GGbw== X-Gm-Message-State: AOJu0YwujEDxCyYWFw6v7109s79RhPemZRTtsbt7S93LJx0Won9XnTiv yVhQcFFivn7BlBcLDlLsIqjAew2VelrOhHagWuV7kS/wMa1wWRSiVtJDMKN22LWFw91vZaMVU71 yAOXWZqLD2h4kUvDbJcqBpT1s0NVUXsTCRFKmOjuxSaeqwuMAK3xnV1jHG7CMxon8vQ1xB5u0CC q3yXJVmJs2LVLwe/lMqb03 X-Google-Smtp-Source: AGHT+IGW/j0yHKFmCI4gIyeZ/IcMKqtdaoz2DNA9kLfX0dB+N4uBsH9FhlY+bBCbFmelGQboajD7Pg== X-Received: by 2002:a05:6870:ac8b:b0:1fb:29d6:b134 with SMTP id ns11-20020a056870ac8b00b001fb29d6b134mr15735694oab.23.1703860649713; Fri, 29 Dec 2023 06:37:29 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:28 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Oleg Nesterov , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Guo Ren , Conor Dooley , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Sami Tolvanen , Jisheng Zhang , Deepak Gupta , Vincent Chen , Heiko Stuebner , Xiao Wang , Haorong Lu , Joel Granados Subject: [v9, 04/10] riscv: sched: defer restoring Vector context for user Date: Fri, 29 Dec 2023 14:36:21 +0000 Message-Id: <20231229143627.22898-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063732_545013_A680FCA4 X-CRM114-Status: GOOD ( 21.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changlog v9: - update comment (Song) Changelog v4: - fix typos and re-add Conor's A-b. Changelog v3: - Guard {get,put}_cpu_vector_context between vstate_* operation and explain it in the commit msg. - Drop R-b from Björn and A-b from Conor. Changelog v2: - rename and add comment for the new thread flag (Conor) --- arch/riscv/include/asm/entry-common.h | 17 +++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/ptrace.c | 5 ++++- arch/riscv/kernel/signal.c | 5 ++++- arch/riscv/kernel/vector.c | 2 +- 8 files changed, 41 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 7ab5e34318c8..19023c430a9b 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,23 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + /* + * We are already called with irq disabled, so go without + * keeping track of riscv_v_flags. + */ + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 574779900bfb..1047a97ddbc8 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -103,12 +103,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 71af3404fda1..961c4e3d1b62 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -193,6 +193,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -200,7 +209,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 2fc145edae3d..8422c881f452 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -117,7 +117,7 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_restore(current, task_pt_regs(current)); + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_disable(); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4a1275db1146..36993f408de4 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -171,6 +171,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -187,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 2afe460de16a..7b93bcbdf9fa 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,8 +99,11 @@ static int riscv_vr_get(struct task_struct *target, * Ensure the vector registers have been saved to the memory before * copying them to membuf. */ - if (target == current) + if (target == current) { + get_cpu_vector_context(); riscv_v_vstate_save(current, task_pt_regs(current)); + put_cpu_vector_context(); + } ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl; diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 88b6220b2608..aca4a12c8416 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,10 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); + get_cpu_vector_context(); riscv_v_vstate_save(current, regs); + put_cpu_vector_context(); + /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); @@ -134,7 +137,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 578b6292487e..66e8c6ab09d2 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Fri Dec 29 14:36:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 583FFC46CD3 for ; Fri, 29 Dec 2023 14:37:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+hIieG5alVHABdmXoSXIF3wc/8qqCtNaqGJkH8HYHzI=; b=2G6a7axqfP7Nf4 IjLmk9lOb0JZp1gRbCh+DHrNScoBD2QQ1MAHrofKWVP1ImUQXA+GYwOSvyCmp8jV6U7xdiV/hg8eR lIS/x0Hip8jgbSHhTwyFGU9PZ4jr+Dz210EzZmR4ALG3miv5SPGlUHMQiQN01Ln0zCse8F6eiyYZn ZAdmhaSWD9M4MitcettCcZqV+Z9YEMi8yaw4fisEv0Ho+c8oz+2K1zAaNbdwWisRM2Wpr8rg6emy5 Qf6m43b1mqRoK6Rr18Dx5S1uI3RUeqIXixX1muFQ+xD82Q5m4OGIvEhLlD/Q6WQFg9vy1diTZ+x94 0uuTXrX27wHXZccTgfKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzg-0012xs-0T; Fri, 29 Dec 2023 14:37:48 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzb-0012vv-2E for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:37:46 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-6d9b37f4804so2507362b3a.1 for ; Fri, 29 Dec 2023 06:37:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860661; x=1704465461; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=vWRlgwSOjDtRNsyl0UctYF1l5cwomAcgVJVuoWlsSQI=; b=dWnuZyDJyIUeybazp+Y/DYdPLfCFLSUKqIdIMoLYv4xR72Z3/NSJf9K7BMDumFQLgU 649BOOYU/7Sn2+fl7bznchrTptgQohFtkFoe+yBHwoGy/VeI0fTcwAIfcQO/wL4hX+iL h0uDhaG/Vfy8UpBp5GhnAc7JR9wObScGnHWoHxr81mVF7yxbLlpqOog0YHFMZoZ4L9Jt Sp4NNAQu7RCxmsju/kiUwRfe8D6DbahDxPhZ1gp4k084lqYTKkEzodU1BYLKBw8HT8lk uC2UX9a2pcFYtfMOHF4lZN/DNidGeYVAvjQz8beVO6JHmjtnI9qD+YC4wP6sRJ71piqb t3jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860661; x=1704465461; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vWRlgwSOjDtRNsyl0UctYF1l5cwomAcgVJVuoWlsSQI=; b=DyU9mTNlVR5fe0ShpuZbXs0UjdP5fINarM57+bFSBJ/4WjyeXfqtR6n/jZ7p4WsHG2 DiunCjhYmTJwcgDQMdnLZQqeoJVkBxY56dvHx9k03xvaWIdYAFZWtw6m3BJdEiLfGeKP b+l+XRLduFaHe9DL5TS9meJkhg8Dd7kS+gJe4DKbTldPsSm3y8nOeVVmo8DJt1pO3TnO Kw2wOFRSsrADIq5hQbtHeL1L0STUK72mtgSoNyQJ7McVNPZhwJkVv8YA2fQJsn5lLfiZ utJlcmus3ERA1Zf+recufsMrf3XprXuUr4aC34UlmyHMwfFFNdOQF2LYq9HqTDxAGufE 5ESQ== X-Gm-Message-State: AOJu0YzPjtXrmwrOytRdKJJnNH3JRWGDP/YK0fXEej0/DR63jhDTjE32 OhwjYgIwdM3uX69cDkIkr1FLGs8Dh1IqU0sEvOLl0g/+8TSCMa+X0QNFusuhxUxO2JQvqrPFb2U ZpwdlNWTjr0hW+rC5NTVOSLCVd/m52ni7NtF8h/F3qSy0SaFaLUWfFVmL/vF8lqgnj4OKaRp6Td qYh5FJvgn+ftGBCIVDlX97 X-Google-Smtp-Source: AGHT+IGgzoxOkULw65eJ9QryqjMf3hxeTXnICwi64OQIwlCtex/QBsNrkHZor/xIJLPz8rx+aQn+xw== X-Received: by 2002:a05:6a20:7291:b0:196:36d2:9baf with SMTP id o17-20020a056a20729100b0019636d29bafmr5101479pzk.46.1703860660664; Fri, 29 Dec 2023 06:37:40 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:39 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Guo Ren , Sami Tolvanen , Han-Kuan Chen , Deepak Gupta , Conor Dooley , Andrew Jones , Heiko Stuebner , Aurelien Jarno , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Bo YU , Alexandre Ghiti , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= Subject: [v9, 05/10] riscv: lib: vectorize copy_to_user/copy_from_user Date: Fri, 29 Dec 2023 14:36:22 +0000 Message-Id: <20231229143627.22898-6-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063743_727079_93153CE9 X-CRM114-Status: GOOD ( 22.35 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch utilizes Vector to perform copy_to_user/copy_from_user. If Vector is available and the size of copy is large enough for Vector to perform better than scalar, then direct the kernel to do Vector copies for userspace. Though the best programming practice for users is to reduce the copy, this provides a faster variant when copies are inevitable. The optimal size for using Vector, copy_to_user_thres, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it. The exception fixup code of the __asm_vector_usercopy must fallback to the scalar one because accessing user pages might fault, and must be sleepable. Current kernel-mode Vector does not allow tasks to be preemptible, so we must disactivate Vector and perform a scalar fallback in such case. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Signed-off-by: Andy Chiu --- Changelog v8: - fix no-mmu build Changelog v6: - Add a kconfig entry to configure threshold values (Charlie) - Refine assembly code (Charlie) Changelog v4: - new patch since v4 --- arch/riscv/Kconfig | 8 ++++ arch/riscv/include/asm/asm-prototypes.h | 4 ++ arch/riscv/lib/Makefile | 6 ++- arch/riscv/lib/riscv_v_helpers.c | 44 ++++++++++++++++++++++ arch/riscv/lib/uaccess.S | 10 +++++ arch/riscv/lib/uaccess_vector.S | 50 +++++++++++++++++++++++++ 6 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/lib/riscv_v_helpers.c create mode 100644 arch/riscv/lib/uaccess_vector.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 95a2a06acc6a..3c5ba05e8a2d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -525,6 +525,14 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_UCOPY_THRESHOLD + int "Threshold size for vectorized user copies" + depends on RISCV_ISA_V + default 768 + help + Prefer using vectorized copy_to_user()/copy_from_user() when the + workload size exceeds this value. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 6db1a9bbff4c..be438932f321 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -11,6 +11,10 @@ long long __ashlti3(long long a, int b); #ifdef CONFIG_RISCV_ISA_V +#ifdef CONFIG_MMU +asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n); +#endif /* CONFIG_MMU */ + void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, const unsigned long *__restrict p2); void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 494f9cd1a00c..c8a6787d5827 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -6,9 +6,13 @@ lib-y += memmove.o lib-y += strcmp.o lib-y += strlen.o lib-y += strncmp.o -lib-$(CONFIG_MMU) += uaccess.o +ifeq ($(CONFIG_MMU), y) +lib-y += uaccess.o +lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o +endif lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o +lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o diff --git a/arch/riscv/lib/riscv_v_helpers.c b/arch/riscv/lib/riscv_v_helpers.c new file mode 100644 index 000000000000..6cac8f4e69e9 --- /dev/null +++ b/arch/riscv/lib/riscv_v_helpers.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2023 SiFive + * Author: Andy Chiu + */ +#include +#include + +#include +#include + +#ifdef CONFIG_MMU +#include +#endif + +#ifdef CONFIG_MMU +size_t riscv_v_usercopy_threshold = CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD; +int __asm_vector_usercopy(void *dst, void *src, size_t n); +int fallback_scalar_usercopy(void *dst, void *src, size_t n); +asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n) +{ + size_t remain, copied; + + /* skip has_vector() check because it has been done by the asm */ + if (!may_use_simd()) + goto fallback; + + kernel_vector_begin(); + remain = __asm_vector_usercopy(dst, src, n); + kernel_vector_end(); + + if (remain) { + copied = n - remain; + dst += copied; + src += copied; + goto fallback; + } + + return remain; + +fallback: + return fallback_scalar_usercopy(dst, src, n); +} +#endif diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index 3ab438f30d13..a1e4a3c42925 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -3,6 +3,8 @@ #include #include #include +#include +#include .macro fixup op reg addr lbl 100: @@ -11,6 +13,13 @@ .endm SYM_FUNC_START(__asm_copy_to_user) +#ifdef CONFIG_RISCV_ISA_V + ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_v, CONFIG_RISCV_ISA_V) + REG_L t0, riscv_v_usercopy_threshold + bltu a2, t0, fallback_scalar_usercopy + tail enter_vector_usercopy +#endif +SYM_FUNC_START(fallback_scalar_usercopy) /* Enable access to user memory */ li t6, SR_SUM @@ -181,6 +190,7 @@ SYM_FUNC_START(__asm_copy_to_user) sub a0, t5, a0 ret SYM_FUNC_END(__asm_copy_to_user) +SYM_FUNC_END(fallback_scalar_usercopy) EXPORT_SYMBOL(__asm_copy_to_user) SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user) EXPORT_SYMBOL(__asm_copy_from_user) diff --git a/arch/riscv/lib/uaccess_vector.S b/arch/riscv/lib/uaccess_vector.S new file mode 100644 index 000000000000..7bd96cee39e4 --- /dev/null +++ b/arch/riscv/lib/uaccess_vector.S @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include +#include +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + + .macro fixup op reg addr lbl +100: + \op \reg, \addr + _asm_extable 100b, \lbl + .endm + +SYM_FUNC_START(__asm_vector_usercopy) + /* Enable access to user memory */ + li t6, SR_SUM + csrs CSR_STATUS, t6 + +loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + fixup vle8.v vData, (pSrc), 10f + fixup vse8.v vData, (pDst), 10f + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + add pDst, pDst, iVL + bnez iNum, loop + +.Lout_copy_user: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + li a0, 0 + ret + + /* Exception fixup code */ +10: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + mv a0, iNum + ret +SYM_FUNC_END(__asm_vector_usercopy) From patchwork Fri Dec 29 14:36:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3323EC4706F for ; Fri, 29 Dec 2023 14:37:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ffuoEMdrovxyufjQG4HqGe2hJknbdY6C1xKF209n7Ao=; b=25jJMdln1WuY9z vgtsHAiuA+uTp4dTWj2mjRywsoWbDat4kxkcT/J4s6FDzKAVaBXjZ5ochfsCmc9nKHAypmMyQaEwW gi+6arkpXBU6S/2rncnWQkfkcHxoXxz/By+xHizoyg3WU01GmwtZJO4BgSZU0Q43mneURcHz/NfGa fen+Dx2fiqEsGcTlMdRMcy9QO8afzlp3lDcx3OFECMO+ZStXRAV/1FCP3di2OcVQFwAw+9iqgI7aK 5XqxKIQbH8cq59gg2BPIvp1ATK5LF9a2torPl4E3wPJ+kDAndrlP8HUyECDpYbq/N3kVveojWIlMF cbQdovai24Z2onrn7jkQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzj-0012zm-31; Fri, 29 Dec 2023 14:37:51 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzh-0012xe-1X for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:37:50 +0000 Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-5cdfbd4e8caso3892299a12.0 for ; Fri, 29 Dec 2023 06:37:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860666; x=1704465466; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=9ObLV7T59RhMl1ztHBcJYna8z6xraXxN5eXN0lzHnI0=; b=OMqu/p6jlM9OEc/y/9uHePaqXJI42bSpmlY2BjbYUSeffEYtOdG0Zsg8E2qlPqPaId 5n4EiaJxm+Vw+SIgEX2Z3uRz6GmBNWo42RxvB4teqDiAsU7bv3zJdv+W3lSzmENEDIvy At2RT2OUVr8ovCChiVFTdDpT9YAwEWqCDXKnw464V7iq1t05T+4t0W8Cwl+bfxoBrS97 azTs+uqorI/Z0mWLHArV0bHcYExo9cWK47s7nEk/CwxYDebl7hp3bHTQOxiX7DF7eQjc uuKvxZz0pEnLzrEWjZFybhEVaUUhQZ6nFmIqxirHGZwWHQlHtR5bO4AYODTNfrbMSOCB TK3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860666; x=1704465466; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9ObLV7T59RhMl1ztHBcJYna8z6xraXxN5eXN0lzHnI0=; b=WrOa6oRta/xl0bykFxy/ARScUdqYfZ2MH8TN/FdWVSI7RUCM2LpIS84LfR0QFm1Uht 1K8OUJUT05NTz0dPFa9DcBOXadoIfjfDF7430pvlHrJ4nyLqIp9aCijJETl2qaHb2iLE QZ03cfFN1JZrZrxLyM2Py5mru1ce8o7+I50rC+nBX78jc1nLluMgxkiXI/LvMPd5alwq /j36fqEWRualxUDLOgDgdUEVZ2IW1wv9nuIXq7C5wip8AOOfMuee5dZSiCZa+ccvzpPW myXS1cCiahSiSIC/otxZH5MBWQ9mzkfMH1eDD2yh3puANTnE/IBfzXR9z1vY63g2ZEND XBCQ== X-Gm-Message-State: AOJu0YwAbOyC0SAIOiR7c9ILjf3pz+27dntTGib7li34mFF4sZg9Jy1k aTnRJHyK4jY1fzvo09NumsKn9Awg1Lx0HXJxdIcCOnRjUjHvmrbEKwjMm2111g0PSnu7AN6S0G/ c2fqAEB9LqUN+cbr/DBz19TvOujS+I/O3EicoLvAknR0z7H30Gtj6brJshHUVUYE6lhcB3APVqM 2k7emRhLp1ATf36RECSOXt X-Google-Smtp-Source: AGHT+IERAfaWIZbVx7vrYsKuOkhL8zwGuFluV3OfH1rmhMG9L2R3JQ2nPrdIWr7XZhkKT15pvB679A== X-Received: by 2002:a05:6a20:7fa1:b0:196:8eb2:43a2 with SMTP id d33-20020a056a207fa100b001968eb243a2mr639856pzj.113.1703860665513; Fri, 29 Dec 2023 06:37:45 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:44 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Heiko Stuebner , Guo Ren , Conor Dooley , Andrew Jones , Jisheng Zhang , Xiao Wang Subject: [v9, 06/10] riscv: fpu: drop SR_SD bit checking Date: Fri, 29 Dec 2023 14:36:23 +0000 Message-Id: <20231229143627.22898-7-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063749_514007_1C7EA1A9 X-CRM114-Status: UNSURE ( 8.66 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org SR_SD summarizes the dirty status of FS/VS/XS. However, the current code structure does not fully utilize it because each extension specific code is divided into an individual segment. So remove the SR_SD check for now. Signed-off-by: Andy Chiu Reviewed-by: Song Shuai Reviewed-by: Guo Ren --- arch/riscv/include/asm/switch_to.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h index f90d8e42f3c7..7efdb0584d47 100644 --- a/arch/riscv/include/asm/switch_to.h +++ b/arch/riscv/include/asm/switch_to.h @@ -53,8 +53,7 @@ static inline void __switch_to_fpu(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - if (unlikely(regs->status & SR_SD)) - fstate_save(prev, regs); + fstate_save(prev, regs); fstate_restore(next, task_pt_regs(next)); } From patchwork Fri Dec 29 14:36:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB6CDC4706F for ; Fri, 29 Dec 2023 14:38:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hkjOVa0TbZa4UcHFXwti5zZbUR1X4cEyxErhBWLD9Gk=; b=VuZu5zZCmTkcmM 5Sy3zKitVpMpWepyAdP7/jsQr2ch38fmtnGCjwv3dClv3JEiqZNr5CwMdXupOZUGNjqAnYPVMH8F1 ugaMXzq8HCVPCztN2yRa6xY3MEr7LlhiyLAoMfUryNOXOsZx8LPQadIlKLml69DUHvNyrc38eHSxA qjOr74BV0kRrcheJGQVLKNnhJ2LtRGlYcjdUWHlsElmGTk8RFbH9E/JW5jQsPdLEigAJZeLVCLzF1 qw/6GJMV14iZa/5aQdDcqYKrE6+FPtjJk1l8ePJPCjpPxaoMmMObaarls1Xr/rxBp08a8Z43vFqtP c/YBWfldB43jzc4f3AAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzs-00134G-2f; Fri, 29 Dec 2023 14:38:00 +0000 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzq-00132D-1h for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:38:00 +0000 Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-6d9aa51571fso3622141b3a.3 for ; Fri, 29 Dec 2023 06:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860675; x=1704465475; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=R71ZbdXB/xy6waoYzA8y4C86dv9lPv8L1sbMaRUXrl0=; b=kRmq6Hk0MSCvth4f+mDe42UKKujhSmtnlBswWyjq2z/t7x2Gi38jIWrvwnWH8BC76Y NBRblKGrbcFHQk7pkCroYQdVjm/DpBBDlKgTrd+bWeeNPUbeogp7PfSBvaaZRmbBXjFH CthDX2WH5oY0qlIbJb2GJdXLWDW7YSyFWq1RAm04nE3jACM3k+mJMP7nTBl1V2TtRFSX L93VAlO98VSaN4lLLNGZSFXazYQosO7iXg7GQDRKus+BDsah3vYQdf0h6M9F140vuVTR m7ofhC6321m9XXjxbghsw1xLuBPiJt7Amuy2REaHvoCztYgYdKuq9GZbURu4JQUN6zzI kJWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860675; x=1704465475; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=R71ZbdXB/xy6waoYzA8y4C86dv9lPv8L1sbMaRUXrl0=; b=emhj6EFQ0+FWhz24q+xs/oDlhGOxCTht2upcX/YRj9pifrf7aQBDumW9TNUaMy7cBg XJYqWE82gL7YqMdtrefcOGf9rnMX1nR1vyXUHN8p4T8uTZSFwRQ581frYOlPJtnT+o1B RLZcFwaEtwnMS0SOCPASUuxLosSZjNuH+wS+9b4nzAW+YpWdnsANMrJ+eaaoyRE0rxel M3x70Yv1097szB0TgMUuyfBWFjXeXX4vYkSNJZ22Zp3Osu4Od6Vta/uUv4CdY0dxTRTs gFR2C/toYpwLymda8itarpqMlKuSo2lJpmmYwTRnbOQXvjAFm5RiMPvRmHlALToJEMhC 44Fg== X-Gm-Message-State: AOJu0YxQee1LKFwTuNij/nurEW/mU3JjGKwGsBkjYMu6NWHY0KqwPbQJ 8N6jdaoKc1lg4+eQZVgkMQsTAYzJkoltDQ9A6/+R+vWWh7EjYSvGCD7vovIaE3LPJCQNY/KpKLD uMsYideuBgxiZpWlG4LP9Mn9ssBWxfLhJNYEuGYdC9ibv/gFNFNw5lb99dMK8jO7L/lOwcuXjKi HFaLdsxFSIV0vmmn1GqFGE X-Google-Smtp-Source: AGHT+IFxBr30GeDVa+WvAQpGxFEv0kYtGxm/wlOiNQAZDE5H75Sw42/k10o9SCqDwfLDHtjb6MQ0Xg== X-Received: by 2002:a05:6a20:8f01:b0:196:39c2:d48e with SMTP id b1-20020a056a208f0100b0019639c2d48emr4057570pzk.65.1703860675359; Fri, 29 Dec 2023 06:37:55 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:54 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Oleg Nesterov , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Guo Ren , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Vincent Chen , Heiko Stuebner , Xiao Wang , Haorong Lu , Mathis Salmen Subject: [v9, 07/10] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Fri, 29 Dec 2023 14:36:24 +0000 Message-Id: <20231229143627.22898-8-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063758_561037_87E54F8C X-CRM114-Status: GOOD ( 11.87 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v6: - re-added for v6 Changelog v3: - save V context after get_cpu_vector_context Changelog v2: - fix build fail that get caught on this patch (Conor) --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 19023c430a9b..2293e535f865 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -16,7 +16,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, * We are already called with irq disabled, so go without * keeping track of riscv_v_flags. */ - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 961c4e3d1b62..d75079520629 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -171,23 +171,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -208,7 +204,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(&prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -226,8 +222,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 8422c881f452..241a8f834e1c 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -97,7 +97,7 @@ void kernel_vector_begin(void) get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); riscv_v_enable(); } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 7b93bcbdf9fa..e8515aa9d80b 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -101,7 +101,7 @@ static int riscv_vr_get(struct task_struct *target, */ if (target == current) { get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); put_cpu_vector_context(); } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index aca4a12c8416..5d69f4db9e8f 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -87,7 +87,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); get_cpu_vector_context(); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); put_cpu_vector_context(); /* Copy everything of vstate but datap. */ From patchwork Fri Dec 29 14:36:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1ECB4C46CD3 for ; Fri, 29 Dec 2023 14:38:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DBoi3TludzXQKB4y5mbYXIWz1RaUEj0UxOrD0WXy9GU=; b=p0Np05bAaHKzXj SDgQuA5oUy7FkGxO/oiqvGmBj2M0G1g3MKQtDsZk5ZiyTw8Wo0umTD3CFLICUJtYF4kO0F9J2jY2u RliYHbC2liG4+ZiP+kv86p9mU2glyEc/GwG86dxZ/+hp2+vyIjaZ0iDFmFZtwTFdgw0XuYgxyJCMz A6limvrf/U54NFoO5+LwuG1H8ytdPRAbQrfo5+9x2zbtlFACIk4t3vLXePxTLAc/YaJ+wJ0YcUEEb FoCj3Cf5LeHHvC5Yt/buQHx38Gj40X2p3oQAH4rWQuQruk6dwfw/avkVUSS60I7TDVuJ6UmQrQO0A xqhWe285mWQc/ADJe0iQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzz-00139M-2o; Fri, 29 Dec 2023 14:38:07 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJDzv-001356-1j for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:38:05 +0000 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-6d9b267007fso1505247b3a.3 for ; Fri, 29 Dec 2023 06:38:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860680; x=1704465480; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=ZAHVGIzYcld82P9wvgk6s02G+QSu3XRRT3GBW8nCSDV4WKSk5A8lWluIgZlLCwasqb dbku65x/zS0il3NHd/x+2QFuxT2MuunhK1v9/dG+ctl/75rkyfF+FMk//w0U6NKAJDsJ PSrJZc7vlVL0EQIeiNAFsvLWschpGiulhfaWcCKN0KqJH4Bzc/zEspkyR49m6bDvBmp/ qmr7E+FU+CO/90xt2odGyXwJEchu82B13qVzLik0efeL1SeDOZ0QQJ3PukspttcjUVAJ SKli43FNFDC5EVZ53wlqq8McoXfHFJaGU2ZsIkGX7lNLBtj4iGnK9sj3sWfsY7obQO91 A6Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860680; x=1704465480; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=jPQBGJOjIpSasUMBLT2cs5Km4OorSElV6WQlXnIraT+b0ODbf20nVrSuxrCWAMxt08 GjKMNT+S6lGiAZ8NKkkApeY3PAWjiyoWVvN2MF29/dStF2sudbh9RaIvBS+iRJVn7JmE vO8sTNcZf7cdVbmcCUX+PxABtnSuRC6qIMIQ//nFRJJKv0v7+0rejGfpem7dn/Ozw6C1 w7U61wEFi7dNj0hqsqAsiMlnzRLJ2y/Hhuj12olL7HYN1MLeENZ1tsYMAtOJoL2SJ8E3 jeKCGB8t3MAYy4oAANZFUczp0eo+jgAAnTStShIpZwlGZNn3icSOnZdXTupYm82aZLZB L/aQ== X-Gm-Message-State: AOJu0Yw8L9eoJ3y0QkjZHBfw0Ii3dag7kE+ORm2b9Y3ytnxFM26rNazl u/DT9n8sWij8Stoh4T2k6OyvkkwI+/ytzA44A/E31AP3ObQ2gqZ9z06rf0eJBAdnXr+Juzqcf+w 7d92Db1n9tBFpVajTzJ5l/o6PztDW1r9QtmZxa2ynns91GwyRLKAFwOAhgVkUF1UK7iiUwnNhSp JzYK/6iRyhFsF7tNyZh2RU X-Google-Smtp-Source: AGHT+IG41ytGn7VITPy8bf2YNNnKQHG3TCNTTLkGR2XfGytSSp3X4wHm+SmUglrCJRjRU2Y1auGS/g== X-Received: by 2002:a05:6a00:d0:b0:6d9:aa6d:e3d3 with SMTP id e16-20020a056a0000d000b006d9aa6de3d3mr4183497pfj.66.1703860680401; Fri, 29 Dec 2023 06:38:00 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.37.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:37:59 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Vincent Chen , Conor Dooley , Joel Granados Subject: [v9, 08/10] riscv: vector: use a mask to write vstate_ctrl Date: Fri, 29 Dec 2023 14:36:25 +0000 Message-Id: <20231229143627.22898-9-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063803_577894_AD85211E X-CRM114-Status: UNSURE ( 8.31 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_ctrl_set() should only touch bits within PR_RISCV_V_VSTATE_CTRL_MASK. So, use the mask when we really set task's vstate_ctrl. Signed-off-by: Andy Chiu --- Changelog v6: - splitted out from v3 --- arch/riscv/kernel/vector.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 66e8c6ab09d2..c1f28bc89ec6 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void) From patchwork Fri Dec 29 14:36:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E708C4706F for ; Fri, 29 Dec 2023 14:38:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HmZ1Hr9ExoLYpRD3aCZ7gF0PITy8IJnVXm49xkG1OYg=; b=KtBwEXkzD3lQ1d m8EJXlZPTXObmvUzBvQNbi1TgrqrefJwJ/efshbAT2eUdJ1taZYB1JB8dnruBcjib/3l3cp6NWwZ9 855ZAC00KwjZimr9gg61thUnXR4qoqHSOSmE6198fapp74Wr/H5a82H4rSwsnIQkeKXG8hw8VUqzo hKkcnJelPGxY0ylm5pJv9WPifu6GpC63SlVGrZ18n21Ymuv4CfJCUm2RAGZ1ngGRNlMpfexUFkRV9 I4bbCa+3WUXgKSq9/0LsY3Zz+9KirRq9n6QtrEnZEO6v9qwyH3TpsMAX/bj7n44fM+F3d9tY+nIwI w6/Cyhl4mhlgXdJ1msHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJE04-0013Cm-2v; Fri, 29 Dec 2023 14:38:12 +0000 Received: from mail-pf1-x433.google.com ([2607:f8b0:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJE02-0013AM-1p for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:38:12 +0000 Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6d99980b2e0so4409684b3a.2 for ; Fri, 29 Dec 2023 06:38:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860688; x=1704465488; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=nNt6uSdbZ7DbOtq3KjOLhDCh+Jm85/iAkPBdV5go9e0=; b=HXp0uTK30R1pJxBt+o/9Z911NkJHis33OQPuMagNNMW3s4xSurrcETqu9Loz5dXYMH /6U8OMUmsqEsQjz5GaGrZdox4yUe5S1wYbjDWyK9tyywewPn7gtGQv9foJNhwqrUHO4h Fy6Je94Udza3ZPbizh0NEZsane/e4a3D9sXhN5belY63/TJnH9OkOJSNNpzK4hjl3Wu5 316dr9zJLt1bO9e2W5bsCZJWkiRdOt+TQATVmiWbwz5di60XCNQ7ZIBTOGczi5E6Qw72 QOdi/HVVJN+fZ1/TnCImDOhiG2iCyZ59pdpWLuUKRHwmzelbO+qTbfN5hpvxW+d8Hc1m Drjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860688; x=1704465488; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nNt6uSdbZ7DbOtq3KjOLhDCh+Jm85/iAkPBdV5go9e0=; b=kVDB5hjmdROlzvuUvNbBKEgYnaKrZfwHSibznH5JblplN6OirsbpuwwlGZQU0gIV8X PsOrcoF6Vb6d896zmPfK5D2U4eCMgnD1ccbX3ajFnjluYpgj3A6V4OvppBiScCe5WQHz /zqfaSR/IE25o34WutVnLdqKoxADhlMvHlKQRWdDzdm6RUvyBopvFFkLt0Nk1OQgxsRx Lnhc0JVn2E6dUE/V2Sb+JJFmFkWjvOFnIPHXvVrnL8ywW601JJWNZH5gK34OEx/bEzm+ OVakuptOFWmaLeibKLFcxN3yGZBnP/dnmn11Ws7ztRsBq8VMAnx4JsL5jPfWc2HKIVY+ N0Sw== X-Gm-Message-State: AOJu0YyBLXakNDzVT59lNGB/y3lDHGC/3PdwGCuHSc5+JAQW4Dagl5qL 4E/ryD0STnVdGAABA29AStpm7NpzZOLoYkxpOfldqGGmXbA3QtupUWYIhsZqzV+E4pVDVTqRJnL 2yxPhN0FsZg9jG5dH/0bLF2QK8HOIKjd6Tuht28bpNfq0NcdOQKH9c0w4QqEkH7XI+Zd2whihp1 mknbzMq/5yDO0x46L1QVYp X-Google-Smtp-Source: AGHT+IFYCcSqLW9MJ3dEVyvl+9csTMjFUzR9Ir23/0M/txBT1i5GGeT5KnaQWuDL6pkvqYocl4sFUw== X-Received: by 2002:a05:6a20:748b:b0:196:6b10:f3f4 with SMTP id p11-20020a056a20748b00b001966b10f3f4mr2108113pzd.74.1703860688426; Fri, 29 Dec 2023 06:38:08 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.38.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:38:07 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Guo Ren , Xiao Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Jisheng Zhang , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Conor Dooley , Joel Granados Subject: [v9, 09/10] riscv: vector: use kmem_cache to manage vector context Date: Fri, 29 Dec 2023 14:36:26 +0000 Message-Id: <20231229143627.22898-10-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063810_606995_91BA3A9B X-CRM114-Status: GOOD ( 11.83 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu --- Changelog v6: - new patch since v6 --- arch/riscv/include/asm/vector.h | 4 ++++ arch/riscv/kernel/process.c | 7 ++++++- arch/riscv/kernel/vector.c | 16 +++++++++++++++- 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index d75079520629..7b316050f24f 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -26,6 +26,8 @@ void kernel_vector_begin(void); void kernel_vector_end(void); void get_cpu_vector_context(void); void put_cpu_vector_context(void); +void riscv_v_thread_free(struct task_struct *tsk); +void __init riscv_v_setup_ctx_cache(void); static inline u32 riscv_v_flags(void) { @@ -227,6 +229,8 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) +#define riscv_v_thread_free(tsk) do {} while (0) +#define riscv_v_setup_ctx_cache() do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 36993f408de4..862d59c3872e 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -179,7 +179,7 @@ void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ if (has_vector()) - kfree(tsk->thread.vstate.datap); + riscv_v_thread_free(tsk); } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -228,3 +228,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; } + +void __init arch_task_cache_init(void) +{ + riscv_v_setup_ctx_cache(); +} diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index c1f28bc89ec6..1fe140e34557 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -21,6 +21,7 @@ #include static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); +static struct kmem_cache *riscv_v_user_cachep; unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -47,6 +48,13 @@ int riscv_v_setup_vsize(void) return 0; } +void __init riscv_v_setup_ctx_cache(void) +{ + riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", + riscv_v_vsize, 16, SLAB_PANIC, + 0, riscv_v_vsize, NULL); +} + static bool insn_is_vector(u32 insn_buf) { u32 opcode = insn_buf & __INSN_OPCODE_MASK; @@ -84,7 +92,7 @@ static int riscv_v_thread_zalloc(void) { void *datap; - datap = kzalloc(riscv_v_vsize, GFP_KERNEL); + datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); if (!datap) return -ENOMEM; @@ -94,6 +102,12 @@ static int riscv_v_thread_zalloc(void) return 0; } +void riscv_v_thread_free(struct task_struct *tsk) +{ + if (tsk->thread.vstate.datap) + kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +} + #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) #define VSTATE_CTRL_GET_NEXT(x) (((x) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2) #define VSTATE_CTRL_MAKE_NEXT(x) (((x) << 2) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) From patchwork Fri Dec 29 14:36:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13506547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2F665C46CD3 for ; Fri, 29 Dec 2023 14:38:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bbsbIfz4hSRNhHy2/rstJUnKYPvzEVSAkOrgaytI6gA=; b=pOIrQFaxMAJmmp CS11A1qS0hU/Qelr+74ZTbZv4mrHGLCRDtPayzeLPVwptVg6XjdgWZgoczBkSKMHoGdgLGBq9BNdm 9SDeNtDnUYpo0ZA+8WcVdvSEf1+QOgReHvZ+I4v/C5CRCHzT1Iu2xydzKsh0V2NOLLATJXBzLuTMI tT1nvVcwKoB7fdvzCK6qpVDw5Il7zekBRwn9ALi+w7nKTp4fJlh5MLNCzQXSLKLtRhUwwfPkK9y2K 9m7bdgrduhizuv2eQIApWTKPs4mfzhcJfmLCMpZAzdgtU60aMS9iCR0Q2bHSxu4RV1XgAknwvx2TW NSZqqnNtYI5kf2r8sHmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rJE0L-0013Lv-2f; Fri, 29 Dec 2023 14:38:29 +0000 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rJE0G-0013J3-2t for linux-riscv@lists.infradead.org; Fri, 29 Dec 2023 14:38:28 +0000 Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-6d9344f30caso4615346b3a.1 for ; Fri, 29 Dec 2023 06:38:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703860703; x=1704465503; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oznqbLMDdGHrQ19SUj0nMg0tzTXsYjhJvx/I4YIfPTM=; b=SMjX0+t3lFLAdD0kQA3gwRJGFr+Kl8pTlM1jliePjr8QqLwhFjtrvAZqDTFTVqmxXP 1wLeaM+ntNWHV7vHXdSyCCjhs97akmr+GB3OK3zYfxWhHc38TYitozAyAgTYCi+McXK5 DC4d/PbtZcpxFpJKwr8aryS8tOTYnLEeZqcPRruXq3zudezFap79i8y7sAz4lxySpmOl 1lNBcqcLyA9Zi8TvGhVKaxok6hCYTHLF6kc18vP3KnBFs6BjB5Fw8T/pzXG4jknGBBU6 iYF1a7gWWoeBbHJI5Czb2/Q6O0fmtKh10RzsAHkFWvpRU557ZmcmmX/fLCA9x3WxnZeD lgvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703860703; x=1704465503; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oznqbLMDdGHrQ19SUj0nMg0tzTXsYjhJvx/I4YIfPTM=; b=wA5e1T/mKcHDNXqMaQQ2GAqN3Zh9oKV/rPttryWmAJ0pcpcRRSwlfraX1jD7/DqMK1 KAOMdcb1d1TNneAhJNinL7r78HzOc/iJlLr9K7rJnQ3O+U04LNnoM6hk0AIb/m4Wl9Fv v+tGQWdEsTO5uKCO6MCIQF0A/pf9vzlEdt/D+YZeGg0wWWs9aPgpB/yPDCmWICTObQec FU7El5XM7H9I2mld5sJI1uECMzOC9fmztBvY6lo4N3AI592qnyDwpBRsBdLU6Kzabcpz jDpweD0fksFTLe/5E2TSZq2ff1NpYGWfgcVlxhLqrhApXIAKWW00LZEU5QWo3H5V/mkl 3S6g== X-Gm-Message-State: AOJu0YxobY/eel1XLxgdpz9pER4BeguA0FlWAyXlJg+WKcn2PdNETAOr WeCugar7b1QlD+urAjaJydLBm+ldyVLzq3b7JF+LUaFypNOqScJqHG1Wk7JFJ42ojbkU9lHMOOh p2i1/rCBO5SKikldiIpbMevAn1B1wa72H+Wj6+1fsbcprSrtpyv+lkwhqLsxsuuag7Fju3vd0lZ bJzPKPCvjdJyyaILsZt+7O X-Google-Smtp-Source: AGHT+IF1OmXBvjX4lNCWN/5hIpyaUlp5IRi1OWg1VnMbesG9RyVGSpP9QHtcW/uzAPl0CBVrW9uMOw== X-Received: by 2002:a05:6a00:62c4:b0:6d9:e443:e24d with SMTP id gc4-20020a056a0062c400b006d9e443e24dmr5510182pfb.1.1703860702955; Fri, 29 Dec 2023 06:38:22 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id y16-20020aa793d0000000b006d99c6c0f1fsm11544727pff.100.2023.12.29.06.38.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 06:38:22 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Guo Ren , Han-Kuan Chen , Sami Tolvanen , Deepak Gupta , Vincent Chen , Heiko Stuebner , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , =?utf-8?b?QmrDtnJu?= =?utf-8?b?IFTDtnBlbA==?= , Xiao Wang , Nathan Chancellor , Jisheng Zhang , Conor Dooley , Joel Granados Subject: [v9, 10/10] riscv: vector: allow kernel-mode Vector with preemption Date: Fri, 29 Dec 2023 14:36:27 +0000 Message-Id: <20231229143627.22898-11-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231229143627.22898-1-andy.chiu@sifive.com> References: <20231229143627.22898-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231229_063824_940388_FB37CE1B X-CRM114-Status: GOOD ( 32.96 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu --- Changelog v9: - Separate context depth tracking out to a individual bitmap. - Use bitwise to mask on/off the preempt_v status and drop unused masks - Do not turn off bh on success path of preempt_v (To make preempt_v available for task context that turns off irq). - Remove and test lockdep assertion. Changelog v8: - fix -Wmissing-prototypes for functions with asmlinkage Changelog v6: - re-write patch to handle context nesting for softirqs - drop thread flag and track context instead in riscv_v_flags - refine some asm code and constraint it into C functions - preallocate v context for preempt_v - Return non-zero in riscv_v_start_kernel_context with non-preemptible kernel-mode Vector Changelog v4: - dropped from v4 Changelog v3: - Guard vstate_save with {get,set}_cpu_vector_context - Add comments on preventions of nesting V contexts - remove warnings in context switch when trap's reg is not pressent (Conor) - refactor code (Björn) Changelog v2: - fix build fail when compiling without RISCV_ISA_V (Conor) - 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor) - merge Kconfig patch into this oine (Conor). - 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor) - fix some typos (Conor) - enclose assembly with RISCV_ISA_V_PREEMPTIVE. - change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor) - 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/' --- arch/riscv/Kconfig | 14 +++ arch/riscv/include/asm/asm-prototypes.h | 5 + arch/riscv/include/asm/processor.h | 30 +++++- arch/riscv/include/asm/simd.h | 26 ++++- arch/riscv/include/asm/vector.h | 68 +++++++++++- arch/riscv/kernel/entry.S | 8 ++ arch/riscv/kernel/kernel_mode_vector.c | 137 ++++++++++++++++++++++-- arch/riscv/kernel/process.c | 3 + arch/riscv/kernel/vector.c | 31 ++++-- 9 files changed, 300 insertions(+), 22 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3c5ba05e8a2d..0a03d72706b5 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -533,6 +533,20 @@ config RISCV_ISA_V_UCOPY_THRESHOLD Prefer using vectorized copy_to_user()/copy_from_user() when the workload size exceeds this value. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Usually, in-kernel SIMD routines are run with preemption disabled. + Functions which envoke long running SIMD thus must yield core's + vector unit to prevent blocking other tasks for too long. + + This config allows kernel to run SIMD without explicitly disable + preemption. Enabling this config will result in higher memory + consumption due to the allocation of per-task's kernel Vector context. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index be438932f321..cd627ec289f1 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -30,6 +30,11 @@ void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, const unsigned long *__restrict p4, const unsigned long *__restrict p5); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +asmlinkage void riscv_v_context_nesting_start(struct pt_regs *regs); +asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs); +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + #endif /* CONFIG_RISCV_ISA_V */ #define DECLARE_DO_ERROR_INFO(name) asmlinkage void name(struct pt_regs *regs) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index e76839789067..b503fd34728d 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -81,8 +81,35 @@ struct pt_regs; * activation of this state disables the preemption. On a non-RT kernel, it * also disable bh. Currently only 0 and 1 are valid value for this field. * Other values are reserved for future uses. + * - bits 8: is used for tracking preemptible kernel-mode Vector, when + * RISCV_ISA_V_PREEMPTIVE is enabled. Calling kernel_vector_begin() does not + * disable the preemption if the thread's kernel_vstate.datap is allocated. + * Instead, the kernel set this bit field. Then the trap entry/exit code + * knows if we are entering/exiting the context that owns preempt_v. + * - 0: the task is not using preempt_v + * - 1: the task is actively using preempt_v. But whether does the task own + * the preempt_v context is decided by bits in RISCV_V_CTX_DEPTH_MASK. + * - bit 16-23 are RISCV_V_CTX_DEPTH_MASK, used by context tracking routine + * when preempt_v starts: + * - 0: the task is actively using, and own preempt_v context. + * - non-zero: the task was using preempt_v, but then took a trap within. + * Thus, the task does not own preempt_v. Any use of Vector will have to + * save preempt_v, if dirty, and fallback to non-preemptible kernel-mode + * Vector. + * - bit 30: The in-kernel preempt_v context is saved, and requries to be + * restored when returning to the context that owns the preempt_v. + * - bit 31: The in-kernel preempt_v context is dirty, as signaled by the + * trap entry code. Any context switches out-of current task need to save + * it to the task's in-kernel V context. Also, any traps nesting on-top-of + * preempt_v requesting to use V needs a save. */ -#define RISCV_KERNEL_MODE_V 0x1 +#define RISCV_V_CTX_DEPTH_MASK 0x00ff0000 + +#define RISCV_V_CTX_UNIT_DEPTH 0x00010000 +#define RISCV_KERNEL_MODE_V 0x00000001 +#define RISCV_PREEMPT_V 0x00000100 +#define RISCV_PREEMPT_V_DIRTY 0x80000000 +#define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000 /* CPU-specific state of a task */ struct thread_struct { @@ -96,6 +123,7 @@ struct thread_struct { u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index 4d699e16c9a9..54efbf523d49 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -12,6 +12,7 @@ #include #include #include +#include #include @@ -28,12 +29,27 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. - * - * Kernel-mode Vector temporarily disables bh. So we must not return - * true on irq_disabled(). Otherwise we would fail the lockdep check - * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + if (in_hardirq() || in_nmi()) + return false; + + /* + * Nesting is acheived in preempt_v by spreading the control for + * preemptible and non-preemptible kernel-mode Vector into two fields. + * Always try to match with prempt_v if kernel V-context exists. Then, + * fallback to check non preempt_v if nesting happens, or if the config + * is not set. + */ + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) { + if (!riscv_preempt_v_started(current)) + return true; + } + /* + * Non-preemptible kernel-mode Vector temporarily disables bh. So we + * must not return true on irq_disabled(). Otherwise we would fail the + * lockdep check calling local_bh_enable() + */ + return !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 7b316050f24f..d69844906d51 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -28,10 +28,11 @@ void get_cpu_vector_context(void); void put_cpu_vector_context(void); void riscv_v_thread_free(struct task_struct *tsk); void __init riscv_v_setup_ctx_cache(void); +void riscv_v_thread_alloc(struct task_struct *tsk); static inline u32 riscv_v_flags(void) { - return current->thread.riscv_v_flags; + return READ_ONCE(current->thread.riscv_v_flags); } static __always_inline bool has_vector(void) @@ -200,14 +201,72 @@ static inline void riscv_v_vstate_set_restore(struct task_struct *task, } } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline bool riscv_preempt_v_dirty(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + return !!(val & RISCV_PREEMPT_V_DIRTY); +} + +static inline bool riscv_preempt_v_restore(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + return !!(val & RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_preempt_v_clear_dirty(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags &= ~RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_set_restore(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags |= RISCV_PREEMPT_V_NEED_RESTORE; +} + +static inline bool riscv_preempt_v_started(struct task_struct *task) +{ + return !!(READ_ONCE(task->thread.riscv_v_flags) & RISCV_PREEMPT_V); +} + +static inline bool riscv_preempt_v_need_save(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + /* preempt_v has started and the state is dirty */ + return !!((val & RISCV_PREEMPT_V_DIRTY) && (val & RISCV_PREEMPT_V)); +} +#else /* !CONFIG_RISCV_ISA_V_PREEMPTIVE */ +static inline bool riscv_preempt_v_dirty(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_restore(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_started(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_need_save(struct task_struct *task) { return false; } +#define riscv_preempt_v_clear_dirty(tsk) do {} while (0) +#define riscv_preempt_v_set_restore(tsk) do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(&prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (riscv_preempt_v_need_save(prev)) { + __riscv_v_vstate_save(&prev->thread.kernel_vstate, + prev->thread.kernel_vstate.datap); + riscv_preempt_v_clear_dirty(prev); + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (riscv_preempt_v_started(next)) + riscv_preempt_v_set_restore(next); + else + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); @@ -231,6 +290,7 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vstate_on(regs) do {} while (0) #define riscv_v_thread_free(tsk) do {} while (0) #define riscv_v_setup_ctx_cache() do {} while (0) +#define riscv_v_thread_alloc(tsk) do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 54ca4564a926..9d1a305d5508 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -83,6 +83,10 @@ SYM_CODE_START(handle_exception) /* Load the kernel shadow call stack pointer if coming from userspace */ scs_load_current_if_task_changed s5 +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_start +#endif move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -138,6 +142,10 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_end +#endif REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 241a8f834e1c..22580d36fd08 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -14,10 +14,13 @@ #include #include #include +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +#include +#endif static inline void riscv_v_flags_set(u32 flags) { - current->thread.riscv_v_flags = flags; + WRITE_ONCE(current->thread.riscv_v_flags, flags); } static inline void riscv_v_start(u32 flags) @@ -27,12 +30,14 @@ static inline void riscv_v_start(u32 flags) orig = riscv_v_flags(); BUG_ON((orig & flags) != 0); riscv_v_flags_set(orig | flags); + barrier(); } static inline void riscv_v_stop(u32 flags) { int orig; + barrier(); orig = riscv_v_flags(); BUG_ON((orig & flags) == 0); riscv_v_flags_set(orig & ~flags); @@ -75,6 +80,121 @@ void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline void riscv_preempt_v_set_dirty(void) +{ + current->thread.riscv_v_flags |= RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_reset_flags(void) +{ + current->thread.riscv_v_flags &= ~(RISCV_PREEMPT_V_DIRTY | RISCV_PREEMPT_V_NEED_RESTORE); +} + +static __always_inline volatile u32 *riscv_v_flags_ptr(void) +{ + return ¤t->thread.riscv_v_flags; +} + +static inline void riscv_v_ctx_depth_inc(void) +{ + *riscv_v_flags_ptr() += RISCV_V_CTX_UNIT_DEPTH; + barrier(); +} + +static inline void riscv_v_ctx_depth_dec(void) +{ + barrier(); + *riscv_v_flags_ptr() -= RISCV_V_CTX_UNIT_DEPTH; +} + +static inline u32 riscv_v_ctx_get_depth(void) +{ + return riscv_v_flags() & RISCV_V_CTX_DEPTH_MASK; +} + +static int riscv_v_stop_kernel_context(void) +{ + if (riscv_v_ctx_get_depth() != 0 || !riscv_preempt_v_started(current)) + return 1; + + riscv_v_stop(RISCV_PREEMPT_V); + return 0; +} + +static int riscv_v_start_kernel_context(bool *is_nested) +{ + struct __riscv_v_ext_state *kvstate, *uvstate; + + kvstate = ¤t->thread.kernel_vstate; + if (!kvstate->datap) + return -ENOENT; + + if (riscv_preempt_v_started(current)) { + WARN_ON(riscv_v_ctx_get_depth() == 0); + *is_nested = true; + if (riscv_preempt_v_dirty(current)) { + get_cpu_vector_context(); + __riscv_v_vstate_save(kvstate, kvstate->datap); + riscv_preempt_v_clear_dirty(current); + put_cpu_vector_context(); + } + get_cpu_vector_context(); + riscv_preempt_v_set_restore(current); + return 0; + } + + riscv_v_start(RISCV_PREEMPT_V); + if ((task_pt_regs(current)->status & SR_VS) == SR_VS_DIRTY) { + uvstate = ¤t->thread.vstate; + riscv_preempt_v_set_dirty(); + __riscv_v_vstate_save(uvstate, uvstate->datap); + __riscv_v_vstate_clean(task_pt_regs(current)); + riscv_preempt_v_clear_dirty(current); + } + return 0; +} + +/* low-level V context handling code, called with irq disabled */ +asmlinkage void riscv_v_context_nesting_start(struct pt_regs *regs) +{ + int depth; + + if (!riscv_preempt_v_started(current)) + return; + + depth = riscv_v_ctx_get_depth(); + if (depth == 0 && (regs->status & SR_VS) == SR_VS_DIRTY) + riscv_preempt_v_set_dirty(); + + riscv_v_ctx_depth_inc(); +} + +asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs) +{ + struct __riscv_v_ext_state *vstate = ¤t->thread.kernel_vstate; + u32 depth; + + WARN_ON(!irqs_disabled()); + + if (!riscv_preempt_v_started(current)) + return; + + riscv_v_ctx_depth_dec(); + depth = riscv_v_ctx_get_depth(); + if (depth == 0) { + if (riscv_preempt_v_restore(current)) { + __riscv_v_vstate_restore(vstate, vstate->datap); + __riscv_v_vstate_clean(regs); + } + riscv_preempt_v_reset_flags(); + } +} +#else +#define riscv_v_start_kernel_context(nested) (-ENOENT) +#define riscv_v_stop_kernel_context() (-ENOENT) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context @@ -90,14 +210,20 @@ void put_cpu_vector_context(void) */ void kernel_vector_begin(void) { + bool nested = false; + if (WARN_ON(!has_vector())) return; BUG_ON(!may_use_simd()); - get_cpu_vector_context(); + if (riscv_v_start_kernel_context(&nested)) { + get_cpu_vector_context(); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + } - riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + if (!nested) + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_enable(); } @@ -117,10 +243,9 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_set_restore(current, task_pt_regs(current)); - riscv_v_disable(); - put_cpu_vector_context(); + if (riscv_v_stop_kernel_context()) + put_cpu_vector_context(); } EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 862d59c3872e..92922dbd5b5c 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -188,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; @@ -224,6 +225,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.s[0] = 0; } p->thread.riscv_v_flags = 0; + if (has_vector()) + riscv_v_thread_alloc(p); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 1fe140e34557..f9769703fd39 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -22,6 +22,9 @@ static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); static struct kmem_cache *riscv_v_user_cachep; +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static struct kmem_cache *riscv_v_kernel_cachep; +#endif unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -53,6 +56,11 @@ void __init riscv_v_setup_ctx_cache(void) riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", riscv_v_vsize, 16, SLAB_PANIC, 0, riscv_v_vsize, NULL); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_kernel_cachep = kmem_cache_create("riscv_vector_kctx", + riscv_v_vsize, 16, + SLAB_PANIC, NULL); +#endif } static bool insn_is_vector(u32 insn_buf) @@ -88,24 +96,35 @@ static bool insn_is_vector(u32 insn_buf) return false; } -static int riscv_v_thread_zalloc(void) +static int riscv_v_thread_zalloc(struct kmem_cache *cache, + struct __riscv_v_ext_state *ctx) { void *datap; - datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); + datap = kmem_cache_zalloc(cache, GFP_KERNEL); if (!datap) return -ENOMEM; - current->thread.vstate.datap = datap; - memset(¤t->thread.vstate, 0, offsetof(struct __riscv_v_ext_state, - datap)); + ctx->datap = datap; + memset(ctx, 0, offsetof(struct __riscv_v_ext_state, datap)); return 0; } +void riscv_v_thread_alloc(struct task_struct *tsk) +{ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_thread_zalloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate); +#endif +} + void riscv_v_thread_free(struct task_struct *tsk) { if (tsk->thread.vstate.datap) kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + if (tsk->thread.kernel_vstate.datap) + kmem_cache_free(riscv_v_kernel_cachep, tsk->thread.kernel_vstate.datap); +#endif } #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) @@ -177,7 +196,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) * context where VS has been off. So, try to allocate the user's V * context and resume execution. */ - if (riscv_v_thread_zalloc()) { + if (riscv_v_thread_zalloc(riscv_v_user_cachep, ¤t->thread.vstate)) { force_sig(SIGBUS); return true; }