From patchwork Mon Jan 15 05:59:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46FADC3DA79 for ; Mon, 15 Jan 2024 07:00:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NaoTZzb7gWX7J18C1v8XK+WKQ6olPs1636bIs3+NjHI=; b=bCbZZI3CyxE2zZ yPzcv9XX22ukwleCo6mq04grUcXFbbEgZrwphnHVfh1295JoBLEvaAYBNTM4uTG2mh11Lmh//LrV4 ncSkzwxLV1q1O0lWTuGxgkfIwbflRAMWw/rZ9Vzxu2BER44ilPTc3kL73mnCwTOeCJ2DvEV40aauq DnUvNvewsabsnhvP77ZHHYksn4ZUYV4lj3Rh2ihGdwMtA0JkW6YTrCcfFI4w2w/hvpsQC+hY+47jS ssN+FmiOnNmfEyuh/8w1WuV/S2TRkjsE5vL2duh7fx0BoR/VHV4u2HYSqJdtXMEiohbGqP0kJ0vhU SbaB8rcpjtjGg4Lm4nFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPGxH-0081NH-05; Mon, 15 Jan 2024 07:00:19 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG0q-007qDN-26 for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 05:59:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:Content-Type :MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=jq0h2LaQ8fcssQa+y+ckaiZBaoX909gyQ3cg8U3GLk8=; b=WvZT94RDUnf0ylgslthLXnQwG+ mLDz8BXKlEL6VTEUc5AGSpydRUhiAVnnm6EnqnXptJZCIOTH2EMb+Y99nBVtP4CJ8/SV4zL94STXF isZeXLr9AUcswhWt3/fhWODu64qberOsqc3tCPd/XI4K9VaXOQraOpKi0AtlmUdFXHz+GOMqy1bz0 a+Ki/sZH7qTq2pjaDOLMYUthNtoSFaG0quGItSUvDxLti8Yx1Ew5oBGYfxYf4rPo+OitTxwpN1CDB FX7d4uOQE1qKYLP5S1PxLX2X0/xQDPnJzJaQVn516d8H+0XD5B1Iq4+CoEID0yTI20jJn+euufwLm mfrjzTlQ==; Received: from mail-yb1-xb33.google.com ([2607:f8b0:4864:20::b33]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG0m-00D1Yz-2R for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 05:59:55 +0000 Received: by mail-yb1-xb33.google.com with SMTP id 3f1490d57ef6-dbeda700015so7074579276.2 for ; Sun, 14 Jan 2024 21:59:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298390; x=1705903190; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jq0h2LaQ8fcssQa+y+ckaiZBaoX909gyQ3cg8U3GLk8=; b=LFSzm16PYHa15sqBz+T81WU4F95mu3HyoZvvJqu9bCLfk9Sz3YXDp2Vh1ioLHCzJKN gFUYuE2ebe4nqTtidu8BaA2bJrRzT1QyJlYyfoNB10//W+EvtMLmkQF+bNaFETDKGo/0 HdE+Wc9M2eVmdl3TSe39QiLYew7MSxG7D8zjz2h0jf3WKzjMjyVD1CPH0lnZS+yYkaro j1vkWDBs0r4b1prE5OOiViVTcQiXjVsP/QoCc+YXqxQpjwPNTBAxB3MNbKHaCBwErLix DycA1bKZKlLpTiaVz9Ek4UzxNxWQjvpHTZ/Q1LYBbSKaM10FwutPT2EINZvS4tqbSsRX 6nHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298390; x=1705903190; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jq0h2LaQ8fcssQa+y+ckaiZBaoX909gyQ3cg8U3GLk8=; b=SM++EOyE5TfIMujbBPeMVyA4MO2nwzCYeKUn6JT+idLddQUGmOdLCzHWkSHJG0rsiC hTegvJgOCRMQRTl9FiTBCryDLsnKIw2qs+cFS40v3/C3Vt+jZuGSEQbC487j/+29vTun U6rmM/0yYzF8muHDsCh3b2Mu+6HOkPfxFzUL9AAuJFAZDtIIe94MqOPQXlLyKmxD8Al3 8CdvfvAsRRCcC87zLbI/VlRrDuqAC+RGz5tkuJXyBH/UeZSFPdasRgC5AlCFi4bj9J7n t3fcYcm64cptScnXmeYo0qUa1c4oK0Ea1fzL5h2rYUQFYSDP+N0OBSlvhBrcVHCNmerk Q6TQ== X-Gm-Message-State: AOJu0Ywi6LZNlhlaT8I071QSStFTX8zoVptXU+tJM6/8mIY8+1TpiioP ZOEldUfLv+6r/L2ZqxeQ0PgaI7qZ0BvIZoml1nltV/4IJGwKV74zsOBzdcL+1akEm3x6W0VCeim EliygnVAQWFVDbbAe3zct6xHHAuF1YhCoKW9qD6/DIC5eX51hDye5OJ7aZnwDSAXV4l4djhCZwM woIQUbDy8zA1L4V0/MWTM1 X-Google-Smtp-Source: AGHT+IGYfEwW7xuhy7FqrmA7tn/iqapaLR2CJJlX1WHD4olzgcjOW9mSfOH8LvDojWgeFW/eUCQC/Q== X-Received: by 2002:a25:6b41:0:b0:dbe:9f26:2159 with SMTP id o1-20020a256b41000000b00dbe9f262159mr1924373ybm.48.1705298389863; Sun, 14 Jan 2024 21:59:49 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.21.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 21:59:49 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Vincent Chen , Andy Chiu , Albert Ou , Guo Ren , Baoquan He , =?utf-8?b?Q2w=?= =?utf-8?b?w6ltZW50IEzDqWdlcg==?= , Eric Biggers , Heiko Stuebner , Xiao Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Alexandre Ghiti , Jisheng Zhang , Evan Green , Anup Patel , Sami Tolvanen Subject: [v11, 01/10] riscv: Add support for kernel mode vector Date: Mon, 15 Jan 2024 05:59:20 +0000 Message-Id: <20240115055929.4736-2-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_055953_141611_654841D1 X-CRM114-Status: GOOD ( 24.63 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu Reviewed-by: Eric Biggers --- Changelog v10: - update comment (Eric) Changelog v9: - use bitwise to mask on/off the use of Vector (Eric, Charlie) - BUG_ON when reentrant enablement of Vector happens (Charlie) - Move compiler barrier to the premept_v patch (Eric) Changelog v8: - Refactor unnecessary whitespace change (Eric) Changelog v7: - fix build fail for allmodconfig Changelog v6: - Use 8 bits to track non-preemptible vector context to provide better WARN coverage. Changelog v4: - Use kernel_v_flags and helpers to track vector context. Changelog v3: - Reorder patch 1 to patch 3 to make use of {get,put}_cpu_vector_context later. - Export {get,put}_cpu_vector_context. - Save V context after disabling preemption. (Guo) - Fix a build fail. (Conor) - Remove irqs_disabled() check as it is not needed, fix styling. (Björn) Changelog v2: - 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor) - export may_use_simd to include/asm/simd.h --- arch/riscv/include/asm/processor.h | 12 ++- arch/riscv/include/asm/simd.h | 44 ++++++++++ arch/riscv/include/asm/vector.h | 9 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 116 +++++++++++++++++++++++++ arch/riscv/kernel/process.c | 1 + 6 files changed, 182 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index e1944ff0757a..3b59fc5c9545 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -73,6 +73,15 @@ struct task_struct; struct pt_regs; +/* + * We use a flag to track in-kernel Vector context. Currently the flag has the + * following meaning: + * + * - bit 0: indicates whether the in-kernel Vector context is active. The + * activation of this state disables the preemption. + */ +#define RISCV_KERNEL_MODE_V 0x1 + /* CPU-specific state of a task */ struct thread_struct { /* Callee-saved registers */ @@ -81,7 +90,8 @@ struct thread_struct { unsigned long s[12]; /* s[0]: frame pointer */ struct __riscv_d_ext_state fstate; unsigned long bad_cause; - unsigned long vstate_ctrl; + u32 riscv_v_flags; + u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; }; diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h new file mode 100644 index 000000000000..ef8af413a9fc --- /dev/null +++ b/arch/riscv/include/asm/simd.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2023 SiFive + */ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include +#include +#include +#include +#include + +#include + +#ifdef CONFIG_RISCV_ISA_V +/* + * may_use_simd - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_simd(void) +{ + /* + * RISCV_KERNEL_MODE_V is only set while preemption is disabled, + * and is clear whenever preemption is enabled. + */ + return !in_hardirq() && !in_nmi() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd(void) +{ + return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 87aaef656257..71af3404fda1 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,15 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +void kernel_vector_begin(void); +void kernel_vector_end(void); +void get_cpu_vector_context(void); +void put_cpu_vector_context(void); + +static inline u32 riscv_v_flags(void) +{ + return current->thread.riscv_v_flags; +} static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index a1f5dc145574..5a66432eb520 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -64,6 +64,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..114cf4f0a0eb --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include +#include + +static inline void riscv_v_flags_set(u32 flags) +{ + current->thread.riscv_v_flags = flags; +} + +static inline void riscv_v_start(u32 flags) +{ + int orig; + + orig = riscv_v_flags(); + BUG_ON((orig & flags) != 0); + riscv_v_flags_set(orig | flags); +} + +static inline void riscv_v_stop(u32 flags) +{ + int orig; + + orig = riscv_v_flags(); + BUG_ON((orig & flags) == 0); + riscv_v_flags_set(orig & ~flags); +} + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +void get_cpu_vector_context(void) +{ + preempt_disable(); + + riscv_v_start(RISCV_KERNEL_MODE_V); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +void put_cpu_vector_context(void) +{ + riscv_v_stop(RISCV_KERNEL_MODE_V); + + preempt_enable(); +} + +/* + * kernel_vector_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_simd() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_vector_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_vector_end() is + * called. + */ +void kernel_vector_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + BUG_ON(!may_use_simd()); + + get_cpu_vector_context(); + + riscv_v_vstate_save(current, task_pt_regs(current)); + + riscv_v_enable(); +} +EXPORT_SYMBOL_GPL(kernel_vector_begin); + +/* + * kernel_vector_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_vector_begin() was previously + * called, with no call to kernel_vector_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_vector_begin() is called again in the meantime. + */ +void kernel_vector_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + riscv_v_vstate_restore(current, task_pt_regs(current)); + + riscv_v_disable(); + + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4f21d970a129..4a1275db1146 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -221,6 +221,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + p->thread.riscv_v_flags = 0; p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; From patchwork Mon Jan 15 05:59:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A9E1C47258 for ; Mon, 15 Jan 2024 07:00:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1HO743LQvYKySftn7m74260+utqE81yzv/hcE9W5LI0=; b=vf84dtjZm7uwLU qcF3rMwNldDKBTpx0gk5pJM1qGdHQQfE+mxQjkZMQF5+ZTVHlu9FKbPgEigIuOVG+1Q4C5Yj9aPtp B1LgevwApBJL4tKsntV4zHJd1+c+8qCBGTofq8JS0mIInGIzRUb7FmMwHltOyaboXawYc5YQTh6mE qpgY3HpELnA8mQCzZDvMKkFNj8R+ouyRhAgNy/5F4R7XgZl6bu6jhnrOtjb9z5XGAOLDyIJt+JOms CjeiFNuZzvl6ricqV4I/qqsiymAyUw/ImU/55BvubqYuB9M7xnde9wzb7C0lJNsxUg33BO33t9AC2 Kih6oYRaiMmM+dvGnIvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPGxH-0081NQ-1v; Mon, 15 Jan 2024 07:00:19 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG0z-007qGJ-0J for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=References:In-Reply-To:Message-Id:Date :Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=PTE5s3JevRaI0ONcRbV59vdIOJW1xXVURdjLv83lPaQ=; b=EQqwwESHr/iQ5MdcQOT0eEVLoE R2a2PJH6ASppMXlCs+O+y0j3VjSmG+m9iPFK8UC5N6A5YGshohKpjAK9QN4ZKTDYEf9bts0vERz7H HNPEERSVRSLG0a6BqbqF58Acp52a2geSPj/kR39E/g6KIJ8nWmsYobmMr/bFeVJTrwM6NHiaO/JrL jFwNmuZmiJrJJTPqh5L4h4eNcivC1HBgKL/umod3N9lHDnPGS+hKnaBomc+bxlDdopJDnz4oV2my3 DVB9LAb/Nl5PC+gPdc8cHItXMgWYcH3UZXGmKwgzAnRtliehZwLFDwVYISYWXTWz2VyYhYzAdvfwA F4RXcxDQ==; Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG0v-00D1Zs-0y for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:03 +0000 Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-5ce10b5ee01so5943742a12.1 for ; Sun, 14 Jan 2024 22:00:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298398; x=1705903198; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=PTE5s3JevRaI0ONcRbV59vdIOJW1xXVURdjLv83lPaQ=; b=m4TEWCTcEx391iFkauYeQ5rOuMTn3+IYDo4Z8hdxLbXyPtN6EIHkbxqfrNY9W9eydt 9/RnTXUF7HcOxjv9DKBcupmL1uVjR2juqBY3mBX22ZHTNtPAjQ6ESFYEdtoHV/BhBrxT SqXA7VtaSqle7DMdFhMKURhNa2Jz1b5c+/LpuNKQ3RgV2pK77m2gEltUHPP7yTf5hU+C G5WMr09EOov4GH35izfDbBgeal2thSbhSMV+YqTCsUffGoGgHkG1NIxFQxHqXpjJQ4qu 0s6pMLIS+FQsR1TU3PDDfOdXC9E/lwKrfABL4DscvBVThfHJUgfkw6HjprGNbe6JDDM/ JIPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298398; x=1705903198; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PTE5s3JevRaI0ONcRbV59vdIOJW1xXVURdjLv83lPaQ=; b=dWA6qjMmKPPQoavWfMkEMvZjdgFcTlBeC/y1cQswglavavCeGTOOvwN6Zaggn6A/1i +eIIou8ItLHLIY4I1hGjLa/9XuZ9AdHagXUcD1fQ4clniVwkzrU24SOvyKz2wdVOWXxm 5H36vTjZQeRQ3DgSEpbN+AFBOlMN/c7TdZr0bNUQyD9c/tco5s6TmH7v09kt5kVQwC25 L1d/huyjVX70NecXnUsnx70+BHmnlJwpQOdm6VK//C3TPHy+8xD8W6ldKqJ/sigffW6o HBxFweBxOLfaKjyoPdulrQMn4JzDSnJWRi06pyTS3RgnXYrR8eRjASnpZjfgAybajgTC 9tdA== X-Gm-Message-State: AOJu0Yw6f7NoepwN5Ee/Ia9mqeHeISwNw3sIvw8/YqjqGaB+xNxjCa5i VbRp0nLiVWM4l6CjWCxwt7jM8OO61g+Tc+QOTIVr7fwvKg6EE8WGAOQ+5TRjXQzbE8jOHz9tjC3 O2fSCfiTajx3031cx6YSYRCMCIYDkCAiMWlh3z2ln2bR3iWu1lMd18nCKDV4aeQ9/oQK54XT6vs +825qs7/g0kPicCeRi+5Sh X-Google-Smtp-Source: AGHT+IHBKxKSQEOhuPOiNKtgQJEchZSFIbGxPlE0dO0a1I8QijfgmZd4mZgMH/JMr0wHrp2g2UqPBQ== X-Received: by 2002:a05:6a20:918c:b0:19a:e201:3b1d with SMTP id v12-20020a056a20918c00b0019ae2013b1dmr2450560pzd.7.1705298397974; Sun, 14 Jan 2024 21:59:57 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.21.59.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 21:59:57 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Vincent Chen , Guo Ren , Heiko Stuebner , Baoquan He , Conor Dooley , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers Subject: [v11, 02/10] riscv: vector: make Vector always available for softirq context Date: Mon, 15 Jan 2024 05:59:21 +0000 Message-Id: <20240115055929.4736-3-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060001_847717_178AB8AA X-CRM114-Status: GOOD ( 16.18 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The goal of this patch is to provide full support of Vector in kernel softirq context. So that some of the crypto alogrithms won't need scalar fallbacks. By disabling bottom halves in active kernel-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. So, softirq context is able to use Vector whenever it runs. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu Reviewed-by: Eric Biggers --- Changelog v8: - refine comments, fix typos (Eric) Changelog v4: - new patch since v4 --- arch/riscv/include/asm/processor.h | 3 ++- arch/riscv/include/asm/simd.h | 6 +++++- arch/riscv/kernel/kernel_mode_vector.c | 14 ++++++++++++-- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 3b59fc5c9545..27ceedf357bf 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -78,7 +78,8 @@ struct pt_regs; * following meaning: * * - bit 0: indicates whether the in-kernel Vector context is active. The - * activation of this state disables the preemption. + * activation of this state disables the preemption. On a non-RT kernel, it + * also disable bh. */ #define RISCV_KERNEL_MODE_V 0x1 diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index ef8af413a9fc..4d699e16c9a9 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -28,8 +28,12 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. + * + * Kernel-mode Vector temporarily disables bh. So we must not return + * true on irq_disabled(). Otherwise we would fail the lockdep check + * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 114cf4f0a0eb..2fc145edae3d 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -46,7 +46,14 @@ static inline void riscv_v_stop(u32 flags) */ void get_cpu_vector_context(void) { - preempt_disable(); + /* + * disable softirqs so it is impossible for softirqs to nest + * get_cpu_vector_context() when kernel is actively using Vector. + */ + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_disable(); + else + preempt_disable(); riscv_v_start(RISCV_KERNEL_MODE_V); } @@ -62,7 +69,10 @@ void put_cpu_vector_context(void) { riscv_v_stop(RISCV_KERNEL_MODE_V); - preempt_enable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_enable(); + else + preempt_enable(); } /* From patchwork Mon Jan 15 05:59:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 537D1C3DA79 for ; Mon, 15 Jan 2024 07:15:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xaq2TRk3wXbQZSmYt3X3wTn1lW/bK0rQJZnzB7U/D/M=; b=Zhfg6HKWKAyo9o 4RtfXjdPGiin0bPjoNOzPH7ZoYorpMJmtl4ooFXntqn4Mwh3vOtg3VvxE+02ajlmGupNsevStXoDe a07OrHLIyr2QNqgskbwl44UJs6a7WDk81Vzx0rsrdIUSMQRCvNSScO7L55ioH94gYzjGs142CIVUz 2dc/ZQ53wCgNmED7q2tPhtrQZytqaPd43q8kxpNhDdLebUuFGb5MzF9GIz2wMY+i68yBt+9oVWFH+ 0KVt6PgnyRRwG3BN1fuY4owyg6ngqviXFRvDKqfY14fAWlAcT7b8M/hFhexpVL0dEOFxl086guw3J FkG02X00Qi3Abqy3d7Ww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPHBl-0083Cm-0p; Mon, 15 Jan 2024 07:15:17 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1F-007qLS-31 for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:22 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=References:In-Reply-To:Message-Id:Date :Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=NFnYzFltuLxaA5kymfUv4YQXh/w1s0UKs0KJQeLv+3U=; b=izfqPcsuYMr1yRDiqXHtUA2MbX Q7uvcxOI1kChL+YIJTteZVyICft21g9fmJrqv/fu4mWUxpqrjc+AKjmzzc/rGSbwatRncAFF4MghY 9U6I0qEZ8oH0tF1MPJsaCtyef7YN+kB5k1jQtIyFtXMqGuEsZ15j+7RxGb5Ho0OlDDHqiTo93WMg7 /Cn+qDNSlhRN/LNVkm51FgkkvgEBqGUyqBSjcAZsHgCsvCrtY1RNdXClO5vWjiKnzZ10Kdm6tUTNk 96zyqxSe4OmP3eTv7YH0uxkcat+6E5kjDTsGNGJlfUpWkr6jaqtngHojU2tsXP4jc+7WOcjcK4b2S VCJOy1Qw==; Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG14-00D1bH-35 for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:13 +0000 Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6db81c6287dso406378b3a.0 for ; Sun, 14 Jan 2024 22:00:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298408; x=1705903208; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=NFnYzFltuLxaA5kymfUv4YQXh/w1s0UKs0KJQeLv+3U=; b=gUXd1u8dVggzZ1myg9QA8VxGZAnl83eQdeSGI2STyIS36ZFsP0AmL4LdQTf/whlmht o43T9ULFLCRU1GmJMy0ckAQlX4Qqa/lkrwrg/qdFhZ2nkQttuv/DwIhYgHRGbeavMHw5 kCcw0cekRTGLHJHfUdJwA4Wk1RTpYHtKI6C6sbi84zDSlUHe+c4Qx7VYrgCT268OdJv8 Y57Rw5Ftftb0zgy9dAWYSzTD1Ff/qpyIQyTpFj3wGQ0oVRLLsqbJQ7ceJTIdA77izw2t kVSiqf00epk+9h5EgDKCLJCYRHNQ4SsWqS8xTCFWbhT2fB9dcMN+q94h1DGscF2jgE3s A7ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298408; x=1705903208; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NFnYzFltuLxaA5kymfUv4YQXh/w1s0UKs0KJQeLv+3U=; b=UQQdBrUIUssfOHJprxzFGesvSs0eoF27ovCcwgAt7S6JSi/2iWtiV+Wr8Q4wQd0yQz cMjvNGKrAJpTmvIUykITTmcfY41QPn5NXYp/+bnIHriPh06OTNJQm46SmhgUE6NvghAq 5+XJIF3f0tFItC+/e0vBS/HapDlLhzWkhznb3o070I3l4adgJaLJmImpecyxFMnmSOdz XDpNsNNIbaG22Ci0mphK4MfhqAiRLxQry59PWHBZauDt5a1Os47ZUPzkPFJS0ldMS0oR s+WOGLlSWpVv0n62G776vLyXcH5agdw/1XrA8GqomSjtPAL2sYvivPcUPgbeLhzTGXoN Lmvg== X-Gm-Message-State: AOJu0Yy14T/nKhHrbY+8d3vaPykH0uJWcYpVGlpQlpBjUXxqyLhxvl2i VstaxDBYDRKgbHWuBx1REcWyKY3spHjhrPsv6GN9dTWL3xYMIptO8eYvf4lrz7Gp9tFLNMJ8r4g +4bMkXY0CaVtpn9Gopx4Bc7LQPL5jQDP3ZO9L98HuE8Thnz9cgZ5d/+TeuXY0q/MnOKLg8aal6Q v1K/mY3UU1JiIN0bibx9x8 X-Google-Smtp-Source: AGHT+IFQ9o4Beuw84DJQv0CcxnQM9UIRHHO0VCbgTgB+5nccklvOx/eNcKBm9DdqVJnj1VDg1MkJIA== X-Received: by 2002:a05:6a20:a8af:b0:19a:260b:ec26 with SMTP id ca47-20020a056a20a8af00b0019a260bec26mr5467308pzb.61.1705298407715; Sun, 14 Jan 2024 22:00:07 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:06 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Han-Kuan Chen , Andy Chiu , Albert Ou , Guo Ren , Jerry Shih , Sami Tolvanen , Deepak Gupta , Conor Dooley , Nick Knight , Andrew Jones , Heiko Stuebner Subject: [v11, 03/10] riscv: Add vector extension XOR implementation Date: Mon, 15 Jan 2024 05:59:22 +0000 Message-Id: <20240115055929.4736-4-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060011_192289_4D1A63AB X-CRM114-Status: GOOD ( 16.17 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- Changelog v8: - wrap xor function prototypes with CONFIG_RISCV_ISA_V Changelog v7: - fix build warning message and use proper entry/exit macro for assembly. Drop Conor's A-b Changelog v2: - 's/rvv/vector/' (Conor) --- arch/riscv/include/asm/asm-prototypes.h | 18 ++++++ arch/riscv/include/asm/xor.h | 68 +++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++ 4 files changed, 168 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 36b955c762ba..6db1a9bbff4c 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -9,6 +9,24 @@ long long __lshrti3(long long a, int b); long long __ashrti3(long long a, int b); long long __ashlti3(long long a, int b); +#ifdef CONFIG_RISCV_ISA_V + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +#endif /* CONFIG_RISCV_ISA_V */ #define DECLARE_DO_ERROR_INFO(name) asmlinkage void name(struct pt_regs *regs) diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..96011861e46b --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include +#include + +static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_vector_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_vector_end(); +} + +static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_vector_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_vector_end(); +} + +static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_vector_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_vector_end(); +} + +static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_vector_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_vector_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..b28f2430e52f --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +SYM_FUNC_START(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +SYM_FUNC_END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +SYM_FUNC_START(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +SYM_FUNC_END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +SYM_FUNC_START(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +SYM_FUNC_END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +SYM_FUNC_START(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +SYM_FUNC_END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Mon Jan 15 05:59:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E4F5C3DA79 for ; Mon, 15 Jan 2024 06:00:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fwCTTNVqZoHhbFl+FR/cOtEmFCMnQd1GVRoEULW+d0c=; b=sOH5K8SLBzaag3 pGZcV9o1uQwIU/JtppTjpCwj466HEGn6hIxhgiGQASECJtDao6pC3n65OStBEMdP9VTqonPg7g97A NwXjiV25x/Oltzy73AU6KFSkhGASyycWWW2XO0vjQEb0TBGKCJBYfCjs7fa22hPZPldz5Y82GHKhz mNtkUnoqz9jReFIW2uOGdebIRFO2+yVc6r8JYVilVwgeinJtBLMb3iUWpy2h3/mMBFfIq+S9u/nKl SvvssHD1cidGy42302hiZdf2wNE45yGbKR3A+yp5W1IvwBd22OeatqSjWTbLZ36Ic62v3YaBIwJj8 XMkGWHDYzHc0c9GG5R6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1T-007qPl-2A; Mon, 15 Jan 2024 06:00:35 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1S-007qPD-0y for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:Content-Type :MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=pxVEmIaJ7fQfkbY5hQr4DL8oB+uGh8ELXfBRuaEAgY8=; b=qv8hLefNKMgUOdJKMEH/uhZNdf FwJ0SRLszL7q7gZDSzd5pdPjVTPGRuqDZYXyc9/kHLXA6TUhGE6qr6CyYEcQqa4vptkUBQJqJ3VZp AikTmc02nQqpyn1TotXN9TBjFcfEy/tpZyBHbLW/Myrd478O1Mkegp4QzRrTjm/0FXyvDe30oU5zh OxQ+oPJZGMRxuN9gzuATHWPqGthb/44u0tjki1aIjbooMGuXJWX+39xLAqni0/xKfV0EI7DPjK2sB qzbw9xGTWZVtsPCu9LysOw4O3oLsM/p9rttjqSBRgtQXhbcbGiux+y/23Hdfx098ETWcXzcn9LMeB 9L1IWgVA==; Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1O-00D1dk-1b for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:32 +0000 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-5ce10b5ee01so5944030a12.1 for ; Sun, 14 Jan 2024 22:00:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298421; x=1705903221; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pxVEmIaJ7fQfkbY5hQr4DL8oB+uGh8ELXfBRuaEAgY8=; b=YLCXnN/AOyZ2H9kRvZe1pMEeiU61TKhJlN+FIJ6DkPVDyWqI4By0rjJO3sgplSo2pP BbB7+3gOr8YzG5QpqjyXKlDfVun8TBrdWIMpbCj4VIu7dRLavIu60FOsHQCN9ro40JV4 /kQ8/2GN466Vv9WuxIYa53lAtvb1xvVSGH2TCCny34OnoUGvBrcQwPcniQFCEZaYduQ3 +JnK7bZ5ZuWakhY75c1oohko0PRsswCqoS4VB3rZX7x84FT6L0NDC8O06RTfHkdDPIhR sY5jZocIj97OIC8hx9UXl/AFjnoFbEfPxAklLGKTSAW5Ug4RKMxfus+6IhXyyYzG4vvr KzmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298421; x=1705903221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pxVEmIaJ7fQfkbY5hQr4DL8oB+uGh8ELXfBRuaEAgY8=; b=FJrDcBBUbP5+ubC2Z2aW1kAdnvG6TrDKbsUW+PdX0Ft1MyzYXSJLBSZtYMvr857ZN6 jUQLkTz/81jJO+EToIXbQuvxPFTkssqJHwepK8NlJ8H+Br4xTEchko1Ld1t5EDd2UkIf 08B/bmtCqQSw4f01KaRMXqnErPq+yV5JWw7JBBYaoX1mWfOJ6bbBwIA+uyqWXOa/78ia Gm7rHjGcoSAKE5uAJSvzgdqZ02b9CxoBcEMjEUlJjkIumNoxGUsSNy8Fg6B5IeLqW79q BsDECm0PM2VGuSI0Fa3HIGbxwpdT6Jtvg0ZqE6n/ouoMT86xPHcyYSM9o2t9s2DuZ2l4 rBXA== X-Gm-Message-State: AOJu0YxpbL24+7uYBEUY3EfmxgZ4AQJZD9QLGRGoSRsNTO5XUIm1vR2e JRr20uTivKLd5WotpLHoHn+5WFmmu9c0pSnqInxDR2HaegVvBcG/BxUkz7kTlaHt4k714S0Ifq6 ruKtRrQ8QxvvCxuYT+sKeiJkh4i4lEAY4hRpcEfXowGbXPsST5+LunOshpdZW7VobZCv9iCg1x+ ILXJ7s3GhaDcbFvX5n5GHd X-Google-Smtp-Source: AGHT+IHJiFjSbkCDIJRyWEaEGbnzngccZ+BZd0MMKeJNL4tZOCCj7z1RneTq5lokcfb7iBDiqNbRiw== X-Received: by 2002:a05:6a20:9f95:b0:19a:fad2:51d6 with SMTP id mm21-20020a056a209f9500b0019afad251d6mr1407006pzb.14.1705298420488; Sun, 14 Jan 2024 22:00:20 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:19 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Oleg Nesterov , Conor Dooley , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Guo Ren , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Sami Tolvanen , Jisheng Zhang , Deepak Gupta , Vincent Chen , Heiko Stuebner , Xiao Wang , Eric Biggers , Mathis Salmen , Haorong Lu , Joel Granados Subject: [v11, 04/10] riscv: sched: defer restoring Vector context for user Date: Mon, 15 Jan 2024 05:59:23 +0000 Message-Id: <20240115055929.4736-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060030_813374_4E3CAC86 X-CRM114-Status: GOOD ( 21.28 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changlog v9: - update comment (Song) Changelog v4: - fix typos and re-add Conor's A-b. Changelog v3: - Guard {get,put}_cpu_vector_context between vstate_* operation and explain it in the commit msg. - Drop R-b from Björn and A-b from Conor. Changelog v2: - rename and add comment for the new thread flag (Conor) --- arch/riscv/include/asm/entry-common.h | 17 +++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/ptrace.c | 5 ++++- arch/riscv/kernel/signal.c | 5 ++++- arch/riscv/kernel/vector.c | 2 +- 8 files changed, 41 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 7ab5e34318c8..19023c430a9b 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,23 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + /* + * We are already called with irq disabled, so go without + * keeping track of riscv_v_flags. + */ + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 4856697c5f25..5d473343634b 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -102,12 +102,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 71af3404fda1..961c4e3d1b62 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -193,6 +193,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -200,7 +209,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 2fc145edae3d..8422c881f452 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -117,7 +117,7 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_restore(current, task_pt_regs(current)); + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_disable(); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4a1275db1146..36993f408de4 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -171,6 +171,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -187,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 2afe460de16a..7b93bcbdf9fa 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,8 +99,11 @@ static int riscv_vr_get(struct task_struct *target, * Ensure the vector registers have been saved to the memory before * copying them to membuf. */ - if (target == current) + if (target == current) { + get_cpu_vector_context(); riscv_v_vstate_save(current, task_pt_regs(current)); + put_cpu_vector_context(); + } ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl; diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 33dfb5078301..f571af3d5f78 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,10 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); + get_cpu_vector_context(); riscv_v_vstate_save(current, regs); + put_cpu_vector_context(); + /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); @@ -134,7 +137,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 578b6292487e..66e8c6ab09d2 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Mon Jan 15 05:59:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8FFD1C47DA2 for ; Mon, 15 Jan 2024 07:15:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lLeVK/4uFHP8gtCAbJiPXzwAo93z9UnWw6Bcs0rNUO0=; b=w8Q/RdQXxYYxYI kQZq2pYLDwkIFtIYGqKOsxByw3FIOzqEd4NcAGfi+J+0C4kgZc5U0JcCapKYb+/4HzQN2MoQShxON XDVhfRXsUjyqmXPipzduPb/9KCcNDlXY6f5GIT957UgCyItrMtHGcrutr08NZBcuQyS08MuA803/O UJBM52JU7aWpAg6TYkEh/DEAG/VKEfRkkX4RvHC4WInyGiA3VE07X+dPkTqXw+9CpAEDacQP2P4SR 1mnKZxNSDJgUVCfeQGMHDX0Mdyqc5SJee858BQhsp84RRgXnzgtS93QnTXRVIFFL4tTxZ4vAQC8Re LbifD1O0oEMUGopliZ1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPHBl-0083D0-2a; Mon, 15 Jan 2024 07:15:17 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1W-007qQb-2K for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=References:In-Reply-To:Message-Id:Date :Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=me9oOiGn5mHkoxh/0N14oRpKAMxeZVEEihh11oDsIlM=; b=EJ30F0B/2YvgwExu2n5xhHdnCb xE0lhiHV7NqbzNzQGaMfzocvvCXQKNfRxybBBlORuu6x6PYoQTs+Kuaj5sQW1iyt/xdfgs1X7ZymA EI2h5PEC1HW/OxlhbKf0LvCAyFMGrqVJknJSK2Ru6no2xgij8I9cJvG9l/+Ihf5p41mD+OFK0bfP0 KS9R2ISRMDuOnC/ZeAVg+lDIZa6Gl99xZ7SoXGEKXl1iCJ92oLBkQ9p1VZhWs+DFQWOSqVSYLpcLt 6reQDdUNN7HFp5VyvA1edNPFpjo81IJBWEdBqJNyi1ts/jZguEc8e1gq5c+XD2DOb2nOY7Dy2XnPz yqAxZBmw==; Received: from mail-il1-x133.google.com ([2607:f8b0:4864:20::133]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1S-00D1eA-1W for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:37 +0000 Received: by mail-il1-x133.google.com with SMTP id e9e14a558f8ab-3606f3f2f37so47248485ab.0 for ; Sun, 14 Jan 2024 22:00:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298431; x=1705903231; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=me9oOiGn5mHkoxh/0N14oRpKAMxeZVEEihh11oDsIlM=; b=b7hlXfogac3PszZH5PdV/nNyYeStR5ocE+cGs+LhDOXrVeLJn60Tcols/+TaladE1E tXoqXO9m+yauDu3xbpePJBvFhD4jwpAUJFYqv8iTH6/f9Wz8+ngGgcCn/U+t7BsZUmWw hyI5P/K92bM54Ah5u34DPkKGbEKFvPk6/oIwKycz5r8sctBDmZxt7rCIrhqWHAHrHQ/Q ZoSUSIII+O6IUhO1/l0X87066L6xgjE6n42A6Mj0Xfd3NXpsddPIcsS1bRVgHHWhseyc 9Zvvn8HXiDW/GR5RExHSFLe7LPrLYnjU96icuUhhXUN7ZKg/R8GoEgPUor1se3KJL49l KNTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298431; x=1705903231; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=me9oOiGn5mHkoxh/0N14oRpKAMxeZVEEihh11oDsIlM=; b=NLB0aiPj3vaQAI0tJVLPcxqJNYkrNe7UXkMCvFYxpmj+OjQp9n653evgjAxP2uLuAl kNUdxC+o1LdHK+vjKq40YBb51d5OwPUP4I09jjwOEY/ncKHU6RCapEoYPtHueUH6G+Bb gZGiK2m2Pegufengsya6thtg5LCKUx1lAq4j5lF3WFus4zE8pFIe0FYKFjshLWU2ojeR iZaAMxxGCFAh3wVLPvriwTXU1EWotoyQlS39J73B0JKrpGQhtw6T5uWB/XLYCTN86+7S Uuqn1OgLpBiNWAhy2p1qGLxOijkNVpEvXlDipkdoJ3+Pq/UQCk/z49Tbcp5ndz3zW5yZ Vm8Q== X-Gm-Message-State: AOJu0Yx4fDZxcbCV6d9TFF4NSerWMT47AC9eg1rpn8R/rREYUVt0nbiy I8JkpnBYtlPiE3jrk6+YoEd/obkas82T832L9iRSO5qjuNX81/e+uP4HeO0J4E21MxqhSPxmNjz yZQAyboQ5kofi4OtdsNRUZFEnqJPIJOZZnGBbKnHKLmaZZNKI246l91KxsC/2dgyXm0+rAui+rJ tyPS/DkkpiY4DyIMQeTbxw X-Google-Smtp-Source: AGHT+IEooaOiNLnMKLRRHGCaWDCOnCAgW3Sh4OwaDZPSOlfMFBnwP+KJRS/zmnhxhB49oEBPXDxTIQ== X-Received: by 2002:a92:c108:0:b0:360:5d99:65b5 with SMTP id p8-20020a92c108000000b003605d9965b5mr4443684ile.21.1705298431173; Sun, 14 Jan 2024 22:00:31 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:30 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Jerry Shih , Nick Knight , Albert Ou , Guo Ren , Sami Tolvanen , Han-Kuan Chen , Deepak Gupta , Andrew Jones , Conor Dooley , Heiko Stuebner , Aurelien Jarno , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Al Viro , Alexandre Ghiti Subject: [v11, 05/10] riscv: lib: vectorize copy_to_user/copy_from_user Date: Mon, 15 Jan 2024 05:59:24 +0000 Message-Id: <20240115055929.4736-6-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060034_819741_1D43DBA4 X-CRM114-Status: GOOD ( 25.56 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch utilizes Vector to perform copy_to_user/copy_from_user. If Vector is available and the size of copy is large enough for Vector to perform better than scalar, then direct the kernel to do Vector copies for userspace. Though the best programming practice for users is to reduce the copy, this provides a faster variant when copies are inevitable. The optimal size for using Vector, copy_to_user_thres, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it. The exception fixup code of the __asm_vector_usercopy must fallback to the scalar one because accessing user pages might fault, and must be sleepable. Current kernel-mode Vector does not allow tasks to be preemptible, so we must disactivate Vector and perform a scalar fallback in such case. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Co-developed-by: Jerry Shih Signed-off-by: Jerry Shih Co-developed-by: Nick Knight Signed-off-by: Nick Knight Suggested-by: Guo Ren Signed-off-by: Andy Chiu --- Changelog v11: - pass the proper size when falling back to scalar. - Honor the original implementation and authors. - Skip bytes which have been processed by the vector store when falling back to scalar (Guo) Changelog v10: - remove duplicated code (Charlie) Changelog v8: - fix no-mmu build Changelog v6: - Add a kconfig entry to configure threshold values (Charlie) - Refine assembly code (Charlie) Changelog v4: - new patch since v4 --- arch/riscv/Kconfig | 8 ++++ arch/riscv/include/asm/asm-prototypes.h | 4 ++ arch/riscv/lib/Makefile | 6 ++- arch/riscv/lib/riscv_v_helpers.c | 45 +++++++++++++++++++++ arch/riscv/lib/uaccess.S | 10 +++++ arch/riscv/lib/uaccess_vector.S | 53 +++++++++++++++++++++++++ 6 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/lib/riscv_v_helpers.c create mode 100644 arch/riscv/lib/uaccess_vector.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d42155c29a55..ff48dc2d0dcc 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -527,6 +527,14 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_UCOPY_THRESHOLD + int "Threshold size for vectorized user copies" + depends on RISCV_ISA_V + default 768 + help + Prefer using vectorized copy_to_user()/copy_from_user() when the + workload size exceeds this value. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 6db1a9bbff4c..be438932f321 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -11,6 +11,10 @@ long long __ashlti3(long long a, int b); #ifdef CONFIG_RISCV_ISA_V +#ifdef CONFIG_MMU +asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n); +#endif /* CONFIG_MMU */ + void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, const unsigned long *__restrict p2); void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 494f9cd1a00c..c8a6787d5827 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -6,9 +6,13 @@ lib-y += memmove.o lib-y += strcmp.o lib-y += strlen.o lib-y += strncmp.o -lib-$(CONFIG_MMU) += uaccess.o +ifeq ($(CONFIG_MMU), y) +lib-y += uaccess.o +lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o +endif lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o +lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o diff --git a/arch/riscv/lib/riscv_v_helpers.c b/arch/riscv/lib/riscv_v_helpers.c new file mode 100644 index 000000000000..be38a93cedae --- /dev/null +++ b/arch/riscv/lib/riscv_v_helpers.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2023 SiFive + * Author: Andy Chiu + */ +#include +#include + +#include +#include + +#ifdef CONFIG_MMU +#include +#endif + +#ifdef CONFIG_MMU +size_t riscv_v_usercopy_threshold = CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD; +int __asm_vector_usercopy(void *dst, void *src, size_t n); +int fallback_scalar_usercopy(void *dst, void *src, size_t n); +asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n) +{ + size_t remain, copied; + + /* skip has_vector() check because it has been done by the asm */ + if (!may_use_simd()) + goto fallback; + + kernel_vector_begin(); + remain = __asm_vector_usercopy(dst, src, n); + kernel_vector_end(); + + if (remain) { + copied = n - remain; + dst += copied; + src += copied; + n = remain; + goto fallback; + } + + return remain; + +fallback: + return fallback_scalar_usercopy(dst, src, n); +} +#endif diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index a9d356d6c03c..bc22c078aba8 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -3,6 +3,8 @@ #include #include #include +#include +#include .macro fixup op reg addr lbl 100: @@ -11,6 +13,13 @@ .endm SYM_FUNC_START(__asm_copy_to_user) +#ifdef CONFIG_RISCV_ISA_V + ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_v, CONFIG_RISCV_ISA_V) + REG_L t0, riscv_v_usercopy_threshold + bltu a2, t0, fallback_scalar_usercopy + tail enter_vector_usercopy +#endif +SYM_FUNC_START(fallback_scalar_usercopy) /* Enable access to user memory */ li t6, SR_SUM @@ -181,6 +190,7 @@ SYM_FUNC_START(__asm_copy_to_user) sub a0, t5, a0 ret SYM_FUNC_END(__asm_copy_to_user) +SYM_FUNC_END(fallback_scalar_usercopy) EXPORT_SYMBOL(__asm_copy_to_user) SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user) EXPORT_SYMBOL(__asm_copy_from_user) diff --git a/arch/riscv/lib/uaccess_vector.S b/arch/riscv/lib/uaccess_vector.S new file mode 100644 index 000000000000..51ab5588e9ff --- /dev/null +++ b/arch/riscv/lib/uaccess_vector.S @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include +#include +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + + .macro fixup op reg addr lbl +100: + \op \reg, \addr + _asm_extable 100b, \lbl + .endm + +SYM_FUNC_START(__asm_vector_usercopy) + /* Enable access to user memory */ + li t6, SR_SUM + csrs CSR_STATUS, t6 + +loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + fixup vle8.v vData, (pSrc), 10f + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + fixup vse8.v vData, (pDst), 11f + add pDst, pDst, iVL + bnez iNum, loop + + /* Exception fixup for vector load is shared with normal exit */ +10: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + mv a0, iNum + ret + + /* Exception fixup code for vector store. */ +11: + /* Undo the subtraction after vle8.v */ + add iNum, iNum, iVL + /* Make sure the scalar fallback skip already processed bytes */ + csrr t2, CSR_VSTART + sub iNum, iNum, t2 + j 10b +SYM_FUNC_END(__asm_vector_usercopy) From patchwork Mon Jan 15 05:59:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB02AC47258 for ; Mon, 15 Jan 2024 07:15:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Hxvhow+Jpr+i0B+a/HAAJPGW5U+waNcv65wrqDFbFLA=; b=iRmvCcnvBaY3rY WyIRTKdaMhN4WEGBlSy7R2H9pLBVGriCx/ChfRuIObPUpS9QOr182eyK2ZRqgVom0gv2dqEPhIV+f VM3e4I0pF6JLUA/erOukqV2DXHvf9FWvxpRWNeDfL+kA+OBmAfBCZ1+A/Rw/cxFQSbF6nSXtQGpjc zLri5DH1wYSdsJJOBYClerwHafXY3nrMEAVZl/8tbfhlsakoBRaqjJyVjoaKTWBRr9qmGQhh47qD3 kvn5VDdUFofhxqD34KDTn96KQ7IS8d9NAF0gMJgDbvHIMVrkGbXGihoVZJHwlnFXVfiWYN/qZeBVh KSf1GLU31M/OuQa/7RAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPHBm-0083DG-1O; Mon, 15 Jan 2024 07:15:18 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1a-007qRw-0s for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=References:In-Reply-To:Message-Id:Date :Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=t3xo3x8zv1BqYEabf/miBjogWe62kYZ8Sg/4Io2Mwxk=; b=JkhqTwT26I3sx9SNRa6ZARch33 NGOnm8QY9vQUdm5nHRctgVp1PuFHiERYjb/kLc2QH0YjvVSX+IK2EWBHGBw/xweO0pSKsPCowF8sp DbcMluT8ysPnO4/GMuwAOBNj9nPkiZaYoYKncOfOzRXMgTv2xoivcoEgk/marwejPdGLvhVonnn5T FHona5Y3JwHfrRxoqcjV4IcapTc4kBGF8HRv23suzXkshtgqqMeLrh/H6B41HIg4wlSlbogsVQd03 Idy5zj4yJBP63um5qCR02QiDQUHYnmP4FlO+5IayyjDp4GJPrpwYcBzvmGXPgnKEWh6+VadRR/eKX Dkdrbm5A==; Received: from mail-oi1-x232.google.com ([2607:f8b0:4864:20::232]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1X-00D1et-0F for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:41 +0000 Received: by mail-oi1-x232.google.com with SMTP id 5614622812f47-3bbc649c275so3810920b6e.0 for ; Sun, 14 Jan 2024 22:00:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298436; x=1705903236; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=t3xo3x8zv1BqYEabf/miBjogWe62kYZ8Sg/4Io2Mwxk=; b=k8JhiupWJFIc/P5pxZ0h1ix/c5VlQ60mfkgv4mrlZfxS9MUkyVFlGYFxeArNJpQrED fvu+Teg/3IDHrnthekUAuaea6BsZfkkI/Jl2lxw33fMK+QXGUI6rcK+igdozoazvyh6u rJNlYIhbl30cjHfS4mv7p7LelGb5yn+gchYKqOoxrtQ/fNtBRyqZeKFUxfa7nEVLDYEm BNOTJIb2+/u1sPno6czu/8lJeXBMIrsouUv7trKtaNx9QGVx2Z2Mge7xRbZM/DZt2+6c me5CHuVfyAUTu9Ld4AtEQrBmkFS161k5OwdyLwIGqXenmtt6UiVxi62QSCMePyppOWN/ 4pHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298436; x=1705903236; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t3xo3x8zv1BqYEabf/miBjogWe62kYZ8Sg/4Io2Mwxk=; b=c3k8D9F7ykSlR3xvW9HOIoHoUa/qclnNtv1WJAV4UwADwVxFOZs6Tax7xVNb0U2tTk 8fFTyKUWuXDFe8OE7ztFdDjjv//J3RHP9TBSPBKcHTV6KopyMTfyOL+fLtivTgGU6zqo 4jzPbZTGdcsHbbAJswxhQBA2d5t8o327ty/h4OauvQrW8hy+//iS5zskDTqZNusotaBo danX9WRWbzQaqVdDgl61y/jWmlwu3n90vNhAA2aTqi+nmvXt+XMiMrYS6Tcah9OI3gIX nYqH8vYgO39wD2JfdztD8EI9BKZujnK1B+paquvmEE/LGLaFV8A6DXNDkz2GU81uDuOg aSuA== X-Gm-Message-State: AOJu0YyExEwf21yz3RjM+VZEEamDPGRyMyEP/8WomADk84I+hyOjXqXb DZoCUwC4EAz2ZnkY1/loBkDf6Y57O9WmMn1QMK9mBXg3j5UBfR/q5EekmHFdE1xtrcBce8bPgoM t84udrtQ0NoNvGG6EsUXdqKjAfV1TFKy05KlViRd8qczO2NM+Ib9CC1/q2M12IJUyY2pG0dhBXr llIBDLEhmQ8F0U11S6t1EW X-Google-Smtp-Source: AGHT+IH0sco5M9ip8tZThrl+ybvw4fe2WWfUwPeeCXU3QsjF179tE8wZ9uLa6lVOBmWKjvRFtd2ghQ== X-Received: by 2002:a05:6808:107:b0:3bd:40af:a93e with SMTP id b7-20020a056808010700b003bd40afa93emr2348371oie.119.1705298435994; Sun, 14 Jan 2024 22:00:35 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:35 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Guo Ren , Heiko Stuebner , Conor Dooley , Andrew Jones , Xiao Wang , Jisheng Zhang Subject: [v11, 06/10] riscv: fpu: drop SR_SD bit checking Date: Mon, 15 Jan 2024 05:59:25 +0000 Message-Id: <20240115055929.4736-7-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060039_257049_778B773F X-CRM114-Status: UNSURE ( 8.53 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org SR_SD summarizes the dirty status of FS/VS/XS. However, the current code structure does not fully utilize it because each extension specific code is divided into an individual segment. So remove the SR_SD check for now. Signed-off-by: Andy Chiu Reviewed-by: Song Shuai Reviewed-by: Guo Ren --- arch/riscv/include/asm/switch_to.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h index f90d8e42f3c7..7efdb0584d47 100644 --- a/arch/riscv/include/asm/switch_to.h +++ b/arch/riscv/include/asm/switch_to.h @@ -53,8 +53,7 @@ static inline void __switch_to_fpu(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - if (unlikely(regs->status & SR_SD)) - fstate_save(prev, regs); + fstate_save(prev, regs); fstate_restore(next, task_pt_regs(next)); } From patchwork Mon Jan 15 05:59:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EEE05C47422 for ; Mon, 15 Jan 2024 07:15:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yjVBovBgCiUfRujKpg3GrA4aI5sgmGXvVv/0hdNpiCc=; b=Jt+bqo6cW4GRtD tbrhiDkUPFQx0t/yknOnWM0sVRN5LeWHgpzkDNy4URSMaFqlBLRRL9hbidsPTkIIVt9bMFhsqvDGw 8VF0Tupd0DdTTPgEpBp80wAdIqrgv/XexA48aTVRu4UQw7TCl3PJdWP3QrZb7BGgEmcNHdiqigReK KRUgzjSCJXYU+ZWy+Sy4Dkyt9bykt0blbrZ3xlR57m/hli596RplsU07Hng07yrr7zJscaLwAkDpo 4LPhEo6iUSQqMtQnlnp8xMRUeg4onSK8o2XqRrjGbaHom8CNUQZ6JHDdvKygzpCXfM3hjj6RRAYnq 9vw8HOMeoZ0l7HRSIWLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPHBm-0083DU-3C; Mon, 15 Jan 2024 07:15:18 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1l-007qV7-0R for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=References:In-Reply-To:Message-Id:Date :Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ZQMUwmhftjVkzfkK+iZc7sWHSuUCRYxtugVVE3q+kKk=; b=rfX3svGe5bE8NuFDgz/I7IgZqM 65XY2tz3zRhECO1mlUio/wCsNOjcCR+sCtLZj7TmNu5MSb6GXaQleDLCVNcqCdmO+29FE0l0tU/a6 5EYox+0ggwqFcFl8XAi99wzCHWQr1FxdKbfxF4VJPH6xtmNWk1T/KqKSkqN7z4vvi/8PooeW9/I2X kY7wwlW3JDSMfFHX28xaEoT1HJmmBtzZbKtV+KA0L9QJviU9igPjBJU8HPD1JrZli9CnjoW+jh9bb ETrmFtxdsWzeodW6yA0jXCnsBxQQNSy5MuiHXexARipn0WGGmGiMZa7zwZPgKXxQshqoJAs0Uzzzk D5/D8jrA==; Received: from mail-oa1-x33.google.com ([2001:4860:4864:20::33]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1h-00D1fb-0q for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:51 +0000 Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-2041e117abaso4296355fac.0 for ; Sun, 14 Jan 2024 22:00:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298445; x=1705903245; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=ZQMUwmhftjVkzfkK+iZc7sWHSuUCRYxtugVVE3q+kKk=; b=ntg5d4xKW1Rba5a6E5dxMHBbKbg7mXxVMXJHyI256XkJyGoSpPsWnGCk64RX6ZifW1 1RmRRovfGlg4KtudwwlmVJLSd5oTaMBRXteO2opcgcpvVeN+m/fEs0DthxHxQw3jRxLd L8nQiwQo6LQIe4IlB5gI/2RCbI+CjTP6KBzsv/0VrdO1KHqeDOMNGZQP+mVLn9ubbpvp tLx5nXlYamuOV9zMjYOE3iKA/StzIWIhlr0Vtv0M7PCtWmpDrhbtOGy2bAeOgo/rbt6G fKDXiyoYTwVb8Ljcijo/PJEcJ5Wc7KIgHsyzCNo66Frswr6KcRlBX1fCgGQiYOMqkWg9 FU0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298445; x=1705903245; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZQMUwmhftjVkzfkK+iZc7sWHSuUCRYxtugVVE3q+kKk=; b=NDL5pHNU4EOxIeOeEd37T8lU/nL2yO8f3CUZ3yHnP9vi/f+3t8yEmEZ/oC2/yOf65V MHGqshdW7E2Ht41ijIxg28kaA9KkBxu0ImO1W/eDCaTR/FUmNcXAyM+Wd4ePocnvd38f nW6YZkR+iCWWEbgtaQr7Gpvo7at5CNKuOS/fr+XWB2yy7TqVSuCD7F0/wILvomMU4ApV 2bFTuHxOMfYyG28XraT6kBPMiUZW0DzDYqOaBRiJ2lp3JNZQASJbYPSpus2N4NlK8iLr JFYJ7n3OMLbqeAii+Se6kbVmSHRZWpk6hARzWZTwS6le/uxuRXO2ScUzNoMNXzzHzSWp 4Z2g== X-Gm-Message-State: AOJu0YxkhfvDkxcuP7+hAeUA+7m5p/Ph6tkIO/A0g15xtRekG0ouj0Ru PBBlw5TIFxwLVTijIuNTSlGjD/qUbEckFzgWda9uAu9fxVlpEurs27vbgOWysKFX9Vs3xmSXjZb eOzRSp7z+8Wa2qyeTzL1Jdku1bY7jL/ghImGNOaBF8DqpekFiG6+F+bCezvcy10HEzW/IMDSETv 5RkrPT8xKBwYyZXZInTg0t X-Google-Smtp-Source: AGHT+IHAwJflWRf548X+GlADLq2u18XIDNi5KZhxi4eX006z24otABvdv1ZQjEaiDgunP4v3sr60xQ== X-Received: by 2002:a05:6870:d8cb:b0:206:9edf:486e with SMTP id of11-20020a056870d8cb00b002069edf486emr4737710oac.42.1705298445449; Sun, 14 Jan 2024 22:00:45 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:44 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Oleg Nesterov , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Vincent Chen , Heiko Stuebner , Xiao Wang , Eric Biggers , Haorong Lu Subject: [v11, 07/10] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Mon, 15 Jan 2024 05:59:26 +0000 Message-Id: <20240115055929.4736-8-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060049_720468_E1A87C13 X-CRM114-Status: GOOD ( 11.85 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v6: - re-added for v6 Changelog v3: - save V context after get_cpu_vector_context Changelog v2: - fix build fail that get caught on this patch (Conor) --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 19023c430a9b..2293e535f865 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -16,7 +16,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, * We are already called with irq disabled, so go without * keeping track of riscv_v_flags. */ - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 961c4e3d1b62..d75079520629 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -171,23 +171,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -208,7 +204,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(&prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -226,8 +222,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 8422c881f452..241a8f834e1c 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -97,7 +97,7 @@ void kernel_vector_begin(void) get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); riscv_v_enable(); } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 7b93bcbdf9fa..e8515aa9d80b 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -101,7 +101,7 @@ static int riscv_vr_get(struct task_struct *target, */ if (target == current) { get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); put_cpu_vector_context(); } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index f571af3d5f78..501e66debf69 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -87,7 +87,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); get_cpu_vector_context(); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); put_cpu_vector_context(); /* Copy everything of vstate but datap. */ From patchwork Mon Jan 15 05:59:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ADEFBC3DA79 for ; Mon, 15 Jan 2024 06:01:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DBoi3TludzXQKB4y5mbYXIWz1RaUEj0UxOrD0WXy9GU=; b=wx1nNzo/tga2qs LRuw5Zs+zVRm8cAMQv33jSRqIV/N3sp6dOKw+njmz6yBuE3xNYFIa8NiPvL25CWFd49++XwereDQf xPD/o3YCmF+ibwcjdyxUgtkvAQvfMhOBX6hDvn7ljbi9WoIeRMylCPL3mg3HW4Rpf+diHQ7Km0yBP ReZt8uilJRgrm3suz/aW5OHFFLqsFltRamLt0kUMLea1OMvB/Oe5KMI30OS3UdezR9h8XdQIGQX5f GCC3SyLAs92ChCpjUNja19bRkSnUmP20Be2PB1H3tdBRDoUL/U8yw0cYvJmJuGjo9qkOBfWhnIl+P O3Hy0d2sxVKbIm7gdX5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1p-007qWn-0N; Mon, 15 Jan 2024 06:00:57 +0000 Received: from mail-oo1-f47.google.com ([209.85.161.47]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1l-007qUr-0t for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:54 +0000 Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-595d24ad466so4739663eaf.0 for ; Sun, 14 Jan 2024 22:00:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298450; x=1705903250; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=bXF+toU2MoS0KKkyJuhWpreG+X00SoASTsHaDWfKLbF/PmBED4/4RtNGBtr7XlQdxa grN90aFvJhc15ushD9XYdKeVJhuZh+vIme9gpBWgBIRGC4/B/cfkR5qPSRwwxfNhEOHY 1Ib+txuef8WOw8mJri8NG/IyYolfwkzfxKn2PJKNkBVb3RhdsBhjV0AA6fpH662KVLRY l7bZUr9GHN3oAZv2AQYRoNqsO4t2wFvgAJvWUI44IZyy9FHTNZ9gQCF4/j5RHOJKQVZx oK8xV48D5URD9aQB4YbTBaay8glIpkpFuO3xDqJMgIFymAJj0w8b5IclBioxcS05N3tH 4ekw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298450; x=1705903250; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=wVkQIqpBGg5qb5ooAJrYs5JOdkUV7J1ru+fJmHLyKRrL1YmSTOUWhv9RzUm1znxvHF EjGU19p3aZCmRuLPJXtM+6jd9C7KMcLe4YkPBy9aQq0h0TanbdPqia2oYD5GkFzvSFkW 7jHDT5fRZhMy1xnMQS6tXRlwZkdlFx6dQ3AEku3FFsvtI4Qfqmf0NbO2mjMtoNYkUY8+ viGYXJLi9eWjcn1lMZ8w6Lh+/AF+srldXhv7A077bU7Hc3MRhy80KXOSFIO/pnXWQUBj 4wYN1LRiIOKoLEsCSkZLrZzOo8rwUdEN/y48aPgd3vayAPL2D+Bor88hBW7GhpoOK3tx Yd8A== X-Gm-Message-State: AOJu0YwVtMLVTDJgTtm6qIU2up/eFCVHKcZhWYy0qhZ+lrHhuXN986qu aTT3GtvDhx7TBCLZ2sL9SD00AX2JBOUGlbUGOqdmn13ZVhHBLStOd3zGy9q1B7IKtK1nLE1aKtR kJHto4EMy420jSrmJGmMU+tTOhRAPvIvfh5IBLrro8/ZvpuKKzRc6zxP3xBzkWWoLdLvIF6p82r bVM622CoQV0fKKt70rEIZm X-Google-Smtp-Source: AGHT+IEn7xU6IC+M/5r6sSIWu4YSO95H2j5r3MOp4KcjXEWtIIZuD7gRtbox/7/r1kJguGHn1J/Qdw== X-Received: by 2002:a05:6359:3516:b0:172:ea55:1ae2 with SMTP id un22-20020a056359351600b00172ea551ae2mr3693788rwb.37.1705298450327; Sun, 14 Jan 2024 22:00:50 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:49 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Vincent Chen , Conor Dooley , Joel Granados Subject: [v11, 08/10] riscv: vector: use a mask to write vstate_ctrl Date: Mon, 15 Jan 2024 05:59:27 +0000 Message-Id: <20240115055929.4736-9-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240114_220053_315420_8A9D102A X-CRM114-Status: UNSURE ( 8.72 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_ctrl_set() should only touch bits within PR_RISCV_V_VSTATE_CTRL_MASK. So, use the mask when we really set task's vstate_ctrl. Signed-off-by: Andy Chiu --- Changelog v6: - splitted out from v3 --- arch/riscv/kernel/vector.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 66e8c6ab09d2..c1f28bc89ec6 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void) From patchwork Mon Jan 15 05:59:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3A3FC47DA6 for ; Mon, 15 Jan 2024 07:15:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8Vs9LroMK+DKRWFwJgakEJ6k5WFx7T8j+5ZANt3hQ5A=; b=akWW5opX5nQ59M V1erqzFJiTWl1VFY8pQRyjm2iGHWTM4CyGwmZyEDB+KHfggMbDEPCEfXNUdZtY/W9scKtRA/eQSGk 0KUibyQrMMwfvKrBudjgVn9Kwj5qEsyJLlPEny1W3xlH6lLwoQe/8K7heBdzZhShQOyk1ROyDCRWT m6ysdVxhZamcjJxhQZUknAa2WtnAn7zNgkoULf9CQ+cCoBF2VdIsP8rv99YCSezb6lqiHO2YMPLak fo2AtkdpnWsNFLLPpmW3edUBLUmVt3gk5pxSFWquuNxIfXmgn/HjMeHkNyLoEABWvemL7lfQ/JLP0 /ifnK1B7iME66vBHrYuw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPHBn-0083Dj-1q; Mon, 15 Jan 2024 07:15:19 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1x-007qZT-0E for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:01:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=References:In-Reply-To:Message-Id:Date :Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=8MWrqoU0Ka6AL1F0Cvls3jUzvKOCNug5V0O39ZvxzlU=; b=Npa3FI3cLg/X3oCC+9OjSP0wWg pLrufgN6pSZk1TNmvRfI4r0ru5CEv2Jf3LPzfTSQKzuDeF9n8FuiICpjadcx4d0+r1AoLMv1rsvo7 Gw5l0znnLBrfcELZTnubBVM3xgREY1esExxkeegcDdpwVjYFfdFCQf9I6bdFBSQ45kTBjaxxCmgjz aT9lCaYaZRZZn7SRKycL2yJnOhf45ZQ4PvcV3LZ8oDUrWXoUgya4reR1eu0KYUWTuTZsJkzObI8nE W+1pcsKhyDRjeUUoAdrK76j57c0KQ2JfgAGnZwjaIyhCQmIYxLaK+So6p/9qrafbpX+7ZhzpB+f20 5KY6ep3Q==; Received: from mail-il1-x135.google.com ([2607:f8b0:4864:20::135]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1s-00D1gj-2O for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:01:03 +0000 Received: by mail-il1-x135.google.com with SMTP id e9e14a558f8ab-3606e2120bdso62166915ab.0 for ; Sun, 14 Jan 2024 22:01:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298458; x=1705903258; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=8MWrqoU0Ka6AL1F0Cvls3jUzvKOCNug5V0O39ZvxzlU=; b=fBRf5d13F2Mr6RhEDLji1DOFCx5qzpNDZXPPlPiR1TgSjLzLJAQHSc3KOkKhCsDuLh I9anUBuJO8/CPqfEesbgYWs7rcSup+QNpeCOokQQkNV/3nxFobOKkeI/IXeBIS+6+UkN ziSHQdngdhMcvO/QqYEJB1DI0ubDGJ/H+0WGsPfsxK5fTiDzBfQ2NQHwzaiqeMUyxavM uvAMm7wJXStEFIgnHjYYsJogHWwuT+PKNyITFq4pRzmu0StcHBK61z9m6DZhm5nO+Ulu iJlr66yD40U1uuvBnmq0efqiIV4fQgERuAWnmUea5vATEHHhYG6ftaGMCvxD+RuODS/l Lc0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298458; x=1705903258; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8MWrqoU0Ka6AL1F0Cvls3jUzvKOCNug5V0O39ZvxzlU=; b=UdSB9NwNm4t9nm6VIk6/H70FMWMI+Ny6YOm2zuLX53ue+4L1vkR2n6yB6yXMbMqc2M 90rLUAoZURNsuBWL76ttkNzh460+4FjY7EfBpd+2RVoEbD6Y0IjOqL7hJw4S9Y55+VQM JAr34TsyNcMBw3tInJVwn9BNrU9ZWLUoYEpTScFTebXh9wqcfLp4PFnoCb+82pbFLilE BTe5Gm/JD0HFeGw+L9JAjMm1sbX71Q4VlqeQ1lLLiDTTW5wOOSLq+P0rW58Svqp/xxnA Dj+JDFC+ZAzjLYIXCLCZZAlNoA3UU0Ez1m78E9JFcyeqbhAEOzLFUDvPG9kJUlWvIaCM EsQA== X-Gm-Message-State: AOJu0YxX2wqMREebKi0rA6oZ18nD1hSRYOS217GUnRWa7G5mxL2pvG79 qEERWjrkAXnkAQByoaqO+3cl3VzIQroGau7MeuBiftsrybKQcUd1CgygotZOJKCPPmjFcKv3yiO Sh2jh0rXujDDpCmnWIcibUQCLoo2z2wbLtj3GjG63JXY8nEFzfZCSWx78iBIWJEXpfeU+PrK/kW nSblxyJXJrtNyovTJM043c X-Google-Smtp-Source: AGHT+IHQ8FPxlMY544qtu8IIfM32jd7dTIw0cjVoBM9ni+Q8yDW6sKzRDMemN7xi/5OAql9srQy77w== X-Received: by 2002:a05:6e02:1a47:b0:35f:eec4:ed55 with SMTP id u7-20020a056e021a4700b0035feec4ed55mr7754210ilv.62.1705298457894; Sun, 14 Jan 2024 22:00:57 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:57 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Xiao Wang , Jisheng Zhang , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Conor Dooley , Joel Granados Subject: [v11, 09/10] riscv: vector: use kmem_cache to manage vector context Date: Mon, 15 Jan 2024 05:59:28 +0000 Message-Id: <20240115055929.4736-10-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060101_228780_0F3FB829 X-CRM114-Status: GOOD ( 12.30 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu --- Changelog v11: - proceed allocation only if has_vector() returns true. Changelog v6: - new patch since v6 --- arch/riscv/include/asm/vector.h | 4 ++++ arch/riscv/kernel/process.c | 7 ++++++- arch/riscv/kernel/vector.c | 19 ++++++++++++++++++- 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index d75079520629..7b316050f24f 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -26,6 +26,8 @@ void kernel_vector_begin(void); void kernel_vector_end(void); void get_cpu_vector_context(void); void put_cpu_vector_context(void); +void riscv_v_thread_free(struct task_struct *tsk); +void __init riscv_v_setup_ctx_cache(void); static inline u32 riscv_v_flags(void) { @@ -227,6 +229,8 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) +#define riscv_v_thread_free(tsk) do {} while (0) +#define riscv_v_setup_ctx_cache() do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 36993f408de4..862d59c3872e 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -179,7 +179,7 @@ void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ if (has_vector()) - kfree(tsk->thread.vstate.datap); + riscv_v_thread_free(tsk); } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -228,3 +228,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; } + +void __init arch_task_cache_init(void) +{ + riscv_v_setup_ctx_cache(); +} diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index c1f28bc89ec6..f7b4aeb9e457 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -21,6 +21,7 @@ #include static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); +static struct kmem_cache *riscv_v_user_cachep; unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -47,6 +48,16 @@ int riscv_v_setup_vsize(void) return 0; } +void __init riscv_v_setup_ctx_cache(void) +{ + if (!has_vector()) + return; + + riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", + riscv_v_vsize, 16, SLAB_PANIC, + 0, riscv_v_vsize, NULL); +} + static bool insn_is_vector(u32 insn_buf) { u32 opcode = insn_buf & __INSN_OPCODE_MASK; @@ -84,7 +95,7 @@ static int riscv_v_thread_zalloc(void) { void *datap; - datap = kzalloc(riscv_v_vsize, GFP_KERNEL); + datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); if (!datap) return -ENOMEM; @@ -94,6 +105,12 @@ static int riscv_v_thread_zalloc(void) return 0; } +void riscv_v_thread_free(struct task_struct *tsk) +{ + if (tsk->thread.vstate.datap) + kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +} + #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) #define VSTATE_CTRL_GET_NEXT(x) (((x) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2) #define VSTATE_CTRL_MAKE_NEXT(x) (((x) << 2) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) From patchwork Mon Jan 15 05:59:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B768EC3DA79 for ; Mon, 15 Jan 2024 07:15:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IBE49zdtc5hszi8asIojfxqpT/zlD0uEqSFnwsq/6sg=; b=Q040j4zxWromyH 0dFPjqbe2dC8KL3W3mNSnmWzPjYSskdgCNZUEL8aOUoEJ6alSmVVfr7L+oBsIwdOyVz6SCJaYkoIy sE0zgBRXCU6zwLl/+rvAeqh4HkSxlLnhCQAHZhkunvdaG65btR3AIaTxYOVvYxW9C6MsPxKFP9VhM E8Eca0YPVhV9kImfT+MAkB8+ym6lBBzUX/pDt0LuaPBox7x2Doe7T/z4KQbAZXbgPokIK9umgismc WEBtAudsVrkr5P7UMCubmXXuaMYdxcjWVIe7d+338meW3mFEfmvhAd4cQk+NBcpnOdYHs9Rk6q4I9 WAM9Q6s/F7l6JiEsqtLg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPHBo-0083EH-0c; Mon, 15 Jan 2024 07:15:20 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG2G-007qga-1t for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:01:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:Content-Type :MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=AdrBqQOViooPbpmudVlzxcoNos/a4N4Ink9f9rqlykQ=; b=N3OkFxypebLsWgofSM3x90ocKG MBVYTD+FoE88hTqZZ5eYytqiLel0QhsbfN5S2itheb96f3lXShQ58R3vtKehGYLnTk1IiS+DxGpds CPYvXTSDhSfaLrccVO+j+65OA7blUIIveD5ISyIlzI7W8Pc1e/jFsikOov0A1Gjs8pNVPwQH3ruP2 ukwD9EhsHYPtWdAg3ZsqFSG7RncFNox3zQ+TUcEAvspIPRif71yqLB+aQOPURjeOAWEp9ErdGXEAh 9hws+DM76VXxqXqPznj8FVwQbKh7bu3MOZmXRizktuD1/48jSdn9eSujo6CJ//10W2UCIEHd4y+QZ s2j/sK+A==; Received: from mail-oo1-xc30.google.com ([2607:f8b0:4864:20::c30]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG29-00D1ii-0l for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:01:23 +0000 Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-5957ede4deaso5032523eaf.1 for ; Sun, 14 Jan 2024 22:01:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298472; x=1705903272; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AdrBqQOViooPbpmudVlzxcoNos/a4N4Ink9f9rqlykQ=; b=gYSzMyPQ1TPK2kB4usUFjBHfQfjlyGkOmY0FB+Kct9TRvMLjIwVyNsyxqph6q2a46a DIPWbSsJ4dxI71F2IiExVh3LdGT+0hW3B1K53AtsCvAxOdbRwA2dXWqUFHfCJTUqYJ23 SKmay0HoAkHCeBv0ZIifEj/nO23+n3dEBH3IDuh5QiHggzHeqAF0RXGisvv86cMCasZn pFQENMT7ApiznQgkJJhCKF4eMZvXBIgiKJBFOi0IvR7ASVdzJJV4AfFLAlibqbAIZyw/ XgYuUfODkRGaGfKiVtAPUwkDcuMkUNyOjMxdPfjbU6CcokdR+mf1c9TEJheEHATe2nZn NpGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298472; x=1705903272; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AdrBqQOViooPbpmudVlzxcoNos/a4N4Ink9f9rqlykQ=; b=IjMKP+a+0i+f4/P/zBC9PxyRGYV6P6KfbvPHq5cyEAI3/rZONzIYKJgShgdgiYarri ImbI2t1abG961QkIEJ9WZQikz4T4ySsJLqep44yxmNm7cHZQMhT4moo3WaJmOTAkGg5n HrkkLuNKu9datXQTICxHRxN8o742BaX/lzso+01g9yiFX2biyfiH+uQzEbZnIUaFvl7w hHqVKhkr2jwYYRzEWsToTPHLsBMqu7UFBUgVjw8PZJt1cz6jygBsf08VLBUKoG2FfoVV qqpsEd+D7SGUAr7BXChdZJ17crZgYZ7KetfmizTTYai79Xa9qVYbZ622T0w3UFwZerba bTUw== X-Gm-Message-State: AOJu0YwhI+//DmdrMsL3qQVpUrgfDz+32A92Ks7QGBOj4diOSm7YaB6b zMPMT/YV2rMqXUNlsbddz0o3C0Ihl9Bzsw/whY9QWPqN1wTSzBWw+AFvdWvO/LFKOzzI+AoUNs3 L6poYCrQdxvzAnn/K1bK0GoQqyDlV2vclbLQVe37V6TglteHsdT2jXhVRYE1WePFjachxQoeRAo 1LnjnCP42ru8uQICjipqXq X-Google-Smtp-Source: AGHT+IE51pNM/9H8SK/VE83zKARl4Gn98HA3ZVH/cuj2Q0Il5Wod7KUH5U97ZdDJvSZNBzfr63GyXw== X-Received: by 2002:a05:6358:7054:b0:175:7a4d:30f3 with SMTP id 20-20020a056358705400b001757a4d30f3mr2692519rwp.22.1705298471716; Sun, 14 Jan 2024 22:01:11 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.01.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:01:10 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Guo Ren , Nick Knight , Sami Tolvanen , Deepak Gupta , Vincent Chen , Heiko Stuebner , Conor Dooley , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Xiao Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Nathan Chancellor , Jisheng Zhang , Joel Granados Subject: [v11, 10/10] riscv: vector: allow kernel-mode Vector with preemption Date: Mon, 15 Jan 2024 05:59:29 +0000 Message-Id: <20240115055929.4736-11-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060119_498984_3549B1D2 X-CRM114-Status: GOOD ( 32.56 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu --- Changelog v10: - Use one get_* instead of get/put/get. (Xiao) - Dont save user's V as long as preempt_v has started during context switch. - Optimize unnecessary compiler barriers. - Clear dirty bit when stopping preempt_v context. (Xiao) - Only clear both dirty & restore flag when NEED_RESTORE is flagged. - Fix preempt_v user context save in _start_kernel_context(). Changelog v9: - Separate context depth tracking out to a individual bitmap. - Use bitwise to mask on/off the preempt_v status and drop unused masks - Do not turn off bh on success path of preempt_v (To make preempt_v available for task context that turns off irq). - Remove and test lockdep assertion. Changelog v8: - fix -Wmissing-prototypes for functions with asmlinkage Changelog v6: - re-write patch to handle context nesting for softirqs - drop thread flag and track context instead in riscv_v_flags - refine some asm code and constraint it into C functions - preallocate v context for preempt_v - Return non-zero in riscv_v_start_kernel_context with non-preemptible kernel-mode Vector Changelog v4: - dropped from v4 Changelog v3: - Guard vstate_save with {get,set}_cpu_vector_context - Add comments on preventions of nesting V contexts - remove warnings in context switch when trap's reg is not pressent (Conor) - refactor code (Björn) Changelog v2: - fix build fail when compiling without RISCV_ISA_V (Conor) - 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor) - merge Kconfig patch into this oine (Conor). - 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor) - fix some typos (Conor) - enclose assembly with RISCV_ISA_V_PREEMPTIVE. - change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor) - 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/' --- arch/riscv/Kconfig | 14 +++ arch/riscv/include/asm/asm-prototypes.h | 5 + arch/riscv/include/asm/processor.h | 30 +++++- arch/riscv/include/asm/simd.h | 26 ++++- arch/riscv/include/asm/vector.h | 58 ++++++++++- arch/riscv/kernel/entry.S | 8 ++ arch/riscv/kernel/kernel_mode_vector.c | 133 ++++++++++++++++++++++-- arch/riscv/kernel/process.c | 3 + arch/riscv/kernel/vector.c | 31 ++++-- 9 files changed, 286 insertions(+), 22 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index ff48dc2d0dcc..b834d2daf95e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -535,6 +535,20 @@ config RISCV_ISA_V_UCOPY_THRESHOLD Prefer using vectorized copy_to_user()/copy_from_user() when the workload size exceeds this value. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Usually, in-kernel SIMD routines are run with preemption disabled. + Functions which envoke long running SIMD thus must yield core's + vector unit to prevent blocking other tasks for too long. + + This config allows kernel to run SIMD without explicitly disable + preemption. Enabling this config will result in higher memory + consumption due to the allocation of per-task's kernel Vector context. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index be438932f321..cd627ec289f1 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -30,6 +30,11 @@ void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, const unsigned long *__restrict p4, const unsigned long *__restrict p5); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +asmlinkage void riscv_v_context_nesting_start(struct pt_regs *regs); +asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs); +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + #endif /* CONFIG_RISCV_ISA_V */ #define DECLARE_DO_ERROR_INFO(name) asmlinkage void name(struct pt_regs *regs) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 27ceedf357bf..a8509cc31ab2 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -80,8 +80,35 @@ struct pt_regs; * - bit 0: indicates whether the in-kernel Vector context is active. The * activation of this state disables the preemption. On a non-RT kernel, it * also disable bh. + * - bits 8: is used for tracking preemptible kernel-mode Vector, when + * RISCV_ISA_V_PREEMPTIVE is enabled. Calling kernel_vector_begin() does not + * disable the preemption if the thread's kernel_vstate.datap is allocated. + * Instead, the kernel set this bit field. Then the trap entry/exit code + * knows if we are entering/exiting the context that owns preempt_v. + * - 0: the task is not using preempt_v + * - 1: the task is actively using preempt_v. But whether does the task own + * the preempt_v context is decided by bits in RISCV_V_CTX_DEPTH_MASK. + * - bit 16-23 are RISCV_V_CTX_DEPTH_MASK, used by context tracking routine + * when preempt_v starts: + * - 0: the task is actively using, and own preempt_v context. + * - non-zero: the task was using preempt_v, but then took a trap within. + * Thus, the task does not own preempt_v. Any use of Vector will have to + * save preempt_v, if dirty, and fallback to non-preemptible kernel-mode + * Vector. + * - bit 30: The in-kernel preempt_v context is saved, and requries to be + * restored when returning to the context that owns the preempt_v. + * - bit 31: The in-kernel preempt_v context is dirty, as signaled by the + * trap entry code. Any context switches out-of current task need to save + * it to the task's in-kernel V context. Also, any traps nesting on-top-of + * preempt_v requesting to use V needs a save. */ -#define RISCV_KERNEL_MODE_V 0x1 +#define RISCV_V_CTX_DEPTH_MASK 0x00ff0000 + +#define RISCV_V_CTX_UNIT_DEPTH 0x00010000 +#define RISCV_KERNEL_MODE_V 0x00000001 +#define RISCV_PREEMPT_V 0x00000100 +#define RISCV_PREEMPT_V_DIRTY 0x80000000 +#define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000 /* CPU-specific state of a task */ struct thread_struct { @@ -95,6 +122,7 @@ struct thread_struct { u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index 4d699e16c9a9..54efbf523d49 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -12,6 +12,7 @@ #include #include #include +#include #include @@ -28,12 +29,27 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. - * - * Kernel-mode Vector temporarily disables bh. So we must not return - * true on irq_disabled(). Otherwise we would fail the lockdep check - * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + if (in_hardirq() || in_nmi()) + return false; + + /* + * Nesting is acheived in preempt_v by spreading the control for + * preemptible and non-preemptible kernel-mode Vector into two fields. + * Always try to match with prempt_v if kernel V-context exists. Then, + * fallback to check non preempt_v if nesting happens, or if the config + * is not set. + */ + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) { + if (!riscv_preempt_v_started(current)) + return true; + } + /* + * Non-preemptible kernel-mode Vector temporarily disables bh. So we + * must not return true on irq_disabled(). Otherwise we would fail the + * lockdep check calling local_bh_enable() + */ + return !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 7b316050f24f..0cd6f0a027d1 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -28,10 +28,11 @@ void get_cpu_vector_context(void); void put_cpu_vector_context(void); void riscv_v_thread_free(struct task_struct *tsk); void __init riscv_v_setup_ctx_cache(void); +void riscv_v_thread_alloc(struct task_struct *tsk); static inline u32 riscv_v_flags(void) { - return current->thread.riscv_v_flags; + return READ_ONCE(current->thread.riscv_v_flags); } static __always_inline bool has_vector(void) @@ -200,14 +201,62 @@ static inline void riscv_v_vstate_set_restore(struct task_struct *task, } } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline bool riscv_preempt_v_dirty(struct task_struct *task) +{ + return !!(task->thread.riscv_v_flags & RISCV_PREEMPT_V_DIRTY); +} + +static inline bool riscv_preempt_v_restore(struct task_struct *task) +{ + return !!(task->thread.riscv_v_flags & RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_preempt_v_clear_dirty(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags &= ~RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_set_restore(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags |= RISCV_PREEMPT_V_NEED_RESTORE; +} + +static inline bool riscv_preempt_v_started(struct task_struct *task) +{ + return !!(task->thread.riscv_v_flags & RISCV_PREEMPT_V); +} + +#else /* !CONFIG_RISCV_ISA_V_PREEMPTIVE */ +static inline bool riscv_preempt_v_dirty(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_restore(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_started(struct task_struct *task) { return false; } +#define riscv_preempt_v_clear_dirty(tsk) do {} while (0) +#define riscv_preempt_v_set_restore(tsk) do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(&prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (riscv_preempt_v_started(prev)) { + if (riscv_preempt_v_dirty(prev)) { + __riscv_v_vstate_save(&prev->thread.kernel_vstate, + prev->thread.kernel_vstate.datap); + riscv_preempt_v_clear_dirty(prev); + } + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (riscv_preempt_v_started(next)) + riscv_preempt_v_set_restore(next); + else + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); @@ -231,6 +280,7 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vstate_on(regs) do {} while (0) #define riscv_v_thread_free(tsk) do {} while (0) #define riscv_v_setup_ctx_cache() do {} while (0) +#define riscv_v_thread_alloc(tsk) do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 54ca4564a926..9d1a305d5508 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -83,6 +83,10 @@ SYM_CODE_START(handle_exception) /* Load the kernel shadow call stack pointer if coming from userspace */ scs_load_current_if_task_changed s5 +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_start +#endif move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -138,6 +142,10 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_end +#endif REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 241a8f834e1c..6afe80c7f03a 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -14,10 +14,13 @@ #include #include #include +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +#include +#endif static inline void riscv_v_flags_set(u32 flags) { - current->thread.riscv_v_flags = flags; + WRITE_ONCE(current->thread.riscv_v_flags, flags); } static inline void riscv_v_start(u32 flags) @@ -27,12 +30,14 @@ static inline void riscv_v_start(u32 flags) orig = riscv_v_flags(); BUG_ON((orig & flags) != 0); riscv_v_flags_set(orig | flags); + barrier(); } static inline void riscv_v_stop(u32 flags) { int orig; + barrier(); orig = riscv_v_flags(); BUG_ON((orig & flags) == 0); riscv_v_flags_set(orig & ~flags); @@ -75,6 +80,117 @@ void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static __always_inline u32 *riscv_v_flags_ptr(void) +{ + return ¤t->thread.riscv_v_flags; +} + +static inline void riscv_preempt_v_set_dirty(void) +{ + *riscv_v_flags_ptr() |= RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_reset_flags(void) +{ + *riscv_v_flags_ptr() &= ~(RISCV_PREEMPT_V_DIRTY | RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_v_ctx_depth_inc(void) +{ + *riscv_v_flags_ptr() += RISCV_V_CTX_UNIT_DEPTH; +} + +static inline void riscv_v_ctx_depth_dec(void) +{ + *riscv_v_flags_ptr() -= RISCV_V_CTX_UNIT_DEPTH; +} + +static inline u32 riscv_v_ctx_get_depth(void) +{ + return *riscv_v_flags_ptr() & RISCV_V_CTX_DEPTH_MASK; +} + +static int riscv_v_stop_kernel_context(void) +{ + if (riscv_v_ctx_get_depth() != 0 || !riscv_preempt_v_started(current)) + return 1; + + riscv_preempt_v_clear_dirty(current); + riscv_v_stop(RISCV_PREEMPT_V); + return 0; +} + +static int riscv_v_start_kernel_context(bool *is_nested) +{ + struct __riscv_v_ext_state *kvstate, *uvstate; + + kvstate = ¤t->thread.kernel_vstate; + if (!kvstate->datap) + return -ENOENT; + + if (riscv_preempt_v_started(current)) { + WARN_ON(riscv_v_ctx_get_depth() == 0); + *is_nested = true; + get_cpu_vector_context(); + if (riscv_preempt_v_dirty(current)) { + __riscv_v_vstate_save(kvstate, kvstate->datap); + riscv_preempt_v_clear_dirty(current); + } + riscv_preempt_v_set_restore(current); + return 0; + } + + /* Transfer the ownership of V from user to kernel, then save */ + riscv_v_start(RISCV_PREEMPT_V | RISCV_PREEMPT_V_DIRTY); + if ((task_pt_regs(current)->status & SR_VS) == SR_VS_DIRTY) { + uvstate = ¤t->thread.vstate; + __riscv_v_vstate_save(uvstate, uvstate->datap); + } + riscv_preempt_v_clear_dirty(current); + return 0; +} + +/* low-level V context handling code, called with irq disabled */ +asmlinkage void riscv_v_context_nesting_start(struct pt_regs *regs) +{ + int depth; + + if (!riscv_preempt_v_started(current)) + return; + + depth = riscv_v_ctx_get_depth(); + if (depth == 0 && (regs->status & SR_VS) == SR_VS_DIRTY) + riscv_preempt_v_set_dirty(); + + riscv_v_ctx_depth_inc(); +} + +asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs) +{ + struct __riscv_v_ext_state *vstate = ¤t->thread.kernel_vstate; + u32 depth; + + WARN_ON(!irqs_disabled()); + + if (!riscv_preempt_v_started(current)) + return; + + riscv_v_ctx_depth_dec(); + depth = riscv_v_ctx_get_depth(); + if (depth == 0) { + if (riscv_preempt_v_restore(current)) { + __riscv_v_vstate_restore(vstate, vstate->datap); + __riscv_v_vstate_clean(regs); + riscv_preempt_v_reset_flags(); + } + } +} +#else +#define riscv_v_start_kernel_context(nested) (-ENOENT) +#define riscv_v_stop_kernel_context() (-ENOENT) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context @@ -90,14 +206,20 @@ void put_cpu_vector_context(void) */ void kernel_vector_begin(void) { + bool nested = false; + if (WARN_ON(!has_vector())) return; BUG_ON(!may_use_simd()); - get_cpu_vector_context(); + if (riscv_v_start_kernel_context(&nested)) { + get_cpu_vector_context(); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + } - riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + if (!nested) + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_enable(); } @@ -117,10 +239,9 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_set_restore(current, task_pt_regs(current)); - riscv_v_disable(); - put_cpu_vector_context(); + if (riscv_v_stop_kernel_context()) + put_cpu_vector_context(); } EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 862d59c3872e..92922dbd5b5c 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -188,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; @@ -224,6 +225,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.s[0] = 0; } p->thread.riscv_v_flags = 0; + if (has_vector()) + riscv_v_thread_alloc(p); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index f7b4aeb9e457..6727d1d3b8f2 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -22,6 +22,9 @@ static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); static struct kmem_cache *riscv_v_user_cachep; +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static struct kmem_cache *riscv_v_kernel_cachep; +#endif unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -56,6 +59,11 @@ void __init riscv_v_setup_ctx_cache(void) riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", riscv_v_vsize, 16, SLAB_PANIC, 0, riscv_v_vsize, NULL); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_kernel_cachep = kmem_cache_create("riscv_vector_kctx", + riscv_v_vsize, 16, + SLAB_PANIC, NULL); +#endif } static bool insn_is_vector(u32 insn_buf) @@ -91,24 +99,35 @@ static bool insn_is_vector(u32 insn_buf) return false; } -static int riscv_v_thread_zalloc(void) +static int riscv_v_thread_zalloc(struct kmem_cache *cache, + struct __riscv_v_ext_state *ctx) { void *datap; - datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); + datap = kmem_cache_zalloc(cache, GFP_KERNEL); if (!datap) return -ENOMEM; - current->thread.vstate.datap = datap; - memset(¤t->thread.vstate, 0, offsetof(struct __riscv_v_ext_state, - datap)); + ctx->datap = datap; + memset(ctx, 0, offsetof(struct __riscv_v_ext_state, datap)); return 0; } +void riscv_v_thread_alloc(struct task_struct *tsk) +{ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_thread_zalloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate); +#endif +} + void riscv_v_thread_free(struct task_struct *tsk) { if (tsk->thread.vstate.datap) kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + if (tsk->thread.kernel_vstate.datap) + kmem_cache_free(riscv_v_kernel_cachep, tsk->thread.kernel_vstate.datap); +#endif } #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) @@ -180,7 +199,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) * context where VS has been off. So, try to allocate the user's V * context and resume execution. */ - if (riscv_v_thread_zalloc()) { + if (riscv_v_thread_zalloc(riscv_v_user_cachep, ¤t->thread.vstate)) { force_sig(SIGBUS); return true; }