From patchwork Mon Jan 15 05:59:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13519325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E4F5C3DA79 for ; Mon, 15 Jan 2024 06:00:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fwCTTNVqZoHhbFl+FR/cOtEmFCMnQd1GVRoEULW+d0c=; b=sOH5K8SLBzaag3 pGZcV9o1uQwIU/JtppTjpCwj466HEGn6hIxhgiGQASECJtDao6pC3n65OStBEMdP9VTqonPg7g97A NwXjiV25x/Oltzy73AU6KFSkhGASyycWWW2XO0vjQEb0TBGKCJBYfCjs7fa22hPZPldz5Y82GHKhz mNtkUnoqz9jReFIW2uOGdebIRFO2+yVc6r8JYVilVwgeinJtBLMb3iUWpy2h3/mMBFfIq+S9u/nKl SvvssHD1cidGy42302hiZdf2wNE45yGbKR3A+yp5W1IvwBd22OeatqSjWTbLZ36Ic62v3YaBIwJj8 XMkGWHDYzHc0c9GG5R6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1T-007qPl-2A; Mon, 15 Jan 2024 06:00:35 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1S-007qPD-0y for linux-riscv@bombadil.infradead.org; Mon, 15 Jan 2024 06:00:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:Content-Type :MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=pxVEmIaJ7fQfkbY5hQr4DL8oB+uGh8ELXfBRuaEAgY8=; b=qv8hLefNKMgUOdJKMEH/uhZNdf FwJ0SRLszL7q7gZDSzd5pdPjVTPGRuqDZYXyc9/kHLXA6TUhGE6qr6CyYEcQqa4vptkUBQJqJ3VZp AikTmc02nQqpyn1TotXN9TBjFcfEy/tpZyBHbLW/Myrd478O1Mkegp4QzRrTjm/0FXyvDe30oU5zh OxQ+oPJZGMRxuN9gzuATHWPqGthb/44u0tjki1aIjbooMGuXJWX+39xLAqni0/xKfV0EI7DPjK2sB qzbw9xGTWZVtsPCu9LysOw4O3oLsM/p9rttjqSBRgtQXhbcbGiux+y/23Hdfx098ETWcXzcn9LMeB 9L1IWgVA==; Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPG1O-00D1dk-1b for linux-riscv@lists.infradead.org; Mon, 15 Jan 2024 06:00:32 +0000 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-5ce10b5ee01so5944030a12.1 for ; Sun, 14 Jan 2024 22:00:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705298421; x=1705903221; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pxVEmIaJ7fQfkbY5hQr4DL8oB+uGh8ELXfBRuaEAgY8=; b=YLCXnN/AOyZ2H9kRvZe1pMEeiU61TKhJlN+FIJ6DkPVDyWqI4By0rjJO3sgplSo2pP BbB7+3gOr8YzG5QpqjyXKlDfVun8TBrdWIMpbCj4VIu7dRLavIu60FOsHQCN9ro40JV4 /kQ8/2GN466Vv9WuxIYa53lAtvb1xvVSGH2TCCny34OnoUGvBrcQwPcniQFCEZaYduQ3 +JnK7bZ5ZuWakhY75c1oohko0PRsswCqoS4VB3rZX7x84FT6L0NDC8O06RTfHkdDPIhR sY5jZocIj97OIC8hx9UXl/AFjnoFbEfPxAklLGKTSAW5Ug4RKMxfus+6IhXyyYzG4vvr KzmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705298421; x=1705903221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pxVEmIaJ7fQfkbY5hQr4DL8oB+uGh8ELXfBRuaEAgY8=; b=FJrDcBBUbP5+ubC2Z2aW1kAdnvG6TrDKbsUW+PdX0Ft1MyzYXSJLBSZtYMvr857ZN6 jUQLkTz/81jJO+EToIXbQuvxPFTkssqJHwepK8NlJ8H+Br4xTEchko1Ld1t5EDd2UkIf 08B/bmtCqQSw4f01KaRMXqnErPq+yV5JWw7JBBYaoX1mWfOJ6bbBwIA+uyqWXOa/78ia Gm7rHjGcoSAKE5uAJSvzgdqZ02b9CxoBcEMjEUlJjkIumNoxGUsSNy8Fg6B5IeLqW79q BsDECm0PM2VGuSI0Fa3HIGbxwpdT6Jtvg0ZqE6n/ouoMT86xPHcyYSM9o2t9s2DuZ2l4 rBXA== X-Gm-Message-State: AOJu0YxpbL24+7uYBEUY3EfmxgZ4AQJZD9QLGRGoSRsNTO5XUIm1vR2e JRr20uTivKLd5WotpLHoHn+5WFmmu9c0pSnqInxDR2HaegVvBcG/BxUkz7kTlaHt4k714S0Ifq6 ruKtRrQ8QxvvCxuYT+sKeiJkh4i4lEAY4hRpcEfXowGbXPsST5+LunOshpdZW7VobZCv9iCg1x+ ILXJ7s3GhaDcbFvX5n5GHd X-Google-Smtp-Source: AGHT+IHJiFjSbkCDIJRyWEaEGbnzngccZ+BZd0MMKeJNL4tZOCCj7z1RneTq5lokcfb7iBDiqNbRiw== X-Received: by 2002:a05:6a20:9f95:b0:19a:fad2:51d6 with SMTP id mm21-20020a056a209f9500b0019afad251d6mr1407006pzb.14.1705298420488; Sun, 14 Jan 2024 22:00:20 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id o18-20020a637e52000000b005b9083b81f0sm7392988pgn.36.2024.01.14.22.00.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 22:00:19 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, ebiggers@kernel.org, Andy Chiu , Albert Ou , Oleg Nesterov , Conor Dooley , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Guo Ren , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Sami Tolvanen , Jisheng Zhang , Deepak Gupta , Vincent Chen , Heiko Stuebner , Xiao Wang , Eric Biggers , Mathis Salmen , Haorong Lu , Joel Granados Subject: [v11, 04/10] riscv: sched: defer restoring Vector context for user Date: Mon, 15 Jan 2024 05:59:23 +0000 Message-Id: <20240115055929.4736-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115055929.4736-1-andy.chiu@sifive.com> References: <20240115055929.4736-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240115_060030_813374_4E3CAC86 X-CRM114-Status: GOOD ( 21.28 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changlog v9: - update comment (Song) Changelog v4: - fix typos and re-add Conor's A-b. Changelog v3: - Guard {get,put}_cpu_vector_context between vstate_* operation and explain it in the commit msg. - Drop R-b from Björn and A-b from Conor. Changelog v2: - rename and add comment for the new thread flag (Conor) --- arch/riscv/include/asm/entry-common.h | 17 +++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/ptrace.c | 5 ++++- arch/riscv/kernel/signal.c | 5 ++++- arch/riscv/kernel/vector.c | 2 +- 8 files changed, 41 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 7ab5e34318c8..19023c430a9b 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,23 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + /* + * We are already called with irq disabled, so go without + * keeping track of riscv_v_flags. + */ + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 4856697c5f25..5d473343634b 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -102,12 +102,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 71af3404fda1..961c4e3d1b62 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -193,6 +193,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -200,7 +209,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 2fc145edae3d..8422c881f452 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -117,7 +117,7 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_restore(current, task_pt_regs(current)); + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_disable(); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4a1275db1146..36993f408de4 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -171,6 +171,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -187,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 2afe460de16a..7b93bcbdf9fa 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,8 +99,11 @@ static int riscv_vr_get(struct task_struct *target, * Ensure the vector registers have been saved to the memory before * copying them to membuf. */ - if (target == current) + if (target == current) { + get_cpu_vector_context(); riscv_v_vstate_save(current, task_pt_regs(current)); + put_cpu_vector_context(); + } ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl; diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 33dfb5078301..f571af3d5f78 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,10 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); + get_cpu_vector_context(); riscv_v_vstate_save(current, regs); + put_cpu_vector_context(); + /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); @@ -134,7 +137,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 578b6292487e..66e8c6ab09d2 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; }