From patchwork Tue Mar 18 06:15:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14020354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19AE0C35FF8 for ; Tue, 18 Mar 2025 06:15:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3SkCb4ZGDjqLh2dsg3oxP5de9jqXmwGj2CWSh2CeTIE=; b=3tRVx+VxIEbHg1 NF+EIRvRzTGgQOQBZqGmDtJp6u4jMxEtzPwbEhYdlqlGJV/pBk2SCnC1B94eDQBfWHGYUr41oYyHY x2rMmWTWqI+XKaPOZo4xtqUtbj72mBqNQ+fMsDq1RhNveFuuL0PEU7sutcQDeFuyql6i3GI06m4Cg Ol3IBNv/TNHaJxRjgnwhUq3WAFF4KaSjumACB1N75zmMha8YPpeGk12La67zc1w9oL416QBR2TseW Q0axs/43dqKSU6BRdQbAnIA2S+WXDPGVd3cWsPeUA4G6EWx4mm3rWiMBK16sMke4ugp8wVqJvkWbt lxH2Ba9McAVwJ0wIklqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEU-00000004ngz-0qbr; Tue, 18 Mar 2025 06:15:22 +0000 Received: from mail-ot1-x329.google.com ([2607:f8b0:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEQ-00000004nfJ-3qGn for linux-riscv@lists.infradead.org; Tue, 18 Mar 2025 06:15:20 +0000 Received: by mail-ot1-x329.google.com with SMTP id 46e09a7af769-72bbd3a3928so1999694a34.2 for ; Mon, 17 Mar 2025 23:15:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278518; x=1742883318; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=dFGa2kto1kHs9lIXOtJtQ/Cx/JbxIB4v8QEEe+ilNCPht1Fo81T92xi8RLo/YWwf7a BEGNGPphdwQRJa0StQH9aMRVP/Gm2MJuYFcN+9qIgS+d5AY+qe3VjWi6RBpuxg8KSa2t /nSSfRvSTm1UA8ou4yVeW1V/8kusJF7PPaKr7JfLCrd2KJrR/JNMgVbv+uwgY+5j0KJj 5AbvOEoVSjgsVyQ1wYUCVvELxzVj06c52NjcPzQAACEYqIx1ScVnLHQh0oAYZHqYAa6Y xab50fVi1cD5MJJolWdwH3Ikj5ZS55LV3Cutqrjv/uMq3SkrIuZyoizV6UAl6o3CN3ms pzAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278518; x=1742883318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=T9g3G8GiRsWUKyqx/xWyTnlqpZTfpmfAUNJPLOopNi7kmDD+vNXqaIfdW6TAOd+DDr zxy5M5R8kpKz6vShH4cAht3+QF+t0Z38WhTAU7Rvdhy2h9+4l6JoGFMnXvlWTgJFc7Zk TACuAaOPNAXS6sJPP67eESx2Kr8FHSiKQhARNYult+AUfTIh6K3EV7vtCs28PmJV4Ww5 eLgJ2c7+wUUlAyqY0dNhI4t8ivK5KIn+ZwwYAdIRjt9GGldEbVwk0Y/XCYGzGZTRUt5Y SXaWm8hw//GGtSPlKjkg1vx+qsPl4bJ59MAr/69Hy9F/3fY7X5hHDQQGqu11/zHpLFv6 N2EA== X-Gm-Message-State: AOJu0Ywojn/3Ysqm6IFEHCl1fjNa/UlCPBo2bqAnFbzOYqCYZ+oZoFP/ ivBA93ukyFxF56S0R3vmJ4/Rq7Hm6Q1pzmTSFz6zq9tuJ5M271GA469v7QrYNw== X-Gm-Gg: ASbGncvScAAXqA126gvnyl33sjyUJu5LETdUQu+GoMUOU2CeZnxG3+W8qncrYtsUR2j 1PjnIRRtece03jd0WeHfGRz95dsB1LbmwKOK/kqb1GDtwQqxpvZ98lBTwOFDRfzKvO7VfQz3fu0 1xP6gPFS4Q3lYXdyaXtEa5imhDCWyuTxuzrcIB+ZKvaxWGZTmxX7j2TC5yPuBZAPZYSn/5NY2b9 jyuwiUJT6qDzQ1UscXoccG+stTQ/MrnafT1jnNnAkXxqMYOR5iZQG7NhsiWoZnVZcK+/ao1m2ww 8EVXrqmAzRJApYuozfFqzAZciITC49Ch+Zdl68MTres1Zf2jjJGW5VVtZ5cw6ZFJJwFybtJbDg= = X-Google-Smtp-Source: AGHT+IE9dCEHhNSEUdS36yalwDgnQ6lSUtajzuROrLjALRXZFS2tcFhxjwVecmB7upfnJriqyJFUDw== X-Received: by 2002:a05:6830:6516:b0:72b:87bd:ad5b with SMTP id 46e09a7af769-72bbc247edcmr8104772a34.4.1742278517914; Mon, 17 Mar 2025 23:15:17 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:17 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org, syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Subject: [PATCH v4 1/5] riscv: save the SR_SUM status over switches Date: Tue, 18 Mar 2025 06:15:10 +0000 Message-Id: <20250318061514.1223111-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250317_231518_982192_AF6E54B7 X-CRM114-Status: GOOD ( 19.52 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Ben Dooks When threads/tasks are switched we need to ensure the old execution's SR_SUM state is saved and the new thread has the old SR_SUM state restored. The issue is seen under heavy load especially with the syz-stress tool running, with crashes as follows in schedule_tail: Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0 Oops [#1] Modules linked in: CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0 Hardware name: riscv-virtio,qemu (DT) epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 ra : task_pid_vnr include/linux/sched.h:1421 [inline] ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264 epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0 gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000 t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0 s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003 a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00 a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0 s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850 s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8 s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2 t5 : ffffffc4043cafba t6 : 0000000000040000 status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f Call Trace: [] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 [] ret_from_exception+0x0/0x14 Dumping ftrace buffer: (ftrace buffer empty) ---[ end trace b5f8f9231dc87dda ]--- The issue comes from the put_user() in schedule_tail (kernel/sched/core.c) doing the following: asmlinkage __visible void schedule_tail(struct task_struct *prev) { ... if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); ... } the put_user() macro causes the code sequence to come out as follows: 1: __enable_user_access() 2: reg = task_pid_vnr(current); 3: *current->set_child_tid = reg; 4: __disable_user_access() This means the task_pid_vnr() is being called with user-access enabled which itself is not a good idea, but that is a separate issue. Here we have a function that /might/ sleep being called with the SR_SUM and if it does, then it returns with the SR_SUM flag possibly cleared thus causing the above abort. To try and deal with this, and stop the SR_SUM leaking out into other threads (this has also been tested and see under stress. It can rarely happen but it /does/ under load) make sure the __switch_to() will save and restore the SR_SUM flag, and clear it possibly for the next thread if it does not need it. Note, test code to be supplied once other checks have been finished. There may be further issues with the mstatus flags with this, this can be discussed further once some initial testing has been done. Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Signed-off-by: Ben Dooks Signed-off-by: Cyril Bur --- arch/riscv/include/asm/processor.h | 1 + arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 8 ++++++++ 3 files changed, 14 insertions(+) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 5f56eb9d114a..0de05d652e0f 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -103,6 +103,7 @@ struct thread_struct { struct __riscv_d_ext_state fstate; unsigned long bad_cause; unsigned long envcfg; + unsigned long flags; u32 riscv_v_flags; u32 vstate_ctrl; struct __riscv_v_ext_state vstate; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index e89455a6a0e5..556ebcbb7e22 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,6 +34,7 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + OFFSET(TASK_THREAD_FLAGS, task_struct, thread.flags); OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); @@ -347,6 +348,10 @@ void asm_offsets(void) offsetof(struct task_struct, thread.s[11]) - offsetof(struct task_struct, thread.ra) ); + DEFINE(TASK_THREAD_FLAGS_RA, + offsetof(struct task_struct, thread.flags) + - offsetof(struct task_struct, thread.ra) + ); DEFINE(TASK_THREAD_F0_F0, offsetof(struct task_struct, thread.fstate.f[0]) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 33a5a9f2a0d4..c278b3ac37b9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -397,9 +397,17 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + + /* save (and disable the user space access flag) */ + li s0, SR_SUM + csrrc s1, CSR_STATUS, s0 + REG_S s1, TASK_THREAD_FLAGS_RA(a3) + /* Save the kernel shadow call stack pointer */ scs_save_current /* Restore context from next->thread */ + REG_L s0, TASK_THREAD_FLAGS_RA(a4) + csrs CSR_STATUS, s0 REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) REG_L s0, TASK_THREAD_S0_RA(a4) From patchwork Tue Mar 18 06:15:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14020356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25E8CC35FF9 for ; Tue, 18 Mar 2025 06:15:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=riKXSI+E2K6CYMT9GP1v7IFtNF9V7NAELowQ2sQ90fY=; b=bdcUeYg3A64w4h +PzWTJmIxshvN48DcJEr+guuX8INDznBh0iWy+k6g5ZqEZosjCFRfd4bmxPS7pVgEIYKBwOHL5P26 n9CarnhE6u6D/gYcopjEd3BBVxxGiICK3ez2lXA56jbOLhlFAVEfQAj10nW40P4uxDyOHwIUyHAOe n+MuHBeOVRb0gDNJSTKqf07WDNUqlDHinCwWKVfRKG9Xqx29I/YPWwYtmdTIV8sPAGDRnIOxfianA SGsNJm8rUvOFsJuMqGpqNROy/N5TFr2oiQlFhMXhYQ83iySJR+Nb7//JfN3F9105faKD6BzMFHFRc yihHJbsj7tNBdrL8pvIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEU-00000004nhE-3LTA; Tue, 18 Mar 2025 06:15:22 +0000 Received: from mail-oi1-x22c.google.com ([2607:f8b0:4864:20::22c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tuQES-00000004nfz-0glQ for linux-riscv@lists.infradead.org; Tue, 18 Mar 2025 06:15:21 +0000 Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3f682a2c3c8so3019320b6e.1 for ; Mon, 17 Mar 2025 23:15:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278519; x=1742883319; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Rz7GbyOPq3wyNoMRzcZSnc3xOAtIOrgSgwyUJM88Lo=; b=DQA5lPoZykGY3SwAr9glNH60GcYd7Qh4N1dq/ZhUijfhX3RtgVAPDOc6Kon3EPN+gV 7TBC3LoenzX+BTCvykS/mL5f2EqFTuOjjuMA4Aj4r1Go8421YGumzgPaKYzoIbuan9Y8 CMxVqfPOHw2XPjOFRi4L7RYrMx5D5ZK/nRaFsHTXNUfZr3bTS91AAu3CMkEmqcJt7ZxL bbvn40PKoc0GOZhS63kXZ/Yv/Sk5BHwNaB+LO7eRWs8yg2WmeoBSVOb4/7eNaS1934UW 6p9Suy0BjAjvwEJEAnUabsU3AtCHjVu9ubv9ikIir9vhV1LA4xke9rz/R5cUUd+zr5D1 j59g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278519; x=1742883319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Rz7GbyOPq3wyNoMRzcZSnc3xOAtIOrgSgwyUJM88Lo=; b=msuAP6IlBdFWarNM7b5ksnHfJNs1LzMRe1B1R5tgA6U4xs9TNsLS0/nGcgumq0/mOn mLUe+pWYUAZzrXkFzLzBWEKA0fq3lT8/gP4KjQw+3vJpkGdspUhMvzJ1++RK6dmOZ/ej zFRp/uWluRbEP1obRT7Ao8BIPo5JnO2T9NtlxfDDu/NMYoP7Gjv2nySDg9lU9kPne5lg fjkB5H2KLhUFkVyHfs4leIj2CMIIuIKsoEqKj5xRzRgV6TyoIDzFQohsC3hTo36CYfJv yMWtKNaAslBP8/Cv5M0zk4OAM5HMxM7AxG/ounXOYQVWxEr+FGwJab2tSEFF2O99N//B YsAQ== X-Gm-Message-State: AOJu0YzBYDq9OhEbxbfmn/3bfOUY8XNt9dFWXgu5rsYOFXSpMxG/tHBI KS8HGlwIOkUTMHEWa+Z6nl9uAmeHWDvOmNc/AhK21hjkPl60oVdKLDwvOESkUQ== X-Gm-Gg: ASbGnctVYDiFrq6Nxofvqq8zclPkWmZJY747BTeGkFO/szT2cVJA6K8Ezj4fArWJOXt 8NPGcC92Po9G4RXTiQUFBs3W3PllLvQp8bQe1CcjasVP8j4X8s1NojwVEFeXPSKz7LR0wfV+Dyg LUT6kFgAhg8NQbk7Rx7Z1xr/KF3d6t3Em0aNvyyRiTJfQRGDizRFBEr8V3SyjnOBzbxV9/jy/Hm nDihaKAfcBTw2T71TUQuowUV9MrYwALIQ36LzIzBN+ezRSTkbfIWZ0I/XOdX/ntumMOi8MFaT8V FVOgFhMWagvI0UQX8srZGpC5Qi3ih0bEkkVZpMKUvivqHtp7iajyV8FNiMqalTk= X-Google-Smtp-Source: AGHT+IHjTlUJ8udAjH/mkkQoHGmHBU7+KcdMu+hMBhRq6iTs6DPSX4yEs5XtvzKJa+RBjf2KhPoFlQ== X-Received: by 2002:a05:6808:3507:b0:3f8:587:dacb with SMTP id 5614622812f47-3fdee93092fmr8770534b6e.9.1742278519108; Mon, 17 Mar 2025 23:15:19 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:18 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 2/5] riscv: implement user_access_begin() and families Date: Tue, 18 Mar 2025 06:15:11 +0000 Message-Id: <20250318061514.1223111-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250317_231520_209562_87EC6803 X-CRM114-Status: GOOD ( 14.84 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 76 ++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index fee56b0c8058..d7fef81b7969 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x, char, \ + __typefits(x, short, \ + __typefits(x, int, \ + __typefits(x, long, 0ULL))))) + +#define __typefits(x, type, not) \ + __builtin_choose_expr(sizeof(x) <= sizeof(type), (unsigned type)0, not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,69 @@ do { \ goto err_label; \ } while (0) +static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr, len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin user_access_begin +#define user_access_end __disable_user_access + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err = 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err = 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) = (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, op, label) \ + while (len >= sizeof(type)) { \ + op(*(type *)(src), (type __user *)(dst), label); \ + dst += sizeof(type); \ + src += sizeof(type); \ + len -= sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst, _src, _len, label) \ +do { \ + char __user *__ucu_dst = (_dst); \ + const char *__ucu_src = (_src); \ + size_t __ucu_len = (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_get_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_get_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_get_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_get_user, label); \ +} while (0) + +#define unsafe_copy_from_user(_dst, _src, _len, label) \ +do { \ + char *__ucu_dst = (_dst); \ + const char __user *__ucu_src = (_src); \ + size_t __ucu_len = (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_put_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_put_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_put_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_put_user, label); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ From patchwork Tue Mar 18 06:15:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14020355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD3E1C35FFA for ; Tue, 18 Mar 2025 06:15:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7PDsACv0gBGi1i4W1yr67QCtiDzTxJAo7GkQoT6zHpo=; b=DkQlCq7Ck4bBUT WIJTOrMnBJch1k+OWuLcKfHI7oP8E8yekw3+C3ku24h5lBWCiKFGp5lOhPL2anCLoT/Ts5zcGzWYZ xpe0SpADe4tnY5TIWxlM9hBoMkIc9+LieyuHvI8R3+WN0jnR0cMU4Rku0aci/B74caOOUWzH62R58 HFZclwgtISVVWcX4k6+dAjelnN7t2Aw/Mb41YGmZX4nNm8PmOTyBOBAgbO5sriE7ZMv8eSTawLEyj InvCQ9Fo71DPebYBxLU1AI7hH/sulP6TD/Wk6T746/9QNwfvyb41QK/8YTXGP3n/FOZLnRFXFtl2u trQhOhLRQtRIXlza1E+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEX-00000004niv-1inI; Tue, 18 Mar 2025 06:15:25 +0000 Received: from mail-oi1-x22b.google.com ([2607:f8b0:4864:20::22b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tuQET-00000004ngU-2Rjr for linux-riscv@lists.infradead.org; Tue, 18 Mar 2025 06:15:22 +0000 Received: by mail-oi1-x22b.google.com with SMTP id 5614622812f47-3f6a92f2328so2580507b6e.0 for ; Mon, 17 Mar 2025 23:15:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278520; x=1742883320; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=STOmRD1NGs9uSMt++nKAUKoj9Ow/rlWCVnbqWbWWLRY=; b=T0OMwn3RItUSEAcDAREIkKqnUzuo+LWjsvqAQgeNYEgnuO6qKvU6QOqbh1f1IBxVfF 17x7zvUDOTKmDqyfPeX0DnejsS90/dfH+mMQm3E8EN8tl/GruPXkp+QaLL4+cTiWxGA7 T2kjHJkhQwKmmTd1GyFfJwMmrCYXj3NqwbmOA8myYhAmYAl/Cm/+AkkV6BsnN4MHfUlS BgCoqx/p6miz2Puks65MDgOAYSB3/UrXk2i50qB5sxRDqAyGRui/cjO++PltJl8gM+1P gJFVnfe2gCeGvP3p85AZWWh+zJ3l6za5wFS8tVtHj1jZBE+nsfBRc33sY8z2RX7CJHUI 4K8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278520; x=1742883320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=STOmRD1NGs9uSMt++nKAUKoj9Ow/rlWCVnbqWbWWLRY=; b=wNWZX+WdgUtOj8caLRVHsRgfrSrRapyBW9xWuTdhC64PD4FC9svtr1MiW5BaRbA6Un B/LMr7I9lcgqYx45W4QCoLXnK3KjSEFzty/XI0tGph+nxHJo+GMeZ8KctxWmgdeOCBJT CpbrrKEeZtmlJRV1YEccsc42jWKD5c5ysmXd2c1zIxHAvAHzdDko2NjnBl5xNbd9EVO3 gHrbgO+fNtRdjnqA7sNU7HtwUj20k/i11W9KuPPKP/jHaELjt1R1gZT9I6nKY3aXyvSg Ac3hY5+8FLN+OBv9wZzi/nKLK4OOwB4bH2CXgaGxYjVYKyi2x2ti/2DG09FhLGROJTGw ccwA== X-Gm-Message-State: AOJu0Yz91Dz0XmsNW8JSPWPfyqjiTHy4GfbmwlfjejPATJBV1JsVuoPn D6IPFBJ/iuC8Xybzr80UREM6HnUE3EtoE9iRU5eoJtGnMDW1b7TYets60v5zHg== X-Gm-Gg: ASbGncuXZODVIBrcKhY+GyysGAuOGr122/DdIpoKd8OAk7FPw57FFxqtlceWa3ILVwQ hEFjsdqCohwwFg3yaqGuytv7/QQhds2SEHxFyDipuavgz1DGIXAxLgnbXpPM3v0jMnQ/gUbkCLh 4rVqeoaJHzg9N/PiS00LxRZvnzvdMDKXCVC1w1VHT54rT1QhgpAPVpY1Ci1Pf4ZYFCQR9AwCKOI CG6DWLSgZVoDALy6MC8FKf0hTzKbJTbN8rvarca+w+PBTlMjn/onn0g+RZWhwtYee5p3vBwLrAM riGTM/Q82NtM/d6Ud/CQJgFf0RKia00qA2N3M7+Cu9V1VC+hrREhgCOTkfQO+hs= X-Google-Smtp-Source: AGHT+IF+QEcnPvq7HOQUCyqTY14BOpN8Y8eX1PyixCC8ZH0S4NtzfP09P+KLDIccR2wqd6pOP+an5g== X-Received: by 2002:a05:6808:38c4:b0:3f8:18a3:b1cc with SMTP id 5614622812f47-3fea2482935mr1337534b6e.12.1742278520647; Mon, 17 Mar 2025 23:15:20 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:19 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 3/5] riscv: uaccess: use input constraints for ptr of __put_user() Date: Tue, 18 Mar 2025 06:15:12 +0000 Message-Id: <20250318061514.1223111-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250317_231521_625337_39C84AC1 X-CRM114-Status: GOOD ( 10.01 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index d7fef81b7969..180d6e21d5b5 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x = x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=m" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x = (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=m" (__ptr[__LSW]), \ - "=m" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ From patchwork Tue Mar 18 06:15:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14020357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ECBB3C282EC for ; Tue, 18 Mar 2025 06:15:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fNMNE30+kv0bGqxEwrg7CYKb2qTS4Wym7WL/Dy8ovuA=; b=mMlFwvGgDKIRQH x+v1JeKWQ9a5F/i5g9FSU5b3vv3OuwdzJ4nzi+NRtYeAPtN5J5dabRKCudZNKDL0hOEX518UEoNEf q89eHsPbA9/dlvZBdP1s8RQV8It6Ymw/koN9Ip3NDS/enI1PjOIBXTzW0kHuoIub184eQNaP7OAp6 je4STNffRric9jNRVph8yD6BN0+KYdDGLIBvQIGrwu2btCZ7wRXS2+/1QKgANoOKiwhbyvdGVv3eW Ynjn/tvl9Q6TXmsfAlCVV8ow4MqPyP/mk9+iHC2BAok7CaOYeRqwyjDKl9Xpn+6RZZIad+EJlaVlH v7GP+94AUKthQSht82IA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEY-00000004njB-04v2; Tue, 18 Mar 2025 06:15:26 +0000 Received: from mail-oi1-x22d.google.com ([2607:f8b0:4864:20::22d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEV-00000004nhD-0UPe for linux-riscv@lists.infradead.org; Tue, 18 Mar 2025 06:15:24 +0000 Received: by mail-oi1-x22d.google.com with SMTP id 5614622812f47-3f68460865aso2841101b6e.0 for ; Mon, 17 Mar 2025 23:15:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278522; x=1742883322; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HccPZpbZyX4ghGGLQ1GggeTzIynZ7PXEL+v/MCdb8aI=; b=QQIYL34SCgQ8Es1CKDANsE1+CCc7eB7k9+bw+bHQSZMpFoFlDbBR0BAQ0sWSulnQMU LRxhDeQSlHKEX75ktvirENnymo6ijTrKmQQBFAbPlS9SlnS3NI6BYP+YoS9HQ8Nwak/u DW4sNr6JLa3iHDwKLr9dwLnrMmzt1d2A/PyKYHU7FsBrE0bRj6aUevFETHg0t+9xqX9l qLmABL4f2pb3B3sdoOAgJYDsNnEAY3U+hVJ2b/BA1T39Dbh1i/DvX6njIuQkd+SI4KxI 73BM7OanPof66gc3xDqwLQxjFhIHbyC0nuUUFKwC7WKGyLLvHBGsn9IBsvFaLWo8T+kd l1dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278522; x=1742883322; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HccPZpbZyX4ghGGLQ1GggeTzIynZ7PXEL+v/MCdb8aI=; b=Bq/dPGjO0kDa1RnBCzCLahSWRcroAhkDqglM87GD6my4RgPLI+BEfJMjuS9uTlplKW HfHZbRbmUraftO7MpvkvDXSDS5UZDGFARaMsuSvxWtr1Rmw/4U4VpY0pfANd+57faFWc gATAhI2F5vdtJjnBgchXxfsSxYpXRr3AYJPY2D4YthCjIVjq2Gx1SUYIw4a/yxHePA9E j4wrEepRZtM0b1+XXDjbTkKvELCse4u0LZgbhm3nTNs/M0JxRtwkWDekrfoRmE3NrjbS QjYpYkP3ufrrHDDbee2PSvMMPWB6Jc0ZKU9bLtuThdYsoWXhAi5Jkwuum1x8cNBoGA5Z kfkg== X-Gm-Message-State: AOJu0YxWS8u08VI1XmMXW9JWLQADxA+3q88mc9x7JYRljZbBtk4uU7Rv KZu8v7eEDbtQGAcNaYmnQevB0aaacEFj7azWS0LIBo+w1Q9r2W3D4ynCkKyIRA== X-Gm-Gg: ASbGncuzBCj4bpw2WpvmlEozFkECtHYj1GxwGLJnrf+X2c9a+VTK/8V5aPy0hzvFxSC LKux2qndYJ/KFeRSmKtfOgsJG3hF7Th9kDvtJqV6T1TL2Y8PkCXB+wfiKcYT3U4xMybF85Kzv+W +3lQiIPtFTENhVIB9PxMKaAGc8L63HmZci/FqgInlp7AJi85ql0rS8+66u88YM0sHmhS6l/6s0n TroCbtuRMvT6TD0QUpCfqk/Im4SCOeo1pjslU7kZCPSEVGMpu/e0y4d1iveMaED4+dYmOxPap0X VmoXgcFCyX3d4gzK/Fx0EJK5xnIgyrXQMnip2IbK6QdFxHUoey/BfHWEJ0VnDL8= X-Google-Smtp-Source: AGHT+IHEfvBRTZGysFXmW3RRQXGtzyNoB5Q4apt6oGZu+ZdRy6GfL5mJC1oNgLlW5pjKpJDDbIHUXw== X-Received: by 2002:a05:6808:201a:b0:3f6:7192:6aaf with SMTP id 5614622812f47-3fdeed0d9b8mr9357410b6e.22.1742278522205; Mon, 17 Mar 2025 23:15:22 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:21 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 4/5] riscv: uaccess: use 'asm goto' for put_user() Date: Tue, 18 Mar 2025 06:15:13 +0000 Message-Id: <20250318061514.1223111-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250317_231523_165407_015DFEE4 X-CRM114-Status: GOOD ( 12.84 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 71 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 38 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index 180d6e21d5b5..ab91dd2fa230 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) = (__force __typeof__(x))0, -EFAULT); \ }) -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x = x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr = (u32 __user *)(ptr); \ u64 __x = (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) = -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err = 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) { @@ -398,12 +397,8 @@ static inline void user_access_restore(unsigned long enabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err = 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) \ - goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) #define unsafe_get_user(x, ptr, label) do { \ long __err = 0; \ From patchwork Tue Mar 18 06:15:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14020358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FC3AC28B28 for ; Tue, 18 Mar 2025 06:15:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DwyHUnH9kqvPqtEdh2PRkVmm3Fmtgh1XB1PHgGBjFU0=; b=rhUcQ/FTW64Otx J1zZsebPcG/qIvtlORH/rmeprhHtL4LPyewCZl1/tCa+t9Pp4rM5g49g1MEzbFzbmPEzTY2314fUr dBGIxlAWNU8RRCEXRrBAoJd/KJCGF3LFJ4dtdMpSFMRLEDxCvonmHNQy3p7ZBDzJHt4GzggKzXooe vN62Z8Uage5o7WHkDqJF0kSp0V28XGguBB/GbowTv8MHcAEfPv7GaG1EP2mINzuDwfo6nRGHJEPdH M3ayXCxWl88dn2iR/T58YUjY+851B4MKhzh8OHvNJUj+OXgR1hkDDqYz3QZ9a3mWqONyXSd5o9b+y gpR0wJUHPRwbdx7pofag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEa-00000004nka-2gcY; Tue, 18 Mar 2025 06:15:28 +0000 Received: from mail-oi1-x231.google.com ([2607:f8b0:4864:20::231]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tuQEW-00000004niE-2vDz for linux-riscv@lists.infradead.org; Tue, 18 Mar 2025 06:15:26 +0000 Received: by mail-oi1-x231.google.com with SMTP id 5614622812f47-3f417de5e25so1972055b6e.3 for ; Mon, 17 Mar 2025 23:15:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278523; x=1742883323; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qB88SWzbKOqS560oNmcGvhVMbrzNnGmMk8r7GYi48TU=; b=RnoL0QVXbyBWa98WO6L0ZREIWovexUY26+8dBMqiRbhfILadUOQEWcgUajDpOgz2gw sXPYGaL+LHWxgWNTYTlyiuoOzi6nI9jjVfht6wZNf+E5ul4E5WAtQ8DBS8aVO5GNo2Dy qNImdorjFXDC3h87TtnM5ewdDEnDSjoxMQUmvb4r/9czmSIQ26HskKZLEQVeT65i5ZS4 JoxUE/qLO0En0bkKh3SYo2/xFSSopXUVNvZh0e3wtoBLgtVGPskpIdc2O1OIP51VDM9l k72x5zD1gi5Kht/qnCZ8u4XtJ1u3h3mV+WzELkbnonitr80NN+yNuBXoR6/C0b44CqDG l+3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278524; x=1742883324; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qB88SWzbKOqS560oNmcGvhVMbrzNnGmMk8r7GYi48TU=; b=oy0D1TAtD8pH3TwpGSJ1GnYmHVDHmgntjSdX7H/NL8cGy80y8sPmbaC3ANRtAI0aj0 X6r9RYBoLzc9GGDGQKQ59IvGHecAqrMOVJl8CLQZVShOf6H5MA+W8naDr90kXkbucTbs iMr3BqvRrIQSM40LITLPBbyTkK8l7S10jMEcNPI+xoY95p/vW0gfUPuDsBkA+s39zBsW /EpGkz4o46r5XwwmFlb7NqzXnYghcAYml6djGwN6ePm+tnBDe1S9MWAPTPt5lWTM1Gn8 G8/YMjbGQhrOIkx1vVirTFnCAD+Z5xQdqF9tfInQ0Ocu5dpjA4dzk3MdcBMFt6udGjf5 Erfg== X-Gm-Message-State: AOJu0YxOYIl4mSg90lNaIx9f2z+pbMwiCgTguuNxsUeILe+PIa3HE8pu nJyMwiPeZ/EZ7es8hZTaqcTcmAglbnFV+9QKZfQU2idOOVu71mIFTYYN+OYzww== X-Gm-Gg: ASbGncuAjsGsPOVKcikb/O0qNC8+c7LVCtikKfSygmxDaigSEHx3bCT70y2yM7Ijeam Cd0HTT6aeWY1qTm6I4fihW7L5cz8/h2wEhd/gPoPAhNaBEzHvrCQvgHIKP1IUejB3KRKSKy0gQX UFnUsK4Eq2Hk9PFg5vhVYq/Y7cTkMCxjivIg5frbD8iZpVkROWQDHq7dNFk3pfURCp+4h2fOYk4 a5sNNuL8w3LJXdE55bKObeUvDGqDbLxyt3a5p4YonXuMlSDQ6TrQY0rRbIBYYOyiKZKHf60Hz6O eANuxipLEAaxUgQxoar364MdSNe4OHC0YkTbbhVSl6jFCnK0PI9zN38Qg2B1drY= X-Google-Smtp-Source: AGHT+IG5rNTfQnLxyptciRc/2MpA8tF5u7O/A4GY0S1qU3Z6Z6RSpie2GzJANFt/uJHp/UYORqcpgQ== X-Received: by 2002:a05:6808:f05:b0:3fb:3be7:ac9c with SMTP id 5614622812f47-3fdf0081260mr8892498b6e.30.1742278523672; Mon, 17 Mar 2025 23:15:23 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:22 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 5/5] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Tue, 18 Mar 2025 06:15:14 +0000 Message-Id: <20250318061514.1223111-6-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250317_231524_738424_C146464F X-CRM114-Status: GOOD ( 12.31 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 95 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index ab91dd2fa230..ef693aca5236 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,58 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne * call. */ -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err = 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=&r" (__x) \ + : "+r" (__gua_err), "=&r" (x) \ : "m" (*(ptr))); \ - (x) = __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr = (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=&r" (__lo), "=r" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label) \ + (x) = (__typeof__(x))((__typeof__((x) - (x)))( \ + (((u64)__hi << 32) | __lo))); \ + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr = (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err = 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +156,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=&r" (__lo), "=r" (__hi) \ + : "+r" (__gu8_err), "=&r" (__lo), "=r" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi = 0; \ - (x) = (__typeof__(x))((__typeof__((x)-(x)))( \ + goto label; \ + } \ + (x) = (__typeof__(x))((__typeof__((x) - (x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err = 0; \ + break; \ +__gu_failed: \ + x = 0; \ + err = -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +225,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr = untagged_addr(ptr); \ long __gu_err = 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) = __gu_val; \ + \ __gu_err; \ }) @@ -369,13 +419,7 @@ unsigned long __must_check clear_user(void __user *to, unsigned long n) } #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err = 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,12 +445,9 @@ static inline void user_access_restore(unsigned long enabled) { } __put_user_nocheck(x, (ptr), label) #define unsafe_get_user(x, ptr, label) do { \ - long __err = 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) \ - goto label; \ } while (0) #define unsafe_copy_loop(dst, src, len, type, op, label) \