From patchwork Thu Mar 20 22:44:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14024669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E5B1C28B30 for ; Thu, 20 Mar 2025 22:44:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3SkCb4ZGDjqLh2dsg3oxP5de9jqXmwGj2CWSh2CeTIE=; b=ZtDFkOM8hrxHKJ YwjRhzlPqpapnBTh7XdZnOG1v7BYFVl80nN5YYlyF324mix0k2yzQG01kXaaV8fr+NyldmobSIHcJ pOdw5XhFEBaZHcho7K2dltOCNK1c8d5UJZgOSrsOxwqEyJQd8keURt3KwUkjEUiQXI6ornTqPROww YOdUSWLoYh7BDMS4j7MExDfe38uwaytCc8eA5yAQfuasuFF7czIz246lHeu5xcs6IllYM63LOZaq/ +z+QHAbcdY+OekGAEvx4mhb/rb7fbv/FEL6Qf0Tcvy9t8IjQHVqoMdhmsWGtrfgK84bzDLNPQJo7e oyCrSzHiHZj6nkDET/6g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcp-0000000DPH9-1U20; Thu, 20 Mar 2025 22:44:31 +0000 Received: from mail-oi1-x22c.google.com ([2607:f8b0:4864:20::22c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcm-0000000DPFu-0N0d for linux-riscv@lists.infradead.org; Thu, 20 Mar 2025 22:44:29 +0000 Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3f9a7cbc8f1so354518b6e.0 for ; Thu, 20 Mar 2025 15:44:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510667; x=1743115467; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=FRCKs1QOPApNJUIMNTa6STGLNQwLbbbMYiKFktcS02SAI5YFxzfuq2mIIDBde9aX5x 6i4J6f8ZjcLXHdfzUZs/0GdpBwDQGWeIgw91tS3YHCoqznQl7LcUWMPIU2ZFdJN1OT83 7keTPIQTQSfsSJ6rJoGb6SfH8aFBnxORxTPHxLR9D8hg/HtFOAFwteBBe9d1smNT8UhB qN8b8rYj5UpmbcVSQ7SaY4MwS99dIlNdEyPcvUFsONKJMBrFKi5G1gk3Bue9qUL6bRSM rbMP3grdJT0ymgQcCaEC+1P4FP+DY2khfJfCn2SznYAi9Mn2S9HPhb7+Fds1O85NL9Lm 4HuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510667; x=1743115467; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=uKsvdz/Vg8oDzSyK0sxqVHPFwmlpA9xrv1VUQNH/Spq3G/3BKXdGA8RplNw7flFSeG wIvp8l4DLMUerShZUUBIrVdTgsBdXyhYi1/qLV1ub3UasaEhADVtQc5TwBdEPlnUaVM8 QP4qjWOrzK7Ay6GRPWQ6+v4ZhCGfsZt0QLrODOyEvRFZNY/PXRvrW+fmZNTcadb/MNIP fRpBs9M95+lU9UmYOMDo8EoKoxW3l9+8PKfXIDwav2zlCwBG3T7HrNKmhyLLqBj705rc VNaJ9DSOINSSdUQTudOWNVz9DNrvOq9YCgFgAL0FBkzyLEmMDHKrFKRNsPvBStFTcUBb 9Y4w== X-Gm-Message-State: AOJu0YxFZ/Lacp6A9GHpkklXYq4xCKAk7TDU4B1dNtMpXUOYko4KgvZh 5i6UXgwZroKZU/wZc/0QK89fke2uMiVDapGKuJvNBlloESWnnh7u0Sgz854g4g== X-Gm-Gg: ASbGncsbt4s3og9pJr8bItGdlRbOK95QJsu4QyMF8Kwgctopv+plRyE3ZcW8c5MElSB NFnb0KCDslhtEi28CqPQe0w98FXPLCNW0D9L4nA0cH/3RjoEJPEyD7Su98sZJnmwO11Jjg3XHg2 f4A71CcXYJnAFgQF4GhqosusMV1a4FjVwpgEwzNrJbmQDt18YgCljdV0+C+qEtUTPg1bug9XZK/ GWZLMBYWEyvUjPHIEW7omLMBjvtaNTaLdI7nrSr1susge2StRtJRR3b6vOH1Pa6ouJeU/8UB0KP Se/km6G7ZH86ISeOdtuhJTdTMDRAJax3EtEl8u37A7s1v+K3jgpI0VvBHJ8o/9k= X-Google-Smtp-Source: AGHT+IFd0nlvhAJDq2aqQ1KFOkNVEh4rXpWJJkMBcG9z9sJUjee94QYnGOMRYiHTi9qWadmuKL5cPg== X-Received: by 2002:a05:6808:250d:b0:3fe:b1fd:527f with SMTP id 5614622812f47-3febf70fc7amr619142b6e.1.1742510667005; Thu, 20 Mar 2025 15:44:27 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:26 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org, syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Subject: [PATCH v5 1/5] riscv: save the SR_SUM status over switches Date: Thu, 20 Mar 2025 22:44:19 +0000 Message-Id: <20250320224423.1838493-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_154428_128132_D9405BFA X-CRM114-Status: GOOD ( 19.53 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Ben Dooks When threads/tasks are switched we need to ensure the old execution's SR_SUM state is saved and the new thread has the old SR_SUM state restored. The issue is seen under heavy load especially with the syz-stress tool running, with crashes as follows in schedule_tail: Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0 Oops [#1] Modules linked in: CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0 Hardware name: riscv-virtio,qemu (DT) epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 ra : task_pid_vnr include/linux/sched.h:1421 [inline] ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264 epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0 gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000 t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0 s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003 a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00 a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0 s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850 s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8 s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2 t5 : ffffffc4043cafba t6 : 0000000000040000 status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f Call Trace: [] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 [] ret_from_exception+0x0/0x14 Dumping ftrace buffer: (ftrace buffer empty) ---[ end trace b5f8f9231dc87dda ]--- The issue comes from the put_user() in schedule_tail (kernel/sched/core.c) doing the following: asmlinkage __visible void schedule_tail(struct task_struct *prev) { ... if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); ... } the put_user() macro causes the code sequence to come out as follows: 1: __enable_user_access() 2: reg = task_pid_vnr(current); 3: *current->set_child_tid = reg; 4: __disable_user_access() This means the task_pid_vnr() is being called with user-access enabled which itself is not a good idea, but that is a separate issue. Here we have a function that /might/ sleep being called with the SR_SUM and if it does, then it returns with the SR_SUM flag possibly cleared thus causing the above abort. To try and deal with this, and stop the SR_SUM leaking out into other threads (this has also been tested and see under stress. It can rarely happen but it /does/ under load) make sure the __switch_to() will save and restore the SR_SUM flag, and clear it possibly for the next thread if it does not need it. Note, test code to be supplied once other checks have been finished. There may be further issues with the mstatus flags with this, this can be discussed further once some initial testing has been done. Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Signed-off-by: Ben Dooks Signed-off-by: Cyril Bur --- arch/riscv/include/asm/processor.h | 1 + arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 8 ++++++++ 3 files changed, 14 insertions(+) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 5f56eb9d114a..0de05d652e0f 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -103,6 +103,7 @@ struct thread_struct { struct __riscv_d_ext_state fstate; unsigned long bad_cause; unsigned long envcfg; + unsigned long flags; u32 riscv_v_flags; u32 vstate_ctrl; struct __riscv_v_ext_state vstate; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index e89455a6a0e5..556ebcbb7e22 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,6 +34,7 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + OFFSET(TASK_THREAD_FLAGS, task_struct, thread.flags); OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); @@ -347,6 +348,10 @@ void asm_offsets(void) offsetof(struct task_struct, thread.s[11]) - offsetof(struct task_struct, thread.ra) ); + DEFINE(TASK_THREAD_FLAGS_RA, + offsetof(struct task_struct, thread.flags) + - offsetof(struct task_struct, thread.ra) + ); DEFINE(TASK_THREAD_F0_F0, offsetof(struct task_struct, thread.fstate.f[0]) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 33a5a9f2a0d4..c278b3ac37b9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -397,9 +397,17 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + + /* save (and disable the user space access flag) */ + li s0, SR_SUM + csrrc s1, CSR_STATUS, s0 + REG_S s1, TASK_THREAD_FLAGS_RA(a3) + /* Save the kernel shadow call stack pointer */ scs_save_current /* Restore context from next->thread */ + REG_L s0, TASK_THREAD_FLAGS_RA(a4) + csrs CSR_STATUS, s0 REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) REG_L s0, TASK_THREAD_S0_RA(a4) From patchwork Thu Mar 20 22:44:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14024668 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1313C36000 for ; Thu, 20 Mar 2025 22:44:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+WRMTm5n9vjpjLY+cA0wBOTzvcLny4cuYR6T6NDo8kQ=; b=ax+cjzcsHZrmhr 0wZT/Z2Rbte0q7EAt7dHxo22nURoeZGjNu9tHMKBRwT3zAERuIxUAfwHKz/lbrXqjYv/0swHexv2O TjBnTuCwwRo3qd26mWD9MlwEpP7vPMMBmEIHGdDjHd/CpnQfq+ENCuiHo948GMSKpJOY8hTmwDsxZ ubx2pQsXrJg+ymsCdz1Km3oiQadNj6l2/x44LRHisGeTvui6X0ecDBrcMZeCNElEY43aUSF7IPs70 2eqE4Pi51eg78x2xUmw44wF2puToGaVGjXidwmQRUBFNqjweXMB2GtOiv3rVBNRQSyG1GZkpljyMu CdY6cZzjGaYiG8tg9Cag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcp-0000000DPHQ-3rLC; Thu, 20 Mar 2025 22:44:31 +0000 Received: from mail-oi1-f171.google.com ([209.85.167.171]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcn-0000000DPGO-29lU for linux-riscv@lists.infradead.org; Thu, 20 Mar 2025 22:44:30 +0000 Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3f3f4890596so755579b6e.2 for ; Thu, 20 Mar 2025 15:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510668; x=1743115468; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vvw7nYWiLBIqfTSpurYIsb0A2VRmihRadD2HqJzvOMM=; b=EC2e8KULGNAiZ7bBGLPT+kxW4n/FQ6dlinbjol0F35iLJ8ZhgwvXlXMP67Swe7O86j JtY4Ai0RFUsp2Ug3u8SDbMPyo1rTVLD7XRxhelLBxzGrIuTc7pFgWcHN8/E046YJeCoq L+pINYMNxuDpD4GuwMKtfiwB11D1uGn7aJwaWrQeokm0j81g7z6B7XEz5JyOOZ7XKQQ6 /cr/sKEyL7c18L2x9sjVfQB2g+EgcsTOom8aQea/KtOK7EJf20FzvgTd07HTPegrbc3w Sm+hyI12QewcqGhwy6VXTZjXIfWE7NRfMYTM39ljnuYM1ThAceuC+6Wma/QlMtt9WgbE KTlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510668; x=1743115468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vvw7nYWiLBIqfTSpurYIsb0A2VRmihRadD2HqJzvOMM=; b=Ob+d1StYdbHptRQ5zk5ES4hme9mdZVCMtlqqnLBX1QLDl9pgKoyWyvZeJOoBfUbSzi ESyRQReeVvLhKK4FOqQJDDfgygiXIz12AYRehtPY9PzjEzaNV3fDO0VL97qlhf0W0qyW mBkgQeJ71IMXVO11qsrKFhKVayKRt2jAKOJOkKBOx6Hh9GiwuGkXxgHjWeD1mAW39Ipy IfPRzqqPeS0mBJ0o/c4WIc1hJRyKKeX/O0EEzwqrTOssgOXoqRycVV10jcwtmjeKXEKj qfiiFQd7qcHRZZQQH/9YrlOz6pRSH1V83+xohYSHA7gNUohhIjmwagmkaa3u6mNb0nP4 x9zw== X-Gm-Message-State: AOJu0YzvDmEO0igH/N4Dreelji0tePdlD3kF5rYgimjfbcBJu2uK8VNS P75Yx11foyzdNlkO0aKVkGR/kE7M4y03QvI6URKxuAdmMbAlm9ldjbsd21UYuw== X-Gm-Gg: ASbGncukqW7ZDKK2TaUYLt/aP46S4uFchSvVeLcMvUDRnoOTI28MZuBXDt698yJjFFm MHyFucghWsBwnRm7xoImDP41mgWmyKl0Z+DsetZhCg05nwzQRV2rbmuQ51ppOdmQBQ5ay2ZrN+M qrbJjB+eYt3ZHYb+gKYdfDMu6adkAzg/YyRmleaSX5370iMUq48rZ48RawjCE7YBEJWE5NGpoDR TM73orKdWFdGpF5pIP7nZPWmkyjzzrykeG5PdDnxead4IUiYU6K64PYSOqFKT1E28LZl3k3dFt/ 4g/xSimCdfBTz/KYBttaEqr7Zrd9VCKlXaoNxCdxQgmERth4eqHEiZil3JnX/ro= X-Google-Smtp-Source: AGHT+IFJedQeXWVCp8Wwl18/3gDvFSbrwpqc13CUW17x0vVkYsJlg/z2sLYvBni60TXj6/O4eMplxg== X-Received: by 2002:a05:6808:1495:b0:3fa:d6c:cdb8 with SMTP id 5614622812f47-3febf79528emr596590b6e.38.1742510668153; Thu, 20 Mar 2025 15:44:28 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:27 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 2/5] riscv: implement user_access_begin() and families Date: Thu, 20 Mar 2025 22:44:20 +0000 Message-Id: <20250320224423.1838493-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_154429_554683_82E995C0 X-CRM114-Status: GOOD ( 14.76 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 76 ++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index fee56b0c8058..c9a461467bf4 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x, char, \ + __typefits(x, short, \ + __typefits(x, int, \ + __typefits(x, long, 0ULL))))) + +#define __typefits(x, type, not) \ + __builtin_choose_expr(sizeof(x) <= sizeof(type), (unsigned type)0, not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,69 @@ do { \ goto err_label; \ } while (0) +static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr, len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin user_access_begin +#define user_access_end __disable_user_access + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err = 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err = 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) = (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, op, label) \ + while (len >= sizeof(type)) { \ + op(*(type *)(src), (type __user *)(dst), label); \ + dst += sizeof(type); \ + src += sizeof(type); \ + len -= sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst, _src, _len, label) \ +do { \ + char __user *__ucu_dst = (_dst); \ + const char *__ucu_src = (_src); \ + size_t __ucu_len = (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_put_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_put_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_put_user, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_put_user, label); \ +} while (0) + +#define unsafe_copy_from_user(_dst, _src, _len, label) \ +do { \ + char *__ucu_dst = (_dst); \ + const char __user *__ucu_src = (_src); \ + size_t __ucu_len = (_len); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u64, unsafe_get_user, label); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u32, unsafe_get_user, label); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u16, unsafe_get_user, label); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u8, unsafe_get_user, label); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ From patchwork Thu Mar 20 22:44:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14024708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C838CC36002 for ; Thu, 20 Mar 2025 23:49:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2gUSezpNG+tv1bxmlSB8jSWXI6cM5+ITA0N549WY/jM=; b=Q8ROsgKxOsVglB tITCCy8Ra7bDY4OAS+t+jNpagrfgNAPRSBZPWPuTrl189Td0T/XSr3JhAI6fIj+jxxq1rQ45OK1kP REanc4gzziVJ2pJvihZQ6T/a3obB7AYjfw4sfZ9xQ3KE0U+y0xakZfUlDMuQ+RLtJbjWG2m1POOY5 u9io812xWYDNzk3KM6f5c4ilWlusWCgZzmc1wLdw4j9mYu6IJGOADBgsymi/Cc/nYP+gOZBqKWbsF aWZfjWo6Dx0Or6fWZ5N5GGhw4eEXdtyh7DMb6NGGW8qVpyD8D5hK/XJ4Xtz3sNAIBQO/MLaecYHDQ RJxt279aRtOA4G47cvHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvPda-0000000DUa6-2m3z; Thu, 20 Mar 2025 23:49:22 +0000 Received: from mail-ot1-x333.google.com ([2607:f8b0:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvOco-0000000DPGk-3kXo for linux-riscv@lists.infradead.org; Thu, 20 Mar 2025 22:44:32 +0000 Received: by mail-ot1-x333.google.com with SMTP id 46e09a7af769-7273f35b201so806331a34.1 for ; Thu, 20 Mar 2025 15:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510670; x=1743115470; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mh8UBXGzUbnvdsif2sw+KTYhRC1BSjBoml8SqEobyIk=; b=attosZ56Vexqq1I/ZpOSUCQeLwJDvNv/9dqJ8lvxe77f1iXJqCS6wRE5WRvdUQ8qZP wXdrTdcG2k6gmtDXTTYjRdpGc6kx6CbwuMBNJwhk6w23lz7DYvis9vlWu43N96HqVF+Y dBVjGluppvOB/RJVuO4NLks9GtYvdkVzQpz1raJDJq++boFqJGoIW7V7hb0BT0KVUcaD fQvdkhPM2X1K/vD864xh8vHnn7g+jElYn4Ojkubgn++s4hnjZHaV4no9gQ/j5Nny7XEO m5ct9TMEFFanMz+OQe47YqJWt55dwIdkbfligvDeikpu/MWPl16EXdTYG4Oi3m82eOTy BtDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510670; x=1743115470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mh8UBXGzUbnvdsif2sw+KTYhRC1BSjBoml8SqEobyIk=; b=ZtapUZIvOASYYOGQpQnPTYZpXy04ZLkTmZxytft8mUuEvyps/RrZuKWU6+SpSp3wd8 Y+Wv/Ao9ojqFsOCkezDVxXevjfJBEEZBp+xakdIzYCmGIE6Rsh0mszJ1F0mQPr/H/Llp NqFBY9/SFebZFrR3PSBn+hZia9cNg2oSwN2a29TrVS6YWXelj2F5Va/JYc/7rRGRB/Fe xbgG63+OiClnZ/rE3khC7bshXrZt/sNUssRVo0cnvsY4YQ0OWiIx7b6dE6caJ2SwWOh1 Wcg6Sd4+9g1zCy2KEGWXwoNIDwjy87aX/AYzo8NuNUaf0Yi8Vnv0FE+20nowJFl37mTu +bwA== X-Gm-Message-State: AOJu0Yw7qkci58brkJbfJr3LEtya/s3eubfu/+/zGpEjW+HH4of+xb6g 7YXiBeHZK5LNNSWPeM7uNw+mfpHpRpSIkmPYqnvic4laLIGqGNzppJFax5ufvA== X-Gm-Gg: ASbGnct8DQ5LiPkSC0awakfLouZ5nAmVHnU3M+Xshg1j7RqqQVlmH9fKnDbLK4tyN3Y 9BAMzuxZ6RLH+0y5FmEwZ/u2OAa1l3aYZvTKqlGoxSPXd9eXfwuBO1Nf5JFJ2C+E20eEgeuSLo3 Un+sASXcVyyEH2VCSEUTzL4UwP6I5rIemUfV3kpmusHY8fCNkLuI7MzvoWS6S1n3SKRkKt+0WdK BTqFi152IwS3LGpVYTdQQTOmiPD7AZlqFOWCQoKBd3XbzJRPH6q86CDXbU/cz5bdaC/015x2tgD hatCkbXvUPthIiLWIRhG0DwII+4SyJXHwF616uLlfufhQOvHqcQQYSWpM/sHTkJYBm8RwHx4UA= = X-Google-Smtp-Source: AGHT+IFPgB4lmDSuUaCrMpZ/2qFcRUoJ3Y+avxEgpvQKbDo7r4YIRe2aaoiARhfstm7Tp4du8Zb0Lg== X-Received: by 2002:a05:6808:1a12:b0:3fb:7ac1:512c with SMTP id 5614622812f47-3febf7481femr654995b6e.17.1742510669653; Thu, 20 Mar 2025 15:44:29 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:28 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 3/5] riscv: uaccess: use input constraints for ptr of __put_user() Date: Thu, 20 Mar 2025 22:44:21 +0000 Message-Id: <20250320224423.1838493-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_154430_934279_AD9ACA13 X-CRM114-Status: GOOD ( 10.13 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index c9a461467bf4..da36057847f0 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x = x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=m" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x = (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=m" (__ptr[__LSW]), \ - "=m" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ From patchwork Thu Mar 20 22:44:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14024670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6B7ABC28B30 for ; Thu, 20 Mar 2025 22:44:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6t0/V6mMYEILSXK1ctPKLebrJwflqHnsgplgtkAgTYc=; b=kpeV0g2H3ESk0g CNDKsi0H6NpVDI9WdqG2tnfFPBMwdkE8ntJuFNbs7hKCX26xH2ltjokXfgQRDZpab0bPRJf6iAI5t Kw5WNb/9hzs8ITPxFD8SqtajnXOECUB8CrUStLjWQ9ryyoBucLagR571rIrCpUD7LNrBKx73pip1d KqCeXPkSCGlMxmkOOQc5sJVTDFTQh3BFvSUk8RQ+br0hOEPsrnmfARa8HqOg7X9ptyy2Rsrm5YsAi q3v5TD1SZMYX0ogV7RbQx6hu6ReBkhaVl97R7G5sP1EzJIIf6MUVPx653pbNkZVgaNsLifSep35Lv LuRFXBZCyCYK1yXYC5xQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvOct-0000000DPK8-2ebq; Thu, 20 Mar 2025 22:44:35 +0000 Received: from mail-oi1-x22d.google.com ([2607:f8b0:4864:20::22d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcq-0000000DPHO-1Lup for linux-riscv@lists.infradead.org; Thu, 20 Mar 2025 22:44:33 +0000 Received: by mail-oi1-x22d.google.com with SMTP id 5614622812f47-3fea0363284so764395b6e.1 for ; Thu, 20 Mar 2025 15:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510671; x=1743115471; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DpC/LMch4vBncOIb4H+uNBUj+A3evJlilB0f5X4S7k8=; b=XsS5VimoAPB1DOlCJ5jFF6lJu4THEPdKoFhreD/a3h1nrkLKgeRf7ImP2GsFRooY4s XOIoaOk4/XtLwGsm2sbz+7pet4s5I90+yOjvDJs1HDnL+n7DtTWbgJPmGEt6rZJyVvyD 0ICfT9abuwJG4oMK9VzUP7OUErs6rM8ukcSFmiYFSQVJLPvBvz0wK2bFeoPMHhv7ORNz fntVL0jzzNn178RzVJ+Oh36cN4u7hOCVncEU2OYsKkuJd3ShbzY9KmtPji5JfiktE57T la5d5zf03aV593eyXduAwjsqd2aShrZ2jQsYZbZEYGyCCQzBhVatZQ0MMiNET8Uzz1tN rppg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510671; x=1743115471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DpC/LMch4vBncOIb4H+uNBUj+A3evJlilB0f5X4S7k8=; b=wOhXrdY1RCze4L3csWWhyE44zL2zvyczmcL9KY9mbYbJ4sGu+cyNGLDXfIk9RJjOBR hofS9k3wMlawUfxywYAJuUl7HRWHaLDSKOYYLUWD1LvYNVuymlv7zrZIXb43T8VfABUc 70WAXBdAeqHoauAIfsedSI3ynMoZ8jHSzwVqVeiDZ60P6FXRq5ey4ReKjwgAiILdUCqG 5QcTBFFJ3n3zCC4lK8g3shaYZqn5A+CT7FFL8jGRJ9L6PbZ9g/Q90Mn9X6q5SpVVGBFP rYyDI6BKdhjSVinIqxLF838jiRkh4Fo7zBROc7XUA7yIZb65Bf+aZtxqdTEpBlRymbyp N0zA== X-Gm-Message-State: AOJu0Yyp34jarvFUGZIontM4kLRRsSrLr8CK5hcJ56hIEPSuKTTkMNiS PQZd7SI8xpEDZv2jNDkw5lP4fGU7mJ4ILZOpVpoi9jucwGH80i1dlMmA8TfNyA== X-Gm-Gg: ASbGncsfVzaF3Ucn/MuJpAbJBKcergwTVm/v65WReJ35YG7B++RAIQO5fqHBsosiUmB V73jQvnNxbjYySpOysctpvuDFym3AZQ0TZjkKmyKRKLyfokKf2ZpINTikRGyzweJ5DIWOdC2R5W GbD2lBH+oEhPNCBVv/fJKmYiX5QNTOnJtf1L3KLH/v+7FGzDbvv4ChbyVRHK1MxhjA6faMZz/Aq nhhMwWcOOHRQDYKBkkEeGRU9Ew9ZtRIjsEhwYQrVShI08z0nI9CWd6b0OJkVUFMPyMbjJWIfsjX gmYSjIxnfSZvGPYkIXaw0D9JpXOwrO3smCK8oHKiCMhA0PV4pf69gqhfTtLiMQc= X-Google-Smtp-Source: AGHT+IE2wGE2Be/0SH2O70UOpsv0bSUloPlhYCg9sxzfrbSR5xy7GPcyoBTaN4dfYwWSgd3IGSKcUA== X-Received: by 2002:a05:6808:1814:b0:3f4:12a:8ca0 with SMTP id 5614622812f47-3febeda6d98mr810784b6e.4.1742510671178; Thu, 20 Mar 2025 15:44:31 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:30 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 4/5] riscv: uaccess: use 'asm goto' for put_user() Date: Thu, 20 Mar 2025 22:44:22 +0000 Message-Id: <20250320224423.1838493-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_154432_377371_EC243F66 X-CRM114-Status: GOOD ( 12.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 71 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 38 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index da36057847f0..719c9179a751 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) = (__force __typeof__(x))0, -EFAULT); \ }) -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x = x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr = (u32 __user *)(ptr); \ u64 __x = (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) = -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err = 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) { @@ -398,12 +397,8 @@ static inline void user_access_restore(unsigned long enabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err = 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) \ - goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) #define unsafe_get_user(x, ptr, label) do { \ long __err = 0; \ From patchwork Thu Mar 20 22:44:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 14024671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1140C36000 for ; Thu, 20 Mar 2025 22:44:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q9d+tiEUFER8rq+XHM0NWO7aMMmA+O4WQVugfBlnoRg=; b=vQGSicJb9FQu6O syDCSqpxCKqYX9qlVse2Zh2kjdXFzKJ4/5yb3iuqbvK33r0y2rkT/cIWFxuPwlUzB4zsOLhcWaesa 6fTA0djzja6QgZ+Wm+24dLSirIg3AxH1qN9IEhmcpcbS7aRoCsvfyFOZwkYE1keVRv3oOrfpe5/aO v2Ne/+7eRMlYGvvL8inj2aGwOUMeBIjqjbed8AZG0vAc/6GV/JFbM0W5nxh0DMdwAZOCk5i9WDHu7 Y4jsvBh4DUq4TkTTQGzAXpA/AOjr79U+3i327kMfpqHXSZA1D27cXYt7VNdsFjccAnuWxexKiw/RD LXawHawrYcpDsB305dcQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcu-0000000DPKn-1QwB; Thu, 20 Mar 2025 22:44:36 +0000 Received: from mail-oi1-x234.google.com ([2607:f8b0:4864:20::234]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvOcr-0000000DPIH-1usQ for linux-riscv@lists.infradead.org; Thu, 20 Mar 2025 22:44:34 +0000 Received: by mail-oi1-x234.google.com with SMTP id 5614622812f47-3fa0eb29cebso1475243b6e.0 for ; Thu, 20 Mar 2025 15:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510672; x=1743115472; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aUM4cL2icpLrYe67mQWolXHTOhfxBr7CeU0SRh8tC7k=; b=Vo+7IBC7ym/8ak8RESNXkL3GYQyvH8N8ufCvuXw3KnSEpiyumYxOEXVLPwNYZMGbsL /FGwpc8YOxRjsl0NarvCURaxI14y9LRsh0qABlsCr8aLXhKfAuik68on2b67MEIk4uqP A7f3lLPcUGns9VEePZsS7x9QHOQUhdH1andR87nJptEMb2zM1eo4hZ5qduF7srni1Fhx up3vAGd7A/6qtddiE5HyawQAo+EDGGnyX37y+BcmeoPq6raBMTO2TaZJSJhwRYYFKym+ ISTSIctBEg08Jq2UKX0gGeEpKx03oAUuk2kueH0ibNlfhUJ88MM7wJ+elWE+LZ8yzxx5 oSTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510672; x=1743115472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aUM4cL2icpLrYe67mQWolXHTOhfxBr7CeU0SRh8tC7k=; b=O57FueeOVxgq/798DS9gQxwBtHRQIAywxiyoi9TWnfx7QFT0nOYXBW6gdTYIDKzjtg BiJZ2BPl43AK9Yv06aJqSGN3xzoI9bJQP2cKpV09y1Cxrl2CoykpZWK7ZYpj/XFl62sF D3Q59xJMHWQDrcRjE8HZeXbDrnBBBx9H6xsecNLDRTOZkVwYQ/9hDpVSQRv1CxJzEC/R jFs/u/KuIs5YuJxtdaF4ZnFM4UPGiByxmMb6DO91CQyoOKWv7k84Hq4OubvaZPXEE8OZ XaSz7SDZ7nkKGj5aQZNPeKw1+iP3QAn1IchBZMYmIXbhmrqiJYj+Mx32VR6+nWr7GbbQ b7Kg== X-Gm-Message-State: AOJu0YwbDoBulnp5sne5FL6wRUA2PY6tWxSK4DAd6NWY+wLx5OMRx/wX 3SZOOMUc8b0VxRX2SUDBNTasg69ju7ypETsVGXx/51zq8+j+FqUYGA9lrpsK6w== X-Gm-Gg: ASbGnctA2Iu+YtKmxYMxTeEKDmR1pCLP+nvLb3Sr7NQPp2GeBI//VlRXoX+0fRJXvS9 9u6mhjBK/8iKdZ2fgflTZfXgJjCS0k4IT/N4BT9HGVkIeB+MRpof8mGUP7yjrf7X3hwbYxPVLvo ATKSSkp1TJ96uLpjHsRjOZTuzwtnHlXWz3wLl5A2FYnU9wvCK0Ok0/I0xShALcp9xh5d5aNg9ji uF7mrnGsX2hNW76zCu/gFVTlsR65yNCu/T74EbyVYu45U8pdhDsr0bKTwKt3Mf50tSyCj+vCQF4 CrWfqc7kaj4kRM94rw4Z97KK/lEGcU54WN8gcT0T0G24FYm+WZOJDy9CD/X35zY= X-Google-Smtp-Source: AGHT+IHChpwGgXIo7oPDkSwy3Z8aUFaMgcTSyEt/gVcKCBKfeFe7Livn0EfqfYGjYKQnsyWXdaZjLg== X-Received: by 2002:a05:6808:444b:b0:3fa:10b4:698c with SMTP id 5614622812f47-3febeea9cf6mr1033217b6e.17.1742510672433; Thu, 20 Mar 2025 15:44:32 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:31 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 5/5] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Thu, 20 Mar 2025 22:44:23 +0000 Message-Id: <20250320224423.1838493-6-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_154433_508064_E762D63E X-CRM114-Status: GOOD ( 12.42 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 95 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index 719c9179a751..8823471b201f 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,58 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne * call. */ -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err = 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=&r" (__x) \ + : "+r" (__gua_err), "=&r" (x) \ : "m" (*(ptr))); \ - (x) = __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr = (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=&r" (__lo), "=r" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label) \ + (x) = (__typeof__(x))((__typeof__((x) - (x)))( \ + (((u64)__hi << 32) | __lo))); \ + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr = (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err = 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +156,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=&r" (__lo), "=r" (__hi) \ + : "+r" (__gu8_err), "=&r" (__lo), "=r" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi = 0; \ - (x) = (__typeof__(x))((__typeof__((x)-(x)))( \ + goto label; \ + } \ + (x) = (__typeof__(x))((__typeof__((x) - (x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err = 0; \ + break; \ +__gu_failed: \ + x = 0; \ + err = -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +225,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr = untagged_addr(ptr); \ long __gu_err = 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) = __gu_val; \ + \ __gu_err; \ }) @@ -369,13 +419,7 @@ unsigned long __must_check clear_user(void __user *to, unsigned long n) } #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err = 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,12 +445,9 @@ static inline void user_access_restore(unsigned long enabled) { } __put_user_nocheck(x, (ptr), label) #define unsafe_get_user(x, ptr, label) do { \ - long __err = 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) \ - goto label; \ } while (0) #define unsafe_copy_loop(dst, src, len, type, op, label) \