From patchwork Mon Apr 7 18:08:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041560 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D9DBC36010 for ; Mon, 7 Apr 2025 18:52:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=7Y517yalD9pQBGYuvKrJ3RIwAbaxrP7pmYi/3Q4XyNk=; b=UMHz0KicLsnGQ4 hLVE5HY7rB93TtUW9RjxOAc4kHTCLXaFUecMFH09X/Fjs1hWucix8TEsnbUO5JRfggVv+Z0JNNXQq UJBoSjaciluEhHc0Terh6/2OSM1I4nIazmTdM8q2PK70D+ikHY8b4+n2Awzy2HMioTLh39RQYSFEo 0i2uOaCTThuZF9xDuF0uVpeeST3SKLLc6n3Y2tOPqyBRCB5CpWEMKYiXE/HuUvwe1VCxqrzhjsQKy xTe3Uwc/9cm36ioT1sBbJP3KF5mzNm23XJhuVzyL38EsUSOJMI+RocmZLqxGuM5u9KgotN9joZsK9 Glnsu9/cRItkPTb2Ek2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1ra6-00000001g6T-3ZoF; Mon, 07 Apr 2025 18:52:26 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1qtx-00000001TQL-3I2x for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:08:55 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-22548a28d0cso64539845ad.3 for ; Mon, 07 Apr 2025 11:08:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049332; x=1744654132; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=OGgUcQY3LpMQzwnxjPDiN9+XewxjYA0t31wP+uhnCCY=; b=P9omRbyhXCGuwNL+LVQDZscn2y8iFQ4z+MHrERx/goIKU7U53xqO2/TjMd3vluvmv4 H68H1ehwDGM/WPI2k+SQ5ryinypiHkHe8m+PONP95HFIBclODakpdftpKntOoRKTGMSf hwVdSaNfmGAzW0GQEDHEjdE3Ci7lbZZ27+NSh046msjg7E4dbLLcSi2uyzLjhdW6TrxS 8qpganVBypkOUrxOz5HZuafwrwXn/IOLpC5/Okup70zE1IEcqnbZwSB1A04VS+wdK2oq /C8/UwBI6QZytakG0mDxBDb2ho3eMnOb794VZokyTtpQnRZZXCB/TQtlY+wCRwh0Zg0u 3pIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049332; x=1744654132; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OGgUcQY3LpMQzwnxjPDiN9+XewxjYA0t31wP+uhnCCY=; b=WxdGe3yY2eEQaohYFqYhMPj74q+WHedFW7LBttMjIDszMhWFzq453d4X3641uk7lyC 6hrwabqubPpGI9y0QNV/hIOAmSAcVqu0FDinKMASs831RJ0wce55vf7tIloPXbFSagzB iwlb6oM6Pn0tT+9e7a6EfScBqUd4VP3ZIY8eHlSrEv0IrQQ8gelTUR0TRFxU7OWwH7Io C4NFV4KX1lmu00Hzzoc5XwQUwhZeYHFRDoYxrWCqdGIQWWyk6W6/80bp40i0VQ6/ZMOD 8AxuHXW/pYhSD9A0RuGPMQxtgIOd83ivGQA3wuq1gg7e7BB0PlkRVh0EK8pV/Yv3WV6y SL8g== X-Gm-Message-State: AOJu0YwWOxRLGbzEMibQvoQrXvI8GIG4KsrpUOVC3Klp9S4imAIF6Iva SfvEnhOmIhBX5tSRspHBOKljqSSximEvZoqAjFN45N8o3wm7njfEIUBiVYzj X-Gm-Gg: ASbGncuQcM8N15qFR/owppFzovdoQ1I2CJueKj/gR12DYhT6XmBR6U8UgV0pymlnK1V I/p22UIznwskarJTJs82XOdJKyw5POHwDuDPbE8H50cOwcfXWaGsaqNg8rkcMnLhkkfwhT6ERsa xGf4ojoX1SQgzZGnqmJeblZKgFvtpY9196iYOHeVPekU3a74n+RC9tdfJY1HJ0OjUVkfnzUrpJ+ cZIZNmZTPifj4Lm1BbvlZCgoeDeqO6GYZP2EeB/o9vNjEB2/6FHXIsDo0xLpy72ioiOtVnHNnIX 89P0OiKrpMk4pN7E+lG7pXhxQzPkD+BnVkVmuGzMD5tut03fc2EqYmv+MmRKEjG/J5/yvtCFYZ0 ftZkauP7t2bYSU4ckXhWLrY7+BftlMrf3fg== X-Google-Smtp-Source: AGHT+IF9o01fgdgPFq84Ese3rGK1eK3K1lWa1HP0jF4cDQEDcmyxsE30VYjB8V+k5xLVlJftQJwEiw== X-Received: by 2002:a17:902:ecd2:b0:223:52fc:a15a with SMTP id d9443c01a7336-22a8a8b80f8mr161642065ad.33.1744049332400; Mon, 07 Apr 2025 11:08:52 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.08.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:08:51 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , Evgenii Shatokhin , Nathan Chancellor , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Palmer Dabbelt , Puranjay Mohan , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, llvm@lists.linux.dev, Mark Rutland , Alexandre Ghiti , Nick Desaulniers , Bill Wendling , Justin Stitt , puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 01/12] riscv: ftrace: support fastcc in Clang for WITH_ARGS Date: Tue, 8 Apr 2025 02:08:25 +0800 Message-Id: <20250407180838.42877-1-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110853_835042_10715371 X-CRM114-Status: GOOD ( 11.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Andy Chiu Some caller-saved registers which are not defined as function arguments in the ABI can still be passed as arguments when the kernel is compiled with Clang. As a result, we must save and restore those registers to prevent ftrace from clobbering them. - [1]: https://reviews.llvm.org/D68559 Reported-by: Evgenii Shatokhin Closes: https://lore.kernel.org/linux-riscv/7e7c7914-445d-426d-89a0-59a9199c45b1@yadro.com/ Fixes: 7caa9765465f ("ftrace: riscv: move from REGS to ARGS") Acked-by: Nathan Chancellor Reviewed-by: Björn Töpel Signed-off-by: Andy Chiu Tested-by: Björn Töpel --- Changelogs v4: - Add a fix tag (Björn, Evgenii) --- arch/riscv/include/asm/ftrace.h | 7 +++++++ arch/riscv/kernel/asm-offsets.c | 7 +++++++ arch/riscv/kernel/mcount-dyn.S | 16 ++++++++++++++-- 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/ftrace.h b/arch/riscv/include/asm/ftrace.h index d627f63ee289..d8b2138bd9c6 100644 --- a/arch/riscv/include/asm/ftrace.h +++ b/arch/riscv/include/asm/ftrace.h @@ -146,6 +146,13 @@ struct __arch_ftrace_regs { unsigned long a5; unsigned long a6; unsigned long a7; +#ifdef CONFIG_CC_IS_CLANG + unsigned long t2; + unsigned long t3; + unsigned long t4; + unsigned long t5; + unsigned long t6; +#endif }; }; }; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 16490755304e..7c43c8e26ae7 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -501,6 +501,13 @@ void asm_offsets(void) DEFINE(FREGS_SP, offsetof(struct __arch_ftrace_regs, sp)); DEFINE(FREGS_S0, offsetof(struct __arch_ftrace_regs, s0)); DEFINE(FREGS_T1, offsetof(struct __arch_ftrace_regs, t1)); +#ifdef CONFIG_CC_IS_CLANG + DEFINE(FREGS_T2, offsetof(struct __arch_ftrace_regs, t2)); + DEFINE(FREGS_T3, offsetof(struct __arch_ftrace_regs, t3)); + DEFINE(FREGS_T4, offsetof(struct __arch_ftrace_regs, t4)); + DEFINE(FREGS_T5, offsetof(struct __arch_ftrace_regs, t5)); + DEFINE(FREGS_T6, offsetof(struct __arch_ftrace_regs, t6)); +#endif DEFINE(FREGS_A0, offsetof(struct __arch_ftrace_regs, a0)); DEFINE(FREGS_A1, offsetof(struct __arch_ftrace_regs, a1)); DEFINE(FREGS_A2, offsetof(struct __arch_ftrace_regs, a2)); diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index 745dd4c4a69c..e988bd26b28b 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -96,7 +96,13 @@ REG_S x8, FREGS_S0(sp) #endif REG_S x6, FREGS_T1(sp) - +#ifdef CONFIG_CC_IS_CLANG + REG_S x7, FREGS_T2(sp) + REG_S x28, FREGS_T3(sp) + REG_S x29, FREGS_T4(sp) + REG_S x30, FREGS_T5(sp) + REG_S x31, FREGS_T6(sp) +#endif // save the arguments REG_S x10, FREGS_A0(sp) REG_S x11, FREGS_A1(sp) @@ -115,7 +121,13 @@ REG_L x8, FREGS_S0(sp) #endif REG_L x6, FREGS_T1(sp) - +#ifdef CONFIG_CC_IS_CLANG + REG_L x7, FREGS_T2(sp) + REG_L x28, FREGS_T3(sp) + REG_L x29, FREGS_T4(sp) + REG_L x30, FREGS_T5(sp) + REG_L x31, FREGS_T6(sp) +#endif // restore the arguments REG_L x10, FREGS_A0(sp) REG_L x11, FREGS_A1(sp) From patchwork Mon Apr 7 18:08:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041561 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CB953C369A1 for ; Mon, 7 Apr 2025 18:52:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1+EAAgtsVxga89o31zieiBM+UvLheoVRVkDIL0mrA9I=; b=jeKfwQK/OcP/Ku YSwnlu8Lhc/flaik0gGWF7ClghML8dWOoLQ8K6bpvyb9dCT507Osekgrp/im+22+0IGFk7r9qEMJ5 uO4T7hemXlRgoAqSoXbGUYnOVcqXqERv+A1wXgVmJ4D3YXXGCCHrBGl1crenONROVY7Cu3iiv9jBc VfVhykf4ETju20eq51eKjRZ+U6K5Sfsl2qZ0Kzhlbyz3vS/Y8ktyWPAo9uLNfXtnFUMHe1SW2KGz5 SG2LzL8f+Y+WIc5VY51viwgBQtPKT9QGn4vTGyv7k3MX0QQwPHZJqNYWdxdALswLFQ3Ipup5h//vC Abf8r3iMzEFski+LcWJg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1ra8-00000001g9p-0wr4; Mon, 07 Apr 2025 18:52:28 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1qu2-00000001TRS-2hCx for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:08:59 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2255003f4c6so42069385ad.0 for ; Mon, 07 Apr 2025 11:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049337; x=1744654137; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6GxvBXScR5G8/guPuwFtybfdLT7Xp8U7BWpl4kN1kr8=; b=mCTleirEoYBIUBwqraA8pgSYYd3jD3TS296CDw2JOBihcnIJrEHcIHqd47CZZkFLyj L1ySbWcyUI5dsw9wbnfbzr5rpui4DrC3tiA7kYkCjuDocGtz5gs+9P3sxdjozTguYbql tieIpiioeNbTtJq4MuXme2AkWtQX3HMD0di809hyZVhMk7EIDlLKeb8RW626ZtdzFU2u BIK7lQ0LUpWdLNLonVIvkVxwS4W1nnxqc9ta52mHXPMECSQjXU93Tjp0Hj7QqrI72XYC kcY4/MPZ75iFyyR1mksLQ1x+vZh4qo6+4CmBK8P68tTjZZJ9aBmlqb+Y6S3R789l1aT5 iM8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049337; x=1744654137; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6GxvBXScR5G8/guPuwFtybfdLT7Xp8U7BWpl4kN1kr8=; b=jpTMRTu+2mZArCmVKXKz/HFSwTwBc0VommzIwy3DGbMk7rV2PNwL64/GQVUmJic/sC Vx0ETmOvSXqCbl2n0jxYvxdxJewzXjnbocfL0iLsXvtq9YZbn5T7fIke+t2qBsN36Ynz pIxMsoQ3cFa4DDI9dZPVqBroqTQSLEwV9AuHOQZ+SdTAAAof+EzwKM2ic/ZG81Vlh/HG 66jhCZW14mkMXCu2hktPE84sWF7KAla//Yr4D2RrSme7Y8B217DG1pHurihlsiVM6e2e cYjI3w1NI0cAn0jMXEnJ8bREqHFi+ulcLVJXYOjPxHyX9F27EDFjnx/ZIguDHOc2Fqrm RGxw== X-Gm-Message-State: AOJu0Ywv0T6KcExWRtmXFiEGgp5cdzcURu8AU5Im/MyEEkR+BjmRYx6H TSFQEv8LVaTVEEynN8TkU2CeI/tZkJGA8p/UtS6KcYL8Ok8ykWlPxNTracVj X-Gm-Gg: ASbGncvCkMyzb16yBln7ceyEywxEVtjOJf+E3B6m2ZC/ze9RHK61fMZ3uEn3wm8GT6P y63NUWrF8mgFbuqGhrRSE6iEXo7UaHwrvMg0eDQZ2q8zyNeS+AaSLv3SVZWQQilgbySZe1F7uU6 HHzzw1upypnYwl0RUZXBMgcDoEHsNLIIizZZwT3gDGW/UIYWiWINNw/AecglUXXyv7EadAnKn5n sFi7OvMRqfXiKbyI621+C5VzErlDirOYDqo5Rm7VdSQuBirjzXvADnPAQ2CXtbzfGZDWFowOfVa 3qDZjwCbFprZ9PoU+ADqksDM0YkNmgACUzcd2uqaHAG/1f9UzewmJm1kaoNSwvTyJsgKXpr/Tgf jxp2kGJOdnnt8hO7ha2uyeaKaBYi/69Gobg== X-Google-Smtp-Source: AGHT+IEWDNqLKG23YHG2EaPKmPmMuBuN2vxb7Eb0UfS1Xfx3xzd34GxBwfA03Pt5nMI3Pt2fEK34Sw== X-Received: by 2002:a17:903:230f:b0:223:517c:bfa1 with SMTP id d9443c01a7336-22a8a0a3a59mr190569785ad.38.1744049337341; Mon, 07 Apr 2025 11:08:57 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.08.53 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:08:56 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Mark Rutland , Alexandre Ghiti , bjorn@rivosinc.com, puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 02/12] riscv: ftrace factor out code defined by !WITH_ARG Date: Tue, 8 Apr 2025 02:08:26 +0800 Message-Id: <20250407180838.42877-2-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110858_679937_4CC35713 X-CRM114-Status: UNSURE ( 9.32 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DYNAMIC_FTRACE selects DYNAMIC_FTRACE_WITH_ARGS and mcount-dyn.S in riscv, so we can remove ifdef jargons of WITH_ARG when it is known that DYNAMIC_FTRACE is true. Signed-off-by: Andy Chiu --- Changelog v4: - Add anew patch that makes ftrace and its asm easier to maintain --- arch/riscv/kernel/ftrace.c | 15 --------------- arch/riscv/kernel/mcount-dyn.S | 34 ---------------------------------- 2 files changed, 49 deletions(-) diff --git a/arch/riscv/kernel/ftrace.c b/arch/riscv/kernel/ftrace.c index 674dcdfae7a1..1fd10555c580 100644 --- a/arch/riscv/kernel/ftrace.c +++ b/arch/riscv/kernel/ftrace.c @@ -210,7 +210,6 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr, } #ifdef CONFIG_DYNAMIC_FTRACE -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { @@ -231,19 +230,5 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, if (!function_graph_enter_regs(old, ip, frame_pointer, parent, fregs)) *parent = return_hooker; } -#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ -extern void ftrace_graph_call(void); -int ftrace_enable_ftrace_graph_caller(void) -{ - return __ftrace_modify_call((unsigned long)&ftrace_graph_call, - (unsigned long)&prepare_ftrace_return, true, true); -} - -int ftrace_disable_ftrace_graph_caller(void) -{ - return __ftrace_modify_call((unsigned long)&ftrace_graph_call, - (unsigned long)&prepare_ftrace_return, false, true); -} -#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ #endif /* CONFIG_DYNAMIC_FTRACE */ #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index e988bd26b28b..3f06b40bb6c8 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -56,8 +56,6 @@ addi sp, sp, ABI_SIZE_ON_STACK .endm -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS - /** * SAVE_ABI_REGS - save regs against the ftrace_regs struct * @@ -149,36 +147,6 @@ mv a3, sp .endm -#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ - -#ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS -SYM_FUNC_START(ftrace_caller) - SAVE_ABI - - addi a0, t0, -FENTRY_RA_OFFSET - la a1, function_trace_op - REG_L a2, 0(a1) - mv a1, ra - mv a3, sp - -SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) - call ftrace_stub - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - addi a0, sp, ABI_RA - REG_L a1, ABI_T0(sp) - addi a1, a1, -FENTRY_RA_OFFSET -#ifdef HAVE_FUNCTION_GRAPH_FP_TEST - mv a2, s0 -#endif -SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) - call ftrace_stub -#endif - RESTORE_ABI - jr t0 -SYM_FUNC_END(ftrace_caller) - -#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ SYM_FUNC_START(ftrace_caller) mv t1, zero SAVE_ABI_REGS @@ -194,8 +162,6 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) jr t1 SYM_FUNC_END(ftrace_caller) -#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ - #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS SYM_CODE_START(ftrace_stub_direct_tramp) jr t0 From patchwork Mon Apr 7 18:08:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041563 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C35FEC36010 for ; Mon, 7 Apr 2025 18:52:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jtSBOLzYvr/BUoKkfnhYFzVroa46KhjhO9zV8LFYa1M=; b=BdQnyl4Sdhcgr9 1XM+ved5oU6fpE3lJge+x2fcsuyS0Xb2kr1qsoxEwp4nf2z3DSEPJ5RbYi0NoXLosTOjHmz/KewxI eueFgtthWggnDNVHDKn1uCZS30WAYMUx0fKvnMpOqqyJs2zTVbggfhXsBSMpUYX3DxUVAG+mpdMEQ aksXspaE8osBa4R/iTOeTgkSWGREu3tShutlmklrr6sBVRwsOsEwUg0EBi3NlN8kDeTbFznstsAkK AlzEHA51LZCHVKjLkS/XgRe5O0QA8G81LASiYSh4Nv6Uw/8r14TyoGg3tAoM8rCgNTsu4OW9uVkFo 7QtP6uuXkD7Zmhi3zuoQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1ra9-00000001gD1-39FN; Mon, 07 Apr 2025 18:52:29 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1qu6-00000001TS4-3LTl for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:04 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-223fd89d036so54977725ad.1 for ; Mon, 07 Apr 2025 11:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049342; x=1744654142; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yUF0nsnlw7RZ2EpqP50Kib/oVdoqhGZKkeMwJl9c7jk=; b=WE0vuWI1ssFrE8GGJOjWVD8JTmSp7SpanSBQKM/WkWv19mC/oRHMuAmt1CiLP5TtMj ZpP4KBvIP2hTZYICE/Ukt5UBn918CUCs2GGV4Zk7lCoaQvJh6YTzmvQUsh1S08rG/w8C AVnkSY6cbs4toWP3Ca/CIXkDQO0jInp92o7U/2LA0DkqMRerN9/HR+RvIyf0082TDkn1 6rd6jNE9ndqkeH/M3KDuG57wHOEnKA6eB8UynMUwomF7ogLfFt7Qt/NBiy4YmSa3Dgdk 8Y0R/FFtmR5TIw41+yFM86xaqIeJmOaQ8nWvaYJkfQc5bnMqxu2uQDOPoGuOACjUNj+Z G0XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049342; x=1744654142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yUF0nsnlw7RZ2EpqP50Kib/oVdoqhGZKkeMwJl9c7jk=; b=Srlk/khKE/pYKfUdYtEmaqjA2wezV0FeirvImZaWyfwtpZXLoVArNwKlMvakzPxyOv lAoibtrnlbEgKjAbM5OP7aYAtKb3N280u3IhpVQ2MD/1+H7q4yViNmUkWDxEliADv8jS 2oGh6i3FQFzHOIZmHtjYQ7LswJ2nezMd52IJt9eMhyAg/jINurNZBtbt0DJlHYyzXmt6 a3gG0b1l8QKDhRx8akBrfcMPystRpmuazYq0PTlSB/ah6ad86pCgSSyN72nspMNT9j9j bP7J6PtCTX+jbP56m/Q9c/IFvqeFxDD7K1xWxTPCYvNZh7vrROVU1CTAIoS2Sz8CziH0 kKpg== X-Gm-Message-State: AOJu0YzOJ5qoI1ingSsYECZPTRfqzcYHnx3gV0VytelQqYaGbAugcdsv cwQQikWjNpftigLpEwfKqgf0N+nw13SqedQPehprESngaQPKH9ORBjpuPB9c X-Gm-Gg: ASbGncsuaMEAihZPbgLyHr3VBiqcDWPWxjhM/46DPa7nfgVNrQdymgke3mvW67YZeLX N290DW2XiQWhKG5vmLXZbiVZbS39r/QHVMODozt8maeqMrWBeW6kq4OwlUQMFYV7ysYwb/hXAm5 IzN0TMomzvK5X9uQK0HDd4dKift8Ap5gwX/qIVd7kgKuZTZKEcxTAYusPNn9dgp5bnVcZGxYK6h 3GXu0+x5vWOoenuCFrY0eEJUjXCw1KaaBGy3ynOJkIRFwmQgq3GmR3T91djKSGx7D76yehths3B apgWA7HK+zqyjagBASRZ8XIU0X9ijg4kT1M9ssdWZkx9kSp70ZyEYzge0omUJy3IFe2tx32i12U M9ZUwMfV4OJO40hV1xFPgShI26h5/yMHR2PFrzbIfhwtA X-Google-Smtp-Source: AGHT+IGoRDgmxduZEMyRWH4rWtb7Cl9qxilgGQO3cjliR6W+8rbaesJEGRduh+4SldUK1SWSm3N/ow== X-Received: by 2002:a17:903:19e8:b0:223:4e54:d2c8 with SMTP id d9443c01a7336-22a8a058646mr194838985ad.21.1744049341737; Mon, 07 Apr 2025 11:09:01 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.08.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:01 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , Evgenii Shatokhin , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, Alexandre Ghiti , puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 03/12] riscv: ftrace: align patchable functions to 4 Byte boundary Date: Tue, 8 Apr 2025 02:08:27 +0800 Message-Id: <20250407180838.42877-3-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110902_834349_978D5268 X-CRM114-Status: GOOD ( 11.64 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Andy Chiu We are changing ftrace code patching in order to remove dependency from stop_machine() and enable kernel preemption. This requires us to align functions entry at a 4-B align address. However, -falign-functions on older versions of GCC alone was not strong enoungh to align all functions. In fact, cold functions are not aligned after turning on optimizations. We consider this is a bug in GCC and turn off guess-branch-probility as a workaround to align all functions. GCC bug id: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88345 The option -fmin-function-alignment is able to align all functions properly on newer versions of gcc. So, we add a cc-option to test if the toolchain supports it. Suggested-by: Evgenii Shatokhin Signed-off-by: Andy Chiu Reviewed-by: Björn Töpel --- arch/riscv/Kconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bbec87b79309..7dbed10843d2 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -150,6 +150,7 @@ config RISCV select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_CONTIGUOUS if MMU select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE) + select FUNCTION_ALIGNMENT_4B if HAVE_DYNAMIC_FTRACE && RISCV_ISA_C select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_ARGS if HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_GRAPH_FUNC @@ -236,6 +237,7 @@ config CLANG_SUPPORTS_DYNAMIC_FTRACE config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=8) + depends on CC_HAS_MIN_FUNCTION_ALIGNMENT || !RISCV_ISA_C config HAVE_SHADOW_CALL_STACK def_bool $(cc-option,-fsanitize=shadow-call-stack) From patchwork Mon Apr 7 18:08:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041562 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 36E77C369A3 for ; Mon, 7 Apr 2025 18:52:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TX1DyaO+I4J3sO3/WgTSrOrjSv4iUqkyO9nqa++HZZM=; b=dwCm/IsHiJckZX CkCsZ61AY0mJC0QgXx4kJUqmFUlEiJn3wVVINxu2ZiLXwIKA6eBN2xeRHIiKzQsfvrRKXupQPX0vd v76uVQwGk3dEXTQR9xVD5cq1eDgFEMaYt7aMsqarhfpl9wSVQwPtFa7ehv4BxrQmH1NEEnnrFQ6bY Afb4yue3Lca85wSuY0Ldg/Vht8l+xvUg9CaWkvYnDo2uaXUH1OBpqH9ijEHVj2D1zW0q5oXeEgM4/ enHhtFUe3QpdhddPvF5Y41tITSw4OcOLWH18ir1M8GfsP4WWor4FQb4tiiaDgT1K7nuW1X2Uc2ggW 9ZkJlLgMIAQCxW6BjPmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raA-00000001gFF-3rYU; Mon, 07 Apr 2025 18:52:30 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quB-00000001TSV-1WXX for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:08 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-225477548e1so43792075ad.0 for ; Mon, 07 Apr 2025 11:09:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049347; x=1744654147; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sO5QthxwGi/gITP2al2MfjRxPqfuU0a+mulRSQDMGlk=; b=Slyo4MMPC4Nj7QiGXPoKy35YUUfbLAc3iUp59jeSfrkhjR92NDfYkVimxK44DfjYtE x9U7Idm8amOdZNa/8FuNLu5VH8M++GLTpQjkthZV946CLtaUXrN6F9+Y04EnzXppxuk9 YV43pfV/yZNCiiRCsrEiz41nBe65S2qLlUgM2BlkGuq1XqppzyBvW2GTWJHR8tPz7gWJ TWILdTnsnHX36Svfj1XKmDstWkncxZDFNSK4VzdvD7fvwBAvwNkcF9zMzdIF8RXETF48 WtwRjYsYyZCeUj1mloxto/stSk7AIG7Qo1HuTeZprzn3iRYg5GvEIT50TRI6bcGNzfUN wtPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049347; x=1744654147; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sO5QthxwGi/gITP2al2MfjRxPqfuU0a+mulRSQDMGlk=; b=Zwr9dWBWnmwBcvSe1T397UUi5tjNRzOBDk+ZBKfdMy/tZqGSTZ+VAHP8Rg+wttj2Kj x48k1jAzM5O511dFb+lRnBOHO0laSdc8q0XHPxB9mNMTrmK77L8klN/VLKZOZRZdPE13 x+4kStq4Npspa+6BoDyd7gD7ysur5YewuuD7aFTB0hghtD9//O6y9fI/KRpVgFaEvJL4 YbQUG6+NOa0JTTDXNvy7H00Wr9zNwtVXfj7Ndq29Wbvfn4F9noqSPQ1A2TTCmr60pGIQ n8CESWP0S6+BFdmfyJ9kbm+oJwuXdefP1JlAv/X25lUxwB0uLIXJKK0SOiReMiWDYW0L Ya4g== X-Gm-Message-State: AOJu0YylAkyGo1kKsjSNpLQ2fm8lRxTHwKvkz+d50juolk91o1oz4vub l8IOVcQ8eF0EpNpr01bC2+1H7crhNVzKqFib77pjArOP/PjmBuo/Adjya41E X-Gm-Gg: ASbGncuNdy3yPUCDIXRnw/WqEbevY2lYzkjZCd3yttJWAHuTDVg42G6SBpPUwXqHUAT ypcVNvXVQe0yI2JurzhrXK31DaZDyO6efYMZbXTSiYsChxVUea80Vxi1m/YwVzCucgNeUX2dD3L 669BH7qrAMa/wkIrTExhuJA/pUYpHBmvTD8RheqsB4TLm/UnTFkBm3xnSqTScIjtGTqO8kp6VEu QXb+F9eQlGIuZ/zVd95Cpi2jL2fUFzoxgwxZuaVk0NAsZOReBB+evJ+Mw8HovjpNoqkO882FX5l CyyDBi/zt6xNan/BM4Kj+p1GARgTT6QCkOD50GzfsgiCCbIHQqXDbLLPT6/Lufa+TNM04tdPuQt aHFJxsAOzHFzpJOH001ci4dYY6cOiZjW0hg== X-Google-Smtp-Source: AGHT+IEQoZKg5b+MyAfF5MYRIteTGVeC6PLB8bj9Qd+rDjzyW2lPRPeTqlPF4OleAXMNaZHG2RYIiw== X-Received: by 2002:a17:903:283:b0:21f:1549:a563 with SMTP id d9443c01a7336-22a8a858f64mr170136425ad.2.1744049346653; Mon, 07 Apr 2025 11:09:06 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:06 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Mark Rutland , Mathieu Desnoyers , Alexandre Ghiti , bjorn@rivosinc.com, puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 04/12] kernel: ftrace: export ftrace_sync_ipi Date: Tue, 8 Apr 2025 02:08:28 +0800 Message-Id: <20250407180838.42877-4-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110907_399189_78040C9A X-CRM114-Status: GOOD ( 11.48 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The following ftrace patch for riscv uses a data store to update ftrace function. Therefore, a romote fence is required to order it against function_trace_op updates. The mechanism is similar to the fence between function_trace_op and update_ftrace_func in the generic ftrace, so we leverage the same ftrace_sync_ipi function. Signed-off-by: Andy Chiu --- include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index fbabc3d848b3..0d4eec574707 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -807,6 +807,7 @@ extern void ftrace_call(void); extern void ftrace_regs_call(void); extern void mcount_call(void); +void ftrace_sync_ipi(void *data); void ftrace_modify_all_code(int command); #ifndef FTRACE_ADDR diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index ee662f380b61..d06bd4a046de 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -188,7 +188,7 @@ static void ftrace_pid_func(unsigned long ip, unsigned long parent_ip, op->saved_func(ip, parent_ip, op, fregs); } -static void ftrace_sync_ipi(void *data) +void ftrace_sync_ipi(void *data) { /* Probably not needed, but do it anyway */ smp_rmb(); From patchwork Mon Apr 7 18:08:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041564 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CEE4DC36018 for ; Mon, 7 Apr 2025 18:52:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9gWo4eDzgdTC3Pk/WN9UXhWA1GI2AzjNTnoTOJ0pD1c=; b=2bn4cDPPyHCY0y FnqKopo6PMri1GuXRqO4KGdfx/rWc1iXh9OMLcnTmTMtJ6lNP5AC96kCvWsd4GtI+gmRC1pGEH+Fv NRO8c0jxYSZieMPdKrfaVj+/g4gKLjg5C2Kpfq7AnA5pCqxHNlG7+njpo6lTcjPNeZ4GatkZNrPi6 D83oruSZ5fDYMV/O5tJKhTW3/7VQ+uektZlHXFu2JppVta1D7e/EKzDMzKFQt7jE7/OVxACOvt7Vl iwRMu9c8Dr01hh4EiqeF+A6aWpqD8iNn4nAIwFjrcrms9SlteXeanLS5PG4MW912uX1G4b9x2GjVH 9qxD5HiR/X4egM1J/MGw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raC-00000001gHW-0EVh; Mon, 07 Apr 2025 18:52:32 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quH-00000001TTS-0yMv for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:14 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-22622ddcc35so63392485ad.2 for ; Mon, 07 Apr 2025 11:09:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049352; x=1744654152; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CKIiqtyOydsKt6aFuPMQpVjfhS5wnmRKxjZ5aj4qfCI=; b=Uq7U2OM5Aknuc01n80fweDh48JGIWHFV2+z1oKPsUX3lTLNauMFIKpTXGoXovDkovy +KI8q9hnxhTgJ91qv+0hNcUy1xEY7/0G5twsD8CvHKsBLM8V6Il8EKY6529E+g7q2vh3 pozafoxYjpgJ+lZ/csn1WPjZJ+U7iw2Hc3P+jD00sKKWrQ+MZwPuF+OOPSSknENWfXiA gilvoZuDRgnsDhSOnePw5FQSfOSrDgOhyRjiTN1PjmA1GCtAx0GO8WnMnG+FuoBntWpR /jmZ1ST3Ep+qx3goK1nCP3vPz0dGMryKR4lQkNQ2hm3XkxDEEDsANkaqdopoF3eeuk46 +Xsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049352; x=1744654152; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CKIiqtyOydsKt6aFuPMQpVjfhS5wnmRKxjZ5aj4qfCI=; b=TBZTFHW+KkQjIn687UyQB5EbOLF7ksS+ZwF04yvrzfWVSmRtqJt7fkQFpzsoNWF/zo ezWZ0tPqWo9a4+3ofqkgaS4hVjIhnprDWCj09sj7tzg21BRRj4tALUnr5mwe7BasycqH W6qbeaZZ2vO4OWege6X3DEPZZQlmRcdTw+lcIBRNWKjOGJgx+PGpQqQfYPEE5gxFbnsG MmonO2e06W3czrPHu4SaHTzBbFgd++6Z1XMNKU5IkSL2GVmmlyYMzDA2KUebz3DdPvHX MhbnY4PQSUX+lfM3reymH4yyXn/frI3o3l9CeMpd9sotJjrNfVBu/EKkM997d4G/b86o VBfw== X-Gm-Message-State: AOJu0YwivLIJa+OEnhvdDi0FcYLk98o6UEePb4M/OT6ivzkpUgLHyqfT M3WaBwHfALG4g0oKQF8J6Rhx5982l4vvycMnQOiK5ZIxawY2OkDACs0GR2Hn X-Gm-Gg: ASbGnctCkxuvjRYZcMcmojMssLUpOZ7K7prbyuSmlwmAjjSIxHk+UmppfIE58YpggU5 0x3Qkvt6jU/8JDgIjox53afb0E/7kNV41P3lMTpHuUHPVqJwp9nI4GlrE7CKxlDyWofSR+lq2KS b7Sqwa08UK2Hc6xKu7hVJ4kbFNTFOSJXgKwSsN269543V9SGKt161vLW6HThYehxZP5DpBt4Wjr Fv1z5f4w0QwuLKub8hBtPmq6Kp/Io+d/wPrK7UVkm4ASy+eM5DNAhpDzgoDroy3Vzqxxh2O5/P/ urxwcQ+0ctHyV+aZefY5WcRe9+VPWUQR11bZ8E7o2b+Hv0n6OM6Y2T2z3WrGVsjlpvpqHh4uvCt cle9O3wsFzMNPisnwXDSL6TL4/7j2Va6Rgw== X-Google-Smtp-Source: AGHT+IGpl2ZX/KJd8sDDwi4VaUQ2lbuaEclQ6Z2k9mbwyZpgmsomMKjEgg3NkOT9ap9NMbGdOVe6Ig== X-Received: by 2002:a17:903:22c2:b0:223:6744:bfb9 with SMTP id d9443c01a7336-22a955738c0mr164235845ad.41.1744049352095; Mon, 07 Apr 2025 11:09:12 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:11 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Mark Rutland , Alexandre Ghiti , puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 05/12] riscv: ftrace: prepare ftrace for atomic code patching Date: Tue, 8 Apr 2025 02:08:29 +0800 Message-Id: <20250407180838.42877-5-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110913_277775_6CC2EEBB X-CRM114-Status: GOOD ( 31.91 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Andy Chiu We use an AUIPC+JALR pair to jump into a ftrace trampoline. Since instruction fetch can break down to 4 byte at a time, it is impossible to update two instructions without a race. In order to mitigate it, we initialize the patchable entry to AUIPC + NOP4. Then, the run-time code patching can change NOP4 to JALR to eable/disable ftrcae from a function. This limits the reach of each ftrace entry to +-2KB displacing from ftrace_caller. Starting from the trampoline, we add a level of indirection for it to reach ftrace caller target. Now, it loads the target address from a memory location, then perform the jump. This enable the kernel to update the target atomically. The new don't-stop-the-world text patching on change only one RISC-V instruction: | -8: &ftrace_ops of the associated tracer function. | : | 0: auipc t0, hi(ftrace_caller) | 4: jalr t0, lo(ftrace_caller) | | -8: &ftrace_nop_ops | : | 0: auipc t0, hi(ftrace_caller) | 4: nop This means that f+0x0 is fixed, and should not be claimed by ftrace, e.g. kprobe should be able to put a probe in f+0x0. Thus, we adjust the offset and MCOUNT_INSN_SIZE accordingly. Co-developed-by: Björn Töpel Signed-off-by: Björn Töpel Signed-off-by: Andy Chiu --- Changelog v4: - Include Björn's fix for kprobe - Refactor code for better reading (Robbin, Björn) - Remove make_call_ra and friedns (Björn) - Update comments to match reality (Björn) - Drop code defined by !WITH_ARG - Add a synchronization point when updating ftrace_call_dest (Björn) --- arch/riscv/include/asm/ftrace.h | 49 ++++++------ arch/riscv/kernel/ftrace.c | 130 ++++++++++++++++---------------- arch/riscv/kernel/mcount-dyn.S | 9 +-- 3 files changed, 92 insertions(+), 96 deletions(-) diff --git a/arch/riscv/include/asm/ftrace.h b/arch/riscv/include/asm/ftrace.h index d8b2138bd9c6..6a5c0a7fb826 100644 --- a/arch/riscv/include/asm/ftrace.h +++ b/arch/riscv/include/asm/ftrace.h @@ -20,10 +20,9 @@ extern void *return_address(unsigned int level); #define ftrace_return_address(n) return_address(n) void _mcount(void); -static inline unsigned long ftrace_call_adjust(unsigned long addr) -{ - return addr; -} +unsigned long ftrace_call_adjust(unsigned long addr); +unsigned long arch_ftrace_get_symaddr(unsigned long fentry_ip); +#define ftrace_get_symaddr(fentry_ip) arch_ftrace_get_symaddr(fentry_ip) /* * Let's do like x86/arm64 and ignore the compat syscalls. @@ -57,12 +56,21 @@ struct dyn_arch_ftrace { * 2) jalr: setting low-12 offset to ra, jump to ra, and set ra to * return address (original pc + 4) * + * The first 2 instructions for each tracable function is compiled to 2 nop + * instructions. Then, the kernel initializes the first instruction to auipc at + * boot time (). The second instruction is patched to jalr to + * start the trace. + * + *: + * 0: nop + * 4: nop + * *: - * 0: auipc t0/ra, 0x? - * 4: jalr t0/ra, ?(t0/ra) + * 0: auipc t0, 0x? + * 4: jalr t0, ?(t0) * *: - * 0: nop + * 0: auipc t0, 0x? * 4: nop * * Dynamic ftrace generates probes to call sites, so we must deal with @@ -75,10 +83,9 @@ struct dyn_arch_ftrace { #define AUIPC_OFFSET_MASK (0xfffff000) #define AUIPC_PAD (0x00001000) #define JALR_SHIFT 20 -#define JALR_RA (0x000080e7) -#define AUIPC_RA (0x00000097) #define JALR_T0 (0x000282e7) #define AUIPC_T0 (0x00000297) +#define JALR_RANGE (JALR_SIGN_MASK - 1) #define to_jalr_t0(offset) \ (((offset & JALR_OFFSET_MASK) << JALR_SHIFT) | JALR_T0) @@ -96,26 +103,14 @@ do { \ call[1] = to_jalr_t0(offset); \ } while (0) -#define to_jalr_ra(offset) \ - (((offset & JALR_OFFSET_MASK) << JALR_SHIFT) | JALR_RA) - -#define to_auipc_ra(offset) \ - ((offset & JALR_SIGN_MASK) ? \ - (((offset & AUIPC_OFFSET_MASK) + AUIPC_PAD) | AUIPC_RA) : \ - ((offset & AUIPC_OFFSET_MASK) | AUIPC_RA)) - -#define make_call_ra(caller, callee, call) \ -do { \ - unsigned int offset = \ - (unsigned long) (callee) - (unsigned long) (caller); \ - call[0] = to_auipc_ra(offset); \ - call[1] = to_jalr_ra(offset); \ -} while (0) - /* - * Let auipc+jalr be the basic *mcount unit*, so we make it 8 bytes here. + * Only the jalr insn in the auipc+jalr is patched, so we make it 4 + * bytes here. */ -#define MCOUNT_INSN_SIZE 8 +#define MCOUNT_INSN_SIZE 4 +#define MCOUNT_AUIPC_SIZE 4 +#define MCOUNT_JALR_SIZE 4 +#define MCOUNT_NOP4_SIZE 4 #ifndef __ASSEMBLY__ struct dyn_ftrace; diff --git a/arch/riscv/kernel/ftrace.c b/arch/riscv/kernel/ftrace.c index 1fd10555c580..cf78eef073a0 100644 --- a/arch/riscv/kernel/ftrace.c +++ b/arch/riscv/kernel/ftrace.c @@ -8,10 +8,21 @@ #include #include #include +#include #include #include #include +unsigned long ftrace_call_adjust(unsigned long addr) +{ + return addr + MCOUNT_AUIPC_SIZE; +} + +unsigned long arch_ftrace_get_symaddr(unsigned long fentry_ip) +{ + return fentry_ip - MCOUNT_AUIPC_SIZE; +} + #ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare(void) __acquires(&text_mutex) { @@ -32,51 +43,32 @@ void ftrace_arch_code_modify_post_process(void) __releases(&text_mutex) mutex_unlock(&text_mutex); } -static int ftrace_check_current_call(unsigned long hook_pos, - unsigned int *expected) +static int __ftrace_modify_call(unsigned long source, unsigned long target, bool validate) { + unsigned int call[2], offset; unsigned int replaced[2]; - unsigned int nops[2] = {RISCV_INSN_NOP4, RISCV_INSN_NOP4}; - /* we expect nops at the hook position */ - if (!expected) - expected = nops; + offset = target - source; + call[1] = to_jalr_t0(offset); - /* - * Read the text we want to modify; - * return must be -EFAULT on read error - */ - if (copy_from_kernel_nofault(replaced, (void *)hook_pos, - MCOUNT_INSN_SIZE)) - return -EFAULT; - - /* - * Make sure it is what we expect it to be; - * return must be -EINVAL on failed comparison - */ - if (memcmp(expected, replaced, sizeof(replaced))) { - pr_err("%p: expected (%08x %08x) but got (%08x %08x)\n", - (void *)hook_pos, expected[0], expected[1], replaced[0], - replaced[1]); - return -EINVAL; + if (validate) { + call[0] = to_auipc_t0(offset); + /* + * Read the text we want to modify; + * return must be -EFAULT on read error + */ + if (copy_from_kernel_nofault(replaced, (void *)source, 2 * MCOUNT_INSN_SIZE)) + return -EFAULT; + + if (replaced[0] != call[0]) { + pr_err("%p: expected (%08x) but got (%08x)\n", + (void *)source, call[0], replaced[0]); + return -EINVAL; + } } - return 0; -} - -static int __ftrace_modify_call(unsigned long hook_pos, unsigned long target, - bool enable, bool ra) -{ - unsigned int call[2]; - unsigned int nops[2] = {RISCV_INSN_NOP4, RISCV_INSN_NOP4}; - - if (ra) - make_call_ra(hook_pos, target, call); - else - make_call_t0(hook_pos, target, call); - - /* Replace the auipc-jalr pair at once. Return -EPERM on write error. */ - if (patch_insn_write((void *)hook_pos, enable ? call : nops, MCOUNT_INSN_SIZE)) + /* Replace the jalr at once. Return -EPERM on write error. */ + if (patch_insn_write((void *)(source + MCOUNT_AUIPC_SIZE), call + 1, MCOUNT_JALR_SIZE)) return -EPERM; return 0; @@ -84,22 +76,21 @@ static int __ftrace_modify_call(unsigned long hook_pos, unsigned long target, int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) { - unsigned int call[2]; + unsigned long distance, orig_addr, pc = rec->ip - MCOUNT_AUIPC_SIZE; - make_call_t0(rec->ip, addr, call); - - if (patch_insn_write((void *)rec->ip, call, MCOUNT_INSN_SIZE)) - return -EPERM; + orig_addr = (unsigned long)&ftrace_caller; + distance = addr > orig_addr ? addr - orig_addr : orig_addr - addr; + if (distance > JALR_RANGE) + return -EINVAL; - return 0; + return __ftrace_modify_call(pc, addr, false); } -int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, - unsigned long addr) +int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) { - unsigned int nops[2] = {RISCV_INSN_NOP4, RISCV_INSN_NOP4}; + u32 nop4 = RISCV_INSN_NOP4; - if (patch_insn_write((void *)rec->ip, nops, MCOUNT_INSN_SIZE)) + if (patch_insn_write((void *)rec->ip, &nop4, MCOUNT_NOP4_SIZE)) return -EPERM; return 0; @@ -114,21 +105,38 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, */ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) { - int out; + unsigned long pc = rec->ip - MCOUNT_AUIPC_SIZE; + unsigned int nops[2], offset; + int ret; + + offset = (unsigned long) &ftrace_caller - pc; + nops[0] = to_auipc_t0(offset); + nops[1] = RISCV_INSN_NOP4; mutex_lock(&text_mutex); - out = ftrace_make_nop(mod, rec, MCOUNT_ADDR); + ret = patch_insn_write((void *)pc, nops, 2 * MCOUNT_INSN_SIZE); mutex_unlock(&text_mutex); - return out; + return ret; } +ftrace_func_t ftrace_call_dest = ftrace_stub; int ftrace_update_ftrace_func(ftrace_func_t func) { - int ret = __ftrace_modify_call((unsigned long)&ftrace_call, - (unsigned long)func, true, true); - - return ret; + WRITE_ONCE(ftrace_call_dest, func); + /* + * The data fence ensure that the update to ftrace_call_dest happens + * before the write to function_trace_op later in the generic ftrace. + * If the sequence is not enforced, then an old ftrace_call_dest may + * race loading a new function_trace_op set in ftrace_modify_all_code + * + * If we are in stop_machine, then we don't need to call remote fence + * as there is no concurrent read-side of ftrace_call_dest. + */ + smp_wmb(); + if (!irqs_disabled()) + smp_call_function(ftrace_sync_ipi, NULL, 1); + return 0; } struct ftrace_modify_param { @@ -172,17 +180,11 @@ void arch_ftrace_update_code(int command) int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) { + unsigned long caller = rec->ip - MCOUNT_AUIPC_SIZE; unsigned int call[2]; - unsigned long caller = rec->ip; - int ret; make_call_t0(caller, old_addr, call); - ret = ftrace_check_current_call(caller, call); - - if (ret) - return ret; - - return __ftrace_modify_call(caller, addr, true, false); + return __ftrace_modify_call(caller, addr, true); } #endif diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index 3f06b40bb6c8..8aa554d56096 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -13,7 +13,6 @@ .text -#define FENTRY_RA_OFFSET 8 #define ABI_SIZE_ON_STACK 80 #define ABI_A0 0 #define ABI_A1 8 @@ -62,8 +61,7 @@ * After the stack is established, * * 0(sp) stores the PC of the traced function which can be accessed -* by &(fregs)->epc in tracing function. Note that the real -* function entry address should be computed with -FENTRY_RA_OFFSET. +* by &(fregs)->epc in tracing function. * * 8(sp) stores the function return address (i.e. parent IP) that * can be accessed by &(fregs)->ra in tracing function. @@ -140,7 +138,7 @@ .endm .macro PREPARE_ARGS - addi a0, t0, -FENTRY_RA_OFFSET + addi a0, t0, -MCOUNT_JALR_SIZE // ip (callsite's jalr insn) la a1, function_trace_op REG_L a2, 0(a1) mv a1, ra @@ -153,7 +151,8 @@ SYM_FUNC_START(ftrace_caller) PREPARE_ARGS SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) - call ftrace_stub + REG_L ra, ftrace_call_dest + jalr ra, 0(ra) RESTORE_ABI_REGS bnez t1, .Ldirect From patchwork Mon Apr 7 18:08:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041565 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47397C36010 for ; Mon, 7 Apr 2025 18:52:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jmaVhwoQusstl6GOC3FsobapoJoWoWafBQQ/L+nH37A=; b=IoZLVOAckodm1v CXoEOKlOz6istB5fuVssQxx5BDn3DvnG0rlcbAsi09xXaqyizy5DMnFLss095SyUnioYd5uRNp+6F dAUcDasx5RAyEB7rNsvawDAoJRqu2SPZq3WMVm7tJuC2ouzeDXRvrJSa9k4O0O7/XkQq4PhizBuLp 3frPe5apYvrXw+fPr/l/4t2pcoBQKpv3onb0z92dGrB4wdqt9RqYsDIfhuCxUwAkybUiszKpfzCZU XuLhpbE1qMSGzamClPk9/nG5E0YEOVgV2Fs2fduQY+TtFZ11TrsBuvEOFEUlFY3z34IqWQHx8O8jE 3nh5k41hVlk47IdiiB4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raE-00000001gLR-31AT; Mon, 07 Apr 2025 18:52:34 +0000 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quL-00000001TVL-3h9T for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:19 +0000 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-22a976f3131so19101735ad.3 for ; Mon, 07 Apr 2025 11:09:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049357; x=1744654157; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m9VOOAOCKhHyxTpEvBxwItchQtvdn2LoPZ+ikId4PHc=; b=Ad+4AKNIFaiijnelYUqwm5X4PWJefsvAM2U9ypkChZiGMp5F8qBHbfpbQyQonAQMeN bpKdM5wLiu6/N4tBfFwDRXE8PZpGmRAZVzu1GXOPh38FWZE1eA4GHCcbVkSH3661L2oI zPcCVReLoygbEccxUDXHE3zwsTn9fHSa8/qlvoOhOrUME+HxxQ5LWTLPL4oOjStbm0Vv EWOr4otoCUlFk6Ywtr2xwpS9fDyOl1qQUzA8z94wfo8a1HAn8XrhvYhXhHobnwDEbULF d4tZjHGO9nXcgtaG/GJ42y4UVmzXdsquCRLR6w5DbsFnaymn2Mus/p97vH6a04Qb/hch QKSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049357; x=1744654157; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m9VOOAOCKhHyxTpEvBxwItchQtvdn2LoPZ+ikId4PHc=; b=Q41qkvTSJ+y6QiIGfhd+r0NeEzPIBiJQJwXPbHsQFApZ5aEGJq8Lw0i5oSUl12rn1P mklBordEr/oHQPpcwkUJ1462DuONH8QtdEeHTIaxo0b4r9ZlJTFXRsuhB/Se7KVWpOgA r/0ByzvBQBbUHKCzbLR6dsykpC96L25wz9IGNP4m+h7oSk7hlqJzbce8ZwjP27k3Ag1R JgRD+3Sj3C2xtVMdRfdRdFyRYTHbIK409Kqvzbri55v/LbsXoacHTOnI3b+ehzeAfZCO UCXgU0VYILpq1hpHmL8lpxXyDqwjQPgufoVP7M3f/S6H7Z46+zecaKJ7D3MCDn5YLRZI c8zQ== X-Gm-Message-State: AOJu0YxkZ6mxTsRXmLBjqXxmfSYOfywmmPEIOSU43M+oV36sc+M5KNsU zl58MOwImSSgABmnQlHvdr8gSUJ7RcZMck9Rb4uHaGSSko82kwnRk1MaFXiE X-Gm-Gg: ASbGnctUEdsQhlSqy0NOFtVNcUoAcuKGazH6UdFQMA9ozPfrAkp94fEW512hZglGdtA jTvRR8/BaxbwqPPGW/D4g4GMhocn5XwU9qc+fK8C1b6JqZ/G8e5I7CU9Fc96qggYCOrXMiKMjcV lP8vpP4sQ9jnYc+hr3lB4DdDCc2or/NHzHdxt/V3aMu3BjxgMyl0aADVAt6D+Slr+akhnct534+ 97eMHnL7WT0Z0eYEvypVcWTftSZ+3kThxc3/IzZunmzRLpzMaYHRU83YJUj3sPGt8V0XlAdQ+le gSoG+27U9nJQl9jqxoIwTq6f1lLcspqPjgFg+7iP0xJXanYfgZpG66BWV4OtbLROXrrOAHXmn78 vOiaqx6rrx7gbYfXizc2//G8ynbU8kxippg== X-Google-Smtp-Source: AGHT+IFf6ha6oIioXqe0Dl7spY0xA264EoHLx7/lwi0vWS1NoagPzSr/i5AfZPigVEpWrl+bKtG+9g== X-Received: by 2002:a17:903:1b2d:b0:216:7926:8d69 with SMTP id d9443c01a7336-22a8a8dd69emr158392105ad.47.1744049356897; Mon, 07 Apr 2025 11:09:16 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:16 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Mark Rutland , Alexandre Ghiti , bjorn@rivosinc.com, puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 06/12] riscv: ftrace: do not use stop_machine to update code Date: Tue, 8 Apr 2025 02:08:30 +0800 Message-Id: <20250407180838.42877-6-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110917_921007_9685A8DC X-CRM114-Status: GOOD ( 16.75 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Andy Chiu Now it is safe to remove dependency from stop_machine() for us to patch code in ftrace. Signed-off-by: Andy Chiu --- Changelog v4: - assume ftrace_update_ftrace_func is always called with irqs enabled --- arch/riscv/kernel/ftrace.c | 64 ++++++-------------------------------- 1 file changed, 10 insertions(+), 54 deletions(-) diff --git a/arch/riscv/kernel/ftrace.c b/arch/riscv/kernel/ftrace.c index cf78eef073a0..aca1a322e0aa 100644 --- a/arch/riscv/kernel/ftrace.c +++ b/arch/riscv/kernel/ftrace.c @@ -24,23 +24,13 @@ unsigned long arch_ftrace_get_symaddr(unsigned long fentry_ip) } #ifdef CONFIG_DYNAMIC_FTRACE -void ftrace_arch_code_modify_prepare(void) __acquires(&text_mutex) +void arch_ftrace_update_code(int command) { mutex_lock(&text_mutex); - - /* - * The code sequences we use for ftrace can't be patched while the - * kernel is running, so we need to use stop_machine() to modify them - * for now. This doesn't play nice with text_mutex, we use this flag - * to elide the check. - */ - riscv_patch_in_stop_machine = true; -} - -void ftrace_arch_code_modify_post_process(void) __releases(&text_mutex) -{ - riscv_patch_in_stop_machine = false; + command |= FTRACE_MAY_SLEEP; + ftrace_modify_all_code(command); mutex_unlock(&text_mutex); + flush_icache_all(); } static int __ftrace_modify_call(unsigned long source, unsigned long target, bool validate) @@ -129,51 +119,17 @@ int ftrace_update_ftrace_func(ftrace_func_t func) * before the write to function_trace_op later in the generic ftrace. * If the sequence is not enforced, then an old ftrace_call_dest may * race loading a new function_trace_op set in ftrace_modify_all_code - * - * If we are in stop_machine, then we don't need to call remote fence - * as there is no concurrent read-side of ftrace_call_dest. */ smp_wmb(); - if (!irqs_disabled()) - smp_call_function(ftrace_sync_ipi, NULL, 1); - return 0; -} - -struct ftrace_modify_param { - int command; - atomic_t cpu_count; -}; - -static int __ftrace_modify_code(void *data) -{ - struct ftrace_modify_param *param = data; - - if (atomic_inc_return(¶m->cpu_count) == num_online_cpus()) { - ftrace_modify_all_code(param->command); - /* - * Make sure the patching store is effective *before* we - * increment the counter which releases all waiting CPUs - * by using the release variant of atomic increment. The - * release pairs with the call to local_flush_icache_all() - * on the waiting CPU. - */ - atomic_inc_return_release(¶m->cpu_count); - } else { - while (atomic_read(¶m->cpu_count) <= num_online_cpus()) - cpu_relax(); - - local_flush_icache_all(); - } - + /* + * Updating ftrace dpes not take stop_machine path, so irqs should not + * be disabled. + */ + WARN_ON(irqs_disabled()); + smp_call_function(ftrace_sync_ipi, NULL, 1); return 0; } -void arch_ftrace_update_code(int command) -{ - struct ftrace_modify_param param = { command, ATOMIC_INIT(0) }; - - stop_machine(__ftrace_modify_code, ¶m, cpu_online_mask); -} #endif #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS From patchwork Mon Apr 7 18:08:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041566 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 16209C36010 for ; Mon, 7 Apr 2025 18:52:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=B/c3CXOWD0vCODTIGEyD4XE85CTAFmae0pyKXf6X/Dk=; b=s62NIL2J/1O2xP 2mYEgZ4IpS02fU1ZJLZJ2x/O16a2f3H130gRa7XjA2dDZyk3EfzoAD5ELtGkbeASaNODny/M5IQGL RZg6GkR8VHkrVSgb1/r2ttrewjPhn+v1bsIw1QEEwx5Fx29CAyUl3kw4PylUdIU83JlXdrv5tpx/N R1ARYUZmv5a8WoFSADZ9lAYni+8S9Gaonr8KpSHnDvFhscqyaWWXxZ+rmQTzF2Md6kCY2xWE3ukTF 2qU9hwF6mDpUDviOSvYlEslgpAWnWOzZKvav7ooGmicenzU8k2zgqWrrlH54kNTPqwdG2Fn4TM/Qb SjSKyTt//NDeaeC/V3Mg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raG-00000001gQ5-3k2f; Mon, 07 Apr 2025 18:52:36 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quQ-00000001TXf-2n7H for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:23 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-22435603572so44179625ad.1 for ; Mon, 07 Apr 2025 11:09:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049362; x=1744654162; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NNLP2WcbFnXWmhqiIcXTLGBaX0CpznpLnzRdD+AmWMw=; b=DalthndDJZ3jnDw01Yysz1+eMoePGZ1E6DAPmC7zg86Vbo6beZn15AKCQgoTvGnB6w tPduNCbYcR1BehFqVIQ1YSG9/LOeW26wrVmHY771rR3Wz0PIDd7SmzAWWffCDGK+6LEE TWEDjDNtfZwXgVm8LwMDP4UcU6qLGRVZPW0Z/8Vii0tjyydABkHhtz/JHRzgPAshBr+a R9EPQkX7ACM1f4rVo6lFuqnoRzGllKpFwMnw7usQWrK/lQ73px3NN9TFLR7/jsUuTiGQ 7PW0fVb6MubGqS5LINsVIorkr0TJ8Vdgm8ieIBHoUNaHA9JSGoQXJ9O1BD5P1gkElncu VJIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049362; x=1744654162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NNLP2WcbFnXWmhqiIcXTLGBaX0CpznpLnzRdD+AmWMw=; b=VRErBXB9fKUSEpj2JGKVl2IOxxljAjKBH7Ag614gWVip1nkrN+T4uuoyauFMuEg8kS 4gKiwdj2VG4h9Z5Du98S4JVY6IwQ/MAhX44OxQYH1COIEgEr12X8eAeqHWIEsSZik6aI R3my+y5fIadxYGIjvNG3z6FcssQsJ+cxmYTCO375iqdWrAhK78oMUlvqyYeOnpT0EB9r hOFrzzBqmJNtUZJ3BLMWyP9HBuhY3e4ftdq2Az2D0F9LYV89NTKPiDAVKPiE2rHcUeGQ f7mCeDU293c8Vtx5YjWYcCVBjiAIEnicEiGB2TvRKpIU0xA9FWu0gpqExHtCyLqJl7C4 uQHQ== X-Gm-Message-State: AOJu0Yz6ofH25FNr3REXHVLm6ten9UZkR5lCxq2EWG30gdWPvGz2+AbJ bYBwShPa0plXhOwMgfuXmlRrfJK/A2NpViJ+ZHeTcwMZhHPFMU6yfAmYfZGj X-Gm-Gg: ASbGncvJfysDRugI01t9NKY6ZxBl6vIdGvxXrkZLaCrX3xEO97WFwjgIvJH/aVv9bLr nK/wBw7tfaGxS9H/0pt3tTKLhFs3jH0jjFwJF+4OcwyYidqfuot/CjmxmYXkBbUEeH7RkwewOQ3 E8Y7lFrQsrIy76lV6QbKboJH+vgY8G4nJqKEgEzmJzBL0tBEPsehTrsp53bu2g7l8jl6qjB9eaL e+dljn+bbasdt9wUHYY5XUO9eq00WNrjUL6YAAa/4qmgWZdIVDNQ/Gd9h6d70R/Kut5tLvSfuLr DSO9vmXnAVmYzw+8IBaakwR5wNh+ASX6G650t3JEwNYENK/h7MsS6PZ9zKnMOzVLufEn6So4qx5 97Ey7/lhD71On5V3TBbE1c1396bgoZ7XXCC5pOPhE/Eu7 X-Google-Smtp-Source: AGHT+IFwnBb03K/EWgGxHSZMUCeczsevSxPHhBr5Q9Nlteyu1sSQQHWfAssAlK/1aErAdZs0ChZmog== X-Received: by 2002:a17:902:dac3:b0:224:256e:5e4e with SMTP id d9443c01a7336-22a8a86588cmr183418285ad.16.1744049361576; Mon, 07 Apr 2025 11:09:21 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:21 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , linux-kernel@vger.kernel.org, Alexandre Ghiti , bjorn@rivosinc.com, puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 07/12] riscv: vector: Support calling schedule() for preemptible Vector Date: Tue, 8 Apr 2025 02:08:31 +0800 Message-Id: <20250407180838.42877-7-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110922_703797_F0193CCD X-CRM114-Status: GOOD ( 15.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Andy Chiu Each function entry implies a call to ftrace infrastructure. And it may call into schedule in some cases. So, it is possible for preemptible kernel-mode Vector to implicitly call into schedule. Since all V-regs are caller-saved, it is possible to drop all V context when a thread voluntarily call schedule(). Besides, we currently don't pass argument through vector register, so we don't have to save/restore V-regs in ftrace trampoline. Signed-off-by: Andy Chiu --- arch/riscv/include/asm/processor.h | 5 +++++ arch/riscv/include/asm/vector.h | 22 +++++++++++++++++++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 5f56eb9d114a..9c1cc716b891 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -79,6 +79,10 @@ struct pt_regs; * Thus, the task does not own preempt_v. Any use of Vector will have to * save preempt_v, if dirty, and fallback to non-preemptible kernel-mode * Vector. + * - bit 29: The thread voluntarily calls schedule() while holding an active + * preempt_v. All preempt_v context should be dropped in such case because + * V-regs are caller-saved. Only sstatus.VS=ON is persisted across a + * schedule() call. * - bit 30: The in-kernel preempt_v context is saved, and requries to be * restored when returning to the context that owns the preempt_v. * - bit 31: The in-kernel preempt_v context is dirty, as signaled by the @@ -93,6 +97,7 @@ struct pt_regs; #define RISCV_PREEMPT_V 0x00000100 #define RISCV_PREEMPT_V_DIRTY 0x80000000 #define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000 +#define RISCV_PREEMPT_V_IN_SCHEDULE 0x20000000 /* CPU-specific state of a task */ struct thread_struct { diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index e8a83f55be2b..45c9b426fcc5 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -120,6 +120,11 @@ static __always_inline void riscv_v_disable(void) csr_clear(CSR_SSTATUS, SR_VS); } +static __always_inline bool riscv_v_is_on(void) +{ + return !!(csr_read(CSR_SSTATUS) & SR_VS); +} + static __always_inline void __vstate_csr_save(struct __riscv_v_ext_state *dest) { asm volatile ( @@ -366,6 +371,11 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; if (riscv_preempt_v_started(prev)) { + if (riscv_v_is_on()) { + WARN_ON(prev->thread.riscv_v_flags & RISCV_V_CTX_DEPTH_MASK); + riscv_v_disable(); + prev->thread.riscv_v_flags |= RISCV_PREEMPT_V_IN_SCHEDULE; + } if (riscv_preempt_v_dirty(prev)) { __riscv_v_vstate_save(&prev->thread.kernel_vstate, prev->thread.kernel_vstate.datap); @@ -376,10 +386,16 @@ static inline void __switch_to_vector(struct task_struct *prev, riscv_v_vstate_save(&prev->thread.vstate, regs); } - if (riscv_preempt_v_started(next)) - riscv_preempt_v_set_restore(next); - else + if (riscv_preempt_v_started(next)) { + if (next->thread.riscv_v_flags & RISCV_PREEMPT_V_IN_SCHEDULE) { + next->thread.riscv_v_flags &= ~RISCV_PREEMPT_V_IN_SCHEDULE; + riscv_v_enable(); + } else { + riscv_preempt_v_set_restore(next); + } + } else { riscv_v_vstate_set_restore(next, task_pt_regs(next)); + } } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); From patchwork Mon Apr 7 18:08:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041567 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8706C36010 for ; Mon, 7 Apr 2025 18:52:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=euhDLFwBXC3b4/85FIC23yFqPG4BMZ8W2D1g0Y9sqdE=; b=1RxjKq5Ju9GasK djrF7o2+9gkiZ4IDbQvv1eur6znqZE0Vc5LYcVLUfDjUuQkEtA+iYtsQPYoWt6QWbnSqEYiKbbabw 8F0Z5Cm72dt/rWgvica799hJ8Qde2yfB/JfDF3PcwteF+Nmu8psqB6yUZ8FMstD48DdHE4+TGajyS n1rQc0kayTsS2cC75su0B+lcLWwOc3+2Q3lvtUVr1KqbtW8hJrVzvgVElI4/SRRkS0jM98Ly6qTea 9XlO3dtCRGNW16m0Y38bO9Z6a0hR2WTwAeJBRxCcStxDabEEn/atVqLqnzln/J7sxh7CyXiuq/J+L quFqch1qxajdlMR9Xd4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raK-00000001gVp-1Phq; Mon, 07 Apr 2025 18:52:40 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quU-00000001TYr-3Eug for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:27 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-22580c9ee0aso50094845ad.2 for ; Mon, 07 Apr 2025 11:09:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049366; x=1744654166; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5Z0y0TK+N2j7unfdHAGWrmjHTParwUExlSa0jHt4G78=; b=A//nWHAeMxPgjNuK/GiH/MmMbOptUPI5yGkN1naDh60YMQP0y/dKDMZG/XE1gt32oB IONv5miFdCPoA8DD+tduSZaN1N3HiTeqXH+Zknim7D7v1hQCJ+Ikc2F5yPjo/z7hFcGW Xyvna5odYnOMc4GW39ENdG7ZlMq2tG3n5u90Hq92IQlh540cv4QWteoI6NSx4O/eeCiX oPt5YRAF5VgrUp6tvbt75a+regxPIaDTGiEERxEbceqqhqODcADxVUvokcVPDdLH8xug VT0v8U3duFUynubtKfl+VPV3SIwRIz8xxco+hPJ5KhNEHRsAWErpm1Aqa4+bwA4pdnlC JRpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049366; x=1744654166; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5Z0y0TK+N2j7unfdHAGWrmjHTParwUExlSa0jHt4G78=; b=hC4WXz30tN9KsKrJ2VwRJH12cKOoRAHrkrLV+5PavxNIvl6ML6BQNgyTDosJRycnWK 6L8Pm3qB90x7d4+vMeeT7dTJquLSct0ljd/qLj+f9USdJXshYl/9RTuPngI2Cy31x8Ro vDK9jAKaXtY2n+dv1dCkVREhvzZb2r9aVbA0gR1reX9lAQSRQT06LCDT0Dpi8PW2ybxm OEGoeBsDegk/ZaI2uf4HwKHgk3A/tyNwOL/QHR+lcqBTeZPuUQ76RHU2/5ngqhgewujY pmyJMyk52noyuEThr4Fae2kD1uszmdXjFUvlhcVJp6fe+dC+BiRFce2OLIIxU5T1M4DZ bvtA== X-Gm-Message-State: AOJu0YyYOXAnAijax19zIhi2+J64j9PkFxVjj7dJrUKQVHSdduzrQ9WB xGlz3J3626+9CK0T8qlcHFrDLucMqHgac9RiM5pLWUENrAqOfqPlWJeUhRPy X-Gm-Gg: ASbGncvYvJcnqEJkDGhiUXlXP+AncFl7oSgq09wsn5Zjrn3aMCB7R7CQNrPBYdSKmdh qYvPCqd9lydI3GJ9+sjzB5EPokawHOyfbmFUSshGYF3YW7HXzdmjQZAj3ZGYUvXKCtufkFFnBh+ Hh9SYZID3jSwm/NaAzSJM8tXSMP6NTw5LQlDd22UFjkFzvdx2eqAPBaGMC2OSNyYhU662uWbzDl i7PJW2sT0lYMcj269MhyvBPJ2ZA1zUPl3FEsVwOK0Vn3kcUZEtkPlBoVh84cumJarQjPWOcWyfV +e5TVzHPsx2tvv4aZIJN90SJK2j2ofxdvU88rvWbK/FEeV9wjQO+4+EHKYfJ3KyFxeHnneMtPRj pWbLrF4x+HbRb6kds7uV6Igp/gZnuUSDcxQ== X-Google-Smtp-Source: AGHT+IFgrdURIj5aGIW42rvqFSz+4B3fV+3dwcYfHSLwCG5BImXtPG/2PWupoaefl6dEzTqBihbAPA== X-Received: by 2002:a17:90b:2d4c:b0:2fa:228d:5b03 with SMTP id 98e67ed59e1d1-306a615be9fmr15006956a91.19.1744049365882; Mon, 07 Apr 2025 11:09:25 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:25 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, Alexandre Ghiti , puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 08/12] riscv: add a data fence for CMODX in the kernel mode Date: Tue, 8 Apr 2025 02:08:32 +0800 Message-Id: <20250407180838.42877-8-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110926_811673_5EB55615 X-CRM114-Status: GOOD ( 14.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org RISC-V spec explicitly calls out that a local fence.i is not enough for the code modification to be visble from a remote hart. In fact, it states: To make a store to instruction memory visible to all RISC-V harts, the writing hart also has to execute a data FENCE before requesting that all remote RISC-V harts execute a FENCE.I. Although current riscv drivers for IPI use ordered MMIO when sending IPIs in order to synchronize the action between previous csd writes, riscv does not restrict itself to any particular flavor of IPI. Any driver or firmware implementation that does not order data writes before the IPI may pose a risk for code-modifying race. Thus, add a fence here to order data writes before making the IPI. Signed-off-by: Andy Chiu Reviewed-by: Björn Töpel --- Changelog v4: - Explain more in commit msg and add Björn's R-b --- arch/riscv/mm/cacheflush.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index b81672729887..b2e4b81763f8 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -24,7 +24,20 @@ void flush_icache_all(void) if (num_online_cpus() < 2) return; - else if (riscv_use_sbi_for_rfence()) + + /* + * Make sure all previous writes to the D$ are ordered before making + * the IPI. The RISC-V spec states that a hart must execute a data fence + * before triggering a remote fence.i in order to make the modification + * visable for remote harts. + * + * IPIs on RISC-V are triggered by MMIO writes to either CLINT or + * S-IMSIC, so the fence ensures previous data writes "happen before" + * the MMIO. + */ + RISCV_FENCE(w, o); + + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); From patchwork Mon Apr 7 18:08:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041568 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA4F2C36018 for ; Mon, 7 Apr 2025 18:52:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8fnLjKfihEypHkFwcYJidLnVGWWyUW/YDy3d9euNKcw=; b=JztXU7pfS0kg1s h00h8sozZDBKCVQ6vh6XQtNP26FDUfJmzvhAAa7hsKXxKBFbX6JEvmhAN1gbQU/b0mE20Jf8MfPFk L+e0aoVyKPDdcY5eoNSizvaDQdF64DG7A2RK+gon28Yx2a/CfI1BdEP7C0jD0mf1f/OvKPDzLIhP7 NgZi8L7aKszNPBWA3TcqjfUdWETHCyMAS8yLycJNLXpmXWRK4WZyj5lnEjMkXW6AHpZyFy02ERQKO 7kfDopfdkHywdlwRAigSH23PHJcmu5+OC/gNzYBRffoMD7ZE87dlei171dycZ9LEN0FDEt/+V13u3 Io3rc+pqcAOA+n6hstew==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raM-00000001gYs-1cdA; Mon, 07 Apr 2025 18:52:42 +0000 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quZ-00000001TZd-2BsM for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:32 +0000 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-22409077c06so57709465ad.1 for ; Mon, 07 Apr 2025 11:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049370; x=1744654170; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WMjCLGzG+iKPyMJNKKfkgkaaKsErgB/MOxToztcG4nk=; b=hs/xX7JwuOKkVoOTl9moxBVd/Ka3eIe6xf1wacYWXF96J4Q6sArbUwnGPgiQJ4F3F8 r9l2pZA4+cLTls3h45ztVpWthkRiyd+FjlQ4jkMMs93RQTYcZ936D8gHWrwR660qTJIc iEfbSgHGTUalY3EEu0JjCLYSwMcyZ14dnzAEkOqJhyQ2PAUstD4qVZIcNHmb3hbzZ7di 17zroS723QlBeTLOpFIx0QjcSuOP2BHVYVEO4q+bdr36JVgF4weGmchDltJ6BhsHcoyn FbJXrTgBLw1W09CHypE5xNZ53fMiUJppIeXTJTMMDf0h8q3xxTY6MKYr7eiduQzMH41y ZGRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049370; x=1744654170; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WMjCLGzG+iKPyMJNKKfkgkaaKsErgB/MOxToztcG4nk=; b=Vpf05CN4uo4eVEt725jqiwEKpfgIttUp4Fr6MLRV3Ozw8UkEUrMZhDVDQZm5L7bRwP 1wj9XBDHNeIJwu5LbYdUdTJPvslWKu0kFJgMuPfC/SPbQxg3tQa8WXU5JziU1imZ0CvB FleQ5RJuxRCzL7bhNubf5YXj3kY8NeMiDHwCTyuaW2uALvT/3dDGQrCfXsxrgX/vjsuH JzEaqWNLE3Qz97NmbMH93AD3B8BbVaaqeButDtW7p2xBiwvYY8b5qL7/nKxQLqp+d3Qb FzDeXxMF57pLFisYizq46LJt6IZSKnftAfQZcSW7xI0bjxBQ+meULGbXwdWTX8WGS7Rb Xipw== X-Gm-Message-State: AOJu0YyGrgfRQ9DGL2VvqEyy3+VrsM6g4Ywf90Brb9q/Us1I6wfSecLl lYMeBs/PesuTbuIwzi+1uKURxaE01EMRT5hYrJcabGcfJahitsCZksIvLSxg X-Gm-Gg: ASbGncvIOiL7zJSk8so8J+L+uwZHgvRFo0Bg7AbBgtTq58d8hV6HdIChqMtI8SjvBdk PVbGDNa3crJqBjTnGC++x7W3tbu5OS6gL3wCJ6XQWAYFm8bNpKowS6xsJ+kb940iXbq36Ibg+rd gSw7zd2oA1QqUP+k+puLwUIhXlVWa3Yg1/8BIEIYU0/NBY3ZMFPV9yjp9RbLxLw1tHpeITiWWsV AIdKhMnSQq35JpH+ET1pUZfMvfuFb8rMdEkGXG23r3TWBcsxvynlSw8Y2VIjeigSjiV0ms32Edc Y9awi2Wvajr3LY+whSS97amDZAoR/kZ0lDwLclutN0rB75n3Drk77SZooPJBVQjZq89pr4V0AT6 N9OPKuytKjzl2usTO0mblxAp3gtJrIvGDrw== X-Google-Smtp-Source: AGHT+IFm3nEZ9gY9atSzADAapLMqEef81qmFhhoX5lX79/arRk7VY+zxr+6spU/7xP1WGipqyHp2rw== X-Received: by 2002:a17:903:3bc6:b0:224:fa0:36da with SMTP id d9443c01a7336-22a8a85bf1amr140171225ad.18.1744049370199; Mon, 07 Apr 2025 11:09:30 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:29 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, Alexandre Ghiti , puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 09/12] riscv: ftrace: support PREEMPT Date: Tue, 8 Apr 2025 02:08:33 +0800 Message-Id: <20250407180838.42877-9-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110931_556690_002D491D X-CRM114-Status: UNSURE ( 9.81 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Andy Chiu Now, we can safely enable dynamic ftrace with kernel preemption. Signed-off-by: Andy Chiu Reviewed-by: Björn Töpel --- arch/riscv/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 7dbed10843d2..dc0fc11b6e96 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -157,7 +157,7 @@ config RISCV select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL select HAVE_FUNCTION_GRAPH_TRACER if HAVE_DYNAMIC_FTRACE_WITH_ARGS select HAVE_FUNCTION_GRAPH_FREGS - select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION + select HAVE_FUNCTION_TRACER if !XIP_KERNEL select HAVE_EBPF_JIT if MMU select HAVE_GUP_FAST if MMU select HAVE_FUNCTION_ARG_ACCESS_API From patchwork Mon Apr 7 18:08:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041569 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DD63C36010 for ; Mon, 7 Apr 2025 18:52:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N1ZXfix62lshC9HxvVsKm7jLsscKHsMmlGts8+DrFJg=; b=CpqUV6ctz14NYv r6qSMHh2JPI9Y6EyHchcdB9Y8+uqk9vyzD6FNg7rnDQ56+Gx/Kqflnpo/I5tJoKbqnOzd3NYP5osU kkC0FYVykpnOQ75DD2uo2eytOKTVkFmUCFgHngLpUQuS+kGSa+/nILExUohK2W6n+yZkB9DdesFAN iadfJmdZEPGulRDV1w583FVgLO49wBNxnGwpxSOcrszRtNu14Ay6VrUvngRKvNzy9obK3D/2XQCSp roEfgw1PdQvzGLUExkRt57JMS+jBPtPIvbewh8f8qDpG6k2tv/h8J/SE9VF/jmDesyTRljYWxcym+ qQVkbFzjaY44XSuRrNtw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raN-00000001gbr-2lXk; Mon, 07 Apr 2025 18:52:43 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1que-00000001TaL-3A7t for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:37 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-227b650504fso41767845ad.0 for ; Mon, 07 Apr 2025 11:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049376; x=1744654176; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=03hAqsb3oBHKPV8p492vgGmxv28jGgXuCxUxhKcZY88=; b=WTx932/7n3vUqHtei61GjU0SacfXg4c8w7jvUKaRXbK/IbhnnxjqeDcXIjebpdy9vj gecugSBxfwUTtVoIFOEIjk1l0vKN2zBVXZHKbd38wGdC27LThWKkaY5D2wF6ErTjPO+z S7fvz/zC2ZS6sIXGh0vwSCmBej/sX4/ifokAOgcM1CCf3LgoH58f6dxUbcuicpxLYQV4 lGu9tL+VtMmbJ976r3pPv+uQ7UAOg8yFp6hxGXH5I5BzWFnknQTRdEMlTd+ngpt596Uy fC1MS4cNISUWmUMQSV2Y/HbdNmqCERKM/CTc08NGuF+YO5+NVt8+bIv1UwvpfSevoo9/ 0z5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049376; x=1744654176; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=03hAqsb3oBHKPV8p492vgGmxv28jGgXuCxUxhKcZY88=; b=k85RZytt1zOglU54Vk075re0knu5L4Rwtj9qAZiTkyGtwpbvJnaqO4vt7s1YgtXW1f R8l2M646+Vs54F6t6xQLtLLMR8V9dv8z4rTwBhLdLydZ/rGU0dU6IIPR/hHf/YPEAc8i cMNVG4DltE+0vrZMe1wo9KN6H46vUKR1jh++opHfhRP8eUkMLSl3yN6t+26Bjb9p/cPt grZyLlMzq4I8hkfNlEYSYDQOtCpYDno++V1KLiR+7kHif/Ur7kdaHjx/p0FaB2kcOY5L sd2Spok44Ics67+CgdIfcBvB4ji/vfSwYTs+w/iOG2+8kRRDPkkhY8IunKi0VSkvJfB1 Z9zA== X-Gm-Message-State: AOJu0YwTqc7Avd8LFHcnWtAS75doUTWwVlHjV4LTL08GoofFr8B6CzM4 uqDyHLaHsqpwjoa9bwjAWh6fNvFEWqDO1tc6hh2i3CDV8Vy5fmFyC5j50lJV X-Gm-Gg: ASbGnctfGPp+0lNAbEJuPRbs/mwZ1/Spm+LmTsb0tSHP4MJcNgUh71u+R/4+dhJTE9m 9NYYfwUu+4ccCwdyZ/xH70ONibk0y2a98Qv6A7dzN/2xVBhQLz9YkcoxGJQNoibkTTo8kD6NAwu +/j9wIHpb1XjPJkG6hy4fIojtcwWE07RjhXZZ3xySbmbrwTllNcgydWzeWCJY+0meXE9sfleqlx IT2fBXMvxdWVeDUeXrGHV3aipv2vDSy6fq4GaH0BIEzNsFjqtqA2W2Ac58OafPZb7QNkX0rkFpF Sj5FgVP4Gw1xIfwmS4oM2Met5fA3N3kudRCP919dVqVSWCFnUnsYT4q2LbooL9t+4BDQQC5Dwfj L5wpZZbUHDSHxfWzseqxq/hiTP2aJ1N4T3w== X-Google-Smtp-Source: AGHT+IFyGAlxdHGG370vb/DIpO+e45DzeZTKqsr/viKkGC6+OYzxqciYF2p6d0343L0+h7DTNjZrcg== X-Received: by 2002:a17:902:d4cd:b0:224:e33:889b with SMTP id d9443c01a7336-22a8a05a433mr185510495ad.12.1744049375896; Mon, 07 Apr 2025 11:09:35 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:35 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Puranjay Mohan , Andy Chiu , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Alexandre Ghiti , Mark Rutland , paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 10/12] riscv: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS Date: Tue, 8 Apr 2025 02:08:34 +0800 Message-Id: <20250407180838.42877-10-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110936_812440_C0CFB0AA X-CRM114-Status: GOOD ( 26.41 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Puranjay Mohan This patch enables support for DYNAMIC_FTRACE_WITH_CALL_OPS on RISC-V. This allows each ftrace callsite to provide an ftrace_ops to the common ftrace trampoline, allowing each callsite to invoke distinct tracer functions without the need to fall back to list processing or to allocate custom trampolines for each callsite. This significantly speeds up cases where multiple distinct trace functions are used and callsites are mostly traced by a single tracer. The idea and most of the implementation is taken from the ARM64's implementation of the same feature. The idea is to place a pointer to the ftrace_ops as a literal at a fixed offset from the function entry point, which can be recovered by the common ftrace trampoline. We use -fpatchable-function-entry to reserve 8 bytes above the function entry by emitting 2 4 byte or 4 2 byte nops depending on the presence of CONFIG_RISCV_ISA_C. These 8 bytes are patched at runtime with a pointer to the associated ftrace_ops for that callsite. Functions are aligned to 8 bytes to make sure that the accesses to this literal are atomic. This approach allows for directly invoking ftrace_ops::func even for ftrace_ops which are dynamically-allocated (or part of a module), without going via ftrace_ops_list_func. We've benchamrked this with the ftrace_ops sample module on Spacemit K1 Jupiter: Without this patch: baseline (Linux rivos 6.14.0-09584-g7d06015d936c #3 SMP Sat Mar 29 +-----------------------+-----------------+----------------------------+ | Number of tracers | Total time (ns) | Per-call average time | |-----------------------+-----------------+----------------------------| | Relevant | Irrelevant | 100000 calls | Total (ns) | Overhead (ns) | |----------+------------+-----------------+------------+---------------| | 0 | 0 | 1357958 | 13 | - | | 0 | 1 | 1302375 | 13 | - | | 0 | 2 | 1302375 | 13 | - | | 0 | 10 | 1379084 | 13 | - | | 0 | 100 | 1302458 | 13 | - | | 0 | 200 | 1302333 | 13 | - | |----------+------------+-----------------+------------+---------------| | 1 | 0 | 13677833 | 136 | 123 | | 1 | 1 | 18500916 | 185 | 172 | | 1 | 2 | 22856459 | 228 | 215 | | 1 | 10 | 58824709 | 588 | 575 | | 1 | 100 | 505141584 | 5051 | 5038 | | 1 | 200 | 1580473126 | 15804 | 15791 | |----------+------------+-----------------+------------+---------------| | 1 | 0 | 13561000 | 135 | 122 | | 2 | 0 | 19707292 | 197 | 184 | | 10 | 0 | 67774750 | 677 | 664 | | 100 | 0 | 714123125 | 7141 | 7128 | | 200 | 0 | 1918065668 | 19180 | 19167 | +----------+------------+-----------------+------------+---------------+ Note: per-call overhead is estimated relative to the baseline case with 0 relevant tracers and 0 irrelevant tracers. With this patch: v4-rc4 (Linux rivos 6.14.0-09598-gd75747611c93 #4 SMP Sat Mar 29 +-----------------------+-----------------+----------------------------+ | Number of tracers | Total time (ns) | Per-call average time | |-----------------------+-----------------+----------------------------| | Relevant | Irrelevant | 100000 calls | Total (ns) | Overhead (ns) | |----------+------------+-----------------+------------+---------------| | 0 | 0 | 1459917 | 14 | - | | 0 | 1 | 1408000 | 14 | - | | 0 | 2 | 1383792 | 13 | - | | 0 | 10 | 1430709 | 14 | - | | 0 | 100 | 1383791 | 13 | - | | 0 | 200 | 1383750 | 13 | - | |----------+------------+-----------------+------------+---------------| | 1 | 0 | 5238041 | 52 | 38 | | 1 | 1 | 5228542 | 52 | 38 | | 1 | 2 | 5325917 | 53 | 40 | | 1 | 10 | 5299667 | 52 | 38 | | 1 | 100 | 5245250 | 52 | 39 | | 1 | 200 | 5238459 | 52 | 39 | |----------+------------+-----------------+------------+---------------| | 1 | 0 | 5239083 | 52 | 38 | | 2 | 0 | 19449417 | 194 | 181 | | 10 | 0 | 67718584 | 677 | 663 | | 100 | 0 | 709840708 | 7098 | 7085 | | 200 | 0 | 2203580626 | 22035 | 22022 | +----------+------------+-----------------+------------+---------------+ Note: per-call overhead is estimated relative to the baseline case with 0 relevant tracers and 0 irrelevant tracers. As can be seen from the above: a) Whenever there is a single relevant tracer function associated with a tracee, the overhead of invoking the tracer is constant, and does not scale with the number of tracers which are *not* associated with that tracee. b) The overhead for a single relevant tracer has dropped to ~1/3 of the overhead prior to this series (from 122ns to 38ns). This is largely due to permitting calls to dynamically-allocated ftrace_ops without going through ftrace_ops_list_func. Signed-off-by: Puranjay Mohan [update kconfig, asm, refactor] Signed-off-by: Andy Chiu Tested-by: Björn Töpel --- Changelog v4: - include benchmark result from a real hardware, shout-out to Björn! - new patch copy from Puranjay's RFC implementation - Drop code related to !WITH_ARG && DYNAMIC_FTRACE --- arch/riscv/Kconfig | 2 + arch/riscv/Makefile | 4 +- arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/ftrace.c | 67 +++++++++++++++++++++++++++++++++ arch/riscv/kernel/mcount-dyn.S | 35 +++++++++++++++-- 5 files changed, 105 insertions(+), 6 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index dc0fc11b6e96..ec986c9120e3 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -99,6 +99,7 @@ config RISCV select EDAC_SUPPORT select FRAME_POINTER if PERF_EVENTS || (FUNCTION_TRACER && !DYNAMIC_FTRACE) select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY if DYNAMIC_FTRACE + select FUNCTION_ALIGNMENT_8B if DYNAMIC_FTRACE_WITH_CALL_OPS select GENERIC_ARCH_TOPOLOGY select GENERIC_ATOMIC64 if !64BIT select GENERIC_CLOCKEVENTS_BROADCAST if SMP @@ -152,6 +153,7 @@ config RISCV select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE) select FUNCTION_ALIGNMENT_4B if HAVE_DYNAMIC_FTRACE && RISCV_ISA_C select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG) select HAVE_DYNAMIC_FTRACE_WITH_ARGS if HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_GRAPH_FUNC select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 539d2aef5cab..df57654a615e 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -15,9 +15,9 @@ ifeq ($(CONFIG_DYNAMIC_FTRACE),y) LDFLAGS_vmlinux += --no-relax KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY ifeq ($(CONFIG_RISCV_ISA_C),y) - CC_FLAGS_FTRACE := -fpatchable-function-entry=4 + CC_FLAGS_FTRACE := -fpatchable-function-entry=8,4 else - CC_FLAGS_FTRACE := -fpatchable-function-entry=2 + CC_FLAGS_FTRACE := -fpatchable-function-entry=4,2 endif endif diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 7c43c8e26ae7..2d96197a8abf 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -493,6 +493,9 @@ void asm_offsets(void) DEFINE(STACKFRAME_SIZE_ON_STACK, ALIGN(sizeof(struct stackframe), STACK_ALIGN)); OFFSET(STACKFRAME_FP, stackframe, fp); OFFSET(STACKFRAME_RA, stackframe, ra); +#ifdef CONFIG_FUNCTION_TRACER + DEFINE(FTRACE_OPS_FUNC, offsetof(struct ftrace_ops, func)); +#endif #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS DEFINE(FREGS_SIZE_ON_STACK, ALIGN(sizeof(struct __arch_ftrace_regs), STACK_ALIGN)); diff --git a/arch/riscv/kernel/ftrace.c b/arch/riscv/kernel/ftrace.c index aca1a322e0aa..30bcf60135d8 100644 --- a/arch/riscv/kernel/ftrace.c +++ b/arch/riscv/kernel/ftrace.c @@ -15,6 +15,9 @@ unsigned long ftrace_call_adjust(unsigned long addr) { + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) + return addr + 8; + return addr + MCOUNT_AUIPC_SIZE; } @@ -64,9 +67,52 @@ static int __ftrace_modify_call(unsigned long source, unsigned long target, bool return 0; } +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS +static const struct ftrace_ops *riscv64_rec_get_ops(struct dyn_ftrace *rec) +{ + const struct ftrace_ops *ops = NULL; + + if (rec->flags & FTRACE_FL_CALL_OPS_EN) { + ops = ftrace_find_unique_ops(rec); + WARN_ON_ONCE(!ops); + } + + if (!ops) + ops = &ftrace_list_ops; + + return ops; +} + +static int ftrace_rec_set_ops(const struct dyn_ftrace *rec, + const struct ftrace_ops *ops) +{ + unsigned long literal = rec->ip - 8; + + return patch_text_nosync((void *)literal, &ops, sizeof(ops)); +} + +static int ftrace_rec_set_nop_ops(struct dyn_ftrace *rec) +{ + return ftrace_rec_set_ops(rec, &ftrace_nop_ops); +} + +static int ftrace_rec_update_ops(struct dyn_ftrace *rec) +{ + return ftrace_rec_set_ops(rec, riscv64_rec_get_ops(rec)); +} +#else +static int ftrace_rec_set_nop_ops(struct dyn_ftrace *rec) { return 0; } +static int ftrace_rec_update_ops(struct dyn_ftrace *rec) { return 0; } +#endif + int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) { unsigned long distance, orig_addr, pc = rec->ip - MCOUNT_AUIPC_SIZE; + int ret; + + ret = ftrace_rec_update_ops(rec); + if (ret) + return ret; orig_addr = (unsigned long)&ftrace_caller; distance = addr > orig_addr ? addr - orig_addr : orig_addr - addr; @@ -79,6 +125,11 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) { u32 nop4 = RISCV_INSN_NOP4; + int ret; + + ret = ftrace_rec_set_nop_ops(rec); + if (ret) + return ret; if (patch_insn_write((void *)rec->ip, &nop4, MCOUNT_NOP4_SIZE)) return -EPERM; @@ -99,6 +150,10 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) unsigned int nops[2], offset; int ret; + ret = ftrace_rec_set_nop_ops(rec); + if (ret) + return ret; + offset = (unsigned long) &ftrace_caller - pc; nops[0] = to_auipc_t0(offset); nops[1] = RISCV_INSN_NOP4; @@ -113,6 +168,13 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) ftrace_func_t ftrace_call_dest = ftrace_stub; int ftrace_update_ftrace_func(ftrace_func_t func) { + /* + * When using CALL_OPS, the function to call is associated with the + * call site, and we don't have a global function pointer to update. + */ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) + return 0; + WRITE_ONCE(ftrace_call_dest, func); /* * The data fence ensure that the update to ftrace_call_dest happens @@ -138,8 +200,13 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, { unsigned long caller = rec->ip - MCOUNT_AUIPC_SIZE; unsigned int call[2]; + int ret; make_call_t0(caller, old_addr, call); + ret = ftrace_rec_update_ops(rec); + if (ret) + return ret; + return __ftrace_modify_call(caller, addr, true); } #endif diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index 8aa554d56096..699684eea7f0 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -139,10 +139,34 @@ .macro PREPARE_ARGS addi a0, t0, -MCOUNT_JALR_SIZE // ip (callsite's jalr insn) +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS + /* + * When CALL_OPS is enabled (2 or 4) nops [8B] are placed before the + * function entry, these are later overwritten with the pointer to the + * associated struct ftrace_ops. + * + * -8: &ftrace_ops of the associated tracer function. + *: + * 0: auipc t0/ra, 0x? + * 4: jalr t0/ra, ?(t0/ra) + * + * -8: &ftrace_nop_ops + *: + * 0: nop + * 4: nop + * + * t0 is set to ip+8 after the jalr is executed at the callsite, + * so we find the associated op at t0-16. + */ + mv a1, ra // parent_ip + REG_L a2, -16(t0) // op + REG_L ra, FTRACE_OPS_FUNC(a2) // op->func +#else la a1, function_trace_op - REG_L a2, 0(a1) - mv a1, ra - mv a3, sp + REG_L a2, 0(a1) // op + mv a1, ra // parent_ip +#endif + mv a3, sp // regs .endm SYM_FUNC_START(ftrace_caller) @@ -150,10 +174,13 @@ SYM_FUNC_START(ftrace_caller) SAVE_ABI_REGS PREPARE_ARGS +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS + jalr ra +#else SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) REG_L ra, ftrace_call_dest jalr ra, 0(ra) - +#endif RESTORE_ABI_REGS bnez t1, .Ldirect jr t0 From patchwork Mon Apr 7 18:08:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041570 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7B37DC369A1 for ; Mon, 7 Apr 2025 18:52:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ukEJyd8weOAw77yAFMLdt2TOY3HtptsMRrRwz2e2wEA=; b=Mp/BfNpQKv7vp9 x8q1PN/mvPqpZTdmxSj3uO/njA0StKAXMntQnBDraAwLP2yYLaIiqsCDhGVAgwMqU/THfcCEb0tnU JFNUeEJo6HuQreuxAq+OGniP1O/oon7RUKV2QzeGyYMfvTRLYwRg5Y4ezOsV/2C8ZH6fSjUcv6G1n 4Ix57AymgZJp/M5WxMEq5ZdubeQKIb7sFHOmgU4nci1Ycui5etfphBTv0lQkKZOE34h7QqykqAQdA Tp2LETWjuiW9VknvZ5ub49YoWyMQ1UNIZF/Xq+6SxzC99Y76qQbRdbkbxWk/U8OFCGEczZc2U8e1G 1wSbpcxgBZKZsVhnJX3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raO-00000001gds-2zt8; Mon, 07 Apr 2025 18:52:44 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1quk-00000001Tc2-27Ov for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:43 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-22401f4d35aso51377005ad.2 for ; Mon, 07 Apr 2025 11:09:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049382; x=1744654182; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r2v6JJ+fSgP+xTItzSnoMH89G4xsdIkyOOxFB456+1Q=; b=YBxUSXDJDPVRRX8ZLGDPtaT7Tzmpb2U7oHNyLcYJbKphdpT5cDy57xJq4tRygfaREI /qitzxuMJzT13pldeoelZkH8L/w24/K3pPEM7eJhGXWWKyACSP5QkIcd1WinQnICAodh HEFh2Z2bKp0UkRjpLkEvriuf3hQzwM7SPYobWDMKVGD8M+HmDR3iz7cxoXb5Hm4OfFuI 6nBslIGf11RV1mroki+vQtN0UG66l4L2jNlpgDGqsaQrTfa7LMXc6GeizIBUwQZQHFYP gzZUqgQCq8G6GFCkOtJ4+gwgaIGIKJzu/z0lp+9WQgOO8xhhEa/uXQLBpvIWrgRrMIxF NU1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049382; x=1744654182; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r2v6JJ+fSgP+xTItzSnoMH89G4xsdIkyOOxFB456+1Q=; b=on7MFsmF1v1s8QGA1Gw/MMJTh+PLWLYpH2Fh6fbrQj6sj30qMYmt8uvjyjKkl0pXLy 9mqXG7cvgR2Rz11te3uI3++Vg4sPuvwKZBCmOfJewHc0sDEDQQFWBgoWcykO19TYf/VH kk0Oh9QZMaDzevXHdgTnBc6jqskGfl3FzcHr9Q8l0aJDiL/93YcaC/466cD+bx4QdQr/ ap8XXD+ZtdV57R4zMqm+II+JIbU1Pw0ZdlbFFM6N4cHU5lpx22BgR5v/uaNdtjfc0z59 rxxy39+m/gbpTGUcW9gdQjp/OZyaWqD5sTOhmKULvJoS7h+VERE7SZPcs8iPB/IAlrWq 9A5Q== X-Gm-Message-State: AOJu0YzCeKmsQnb7vrWhyTib6VDh7qPn58MK4RxkkejoBlHx88jCzL21 sRGL6E3rc+bXEt7iuuw6WCHCTmlfAE5IMzACVTmq6Hn0OVlrpekuY4gM6Td3 X-Gm-Gg: ASbGncsZR3OCaO1nFx3WtSR6IvXFF5sPuwanywuqlqKFxnsxv33nHTQ/aJJJ0Lp/JBt znt9HxRvAAPUYvmrqP/bPy5EJqYBJEi2rvYjn9VfRfiD9Z15xeja6bpbRQPBWp2wZhWQINL0GZV qxDx/xGYKDTWoaVdAzYtp82G+ImzLyadrck439xRNbBbsm8tTA2HSffcf56VUDZs/QnqqcxjCdJ DP6iNSvZYe/A5trbnztvgDoXNcmyjUpYDU3Rom5AhgPg2b58U+E3LfemZvNGxEtIM5RhVCTfZHM m+CRLphLUsG0L/4BuucjIzTLALM/RO+w+boIanEwgwACidGizgtrLe2Xb5LbrpzgR98p/3NEVvX 6M1PqtxbbXOjMwR3u7tf5fZDhYevCBlqOqA== X-Google-Smtp-Source: AGHT+IF0udAYv5YxmJjouLsOvzvtc+Qr6BlHLnxA7et5jBerKapGPE4rwrJM9iedDZmRKfzeHULDtQ== X-Received: by 2002:a17:903:1665:b0:223:35cb:e421 with SMTP id d9443c01a7336-22a955a19e8mr145567285ad.49.1744049381669; Mon, 07 Apr 2025 11:09:41 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:41 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Alexandre Ghiti , Mark Rutland , puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 11/12] riscv: ftrace: support direct call using call_ops Date: Tue, 8 Apr 2025 02:08:35 +0800 Message-Id: <20250407180838.42877-11-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110942_545894_2463B196 X-CRM114-Status: GOOD ( 18.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org jump to FTRACE_ADDR if distance is out of reach Co-developed-by: Björn Töpel Signed-off-by: Björn Töpel Signed-off-by: Andy Chiu --- Changelog v4: - New patch since v4 - Include Björn's fix for kprobe (adjusting ftrace address with MCOUNT_INSN_SIZE) - Clean out an unused variable --- arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/ftrace.h | 6 ++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/ftrace.c | 13 ++++----- arch/riscv/kernel/mcount-dyn.S | 51 +++++++++++++++++++++------------ 5 files changed, 48 insertions(+), 27 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index ec986c9120e3..8fdca6345fa3 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -152,7 +152,7 @@ config RISCV select HAVE_DMA_CONTIGUOUS if MMU select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE) select FUNCTION_ALIGNMENT_4B if HAVE_DYNAMIC_FTRACE && RISCV_ISA_C - select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG) select HAVE_DYNAMIC_FTRACE_WITH_ARGS if HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_GRAPH_FUNC diff --git a/arch/riscv/include/asm/ftrace.h b/arch/riscv/include/asm/ftrace.h index 6a5c0a7fb826..22ebea3c2b26 100644 --- a/arch/riscv/include/asm/ftrace.h +++ b/arch/riscv/include/asm/ftrace.h @@ -130,6 +130,9 @@ struct __arch_ftrace_regs { unsigned long sp; unsigned long s0; unsigned long t1; +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + unsigned long direct_tramp; +#endif union { unsigned long args[8]; struct { @@ -223,10 +226,13 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); #define ftrace_graph_func ftrace_graph_func +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr) { arch_ftrace_regs(fregs)->t1 = addr; } +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 2d96197a8abf..b26334075697 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -495,6 +495,9 @@ void asm_offsets(void) OFFSET(STACKFRAME_RA, stackframe, ra); #ifdef CONFIG_FUNCTION_TRACER DEFINE(FTRACE_OPS_FUNC, offsetof(struct ftrace_ops, func)); +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + DEFINE(FTRACE_OPS_DIRECT_CALL, offsetof(struct ftrace_ops, direct_call)); +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ #endif #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS diff --git a/arch/riscv/kernel/ftrace.c b/arch/riscv/kernel/ftrace.c index 30bcf60135d8..d65f06bfb457 100644 --- a/arch/riscv/kernel/ftrace.c +++ b/arch/riscv/kernel/ftrace.c @@ -16,7 +16,7 @@ unsigned long ftrace_call_adjust(unsigned long addr) { if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) - return addr + 8; + return addr + 8 + MCOUNT_AUIPC_SIZE; return addr + MCOUNT_AUIPC_SIZE; } @@ -83,10 +83,9 @@ static const struct ftrace_ops *riscv64_rec_get_ops(struct dyn_ftrace *rec) return ops; } -static int ftrace_rec_set_ops(const struct dyn_ftrace *rec, - const struct ftrace_ops *ops) +static int ftrace_rec_set_ops(const struct dyn_ftrace *rec, const struct ftrace_ops *ops) { - unsigned long literal = rec->ip - 8; + unsigned long literal = ALIGN_DOWN(rec->ip - 12, 8); return patch_text_nosync((void *)literal, &ops, sizeof(ops)); } @@ -117,7 +116,7 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) orig_addr = (unsigned long)&ftrace_caller; distance = addr > orig_addr ? addr - orig_addr : orig_addr - addr; if (distance > JALR_RANGE) - return -EINVAL; + addr = FTRACE_ADDR; return __ftrace_modify_call(pc, addr, false); } @@ -199,15 +198,13 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) { unsigned long caller = rec->ip - MCOUNT_AUIPC_SIZE; - unsigned int call[2]; int ret; - make_call_t0(caller, old_addr, call); ret = ftrace_rec_update_ops(rec); if (ret) return ret; - return __ftrace_modify_call(caller, addr, true); + return __ftrace_modify_call(caller, FTRACE_ADDR, true); } #endif diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index 699684eea7f0..48f6c4f7dca0 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -82,12 +82,9 @@ * +++++++++ **/ .macro SAVE_ABI_REGS - mv t4, sp // Save original SP in T4 addi sp, sp, -FREGS_SIZE_ON_STACK - REG_S t0, FREGS_EPC(sp) REG_S x1, FREGS_RA(sp) - REG_S t4, FREGS_SP(sp) // Put original SP on stack #ifdef HAVE_FUNCTION_GRAPH_FP_TEST REG_S x8, FREGS_S0(sp) #endif @@ -108,9 +105,12 @@ REG_S x15, FREGS_A5(sp) REG_S x16, FREGS_A6(sp) REG_S x17, FREGS_A7(sp) + mv a0, sp + addi a0, a0, FREGS_SIZE_ON_STACK + REG_S a0, FREGS_SP(sp) // Put original SP on stack .endm - .macro RESTORE_ABI_REGS, all=0 + .macro RESTORE_ABI_REGS REG_L t0, FREGS_EPC(sp) REG_L x1, FREGS_RA(sp) #ifdef HAVE_FUNCTION_GRAPH_FP_TEST @@ -139,6 +139,19 @@ .macro PREPARE_ARGS addi a0, t0, -MCOUNT_JALR_SIZE // ip (callsite's jalr insn) +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS + mv a1, ra // parent_ip + REG_L a2, -16(t0) // op + REG_L ra, FTRACE_OPS_FUNC(a2) // op->func +#else + la a1, function_trace_op + REG_L a2, 0(a1) // op + mv a1, ra // parent_ip +#endif + mv a3, sp // regs + .endm + +SYM_FUNC_START(ftrace_caller) #ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS /* * When CALL_OPS is enabled (2 or 4) nops [8B] are placed before the @@ -158,19 +171,17 @@ * t0 is set to ip+8 after the jalr is executed at the callsite, * so we find the associated op at t0-16. */ - mv a1, ra // parent_ip - REG_L a2, -16(t0) // op - REG_L ra, FTRACE_OPS_FUNC(a2) // op->func -#else - la a1, function_trace_op - REG_L a2, 0(a1) // op - mv a1, ra // parent_ip -#endif - mv a3, sp // regs - .endm + REG_L t1, -16(t0) // op Should be SZ_REG instead of 16 -SYM_FUNC_START(ftrace_caller) - mv t1, zero +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* + * If the op has a direct call, handle it immediately without + * saving/restoring registers. + */ + REG_L t1, FTRACE_OPS_DIRECT_CALL(t1) + bnez t1, ftrace_caller_direct +#endif +#endif SAVE_ABI_REGS PREPARE_ARGS @@ -182,10 +193,14 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) jalr ra, 0(ra) #endif RESTORE_ABI_REGS - bnez t1, .Ldirect +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + bnez t1, ftrace_caller_direct +#endif jr t0 -.Ldirect: +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS +SYM_INNER_LABEL(ftrace_caller_direct, SYM_L_LOCAL) jr t1 +#endif SYM_FUNC_END(ftrace_caller) #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS From patchwork Mon Apr 7 18:08:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 14041571 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FC5AC36010 for ; Mon, 7 Apr 2025 18:52:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DUc/CnGM69I2CO5nhDhChWoKHuhjs2gZOxMoofHpHWk=; b=at0nY3eqeNePom /SBsBDXCu2snRFp5n+PaADh4cim7wJcsiAd9RqxmHMh4IaQSEj7Xn9hypCtT1t+iljUY6BWKz8oPV 2XgLdqL/FU1EA0SG6PHKhvvm3LnD7FgzKBm3DbNCGEy7Bkb8GOcnHBOGOMaWNgUgooC049ugq+qcF SCtxI0HRT+0BBQ7whmQpkCSTGwmSSUhd0y4RPfTqOxwEYqeXk5naK/aH+pfbx8cO/9TUzHKhuYm/r Dz/C4H9A9SG8vUb4d9eEFwx9ZR5uDVRkolLF4t0R1gnU10uysQZHLL1QJbmhDE5FcdtU3SZjcVyKl D7O4nji0zM+IUZR8mUjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1raR-00000001ghu-23cn; Mon, 07 Apr 2025 18:52:47 +0000 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by bombadil.infradead.org with esmtps (Exim 4.98.1 #2 (Red Hat Linux)) id 1u1qup-00000001Tdw-1hwU for linux-riscv@lists.infradead.org; Mon, 07 Apr 2025 18:09:48 +0000 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-2255003f4c6so42077045ad.0 for ; Mon, 07 Apr 2025 11:09:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744049386; x=1744654186; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fyM5SZww3BsIxLq5imP6WPb97ECb0VvvV7XlSlZ6uaA=; b=iNxMPko+4rMMJzFUl3cPfQi1VOsT5CDHM7S8r3n7osgNggJ6/HRg8qYldTIv6tmgoa FoxAmxW0QFTa/fTIhkc+BQ1K2afrVaoYh1ghpcHj/l/L/l7DuIGCmjtSq+tLdGP5Duon OeCyTHybavOaZoHIonU9hikkDRoSzr60MqWvFJRFZNVVEptGt8TgPGEc0LCMH+eB+9Gj XQ9o3eYOCe2nPWulLIINFYJSHva3Wy7q544HGO4ByXjcGdLnuyb5UvdyiLW3pP9POYi0 hUZh4/RJtKx6/FvQ0PFKAih86D/A0Mm9o45QF0S6rSR+og4iwrgBcKpmwCIGEdHqJxYn YaVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744049386; x=1744654186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fyM5SZww3BsIxLq5imP6WPb97ECb0VvvV7XlSlZ6uaA=; b=p56cg38U/y0vrOIrE7EYccgquAE/ILcDOEDqN6Q81hIUJx9dQJ/RsXGhkAMlgFKHWk eZR6a+QQTo6vRU7TyVqnc63N1/R+V9Z4icgiKLYLRsnpvWxkOEU/ucGh8on39iLMfwUF eny1R05ira1kZLNAvY2apRIBtUfBHE9YYVMhLqXzVTSvgKGxEOwYhwunjLQZ52Ze/wCy 6y9ZEdoEuvjParERcqdsvCA/6Cz7lfJOm3ePSACqux269aiKeNjJAo9iZcUWZMO3vv+Q /GPPbq94Y5b6/sDEgw4cHT/Z3orn3jvUHgTs1b+uVUr52VC93SEMx/Jbr/ef5MjOoir7 Zpvg== X-Gm-Message-State: AOJu0Yz8l4VGEwFWlp4xnVJhL/BeABz8EjTd5uU+m3elRtE83zIxfCGq 9ujJBJJyv9ag24w5+13mxzucZF4Fj+Gjt0xQzzTuFdTmCzudiahhA4atrlDv X-Gm-Gg: ASbGnctOyUqXvv2J417uamFnX+edLHtzFxT1/NDdIPcdDeFTlmuje3Du+gYM39rq8lg jmsZg700hE9Bo19SEMyo205CQXpVnU8qrmVxCy5aOWmEzVSzOS0P2bK2mZQ0dmrq2PgtYS2odNf TUF/CTiheF2yMPaAK98CnyyhYKnfjFwY2oRs83LOOlijVwU0RJXGacUbiDWM7etVdokGn3YHhXV Mpxr17WWPKpdcPBdiZAJIoqKRPnhjnmE+2CabV0B8++hiEXzxCeqTbFL1NV+0Zh60xUTg7aISId 37+erI6mU12tzlBFw+VTFG1bsAKLKDmNFf2HhRt/z4QWvhe+jcx+pm66aT143F4ege3v46gzBOw uuBzKF5svfvI+cagOehDZF2Yr/xcSp1RTqA== X-Google-Smtp-Source: AGHT+IH4QUToL7D8e8p/6On2iiP6OQv/SkBOfhg4jmTT3eYoxpzbIuK2HeFgMoBbObddEAjzAiLW1w== X-Received: by 2002:a17:902:c40c:b0:224:c47:cbd with SMTP id d9443c01a7336-22a89ebe8e4mr191891385ad.0.1744049386163; Mon, 07 Apr 2025 11:09:46 -0700 (PDT) Received: from localhost.localdomain (118-160-134-247.dynamic-ip.hinet.net. [118.160.134.247]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2297865e03esm84282455ad.146.2025.04.07.11.09.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Apr 2025 11:09:45 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, alexghiti@rivosinc.com, palmer@dabbelt.com Cc: Andy Chiu , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Alexandre Ghiti , bjorn@rivosinc.com, puranjay12@gmail.com, paul.walmsley@sifive.com, greentime.hu@sifive.com, nick.hu@sifive.com, nylon.chen@sifive.com, eric.lin@sifive.com, vicent.chen@sifive.com, zong.li@sifive.com, yongxuan.wang@sifive.com, samuel.holland@sifive.com, olivia.chu@sifive.com, c2232430@gmail.com Subject: [PATCH v4 12/12] riscv: Documentation: add a description about dynamic ftrace Date: Tue, 8 Apr 2025 02:08:36 +0800 Message-Id: <20250407180838.42877-12-andybnac@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20250407180838.42877-1-andybnac@gmail.com> References: <20250407180838.42877-1-andybnac@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250407_110947_442945_D99666EE X-CRM114-Status: GOOD ( 16.18 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add a section in cmodx to describe how dynamic ftrace works on riscv, limitations, and assumptions. Signed-off-by: Andy Chiu --- Documentation/arch/riscv/cmodx.rst | 46 +++++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/Documentation/arch/riscv/cmodx.rst b/Documentation/arch/riscv/cmodx.rst index 8c48bcff3df9..e009873b2d17 100644 --- a/Documentation/arch/riscv/cmodx.rst +++ b/Documentation/arch/riscv/cmodx.rst @@ -10,13 +10,45 @@ modified by the program itself. Instruction storage and the instruction cache program must enforce its own synchronization with the unprivileged fence.i instruction. -However, the default Linux ABI prohibits the use of fence.i in userspace -applications. At any point the scheduler may migrate a task onto a new hart. If -migration occurs after the userspace synchronized the icache and instruction -storage with fence.i, the icache on the new hart will no longer be clean. This -is due to the behavior of fence.i only affecting the hart that it is called on. -Thus, the hart that the task has been migrated to may not have synchronized -instruction storage and icache. +CMODX in the Kernel Space +--------------------- + +Dynamic ftrace +--------------------- + +Essentially, dynamic ftrace directs the control flow by inserting a function +call at each patchable function entry, and patches it dynamically at runtime to +enable or disable the redirection. In the case of RISC-V, 2 instructions, +AUIPC + JALR, are required to compose a function call. However, it is impossible +to patch 2 instructions and expect that a concurrent read-side executes them +without a race condition. This series makes atmoic code patching possible in +RISC-V ftrace. Kernel preemption makes things even worse as it allows the old +state to persist across the patching process with stop_machine(). + +In order to get rid of stop_machine() and run dynamic ftrace with full kernel +preemption, we partially initialize each patchable function entry at boot-time, +setting the first instruction to AUIPC, and the second to NOP. Now, atmoic +patching is possible because the kernel only has to update one instruction. +According to Ziccif, as long as an instruction is naturally aligned, the ISA +guarantee an atomic update. + +By fixing down the first instruction, AUIPC, the range of the ftrace trampoline +is limited to +-2K from the predetermined target, ftrace_caller, due to the lack +of immediate encoding space in RISC-V. To address the issue, we introduce +CALL_OPS, where an 8B naturally align metadata is added in front of each +pacthable function. The metadata is resolved at the first trampoline, then the +execution can be derect to another custom trampoline. + +CMODX in the User Space +--------------------- + +Though fence.i is an unprivileged instruction, the default Linux ABI prohibits +the use of fence.i in userspace applications. At any point the scheduler may +migrate a task onto a new hart. If migration occurs after the userspace +synchronized the icache and instruction storage with fence.i, the icache on the +new hart will no longer be clean. This is due to the behavior of fence.i only +affecting the hart that it is called on. Thus, the hart that the task has been +migrated to may not have synchronized instruction storage and icache. There are two ways to solve this problem: use the riscv_flush_icache() syscall, or use the ``PR_RISCV_SET_ICACHE_FLUSH_CTX`` prctl() and emit fence.i in