From patchwork Wed Dec 11 13:33:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903472 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0396223FD1E; Wed, 11 Dec 2024 13:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924061; cv=none; b=AKPf1P6gkk8HdujHPz0l3CbLU7pJAIZ0JtzGpiJv3YfAbjxgu/L7p9dhOwygeTh8g3IztON5SqAJxsfdsG8Pv87WV/IEwAzloZZdUV5b8RR04zapjrNoTdeVIyyfmVV6GyPFDWmBkfVFxhqMru8p5SjvEtOJvi4+xoQzC7kjIH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924061; c=relaxed/simple; bh=Z7IK1SBqeAl5DybsxkEFcQSaPfCG+Bu6/uH1+f3Yn5Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X+LHo3IoG0kyZNdSqITopwzOmVkxYOx4N6UGkWo4rnwQospZYu7MKaG57zmYoBpF/D5dPSZT+mdk732JKBJKBQHSi30hB2Pt5zF+dVqihK4iKJqskDT+12sOV+3cWNTJ5q9wr46PDpU1TUTZUBJIi7JIurTnSaXVD019GYovBAM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qWd7yEfV; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qWd7yEfV" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 330CCC4CED2; Wed, 11 Dec 2024 13:34:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924060; bh=Z7IK1SBqeAl5DybsxkEFcQSaPfCG+Bu6/uH1+f3Yn5Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qWd7yEfVgO54c8ga+ehSoBNk5mZe1KzZ4jRcRou0UuTHfdK8eap9TWXL0hppYq0BC XS/v/UqiOt+nDTXoxG17NnhVoTQno9NQGPJ5RrHvAwmshr3ePLt/WOYfhzt63wNAEc MZnetHnLk5zTYWT2/ffY3vaz+l1k1CkBEKX+8K3jMETfQYQ7FAwfDRMNubR0UlaBsM R7BmyTvr7JCcUBvY4e6o3vJ9bpR/V0XlcFgTFhAgYmCApiXP7fBsidi5XA1K0ounUv FFkz4gpMQpKkEmb6XwBcEiZIKhLz3AJSuhpY8ZL71UyQoqjr5Yeb/eHZfueBxPf7A8 SyklJ+hNN37Bw== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 01/13] uprobes: Rename arch_uretprobe_trampoline function Date: Wed, 11 Dec 2024 14:33:50 +0100 Message-ID: <20241211133403.208920-2-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net We are about to add uprobe trampoline, so cleaning up the namespace. Signed-off-by: Jiri Olsa Acked-by: Andrii Nakryiko --- arch/x86/kernel/uprobes.c | 2 +- include/linux/uprobes.h | 2 +- kernel/events/uprobes.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 5a952c5ea66b..22a17c149a55 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -338,7 +338,7 @@ extern u8 uretprobe_trampoline_entry[]; extern u8 uretprobe_trampoline_end[]; extern u8 uretprobe_syscall_check[]; -void *arch_uprobe_trampoline(unsigned long *psize) +void *arch_uretprobe_trampoline(unsigned long *psize) { static uprobe_opcode_t insn = UPROBE_SWBP_INSN; struct pt_regs *regs = task_pt_regs(current); diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index e0a4c2082245..09a298e416a8 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -211,7 +211,7 @@ extern bool arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs); extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr, void *src, unsigned long len); extern void uprobe_handle_trampoline(struct pt_regs *regs); -extern void *arch_uprobe_trampoline(unsigned long *psize); +extern void *arch_uretprobe_trampoline(unsigned long *psize); extern unsigned long uprobe_get_trampoline_vaddr(void); #else /* !CONFIG_UPROBES */ struct uprobes_state { diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index fa04b14a7d72..e0e3ebb4c0a1 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1695,7 +1695,7 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area) return ret; } -void * __weak arch_uprobe_trampoline(unsigned long *psize) +void * __weak arch_uretprobe_trampoline(unsigned long *psize) { static uprobe_opcode_t insn = UPROBE_SWBP_INSN; @@ -1727,7 +1727,7 @@ static struct xol_area *__create_xol_area(unsigned long vaddr) init_waitqueue_head(&area->wq); /* Reserve the 1st slot for get_trampoline_vaddr() */ set_bit(0, area->bitmap); - insns = arch_uprobe_trampoline(&insns_size); + insns = arch_uretprobe_trampoline(&insns_size); arch_uprobe_copy_ixol(area->page, 0, insns, insns_size); if (!xol_add_vma(mm, area)) From patchwork Wed Dec 11 13:33:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903473 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1BB622B598; Wed, 11 Dec 2024 13:34:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924071; cv=none; b=JeOwi4oAUVNl8p3Du9LvTYQWmG1pycxqCGQCpS5fLBl5UrNzVbIYpwsuu6eQOOuE3YQsD+iGq7dQpDv/oRIQiGb6LaZXtEfBuZBU40VDmn2/JhZIglOz+i/zyDf2S8RTlwQHs3ncESlMdeGeZ6JbEfpZD0pLnHExttRp3Gpjw4U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924071; c=relaxed/simple; bh=XwvKn2JFR+eGxw1oc/+JL9gG50JFxNqYNkVyrcynD94=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FZwmq02+1dEm4q1GQec1CXZFasguzYsSS9MCfTup3FdpbnTDaY++uY5iiYNLppBZ0742+56upHUD55yGlCN0vycSIdMCekR0Ehm6W1ayxN7RJdauQBYGiABee7taZoV/tx+R3rA03vG3+9yYLStEvPR+rB+xNwRD63YbP/K5mJQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sf/Z1ow7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sf/Z1ow7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08B5CC4CED2; Wed, 11 Dec 2024 13:34:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924071; bh=XwvKn2JFR+eGxw1oc/+JL9gG50JFxNqYNkVyrcynD94=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sf/Z1ow7e05oJ0nWRVHerNcefybnvDi5tQyCZ0A/d2ecGKoLClnKJAWeAvTHBYFC9 4fNkaLbIVXua9XJ6RPn8aJPeQda64DYoLHkSiYDU5BT3fck39YmLfqpPDMc+uXA5Tv Q3kCs9ttB/Q8p8dnNMYka/2GBcB+HDAX2q9FdeyIb8QJ6VszmusOPbzWdanObwSYXg KRkm/z4kQ1dWjYJUcdYQPasGCo/4p/0TQRhvWTnnIxUvP/Db+sPU3u75usIM0+OP0K 5BjkyzykBq2XjudgCrl6Lt1FZkYDig7eI1OlH8PatbbqbRWub3YX9HfpH55jrhsplD Pu+GYnpNGs+Bg== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 02/13] uprobes: Make copy_from_page global Date: Wed, 11 Dec 2024 14:33:51 +0100 Message-ID: <20241211133403.208920-3-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Making copy_from_page global and adding uprobe prefix. Adding the uprobe prefix to copy_to_page as well for symmetry. Signed-off-by: Jiri Olsa Acked-by: Andrii Nakryiko --- include/linux/uprobes.h | 1 + kernel/events/uprobes.c | 16 ++++++++-------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 09a298e416a8..e24fbe496efb 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -213,6 +213,7 @@ extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr, extern void uprobe_handle_trampoline(struct pt_regs *regs); extern void *arch_uretprobe_trampoline(unsigned long *psize); extern unsigned long uprobe_get_trampoline_vaddr(void); +extern void uprobe_copy_from_page(struct page *page, unsigned long vaddr, void *dst, int len); #else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index e0e3ebb4c0a1..61ec91f635dc 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -249,14 +249,14 @@ bool __weak is_trap_insn(uprobe_opcode_t *insn) return is_swbp_insn(insn); } -static void copy_from_page(struct page *page, unsigned long vaddr, void *dst, int len) +void uprobe_copy_from_page(struct page *page, unsigned long vaddr, void *dst, int len) { void *kaddr = kmap_atomic(page); memcpy(dst, kaddr + (vaddr & ~PAGE_MASK), len); kunmap_atomic(kaddr); } -static void copy_to_page(struct page *page, unsigned long vaddr, const void *src, int len) +static void uprobe_copy_to_page(struct page *page, unsigned long vaddr, const void *src, int len) { void *kaddr = kmap_atomic(page); memcpy(kaddr + (vaddr & ~PAGE_MASK), src, len); @@ -277,7 +277,7 @@ static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t * is a trap variant; uprobes always wins over any other (gdb) * breakpoint. */ - copy_from_page(page, vaddr, &old_opcode, UPROBE_SWBP_INSN_SIZE); + uprobe_copy_from_page(page, vaddr, &old_opcode, UPROBE_SWBP_INSN_SIZE); is_swbp = is_swbp_insn(&old_opcode); if (is_swbp_insn(new_opcode)) { @@ -524,7 +524,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, __SetPageUptodate(new_page); copy_highpage(new_page, old_page); - copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + uprobe_copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); if (!is_register) { struct page *orig_page; @@ -1026,7 +1026,7 @@ static int __copy_insn(struct address_space *mapping, struct file *filp, if (IS_ERR(page)) return PTR_ERR(page); - copy_from_page(page, offset, insn, nbytes); + uprobe_copy_from_page(page, offset, insn, nbytes); put_page(page); return 0; @@ -1367,7 +1367,7 @@ struct uprobe *uprobe_register(struct inode *inode, return ERR_PTR(-EINVAL); /* - * This ensures that copy_from_page(), copy_to_page() and + * This ensures that uprobe_copy_from_page(), uprobe_copy_to_page() and * __update_ref_ctr() can't cross page boundary. */ if (!IS_ALIGNED(offset, UPROBE_SWBP_INSN_SIZE)) @@ -1856,7 +1856,7 @@ void __weak arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr, void *src, unsigned long len) { /* Initialize the slot */ - copy_to_page(page, vaddr, src, len); + uprobe_copy_to_page(page, vaddr, src, len); /* * We probably need flush_icache_user_page() but it needs vma. @@ -2287,7 +2287,7 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) if (result < 0) return result; - copy_from_page(page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + uprobe_copy_from_page(page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); put_page(page); out: /* This needs to return true for any variant of the trap insn */ From patchwork Wed Dec 11 13:33:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903474 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CF64229126; Wed, 11 Dec 2024 13:34:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924082; cv=none; b=kDE9OdKY6c5Y9UuN/lco2k9Cnye+Ae5Nb5dvlUu/zMP+e6/eA/ioYLMpI89pNg+zpXFwNDAFfViUgWcu+3pM4kM0DnkfPZtaICEQNQqG7q06Sby2SgvLRUD25a5rhvrWNmXInW1lSdVQOefxdlTCHR/hBwqPcCle7ZUgyYHqb1Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924082; c=relaxed/simple; bh=JATuSHj8SuCkBQq5YKjQDIs8JNzgBKNYnCgW7ApmQfI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XM2reorf9d5khjiLCi8GzxiSE95Ndn/4M4WsgKkJ6j6gbwOk7zrHETonAfO+LM1lCBPbwbZvyQJ+8d7DvYp4CvoZYse9VKJqXs7gRKto6Op9JQ/gzIl5heE+jH1cYCQnt9KudCA7Ui/jXt9fxUJ84FRJUT9WwzzALBZ0sbI0BTk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oY+sRvGL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oY+sRvGL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A1076C4CED2; Wed, 11 Dec 2024 13:34:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924081; bh=JATuSHj8SuCkBQq5YKjQDIs8JNzgBKNYnCgW7ApmQfI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oY+sRvGLFwypjI2ecJCnd0URsdR/C5OeMOXV9wmXgTq3CQIzEZJvG9bhpg4wkS/Yx 7ljZ92jKwP3SAI6Xqt/5g4Q5siPHphJjpO5W4Qqw57R6Iei4P+nXVUUIQYjqOGN3in BPLB0m1Bup6AXlH7T2l+yg5BgUi7svQH+UrH5ZlQ84HWVWhmr5mBCu61G3LZpRnMRw ibhRSpF1s/94SqyUQUuwjPk0gGPNC2MNzBkCosP9KQlknJ1C0sMoK5/Gd2THXHSf13 oexQUqC9z06VqitpMpvSdFdmXHyxACRGLiRLz0d4q+qcKLuu1uzFb0Ch29HTM1ztmK q8LodqZT2UtEQ== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 03/13] uprobes: Add nbytes argument to uprobe_write_opcode Date: Wed, 11 Dec 2024 14:33:52 +0100 Message-ID: <20241211133403.208920-4-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding nbytes argument to uprobe_write_opcode as preparation fo writing longer instructions in following changes. Signed-off-by: Jiri Olsa Acked-by: Andrii Nakryiko --- include/linux/uprobes.h | 3 ++- kernel/events/uprobes.c | 14 ++++++++------ 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index e24fbe496efb..cc723bc48c1d 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -181,7 +181,8 @@ extern bool is_swbp_insn(uprobe_opcode_t *insn); extern bool is_trap_insn(uprobe_opcode_t *insn); extern unsigned long uprobe_get_swbp_addr(struct pt_regs *regs); extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs); -extern int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr, uprobe_opcode_t); +extern int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr, + uprobe_opcode_t *insn, int nbytes); extern struct uprobe *uprobe_register(struct inode *inode, loff_t offset, loff_t ref_ctr_offset, struct uprobe_consumer *uc); extern int uprobe_apply(struct uprobe *uprobe, struct uprobe_consumer *uc, bool); extern void uprobe_unregister_nosync(struct uprobe *uprobe, struct uprobe_consumer *uc); diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 61ec91f635dc..7c2ecf11a573 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -470,7 +470,7 @@ static int update_ref_ctr(struct uprobe *uprobe, struct mm_struct *mm, * Return 0 (success) or a negative errno. */ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, - unsigned long vaddr, uprobe_opcode_t opcode) + unsigned long vaddr, uprobe_opcode_t *insn, int nbytes) { struct uprobe *uprobe; struct page *old_page, *new_page; @@ -479,7 +479,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, bool orig_page_huge = false; unsigned int gup_flags = FOLL_FORCE; - is_register = is_swbp_insn(&opcode); + is_register = is_swbp_insn(insn); uprobe = container_of(auprobe, struct uprobe, arch); retry: @@ -490,7 +490,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (IS_ERR(old_page)) return PTR_ERR(old_page); - ret = verify_opcode(old_page, vaddr, &opcode); + ret = verify_opcode(old_page, vaddr, insn); if (ret <= 0) goto put_old; @@ -524,7 +524,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, __SetPageUptodate(new_page); copy_highpage(new_page, old_page); - uprobe_copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + uprobe_copy_to_page(new_page, vaddr, insn, nbytes); if (!is_register) { struct page *orig_page; @@ -581,7 +581,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, */ int __weak set_swbp(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr) { - return uprobe_write_opcode(auprobe, mm, vaddr, UPROBE_SWBP_INSN); + uprobe_opcode_t insn = UPROBE_SWBP_INSN; + + return uprobe_write_opcode(auprobe, mm, vaddr, &insn, UPROBE_SWBP_INSN_SIZE); } /** @@ -597,7 +599,7 @@ int __weak set_orig_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr) { return uprobe_write_opcode(auprobe, mm, vaddr, - *(uprobe_opcode_t *)&auprobe->insn); + (uprobe_opcode_t *)&auprobe->insn, UPROBE_SWBP_INSN_SIZE); } /* uprobe should have guaranteed positive refcount */ From patchwork Wed Dec 11 13:33:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903475 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1270722C348; Wed, 11 Dec 2024 13:34:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924093; cv=none; b=hCPf5nHYjNOtU81W9wJ3bXLHtBvueRucdu6Nmw3CF2gdbRVLikHtH9pGDAshrwnLWTifc4RoDPaTXV4dkbGaSePFxBdj8ZIidCZhb7ceKr9T0vNRMdSNAwo8IXdBcXfO/2lCEdpYcthq34vKu+OdlDxX8jYh3ueLSammcLNqgq8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924093; c=relaxed/simple; bh=ZP/siBTV/mabKudodcX5uGu2nCMI/KrZ0Kd+sWfYlMM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kpXQ/c0XVq/C+RGcFRxgnkfBBMFewZyMDrZmIPfAJGKMrci/llGEW2ked1KrYkvImUXu+EVGsa79UknRCe5eD7gSybDW0GqFM5ES0fVL0qQ/fmYLWuVC7FN5kkim8q6UnPUmfLnTAGZUjBIx7RHI0hz98duX0F5wpgyc6vUaDz8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H+X7iiGC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H+X7iiGC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67E8DC4CEDD; Wed, 11 Dec 2024 13:34:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924092; bh=ZP/siBTV/mabKudodcX5uGu2nCMI/KrZ0Kd+sWfYlMM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H+X7iiGCVD0RmirccVJMNebhdpnzTIRVKw9UoIw6CI7GjZ1MtRFc3l3IyvCAaGcR+ B4G6YHecYMdXz2YNwz1Dg3GS3UbM22o4iRYwMLf5K5+4cegKNbgiT0b5K04dQ4pCIX USukRza+PiqV/+hIZRMc/ro9KcQKdJs35bAEmRade6winJb6XkF6814OuygKspYmDk YZfX44DafkKxfaRRZyeDE1rN5b9H1+Kg0+g7fFoFSU/Vae77/X2pwwb9oEjAZDFwAD IGkFHmfcbeWE+l4cVx0IzL3P3oQ/c855ViP7yRAHhsCSzTKTM+ioyFjpZb86VGzvxC rssXeuVoOm22Q== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 04/13] uprobes: Add arch_uprobe_verify_opcode function Date: Wed, 11 Dec 2024 14:33:53 +0100 Message-ID: <20241211133403.208920-5-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding arch_uprobe_verify_opcode function, so we can overload verification for each architecture in following changes. Signed-off-by: Jiri Olsa --- include/linux/uprobes.h | 5 +++++ kernel/events/uprobes.c | 19 ++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index cc723bc48c1d..8843b7f99ed0 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -215,6 +215,11 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs); extern void *arch_uretprobe_trampoline(unsigned long *psize); extern unsigned long uprobe_get_trampoline_vaddr(void); extern void uprobe_copy_from_page(struct page *page, unsigned long vaddr, void *dst, int len); +extern int uprobe_verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t *new_opcode); +extern int arch_uprobe_verify_opcode(struct arch_uprobe *auprobe, struct page *page, + unsigned long vaddr, uprobe_opcode_t *new_opcode, + int nbytes); +extern bool arch_uprobe_is_register(uprobe_opcode_t *insn, int nbytes); #else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 7c2ecf11a573..8068f91de9e3 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -263,7 +263,13 @@ static void uprobe_copy_to_page(struct page *page, unsigned long vaddr, const vo kunmap_atomic(kaddr); } -static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t *new_opcode) +__weak bool arch_uprobe_is_register(uprobe_opcode_t *insn, int nbytes) +{ + return is_swbp_insn(insn); +} + +int uprobe_verify_opcode(struct page *page, unsigned long vaddr, + uprobe_opcode_t *new_opcode) { uprobe_opcode_t old_opcode; bool is_swbp; @@ -291,6 +297,13 @@ static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t return 1; } +__weak int arch_uprobe_verify_opcode(struct arch_uprobe *auprobe, struct page *page, + unsigned long vaddr, uprobe_opcode_t *new_opcode, + int nbytes) +{ + return uprobe_verify_opcode(page, vaddr, new_opcode); +} + static struct delayed_uprobe * delayed_uprobe_check(struct uprobe *uprobe, struct mm_struct *mm) { @@ -479,7 +492,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, bool orig_page_huge = false; unsigned int gup_flags = FOLL_FORCE; - is_register = is_swbp_insn(insn); + is_register = arch_uprobe_is_register(insn, nbytes); uprobe = container_of(auprobe, struct uprobe, arch); retry: @@ -490,7 +503,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (IS_ERR(old_page)) return PTR_ERR(old_page); - ret = verify_opcode(old_page, vaddr, insn); + ret = arch_uprobe_verify_opcode(auprobe, old_page, vaddr, insn, nbytes); if (ret <= 0) goto put_old; From patchwork Wed Dec 11 13:33:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903476 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 294DC22C363; Wed, 11 Dec 2024 13:35:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924104; cv=none; b=qyYK7mpbfuhxaDam5A9EaYy83JdNFRaTy5GAGA4z7AyTWlx3TG5Cu9rtSEaiWhfENnXiKsdBipYFJ7Qnv7xnEOJiopU1XtM0pmtRrema5J5cDbgJL8+u3HhEe9v0vMu7JuZFEUl7YOGuYRNNnPNkPR/Hf5klSYVqXZK+EnhZa4k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924104; c=relaxed/simple; bh=8jQBJMTqI5SGCH3HyoZGPUbDCqZg31PSW38saJQZTcA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QCzroADtrsSi7VAJMtZJWoFRIqYMY5XS0XfH6llchzJ9P0PWgMDsIO3OjRv1IT/fuC2Z9S0wMAOpSClZr5Uzl4Z5Sya3zFO/nhdvnGYf1tV71xdk4SENPJ3s4pU4Ff0n46jxt52B34K6boIDpSVoVegmOzSH+O+fGlRvhOiSSiY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GWLeR9XV; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GWLeR9XV" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C1CC5C4CED2; Wed, 11 Dec 2024 13:35:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924104; bh=8jQBJMTqI5SGCH3HyoZGPUbDCqZg31PSW38saJQZTcA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GWLeR9XVPcag6gwCl+YJ8qb8cBu9XPv9ohkYqnBSEYI5YqQREFsCjPbYKx2Hy1srC 9JrgmTaRTldzL/hxnqgl76d49mWPGHptGTnlGWldIMJDV1y/5XTOvs6Ie3HWVIuwV8 8K9ec6QkVX518yRj5hyCGv+0TJJR6UpoRJqY1SdY/AOLOkEyVUVNBkj+efi6XlzG/p U5uQe6ts0JKObq/EFGZLa8S3kjj0ilFBOWnuqd2q/wn9CU41uzufRprulFgfxlwqOo wSOG7tKbSfi/PQ6Ys4xygiqV3WSh7PkgSGEskoji+VjED9vadLdDVFvvznBNQ1gqMg DSRyVmX40mxQg== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 05/13] uprobes: Add mapping for optimized uprobe trampolines Date: Wed, 11 Dec 2024 14:33:54 +0100 Message-ID: <20241211133403.208920-6-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding support to add special mapping for for user space trampoline with following functions: uprobe_trampoline_get - find or add related uprobe_trampoline uprobe_trampoline_put - remove ref or destroy uprobe_trampoline The user space trampoline is exported as architecture specific user space special mapping, which is provided by arch_uprobe_trampoline_mapping function. The uprobe trampoline needs to be callable/reachable from the probe address, so while searching for available address we use arch_uprobe_is_callable function to decide if the uprobe trampoline is callable from the probe address. All uprobe_trampoline objects are stored in uprobes_state object and are cleaned up when the process mm_struct goes down. Locking is provided by callers in following changes. Signed-off-by: Jiri Olsa --- include/linux/uprobes.h | 12 +++++ kernel/events/uprobes.c | 114 ++++++++++++++++++++++++++++++++++++++++ kernel/fork.c | 1 + 3 files changed, 127 insertions(+) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 8843b7f99ed0..c4ee755ca2a1 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -16,6 +16,7 @@ #include #include #include +#include struct uprobe; struct vm_area_struct; @@ -172,6 +173,13 @@ struct xol_area; struct uprobes_state { struct xol_area *xol_area; + struct hlist_head tramp_head; +}; + +struct uprobe_trampoline { + struct hlist_node node; + unsigned long vaddr; + atomic64_t ref; }; extern void __init uprobes_init(void); @@ -220,6 +228,10 @@ extern int arch_uprobe_verify_opcode(struct arch_uprobe *auprobe, struct page *p unsigned long vaddr, uprobe_opcode_t *new_opcode, int nbytes); extern bool arch_uprobe_is_register(uprobe_opcode_t *insn, int nbytes); +extern struct uprobe_trampoline *uprobe_trampoline_get(unsigned long vaddr); +extern void uprobe_trampoline_put(struct uprobe_trampoline *area); +extern bool arch_uprobe_is_callable(unsigned long vtramp, unsigned long vaddr); +extern const struct vm_special_mapping *arch_uprobe_trampoline_mapping(void); #else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 8068f91de9e3..f57918c624da 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -615,6 +615,118 @@ set_orig_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long v (uprobe_opcode_t *)&auprobe->insn, UPROBE_SWBP_INSN_SIZE); } +bool __weak arch_uprobe_is_callable(unsigned long vtramp, unsigned long vaddr) +{ + return false; +} + +const struct vm_special_mapping * __weak arch_uprobe_trampoline_mapping(void) +{ + return NULL; +} + +static unsigned long find_nearest_page(unsigned long vaddr) +{ + struct mm_struct *mm = current->mm; + struct vm_area_struct *vma, *prev; + VMA_ITERATOR(vmi, mm, 0); + + prev = vma_next(&vmi); + vma = vma_next(&vmi); + while (vma) { + if (vma->vm_start - prev->vm_end >= PAGE_SIZE) { + if (arch_uprobe_is_callable(prev->vm_end, vaddr)) + return prev->vm_end; + if (arch_uprobe_is_callable(vma->vm_start - PAGE_SIZE, vaddr)) + return vma->vm_start - PAGE_SIZE; + } + + prev = vma; + vma = vma_next(&vmi); + } + + return 0; +} + +static struct uprobe_trampoline *create_uprobe_trampoline(unsigned long vaddr) +{ + const struct vm_special_mapping *mapping; + struct mm_struct *mm = current->mm; + struct vm_area_struct *vma; + struct uprobe_trampoline *tramp; + + mapping = arch_uprobe_trampoline_mapping(); + if (!mapping) + return NULL; + + vaddr = find_nearest_page(vaddr); + if (!vaddr) + return NULL; + + tramp = kzalloc(sizeof(*tramp), GFP_KERNEL); + if (unlikely(!tramp)) + return NULL; + + atomic64_set(&tramp->ref, 1); + tramp->vaddr = vaddr; + + vma = _install_special_mapping(mm, tramp->vaddr, PAGE_SIZE, + VM_READ|VM_EXEC|VM_MAYEXEC|VM_MAYREAD|VM_DONTCOPY|VM_IO, + mapping); + if (IS_ERR(vma)) + goto free_area; + return tramp; + + free_area: + kfree(tramp); + return NULL; +} + +struct uprobe_trampoline *uprobe_trampoline_get(unsigned long vaddr) +{ + struct uprobes_state *state = ¤t->mm->uprobes_state; + struct uprobe_trampoline *tramp = NULL; + + hlist_for_each_entry(tramp, &state->tramp_head, node) { + if (arch_uprobe_is_callable(tramp->vaddr, vaddr)) { + atomic64_inc(&tramp->ref); + return tramp; + } + } + + tramp = create_uprobe_trampoline(vaddr); + if (!tramp) + return NULL; + + hlist_add_head(&tramp->node, &state->tramp_head); + return tramp; +} + +static void destroy_uprobe_trampoline(struct uprobe_trampoline *tramp) +{ + hlist_del(&tramp->node); + kfree(tramp); +} + +void uprobe_trampoline_put(struct uprobe_trampoline *tramp) +{ + if (tramp == NULL) + return; + + if (atomic64_dec_and_test(&tramp->ref)) + destroy_uprobe_trampoline(tramp); +} + +static void clear_tramp_head(struct mm_struct *mm) +{ + struct uprobes_state *state = &mm->uprobes_state; + struct uprobe_trampoline *tramp; + struct hlist_node *n; + + hlist_for_each_entry_safe(tramp, n, &state->tramp_head, node) + destroy_uprobe_trampoline(tramp); +} + /* uprobe should have guaranteed positive refcount */ static struct uprobe *get_uprobe(struct uprobe *uprobe) { @@ -1787,6 +1899,8 @@ void uprobe_clear_state(struct mm_struct *mm) delayed_uprobe_remove(NULL, mm); mutex_unlock(&delayed_uprobe_lock); + clear_tramp_head(mm); + if (!area) return; diff --git a/kernel/fork.c b/kernel/fork.c index 1450b461d196..b734a172fd6e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1254,6 +1254,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) { #ifdef CONFIG_UPROBES mm->uprobes_state.xol_area = NULL; + INIT_HLIST_HEAD(&mm->uprobes_state.tramp_head); #endif } From patchwork Wed Dec 11 13:33:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903477 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41DF92336AF; Wed, 11 Dec 2024 13:35:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924115; cv=none; b=O1d53SdFGCrXhfXqiJc/MEM24CvlcLCA9HqJlBhklhjZfE+WU/YmHmUl/AdKSsbMaZs/j/6hgVn37QQ95jilKELJdqXmJARfx7u75oW8nmTngwxAmHY/sroWrU1nHAPOAUzqrR9hT5E1snPsuVr3+GrblX4Dk9wYBWqA27ylUe4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924115; c=relaxed/simple; bh=VIt9RfxU3t70Tz8wlMWKsiA5ewmGYP2mYGpmu4K6NcA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JNenlUTnITmT6xsX7Dw6dP8KtlgRhZboMQtzYqWr5yJ4b+GeAZ64zM8RylyoayFC75su28Uib9G/h7TaB6r4dAduN6B5e1IcNlS98vjQFJj0npZtyvC1+JjEoluRgCIfekIt2GBb583eyMvE+yure97gMvVMqX4LQgMda73iOSo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oRPVIdlM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oRPVIdlM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 865E6C4CED2; Wed, 11 Dec 2024 13:35:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924114; bh=VIt9RfxU3t70Tz8wlMWKsiA5ewmGYP2mYGpmu4K6NcA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oRPVIdlMS8l0qIH/hPJSddv+/QbWdk8PFUmsx0rF3/iEA82vMuRYUWQWXYLCu15e/ g4XCnyTsbZFBstrNNqjfC+ak1iS/OqN5KM6FgcsszgxJWo8N75BC6sRVqn7699L7wD D3iggJvFg3vPAQ0P7oltqdK4TO4VdaOPSl70TDfihYcASJALYsGOCjuLlbixYS5b7U hfnvGVNTh9Dh04PGdj1C2rkIA0B2QMEqKLRzu8x635fMaEh6Do1m/i/YGSGjYLpfFS 0vRQczPr7L4q49Mm980mX62sQL58/W0IaSr1tBlNyJd6LB5lxmpCi08D/LZx3ryIuu rvZaNiR/kH3eg== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 06/13] uprobes/x86: Add uprobe syscall to speed up uprobe Date: Wed, 11 Dec 2024 14:33:55 +0100 Message-ID: <20241211133403.208920-7-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding new uprobe syscall that calls uprobe handlers for given 'breakpoint' address. The idea is that the 'breakpoint' address calls the user space trampoline which executes the uprobe syscall. The syscall handler reads the return address of the initial call to retrieve the original 'breakpoint' address. With this address we find the related uprobe object and call its consumers. Adding the arch_uprobe_trampoline_mapping function that provides uprobe trampoline mapping. This mapping is backed with one global page initialized at __init time and shared by the all the mapping instances. We do not allow to execute uprobe syscall if the caller is not from uprobe trampoline mapping. Signed-off-by: Jiri Olsa --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/kernel/uprobes.c | 80 ++++++++++++++++++++++++++ include/linux/syscalls.h | 2 + include/linux/uprobes.h | 1 + kernel/events/uprobes.c | 22 +++++++ kernel/sys_ni.c | 1 + 6 files changed, 107 insertions(+) diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index 5eb708bff1c7..88e388c7675b 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -345,6 +345,7 @@ 333 common io_pgetevents sys_io_pgetevents 334 common rseq sys_rseq 335 common uretprobe sys_uretprobe +336 common uprobe sys_uprobe # don't use numbers 387 through 423, add new calls after the last # 'common' entry 424 common pidfd_send_signal sys_pidfd_send_signal diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 22a17c149a55..23e4f2821cff 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -425,6 +425,86 @@ SYSCALL_DEFINE0(uretprobe) return -1; } +static int tramp_mremap(const struct vm_special_mapping *sm, struct vm_area_struct *new_vma) +{ + return -EPERM; +} + +static struct vm_special_mapping tramp_mapping = { + .name = "[uprobes-trampoline]", + .mremap = tramp_mremap, +}; + +SYSCALL_DEFINE0(uprobe) +{ + struct pt_regs *regs = task_pt_regs(current); + struct vm_area_struct *vma; + unsigned long bp_vaddr; + int err; + + err = copy_from_user(&bp_vaddr, (void __user *)regs->sp + 3*8, sizeof(bp_vaddr)); + if (err) { + force_sig(SIGILL); + return -1; + } + + /* Allow execution only from uprobe trampolines. */ + vma = vma_lookup(current->mm, regs->ip); + if (!vma || vma->vm_private_data != (void *) &tramp_mapping) { + force_sig(SIGILL); + return -1; + } + + handle_syscall_uprobe(regs, bp_vaddr - 5); + return 0; +} + +asm ( + ".pushsection .rodata\n" + ".global uprobe_trampoline_entry\n" + "uprobe_trampoline_entry:\n" + "endbr64\n" + "push %rcx\n" + "push %r11\n" + "push %rax\n" + "movq $" __stringify(__NR_uprobe) ", %rax\n" + "syscall\n" + "pop %rax\n" + "pop %r11\n" + "pop %rcx\n" + "ret\n" + ".global uprobe_trampoline_end\n" + "uprobe_trampoline_end:\n" + ".popsection\n" +); + +extern __visible u8 uprobe_trampoline_entry[]; +extern __visible u8 uprobe_trampoline_end[]; + +const struct vm_special_mapping *arch_uprobe_trampoline_mapping(void) +{ + struct pt_regs *regs = task_pt_regs(current); + + return user_64bit_mode(regs) ? &tramp_mapping : NULL; +} + +static int __init arch_uprobes_init(void) +{ + unsigned long size = uprobe_trampoline_end - uprobe_trampoline_entry; + static struct page *pages[2]; + struct page *page; + + page = alloc_page(GFP_HIGHUSER); + if (!page) + return -ENOMEM; + pages[0] = page; + tramp_mapping.pages = (struct page **) &pages; + arch_uprobe_copy_ixol(page, 0, uprobe_trampoline_entry, size); + return 0; +} + +late_initcall(arch_uprobes_init); + /* * If arch_uprobe->insn doesn't use rip-relative addressing, return * immediately. Otherwise, rewrite the instruction so that it accesses diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index c6333204d451..002f4e1debe5 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -994,6 +994,8 @@ asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int on); asmlinkage long sys_uretprobe(void); +asmlinkage long sys_uprobe(void); + /* pciconfig: alpha, arm, arm64, ia64, sparc */ asmlinkage long sys_pciconfig_read(unsigned long bus, unsigned long dfn, unsigned long off, unsigned long len, diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index c4ee755ca2a1..5e9a33bfb747 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -232,6 +232,7 @@ extern struct uprobe_trampoline *uprobe_trampoline_get(unsigned long vaddr); extern void uprobe_trampoline_put(struct uprobe_trampoline *area); extern bool arch_uprobe_is_callable(unsigned long vtramp, unsigned long vaddr); extern const struct vm_special_mapping *arch_uprobe_trampoline_mapping(void); +extern void handle_syscall_uprobe(struct pt_regs *regs, unsigned long bp_vaddr); #else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f57918c624da..52f38d1ef276 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2729,6 +2729,28 @@ static void handle_swbp(struct pt_regs *regs) rcu_read_unlock_trace(); } +void handle_syscall_uprobe(struct pt_regs *regs, unsigned long bp_vaddr) +{ + struct uprobe *uprobe; + int is_swbp; + + rcu_read_lock_trace(); + uprobe = find_active_uprobe_rcu(bp_vaddr, &is_swbp); + if (!uprobe) + goto unlock; + + if (!get_utask()) + goto unlock; + + if (arch_uprobe_ignore(&uprobe->arch, regs)) + goto unlock; + + handler_chain(uprobe, regs); + + unlock: + rcu_read_unlock_trace(); +} + /* * Perform required fix-ups and disable singlestep. * Allow pending signals to take effect. diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index c00a86931f8c..bf5d05c635ff 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -392,3 +392,4 @@ COND_SYSCALL(setuid16); COND_SYSCALL(rseq); COND_SYSCALL(uretprobe); +COND_SYSCALL(uprobe); From patchwork Wed Dec 11 13:33:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903478 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D540F230274; Wed, 11 Dec 2024 13:35:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924126; cv=none; b=D5JgKih1/Ctly7ihUYoAhDcZFa81Yx40OPeyqoPUNWtsZ/pT6yHiA1TCxMZWJ7TV9DqxpUNeayssBTfh7SKq7Q4FBebvll61ZKqKgbdrpWFh3w/F+hSPNidYAlA2HlRNYS4IF3P5ZCqGmimRrZFVfJ9YH9S3eyT3IxZpkzmY50Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924126; c=relaxed/simple; bh=WwECUbWo0u9vUSB6TwXrs4yLoqM7oB5o5WuTwDjNBCM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ejm07EjOchjhtxELFsiWL4mLRdrWXIbFk2TFSFkZuaMqGvmC4vORA4YtPiXHvadvBvwXEvTyQNA85r59GlZnx1JCoME6aaHRhdjMfyAmCxe+u3OM2UK2QL6E1LVKxXxC2GOTujg3FS1fFl2II9oBVpF1+my6yVrUiS8ZGl2WJ3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oN9LtIBB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oN9LtIBB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 16232C4CED2; Wed, 11 Dec 2024 13:35:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924125; bh=WwECUbWo0u9vUSB6TwXrs4yLoqM7oB5o5WuTwDjNBCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oN9LtIBB7sI/Ztd1K3UWLmFr54pgxIjnHUTVhCuyStuHxcB2q8BkukGzzAh4PgD+Q ta9xVoyWxYaFly9qszDgn1pIEWRtXczcTohOYJ3rI/eZVDwLpXHyl7rjjWnmXgNVWI DE1+0cWyeGtGuZoY/ocEHPZ/mifLhja8bv3oahCjYDpdvooMd8UVLxD+od44gM3fsf RDfAelrothI0JQwMCsotYON3+8aDzPYJQ0Vor5o2cugtCX0fxoZHZTKK8hf4gue0ho s/WAFNj5xlVykkriiqLhVCu19hQwPEa8+T9v370YrMDHMa2Xa8Qm061QNunaPxqD90 PO7Fp5UBLuwzQ== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 07/13] uprobes/x86: Add support to emulate nop5 instruction Date: Wed, 11 Dec 2024 14:33:56 +0100 Message-ID: <20241211133403.208920-8-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding support to emulate nop5 as the original uprobe instruction. This speeds up uprobes on top of nop5 instructions: (results from benchs/run_bench_uprobes.sh) current: uprobe-nop : 3.252 ± 0.019M/s uprobe-push : 3.097 ± 0.002M/s uprobe-ret : 1.116 ± 0.001M/s --> uprobe-nop5 : 1.115 ± 0.001M/s uretprobe-nop : 1.731 ± 0.016M/s uretprobe-push : 1.673 ± 0.023M/s uretprobe-ret : 0.843 ± 0.009M/s --> uretprobe-nop5 : 1.124 ± 0.001M/s after the change: uprobe-nop : 3.281 ± 0.003M/s uprobe-push : 3.085 ± 0.003M/s uprobe-ret : 1.130 ± 0.000M/s --> uprobe-nop5 : 3.276 ± 0.007M/s uretprobe-nop : 1.716 ± 0.016M/s uretprobe-push : 1.651 ± 0.017M/s uretprobe-ret : 0.846 ± 0.006M/s --> uretprobe-nop5 : 3.279 ± 0.002M/s Strangely I can see uretprobe-nop5 is now much faster compared to uretprobe-nop, while perf profiles for both are almost identical. I'm still checking on that. Signed-off-by: Jiri Olsa --- arch/x86/kernel/uprobes.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 23e4f2821cff..cdea97f8cd39 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -909,6 +909,11 @@ static const struct uprobe_xol_ops push_xol_ops = { .emulate = push_emulate_op, }; +static int is_nop5_insn(uprobe_opcode_t *insn) +{ + return !memcmp(insn, x86_nops[5], 5); +} + /* Returns -ENOSYS if branch_xol_ops doesn't handle this insn */ static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn) { @@ -928,6 +933,8 @@ static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn) break; case 0x0f: + if (is_nop5_insn((uprobe_opcode_t *) &auprobe->insn)) + goto setup; if (insn->opcode.nbytes != 2) return -ENOSYS; /* From patchwork Wed Dec 11 13:33:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903479 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C402423236C; Wed, 11 Dec 2024 13:35:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924136; cv=none; b=cd3mbNEOoQ904HyndqImFU0TTtZVzgWfbdjZAJgWRfs6T9mCoMA73W3nYo5NpPn+bTegD1PLT5xkkpbt2Jd4djCH55mFs2jqgOjI75ookSMvct1sr7/cn5QmecJUeTih0KV2hKCQgtkN801t2SUdfaNzrMsmGHcGfApwt7wKAX8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924136; c=relaxed/simple; bh=tU0ZW+9hsH2/3lQywbTok4TSpnaoCEWQXqU1idcN8nc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rRqFxiFw9eBQn6fx9it8L0h+d0kCWJVDvJ2JlFBkYki2IdfjIPImRLk3gyaACFrl41sJ00WYGF36GwTsNPfNil0/ls71e15Bh0eFX6Kd6lUZBBJzcTJR8oPlHmx0d6ciqzyj7JYGKZePq5+RdiHNw8Tih8c/1gfFez0AuRAbN3M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=W2e8uhzh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="W2e8uhzh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E9CDAC4CED2; Wed, 11 Dec 2024 13:35:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924136; bh=tU0ZW+9hsH2/3lQywbTok4TSpnaoCEWQXqU1idcN8nc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W2e8uhzhiL+Au/9oOddSrCGBxgmXJq8GQrQsEV6ZrLNj5ys42qfIiMPwSy9SgGQu6 EYGIjEb2AawEFRa8lgrTL+ho0HOJbYMtB4braG7KcSO0m/GJwhq8cpHpG3v2vixBlU z5xdL63NG7fnxi7o83JO4pnSDwEAoA4FGU7HBb4S42cblOKLTIXh5GL7JZiPoW5Eli w7iIFcOtbtT+tAJZeyBIzyh4XjkF/rmFwcompJI7sBnrTDAIjfYMtajz+/5vaf1UpC goBRFjMFqOci7/w/0XcL2rC2z8PA9+C7KRdHKE8ycgwZ3+Au1Zf1uurqX6lID7jfwy U2SFdCgQsKr0g== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 08/13] uprobes/x86: Add support to optimize uprobes Date: Wed, 11 Dec 2024 14:33:57 +0100 Message-ID: <20241211133403.208920-9-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Putting together all the previously added pieces to support optimized uprobes on top of 5-byte nop instruction. The current uprobe execution goes through following: - installs breakpoint instruction over original instruction - exception handler hit and calls related uprobe consumers - and either simulates original instruction or does out of line single step execution of it - returns to user space The optimized uprobe path - checks the original instruction is 5-byte nop (plus other checks) - adds (or uses existing) user space trampoline and overwrites original instruction (5-byte nop) with call to user space trampoline - the user space trampoline executes uprobe syscall that calls related uprobe consumers - trampoline returns back to next instruction This approach won't speed up all uprobes as it's limited to using nop5 as original instruction, but we could use nop5 as USDT probe instruction (which uses single byte nop ATM) and speed up the USDT probes. This patch overloads related arch functions in uprobe_write_opcode and set_orig_insn so they can install call instruction if needed. The arch_uprobe_optimize triggers the uprobe optimization and is called after first uprobe hit. I originally had it called on uprobe installation but then it clashed with elf loader, because the user space trampoline was added in a place where loader might need to put elf segments, so I decided to do it after first uprobe hit when loading is done. We do not unmap and release uprobe trampoline when it's no longer needed, because there's no easy way to make sure none of the threads is still inside the trampoline. But we do not waste memory, because there's just single page for all the uprobe trampoline mappings. We do waste frmae on page mapping for every 4GB by keeping the uprobe trampoline page mapped, but that seems ok. Attaching the speed up from benchs/run_bench_uprobes.sh script: current: uprobe-nop : 3.281 ± 0.003M/s uprobe-push : 3.085 ± 0.003M/s uprobe-ret : 1.130 ± 0.000M/s --> uprobe-nop5 : 3.276 ± 0.007M/s uretprobe-nop : 1.716 ± 0.016M/s uretprobe-push : 1.651 ± 0.017M/s uretprobe-ret : 0.846 ± 0.006M/s --> uretprobe-nop5 : 3.279 ± 0.002M/s after the change: uprobe-nop : 3.246 ± 0.004M/s uprobe-push : 3.057 ± 0.000M/s uprobe-ret : 1.113 ± 0.003M/s --> uprobe-nop5 : 6.751 ± 0.037M/s uretprobe-nop : 1.740 ± 0.015M/s uretprobe-push : 1.677 ± 0.018M/s uretprobe-ret : 0.852 ± 0.005M/s --> uretprobe-nop5 : 6.769 ± 0.040M/s Signed-off-by: Jiri Olsa --- arch/x86/include/asm/uprobes.h | 7 ++ arch/x86/kernel/uprobes.c | 168 ++++++++++++++++++++++++++++++++- include/linux/uprobes.h | 1 + kernel/events/uprobes.c | 8 ++ 4 files changed, 181 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/uprobes.h b/arch/x86/include/asm/uprobes.h index 678fb546f0a7..84a75ed748f0 100644 --- a/arch/x86/include/asm/uprobes.h +++ b/arch/x86/include/asm/uprobes.h @@ -20,6 +20,11 @@ typedef u8 uprobe_opcode_t; #define UPROBE_SWBP_INSN 0xcc #define UPROBE_SWBP_INSN_SIZE 1 +enum { + ARCH_UPROBE_FLAG_CAN_OPTIMIZE = 0, + ARCH_UPROBE_FLAG_OPTIMIZED = 1, +}; + struct uprobe_xol_ops; struct arch_uprobe { @@ -45,6 +50,8 @@ struct arch_uprobe { u8 ilen; } push; }; + + unsigned long flags; }; struct arch_uprobe_task { diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index cdea97f8cd39..b2420eeee23a 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -18,6 +18,7 @@ #include #include #include +#include /* Post-execution fixups. */ @@ -914,8 +915,37 @@ static int is_nop5_insn(uprobe_opcode_t *insn) return !memcmp(insn, x86_nops[5], 5); } +static int is_call_insn(uprobe_opcode_t *insn) +{ + return *insn == CALL_INSN_OPCODE; +} + +static void relative_insn(void *dest, void *from, void *to, u8 op) +{ + struct __arch_relative_insn { + u8 op; + s32 raddr; + } __packed *insn; + + insn = (struct __arch_relative_insn *)dest; + insn->raddr = (s32)((long)(to) - ((long)(from) + 5)); + insn->op = op; +} + +static void relative_call(void *dest, void *from, void *to) +{ + relative_insn(dest, from, to, CALL_INSN_OPCODE); +} + +static bool can_optimize_vaddr(unsigned long vaddr) +{ + /* We can't do cross page atomic writes yet. */ + return PAGE_SIZE - (vaddr & ~PAGE_MASK) >= 5; +} + /* Returns -ENOSYS if branch_xol_ops doesn't handle this insn */ -static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn) +static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn, + unsigned long vaddr) { u8 opc1 = OPCODE1(insn); insn_byte_t p; @@ -933,8 +963,11 @@ static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn) break; case 0x0f: - if (is_nop5_insn((uprobe_opcode_t *) &auprobe->insn)) + if (is_nop5_insn((uprobe_opcode_t *) &auprobe->insn)) { + if (can_optimize_vaddr(vaddr)) + set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags); goto setup; + } if (insn->opcode.nbytes != 2) return -ENOSYS; /* @@ -1065,7 +1098,7 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret) return ret; - ret = branch_setup_xol_ops(auprobe, &insn); + ret = branch_setup_xol_ops(auprobe, &insn, addr); if (ret != -ENOSYS) return ret; @@ -1306,3 +1339,132 @@ bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check ctx, else return regs->sp <= ret->stack; } + +int arch_uprobe_verify_opcode(struct arch_uprobe *auprobe, struct page *page, + unsigned long vaddr, uprobe_opcode_t *new_opcode, + int nbytes) +{ + uprobe_opcode_t old_opcode[5]; + bool is_call, is_swbp, is_nop5; + + if (!test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags)) + return uprobe_verify_opcode(page, vaddr, new_opcode); + + /* + * The ARCH_UPROBE_FLAG_CAN_OPTIMIZE flag guarantees the following + * 5 bytes read won't cross the page boundary. + */ + uprobe_copy_from_page(page, vaddr, (uprobe_opcode_t *) &old_opcode, 5); + is_call = is_call_insn((uprobe_opcode_t *) &old_opcode); + is_swbp = is_swbp_insn((uprobe_opcode_t *) &old_opcode); + is_nop5 = is_nop5_insn((uprobe_opcode_t *) &old_opcode); + + /* + * We allow following trasitions for optimized uprobes: + * + * nop5 -> swbp -> call + * || | | + * |'--<---' | + * '---<-----------' + * + * We return 1 to ack the write, 0 to do nothing, -1 to fail write. + * + * If the current opcode (old_opcode) has already desired value, + * we do nothing, because we are racing with another thread doing + * the update. + */ + switch (nbytes) { + case 5: + if (is_call_insn(new_opcode)) { + if (is_swbp) + return 1; + if (is_call && !memcmp(new_opcode, &old_opcode, 5)) + return 0; + } else { + if (is_call || is_swbp) + return 1; + if (is_nop5) + return 0; + } + break; + case 1: + if (is_swbp_insn(new_opcode)) { + if (is_nop5) + return 1; + if (is_swbp || is_call) + return 0; + } else { + if (is_swbp || is_call) + return 1; + if (is_nop5) + return 0; + } + } + return -1; +} + +bool arch_uprobe_is_register(uprobe_opcode_t *insn, int nbytes) +{ + return nbytes == 5 ? is_call_insn(insn) : is_swbp_insn(insn); +} + +static void __arch_uprobe_optimize(struct arch_uprobe *auprobe, struct mm_struct *mm, + unsigned long vaddr) +{ + struct uprobe_trampoline *tramp; + char call[5]; + + tramp = uprobe_trampoline_get(vaddr); + if (!tramp) + goto fail; + + relative_call(call, (void *) vaddr, (void *) tramp->vaddr); + if (uprobe_write_opcode(auprobe, mm, vaddr, call, 5)) + goto fail; + + set_bit(ARCH_UPROBE_FLAG_OPTIMIZED, &auprobe->flags); + return; + +fail: + /* Once we fail we never try again. */ + clear_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags); + uprobe_trampoline_put(tramp); +} + +static bool should_optimize(struct arch_uprobe *auprobe) +{ + if (!test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags)) + return false; + if (test_bit(ARCH_UPROBE_FLAG_OPTIMIZED, &auprobe->flags)) + return false; + return true; +} + +void arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vaddr) +{ + struct mm_struct *mm = current->mm; + + if (!should_optimize(auprobe)) + return; + + mmap_write_lock(mm); + if (should_optimize(auprobe)) + __arch_uprobe_optimize(auprobe, mm, vaddr); + mmap_write_unlock(mm); +} + +int set_orig_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr) +{ + uprobe_opcode_t *insn = (uprobe_opcode_t *) auprobe->insn; + + if (test_bit(ARCH_UPROBE_FLAG_OPTIMIZED, &auprobe->flags)) + return uprobe_write_opcode(auprobe, mm, vaddr, insn, 5); + + return uprobe_write_opcode(auprobe, mm, vaddr, insn, UPROBE_SWBP_INSN_SIZE); +} + +bool arch_uprobe_is_callable(unsigned long vtramp, unsigned long vaddr) +{ + long delta = (long)(vaddr + 5 - vtramp); + return delta >= INT_MIN && delta <= INT_MAX; +} diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 5e9a33bfb747..1b14b9f2f8d0 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -233,6 +233,7 @@ extern void uprobe_trampoline_put(struct uprobe_trampoline *area); extern bool arch_uprobe_is_callable(unsigned long vtramp, unsigned long vaddr); extern const struct vm_special_mapping *arch_uprobe_trampoline_mapping(void); extern void handle_syscall_uprobe(struct pt_regs *regs, unsigned long bp_vaddr); +extern void arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vaddr); #else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 52f38d1ef276..a7a3eeec9e51 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2654,6 +2654,11 @@ bool __weak arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c return true; } +void __weak arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vaddr) +{ + return; +} + /* * Run handler and ask thread to singlestep. * Ensure all non-fatal signals cannot interrupt thread while it singlesteps. @@ -2718,6 +2723,9 @@ static void handle_swbp(struct pt_regs *regs) handler_chain(uprobe, regs); + /* Try to optimize after first hit. */ + arch_uprobe_optimize(&uprobe->arch, bp_vaddr); + if (arch_uprobe_skip_sstep(&uprobe->arch, regs)) goto out; From patchwork Wed Dec 11 13:33:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903480 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6746822913C; Wed, 11 Dec 2024 13:35:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924147; cv=none; b=Vi5bgtcJEQ+7Dyb+n/CBJi8zs1LG5OrMgH+DAHKYkHZCjxI433PgpAOSU4QK0djDGwmR9nQw6OfD34z08xibx+7FbrTidcvn/HIr95/JO0qjzYsorVgbbJoSUf5eqT8Hg8sX1KYmRk/Le3mVuhyuPRCC5+Ck/6nunpkLmPXHK60= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924147; c=relaxed/simple; bh=eVJp2tvqlAvwi3XNY+zHgNGDY+OQsQBQRyY5w45290U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oZc35ffCp1VLkAD44THJ85Oa+XeYuNElw8mvEkABEtAfoKzTH4oiFbQbktsAUMnz9AkRs/coArOkrq5JK1jW+LyCcI2EocYqojxKwYCl+iHHhAxrgJqHLpk8fXMVEmzmXolQDjvVyvo90jAdG/6U3AJlinCWw1ARNyQvV7+RKdA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dvWAeSTG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dvWAeSTG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 94E29C4CED2; Wed, 11 Dec 2024 13:35:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924146; bh=eVJp2tvqlAvwi3XNY+zHgNGDY+OQsQBQRyY5w45290U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dvWAeSTG946LLwT9po0SC0DUVCIt8caT7gTXeIdVfjKyuwPEXw3VAWVo3lAz3Da73 I9ri6F8h1I28t+YGBXcMMY3OzxvTD7mapawbthafwpURjX5BkSPfVMXgAMoOoJVpcS AK9qQxUSYUZNQjuUZvBLb0veNWOvhSR0qU7jYC2baYhIhTTQOhaXFAu/v4CifrA/LM 0wr9SKPLuSSC5JlVRl3yuBiRy19pBEOQF6iY4M004A2SJTWfTT2VcLR4SepovLaCm9 YRJgNMzQCvxh3Nm2+lkiDOrwExuGFCnZks+P8prvYGsmIvFa6tGhHuiGrVoOERlmXj MX2w1f/ltJZpg== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 09/13] selftests/bpf: Use 5-byte nop for x86 usdt probes Date: Wed, 11 Dec 2024 14:33:58 +0100 Message-ID: <20241211133403.208920-10-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Using 5-byte nop for x86 usdt probes so we can switch to optimized uprobe them. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/sdt.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/sdt.h b/tools/testing/selftests/bpf/sdt.h index ca0162b4dc57..7ac9291f45f1 100644 --- a/tools/testing/selftests/bpf/sdt.h +++ b/tools/testing/selftests/bpf/sdt.h @@ -234,6 +234,13 @@ __extension__ extern unsigned long long __sdt_unsp; #define _SDT_NOP nop #endif +/* Use 5 byte nop for x86_64 to allow optimizing uprobes. */ +#if defined(__x86_64__) +# define _SDT_DEF_NOP _SDT_ASM_5(990: .byte 0x0f, 0x1f, 0x44, 0x00, 0x00) +#else +# define _SDT_DEF_NOP _SDT_ASM_1(990: _SDT_NOP) +#endif + #define _SDT_NOTE_NAME "stapsdt" #define _SDT_NOTE_TYPE 3 @@ -286,7 +293,7 @@ __extension__ extern unsigned long long __sdt_unsp; #define _SDT_ASM_BODY(provider, name, pack_args, args, ...) \ _SDT_DEF_MACROS \ - _SDT_ASM_1(990: _SDT_NOP) \ + _SDT_DEF_NOP \ _SDT_ASM_3( .pushsection .note.stapsdt,_SDT_ASM_AUTOGROUP,"note") \ _SDT_ASM_1( .balign 4) \ _SDT_ASM_3( .4byte 992f-991f, 994f-993f, _SDT_NOTE_TYPE) \ From patchwork Wed Dec 11 13:33:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903481 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1718A236914; Wed, 11 Dec 2024 13:35:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924158; cv=none; b=eFnOVUElFswc4mblDTvmORfkQU8jeruJuu2cLk1QLJw4DJ9IG6wQpJxZzeDqBFITU76cXG4xTL5Oid+VI26rb5Zufe0Z1ieIhkkeDFRlypLe0ZZDkLx6uSy/d6SN+lTbEgFqz4gQjicbMDUVD7sWJtNoatiVzcFvBSQQnd9YkGo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924158; c=relaxed/simple; bh=09gvuS1npBvqynFZ+ZH4cs0xqN26Ic6eEPXMk6Tzgac=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IabclwVbdT+K8owWOQizsjYI2tDvMcPPPWCAb9+VmLKYPJnrpr/lYNcxz3j4v+yN9L5Y6w8gQc5d82D7I9w3xuMhzA/MwaO6nrClUFkCIq0T6dqzo3Hztk9gb2w1RWFSz6Ou6YWAzHxN/+QzqPZHzRZUl2ZoihAJeOh7TB60uVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WjXePPr6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WjXePPr6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 460F7C4CEDD; Wed, 11 Dec 2024 13:35:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924157; bh=09gvuS1npBvqynFZ+ZH4cs0xqN26Ic6eEPXMk6Tzgac=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WjXePPr6DnNkDJZfFBXKvwY2oipuKFfQ9LpMmyKuZWERQvEpxpCbQbkUQ2hsh7Wvc fBWGjRAqDjx6kPYlIVjbK8qwD7PRch5r4fNJY12DxxF+mksY+lk7+n6qZp9gXL6Uwx vtJBsWu3LrVZIETG/+/eXN9Ojq+92dE4YQ4k25wsgQ27u5bG3AElvOTZ3N83XPegP+ x9tQ9kYZDcdL7YwgAcFBnoyAUwbLnqwwUdL9oWfnefyhHisd3WVOOPRYp4kIScErHM a/QRo/uIx2umuO7A9XD0K9BCx5UOr57erT43bz1o0waDW+PKEYTTmm3/64VA1UP05E rCuKj9rLXZK4A== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 10/13] selftests/bpf: Add uprobe/usdt optimized test Date: Wed, 11 Dec 2024 14:33:59 +0100 Message-ID: <20241211133403.208920-11-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding tests for optimized uprobe/usdt probes. Checking that we get expected trampoline and attached bpf programs get executed properly. Signed-off-by: Jiri Olsa --- .../selftests/bpf/prog_tests/uprobe_syscall.c | 203 ++++++++++++++++++ .../selftests/bpf/progs/uprobe_optimized.c | 29 +++ 2 files changed, 232 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/uprobe_optimized.c diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c index c397336fe1ed..1dbc26a1130c 100644 --- a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +++ b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c @@ -14,6 +14,8 @@ #include #include "uprobe_syscall.skel.h" #include "uprobe_syscall_executed.skel.h" +#include "uprobe_optimized.skel.h" +#include "sdt.h" __naked unsigned long uretprobe_regs_trigger(void) { @@ -350,6 +352,186 @@ static void test_uretprobe_shadow_stack(void) ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK); } + +#define TRAMP "[uprobes-trampoline]" + +static unsigned char nop5[5] = { 0x0f, 0x1f, 0x44, 0x00, 0x00 }; + +noinline void uprobe_test(void) +{ + asm volatile (" \n" + ".global uprobe_test_nop5 \n" + ".type uprobe_test_nop5, STT_FUNC \n" + "uprobe_test_nop5: \n" + ".byte 0x0f, 0x1f, 0x44, 0x00, 0x00 \n" + ); +} + +extern u8 uprobe_test_nop5[]; + +noinline void usdt_test(void) +{ + STAP_PROBE(optimized_uprobe, usdt); +} + +static void *find_nop5(void *fn) +{ + int i; + + for (i = 0; i < 10; i++) { + if (!memcmp(nop5, fn + i, 5)) + return fn + i; + } + return NULL; +} + +static int find_uprobes_trampoline(void **start, void **end) +{ + char line[128]; + int ret = -1; + FILE *maps; + + maps = fopen("/proc/self/maps", "r"); + if (!maps) { + fprintf(stderr, "cannot open maps\n"); + return -1; + } + + while (fgets(line, sizeof(line), maps)) { + int m = -1; + + /* We care only about private r-x mappings. */ + if (sscanf(line, "%p-%p r-xp %*x %*x:%*x %*u %n", start, end, &m) != 2) + continue; + if (m < 0) + continue; + if (!strncmp(&line[m], TRAMP, sizeof(TRAMP)-1)) { + ret = 0; + break; + } + } + + fclose(maps); + return ret; +} + +static void check_attach(struct uprobe_optimized *skel, void (*trigger)(void), void *addr) +{ + void *tramp_start, *tramp_end; + struct __arch_relative_insn { + u8 op; + s32 raddr; + } __packed *call; + + s32 delta; + + /* Uprobe gets optimized after first trigger, so let's press twice. */ + trigger(); + trigger(); + + if (!ASSERT_OK(find_uprobes_trampoline(&tramp_start, &tramp_end), "uprobes_trampoline")) + return; + + /* Make sure bpf program got executed.. */ + ASSERT_EQ(skel->bss->executed, 2, "executed"); + + /* .. and check the trampoline is as expected. */ + call = (struct __arch_relative_insn *) addr; + delta = (unsigned long) tramp_start - ((unsigned long) addr + 5); + + ASSERT_EQ(call->op, 0xe8, "call"); + ASSERT_EQ(call->raddr, delta, "delta"); + ASSERT_EQ(tramp_end - tramp_start, 4096, "size"); +} + +static void check_detach(struct uprobe_optimized *skel, void (*trigger)(void), void *addr) +{ + void *tramp_start, *tramp_end; + + /* [uprobes_trampoline] stays after detach */ + ASSERT_OK(find_uprobes_trampoline(&tramp_start, &tramp_end), "uprobes_trampoline"); + ASSERT_OK(memcmp(addr, nop5, 5), "nop5"); +} + +static void check(struct uprobe_optimized *skel, struct bpf_link *link, + void (*trigger)(void), void *addr) +{ + check_attach(skel, trigger, addr); + bpf_link__destroy(link); + check_detach(skel, trigger, addr); +} + +static void test_uprobe_legacy(void) +{ + struct uprobe_optimized *skel; + struct bpf_link *link; + unsigned long offset; + + skel = uprobe_optimized__open_and_load(); + if (!ASSERT_OK_PTR(skel, "uprobe_optimized__open_and_load")) + return; + + offset = get_uprobe_offset(&uprobe_test_nop5); + if (!ASSERT_GE(offset, 0, "get_uprobe_offset")) + goto cleanup; + + link = bpf_program__attach_uprobe_opts(skel->progs.test_1, + 0, "/proc/self/exe", offset, NULL); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_opts")) + goto cleanup; + + check(skel, link, uprobe_test, uprobe_test_nop5); + +cleanup: + uprobe_optimized__destroy(skel); +} + +static void test_uprobe_multi(void) +{ + struct uprobe_optimized *skel; + struct bpf_link *link; + + skel = uprobe_optimized__open_and_load(); + if (!ASSERT_OK_PTR(skel, "uprobe_optimized__open_and_load")) + return; + + link = bpf_program__attach_uprobe_multi(skel->progs.test_2, + 0, "/proc/self/exe", "uprobe_test_nop5", NULL); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_multi")) + goto cleanup; + + check(skel, link, uprobe_test, uprobe_test_nop5); + +cleanup: + uprobe_optimized__destroy(skel); +} + +static void test_uprobe_usdt(void) +{ + struct uprobe_optimized *skel; + struct bpf_link *link; + void *addr; + + errno = 0; + addr = find_nop5(usdt_test); + if (!ASSERT_OK_PTR(addr, "find_nop5")) + return; + + skel = uprobe_optimized__open_and_load(); + if (!ASSERT_OK_PTR(skel, "uprobe_optimized__open_and_load")) + return; + + link = bpf_program__attach_usdt(skel->progs.test_3, + -1 /* all PIDs */, "/proc/self/exe", + "optimized_uprobe", "usdt", NULL); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_usdt")) + goto cleanup; + + check(skel, link, usdt_test, addr); + +cleanup: + uprobe_optimized__destroy(skel); +} #else static void test_uretprobe_regs_equal(void) { @@ -370,6 +552,21 @@ static void test_uretprobe_shadow_stack(void) { test__skip(); } + +static void test_uprobe_legacy(void) +{ + test__skip(); +} + +static void test_uprobe_multi(void) +{ + test__skip(); +} + +static void test_uprobe_usdt(void) +{ + test__skip(); +} #endif void test_uprobe_syscall(void) @@ -382,4 +579,10 @@ void test_uprobe_syscall(void) test_uretprobe_syscall_call(); if (test__start_subtest("uretprobe_shadow_stack")) test_uretprobe_shadow_stack(); + if (test__start_subtest("uprobe_legacy")) + test_uprobe_legacy(); + if (test__start_subtest("uprobe_multi")) + test_uprobe_multi(); + if (test__start_subtest("uprobe_usdt")) + test_uprobe_usdt(); } diff --git a/tools/testing/selftests/bpf/progs/uprobe_optimized.c b/tools/testing/selftests/bpf/progs/uprobe_optimized.c new file mode 100644 index 000000000000..2441d59960a6 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/uprobe_optimized.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" +#include +#include + +char _license[] SEC("license") = "GPL"; +unsigned long executed = 0; + +SEC("uprobe") +int BPF_UPROBE(test_1) +{ + executed++; + return 0; +} + +SEC("uprobe.multi") +int BPF_UPROBE(test_2) +{ + executed++; + return 0; +} + +SEC("usdt") +int test_3(struct pt_regs *ctx) +{ + executed++; + return 0; +} From patchwork Wed Dec 11 13:34:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903482 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2686B236914; Wed, 11 Dec 2024 13:36:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924168; cv=none; b=ANH+sqJvpSDQfqRcebMygQgE46ducZs0wSWH9Vtkd9uWZTc/R8jJo+IYULEVVdrXLQTdnkpgFvsm1X2FwrlVIeoxaUe3yKi7hlkZnSbHOUXCOZws1J2oT7+B/++oTzeiolkzQRH/HjDJoEakBlh26Qe2NdA7xZLLUyZKhLHF2jk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924168; c=relaxed/simple; bh=9WHW7RegrBjWfp1MlD/1r5aFn90BYcuUDsDRM91xMic=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JZo28hMiUsnbAWFcsG8GcgZ0UwVsJCXiU15V5gR21omk5pCANrOUTNWZXLA4J564Rv8gwW0+Ot+3AZNT5A1bSwdBinVQCI9gnrphJDHU5Grfpoady/TzoVM9H14Bp+LsBlC0qc1Dc5SWcGisdKsmeQpaXvM0IMv2tu1coS/8Tcw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QDNINR25; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QDNINR25" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C1259C4CED2; Wed, 11 Dec 2024 13:36:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924168; bh=9WHW7RegrBjWfp1MlD/1r5aFn90BYcuUDsDRM91xMic=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QDNINR25L96CIschb+h6elOZECzE7h/kUSsVsH3uqAXLbBxIKsiIC7DGaJwlWKP+C 82GioyKpz/HSL//+JxiHcsSGmMXo89/ai2V6T/ctQW4klYBBEc9k0WJa6uZO66NQmY BqRDgHkl5tDKFWVGYVO/dnt7FeDPbW1jtoX0n/jpIqv3HItHdV+FKsYVudkSdpJqCO ysRAS+3dsCYsfgN9v+SfvSMJYDjGcgzPV5rk300DSzV0OWtvG7lhNxA8uzCeqIeGn8 kn0KWpMlq6yIBPkrqRjJ4qwbTjBzPcbSdmI09afgddBbU7e4n/xEN0c4Lc6z7OYbvZ 09RcWWu+m8pwQ== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 11/13] selftests/bpf: Add hit/attach/detach race optimized uprobe test Date: Wed, 11 Dec 2024 14:34:00 +0100 Message-ID: <20241211133403.208920-12-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Adding test that makes sure parallel execution of the uprobe and attach/detach of optimized uprobe on it works properly. Signed-off-by: Jiri Olsa --- .../selftests/bpf/prog_tests/uprobe_syscall.c | 82 +++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c index 1dbc26a1130c..eacd14db8f8d 100644 --- a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +++ b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c @@ -532,6 +532,81 @@ static void test_uprobe_usdt(void) cleanup: uprobe_optimized__destroy(skel); } + +static bool race_stop; + +static void *worker_trigger(void *arg) +{ + unsigned long rounds = 0; + + while (!race_stop) { + uprobe_test(); + rounds++; + } + + printf("tid %d trigger rounds: %lu\n", gettid(), rounds); + return NULL; +} + +static void *worker_attach(void *arg) +{ + struct uprobe_optimized *skel; + unsigned long rounds = 0; + + skel = uprobe_optimized__open_and_load(); + if (!ASSERT_OK_PTR(skel, "uprobe_optimized__open_and_load")) + goto cleanup; + + while (!race_stop) { + skel->links.test_2 = bpf_program__attach_uprobe_multi(skel->progs.test_2, -1, + "/proc/self/exe", "uprobe_test_nop5", NULL); + if (!ASSERT_OK_PTR(skel->links.test_2, "bpf_program__attach_uprobe_multi")) + break; + bpf_link__destroy(skel->links.test_2); + skel->links.test_2 = NULL; + rounds++; + } + + printf("tid %d attach rounds: %lu hits: %lu\n", gettid(), rounds, skel->bss->executed); + +cleanup: + uprobe_optimized__destroy(skel); + return NULL; +} + +static void test_uprobe_race(void) +{ + int err, i, nr_cpus, nr; + pthread_t *threads; + + nr_cpus = libbpf_num_possible_cpus(); + if (!ASSERT_GE(nr_cpus, 0, "nr_cpus")) + return; + + nr = nr_cpus * 2; + threads = malloc(sizeof(*threads) * nr); + if (!ASSERT_OK_PTR(threads, "malloc")) + return; + + for (i = 0; i < nr_cpus; i++) { + err = pthread_create(&threads[i], NULL, worker_trigger, NULL); + if (!ASSERT_OK(err, "pthread_create")) + goto cleanup; + } + + for (; i < nr; i++) { + err = pthread_create(&threads[i], NULL, worker_attach, NULL); + if (!ASSERT_OK(err, "pthread_create")) + goto cleanup; + } + + sleep(4); + +cleanup: + race_stop = true; + for (i = 0; i < nr; i++) + pthread_join(threads[i], NULL); +} #else static void test_uretprobe_regs_equal(void) { @@ -567,6 +642,11 @@ static void test_uprobe_usdt(void) { test__skip(); } + +static void test_uprobe_race(void) +{ + test__skip(); +} #endif void test_uprobe_syscall(void) @@ -585,4 +665,6 @@ void test_uprobe_syscall(void) test_uprobe_multi(); if (test__start_subtest("uprobe_usdt")) test_uprobe_usdt(); + if (test__start_subtest("uprobe_race")) + test_uprobe_race(); } From patchwork Wed Dec 11 13:34:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903483 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E7992368EF; Wed, 11 Dec 2024 13:36:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924179; cv=none; b=msmbkLgBoC2r23+t78qnN5zKEzPQPHI+/vK/BsnIJkR6aAmhORhE4XZTkgCzWIfF0a/xdQ2GxRoC/FzwFkfxA3XK8orivdHZwViSRvjFeaTBviGroLm5TrTc3LQbqvZtDd80LEPSp7jP4OMLrloMVtQoz7KP+KJmAdBVzkIPSQA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924179; c=relaxed/simple; bh=ztxN7nzwUWr4QzxZKHnancBay3JT0i2VtQ+cjXHy2tA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mPp0vLg62PbGcL9lJT0z6bTjCx0W+J312p6o9FXJuvSZWnlbJ0A+Qr7KOqAwqLLk7rg9o3YKw19tLkddBgvr79EkLgJLBAUd5ReiGsNoplBaxSNAvEFnVFHD8/z3L6Pub6aTyXoD79H97AbP4UmU94LXZSaLLp+e3vx4Clw2vvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iwKPDwJL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iwKPDwJL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59C6EC4CED2; Wed, 11 Dec 2024 13:36:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924178; bh=ztxN7nzwUWr4QzxZKHnancBay3JT0i2VtQ+cjXHy2tA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iwKPDwJLQAt1JS8a6Un25ef2oMlp4ycTW4moj9ToixK1Zn4JuGU0BlhHonj9Kld9Y 4Qd/ZDT6Da8naRKdArvgyKBJe/1ksjdNtmwNt6uEnIEw2rr/clVAA4/wFV3DeRWpaH L7I+EnBRE4qw9UCkvwonSBcFSOyA2eFmV3S0dHhVMIv+2m8MUSkPSKinG405/t+iRv ea3LHi2HH36pNXsIHZoqECkL5lZ7XMeMp7A7x6J9VHsRiTctZj7pnxwaiui00eMCby MDpsYkiXfFYUV5+2OihRXPZm9weT9H1bQ5dHJ+s/CkgvNtRqGv0WHyVXVxmAVrBXC/ h6CmdnrEpxFrg== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 12/13] selftests/bpf: Add uprobe syscall sigill signal test Date: Wed, 11 Dec 2024 14:34:01 +0100 Message-ID: <20241211133403.208920-13-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Make sure that calling uprobe syscall from outside uprobe trampoline results in sigill signal. Signed-off-by: Jiri Olsa --- .../selftests/bpf/prog_tests/uprobe_syscall.c | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c index eacd14db8f8d..12631abc7017 100644 --- a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +++ b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c @@ -607,6 +607,40 @@ static void test_uprobe_race(void) for (i = 0; i < nr; i++) pthread_join(threads[i], NULL); } + +#ifndef __NR_uprobe +#define __NR_uprobe 336 +#endif + +static void test_uprobe_sigill(void) +{ + int status, err, pid; + + pid = fork(); + if (!ASSERT_GE(pid, 0, "fork")) + return; + /* child */ + if (pid == 0) { + asm volatile ( + "pushq %rax\n" + "pushq %rcx\n" + "pushq %r11\n" + "movq $" __stringify(__NR_uprobe) ", %rax\n" + "syscall\n" + "popq %r11\n" + "popq %rcx\n" + "retq\n" + ); + exit(0); + } + + err = waitpid(pid, &status, 0); + ASSERT_EQ(err, pid, "waitpid"); + + /* verify the child got killed with SIGILL */ + ASSERT_EQ(WIFSIGNALED(status), 1, "WIFSIGNALED"); + ASSERT_EQ(WTERMSIG(status), SIGILL, "WTERMSIG"); +} #else static void test_uretprobe_regs_equal(void) { @@ -647,6 +681,11 @@ static void test_uprobe_race(void) { test__skip(); } + +static void test_uprobe_sigill(void) +{ + test__skip(); +} #endif void test_uprobe_syscall(void) @@ -667,4 +706,6 @@ void test_uprobe_syscall(void) test_uprobe_usdt(); if (test__start_subtest("uprobe_race")) test_uprobe_race(); + if (test__start_subtest("uprobe_sigill")) + test_uprobe_sigill(); } From patchwork Wed Dec 11 13:34:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 13903484 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FCEE233694; Wed, 11 Dec 2024 13:36:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924189; cv=none; b=amwxcnbCU/kL+9flHJRw/EXOdpdt2EJh5zMuLv9R/7c2sEe9qLJSNhSE1Y0AAuksYH5Zd0Bx8xpkHU2v7inBc+UmaODdU0eG3vDSlpLhpIFE/k9OwpZC173R6D+pL7vatoRVcb4mvd8xMwYGXokdTgdipvZWgg+q3Zhtyg/JF0E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733924189; c=relaxed/simple; bh=Z9Qk+24L6IuNIAa7v/u7RWbB10cU/z7S162cZ7vfMPE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OQ60Et83eL3CinNKoXWLMRDbBda5pLlTGhzUQQPDDXpnSK0VCRRAI/mXCIWllKi/cV+OOwfLxSLfjCbnkN0bhBe5gvbieIlU1m/DKUBIyQnpy3uR+G4bNkFL8QbY8yygUNx1e10TEctenO3GECOY+GrWTqjXWOFxGturEiWqLC4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=o7AsroI/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="o7AsroI/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4AD5BC4CED2; Wed, 11 Dec 2024 13:36:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733924189; bh=Z9Qk+24L6IuNIAa7v/u7RWbB10cU/z7S162cZ7vfMPE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o7AsroI/XzuWfPn9YmrVeSHNOnQAy/Nz0VE4rCIo4MlVhNKcaegztnFJ0jfz6VAxf oKPpMvCVVjk2Kw0KAOR8DYZBn+RLjnslPRClSfOCkaupWcMhTawdTlyy0fmN+tIsl1 kparAfrxHt/ZbmyVBNkJBa8/+FI73pZvI0cmrt75Zz/4No8u2CxujcSk+BUOEy1y4U QgrW0n1a9qGNqwHAg3vWE53GTvwmt6B/hDLMTVpd5zAGFIIFH0yNO/9EjRwV+N7JYI WSERARVyDHmifz0Qqka5ARP/tcKwe2uESZYRKAgO50ZOKu+TIKqjY3j+9V1G+Xazut Kxg74Dp2Wb8oA== From: Jiri Olsa To: Oleg Nesterov , Peter Zijlstra , Andrii Nakryiko Cc: bpf@vger.kernel.org, Song Liu , Yonghong Song , John Fastabend , Hao Luo , Steven Rostedt , Masami Hiramatsu , Alan Maguire , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next 13/13] selftests/bpf: Add 5-byte nop uprobe trigger bench Date: Wed, 11 Dec 2024 14:34:02 +0100 Message-ID: <20241211133403.208920-14-jolsa@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241211133403.208920-1-jolsa@kernel.org> References: <20241211133403.208920-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add 5-byte nop uprobe trigger bench (x86_64 specific) to measure uprobes/uretprobes on top of nop5 instruction. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/bench.c | 12 ++++++ .../selftests/bpf/benchs/bench_trigger.c | 42 +++++++++++++++++++ .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- 3 files changed, 55 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index 1bd403a5ef7b..0fd8c9b0d38f 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -526,6 +526,12 @@ extern const struct bench bench_trig_uprobe_multi_push; extern const struct bench bench_trig_uretprobe_multi_push; extern const struct bench bench_trig_uprobe_multi_ret; extern const struct bench bench_trig_uretprobe_multi_ret; +#ifdef __x86_64__ +extern const struct bench bench_trig_uprobe_nop5; +extern const struct bench bench_trig_uretprobe_nop5; +extern const struct bench bench_trig_uprobe_multi_nop5; +extern const struct bench bench_trig_uretprobe_multi_nop5; +#endif extern const struct bench bench_rb_libbpf; extern const struct bench bench_rb_custom; @@ -586,6 +592,12 @@ static const struct bench *benchs[] = { &bench_trig_uretprobe_multi_push, &bench_trig_uprobe_multi_ret, &bench_trig_uretprobe_multi_ret, +#ifdef __x86_64__ + &bench_trig_uprobe_nop5, + &bench_trig_uretprobe_nop5, + &bench_trig_uprobe_multi_nop5, + &bench_trig_uretprobe_multi_nop5, +#endif /* ringbuf/perfbuf benchmarks */ &bench_rb_libbpf, &bench_rb_custom, diff --git a/tools/testing/selftests/bpf/benchs/bench_trigger.c b/tools/testing/selftests/bpf/benchs/bench_trigger.c index 32e9f194d449..e74289a3270b 100644 --- a/tools/testing/selftests/bpf/benchs/bench_trigger.c +++ b/tools/testing/selftests/bpf/benchs/bench_trigger.c @@ -333,6 +333,20 @@ static void *uprobe_producer_ret(void *input) return NULL; } +#ifdef __x86_64__ +__nocf_check __weak void uprobe_target_nop5(void) +{ + asm volatile (".byte 0x0f, 0x1f, 0x44, 0x00, 0x00"); +} + +static void *uprobe_producer_nop5(void *input) +{ + while (true) + uprobe_target_nop5(); + return NULL; +} +#endif + static void usetup(bool use_retprobe, bool use_multi, void *target_addr) { size_t uprobe_offset; @@ -448,6 +462,28 @@ static void uretprobe_multi_ret_setup(void) usetup(true, true /* use_multi */, &uprobe_target_ret); } +#ifdef __x86_64__ +static void uprobe_nop5_setup(void) +{ + usetup(false, false /* !use_multi */, &uprobe_target_nop5); +} + +static void uretprobe_nop5_setup(void) +{ + usetup(false, false /* !use_multi */, &uprobe_target_nop5); +} + +static void uprobe_multi_nop5_setup(void) +{ + usetup(false, true /* use_multi */, &uprobe_target_nop5); +} + +static void uretprobe_multi_nop5_setup(void) +{ + usetup(false, true /* use_multi */, &uprobe_target_nop5); +} +#endif + const struct bench bench_trig_syscall_count = { .name = "trig-syscall-count", .validate = trigger_validate, @@ -506,3 +542,9 @@ BENCH_TRIG_USERMODE(uprobe_multi_ret, ret, "uprobe-multi-ret"); BENCH_TRIG_USERMODE(uretprobe_multi_nop, nop, "uretprobe-multi-nop"); BENCH_TRIG_USERMODE(uretprobe_multi_push, push, "uretprobe-multi-push"); BENCH_TRIG_USERMODE(uretprobe_multi_ret, ret, "uretprobe-multi-ret"); +#ifdef __x86_64__ +BENCH_TRIG_USERMODE(uprobe_nop5, nop5, "uprobe-nop5"); +BENCH_TRIG_USERMODE(uretprobe_nop5, nop5, "uretprobe-nop5"); +BENCH_TRIG_USERMODE(uprobe_multi_nop5, nop5, "uprobe-multi-nop5"); +BENCH_TRIG_USERMODE(uretprobe_multi_nop5, nop5, "uretprobe-multi-nop5"); +#endif diff --git a/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh b/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh index af169f831f2f..03f55405484b 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh @@ -2,7 +2,7 @@ set -eufo pipefail -for i in usermode-count syscall-count {uprobe,uretprobe}-{nop,push,ret} +for i in usermode-count syscall-count {uprobe,uretprobe}-{nop,push,ret,nop5} do summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) printf "%-15s: %s\n" $i "$summary"