From patchwork Tue Aug 13 04:29:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13761252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C978C52D7B for ; Tue, 13 Aug 2024 04:30:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACCC26B00AD; Tue, 13 Aug 2024 00:30:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A53776B00AC; Tue, 13 Aug 2024 00:30:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89B8B6B00AD; Tue, 13 Aug 2024 00:30:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 64DFD6B00AB for ; Tue, 13 Aug 2024 00:30:22 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1A695C05D0 for ; Tue, 13 Aug 2024 04:30:22 +0000 (UTC) X-FDA: 82445945484.25.DAA29ED Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf04.hostedemail.com (Postfix) with ESMTP id 2C1D840004 for ; Tue, 13 Aug 2024 04:30:18 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VsFyfNFI; spf=pass (imf04.hostedemail.com: domain of andrii@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723523408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0PqwzyKHX14q+atJT7rqGOiqxxOYOUyIwT7vShAS1sY=; b=JhJkd+3E2qFYWC5je3ex7/fDvAJ9ApzmdTFV1K6acRBLH/l27UtB81zf7Iow68vLVRYVKs 6KtJQwfTm6IbVk7LNboiKVOc1GuGTewc0knm71O+LbYnw+H9zNkiSBPdFb4wXNVs5z6jS4 55NCJU4070L+qe93tJzs20M/MwTspbE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VsFyfNFI; spf=pass (imf04.hostedemail.com: domain of andrii@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723523408; a=rsa-sha256; cv=none; b=j22kpQvqCDFRqTuW7K56Gnoq/awFo/xYVQk4NF51XsOHTFWRjSGMB+dmnweXDd9q5WKF2G dsKoaVi11w/EZYUNwDdYbqW9aZORacK6cByvsUmTD84/TiZUXDdKN7wWsucKqZYW6Z7oGi /2Sq2SUjKbNIzFheYC4WwxPyL9/qtfw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 212EECE1181; Tue, 13 Aug 2024 04:30:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2116BC4AF11; Tue, 13 Aug 2024 04:30:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723523413; bh=lE/xzipeIMKlpu3HvFd4L+QxhBM6/Hk0Qs9hd5OSsYM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VsFyfNFI1aH1yYtea8NqaJI8dNdgsqy7CbBuR0QzPL1quQeQDPrXwLZh7di5rL9uS jRz1DW3DAlmVOhkzhAt4IpDSk5aRrst+Gkj55GZpizSIJUhFwV//P2uGNHw++ixzfm 0uZ3uhs8RHOv8cjppbjx20zTHVBGNXxZ8sufUBJXsguCXTtIbe/Z2khO/SRP55lbI1 bjB4AS7YmyAZDbVkW/BPle9cTfX+7DdCW5b4IzgU7Ht3uEkotHkag+cQMm2rkYR9BD yeEZ1V1yoh3bSUXYAgctMa8IAVOv39IoRp9hmW6CX5P/v8fWj7R6UkPdGXIeY0tnbd YvHBquD/4SPMg== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, Andrii Nakryiko Subject: [PATCH RFC v3 10/13] uprobes: implement SRCU-protected lifetime for single-stepped uprobe Date: Mon, 12 Aug 2024 21:29:14 -0700 Message-ID: <20240813042917.506057-11-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813042917.506057-1-andrii@kernel.org> References: <20240813042917.506057-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: anm4mc8aizf9frha4fp98jnhazmq338e X-Rspamd-Queue-Id: 2C1D840004 X-Rspamd-Server: rspam11 X-HE-Tag: 1723523418-616574 X-HE-Meta: U2FsdGVkX18+pF9tIp4zWY+APGtr0whrYxYEcgr2zbOnW6GU5IdD35RlNa59h0VEiXZobLee10DvLiEB8p9YOl9Jh+is9qrPO70py+L+KFADfi6ni677lFyo/XE75g9/tEI2fe8h2KYe3z/o73ygz/2FkihmwhZq4kJNm30IXtTFYOY2LAMAjnERCYcZF+6v5lLSdsyiA6wsn118fjkDtIQKWWo+t+vWR76a8DomzfTeBS+cmDhP4RLiuX1kEHWCKqGLMYXZDPxqUm0G/41yAW/Dj17WMHWE7vphsqFcMcbnlvfDj3C+qUE2qa7Xc6xTHJzZWoAYLc46QwTlcx8pjHIpcdz60AXF85Q3f414LmzvD5SxtZuL0f6ZLIjii5jUZu5HeOeBIF2BTVyC1+vvBtIrtL7vZ7uSJdGupdCfBf59pippXFHzIfFgZZdOyduWcpEEwz0JdmH31fqp/H4UOfnVABfdIPH7N3iPjQGe2ZWV9f0EiMk2rcLcrMX66KRv7cMzN4X2nd8REQiYiDV+/DUcrGP9sTssT2FL37M/A//rch+xkK+3HyDzyo4v26jd9K/kIiCBR6lbBIeF89d8HmIOqIcGNmM7f0KrZkWVfgG5/L2IgDwhaVlEqlj58PCsDKCppnLmKYwEuvs8FP3jMR+y6aZUtwaEfbMh8m7epJ1KMsQ0GlD+g4RxWRfDVh/rATNQS7y5MyYv7L/3QQxdRFzZnhoCvMOJpZ/RV3hWtQFqaOwhRtzDYDn3zPQr0X8lcdkkTIEw+vtueXpBUqiMs0T8VlGk7hg9AYZBhcfAJRAZNawMu5W+mJGZYj0NJof8NxRX74LoPImxyjpSFTK+zQIWhTH7C10qTiWEi/nnj7weizFYubgcTo9XLv+/qPMkEt5XEFi3Q6VCsQlHNWgtoy2hd2g9NK+ri2SreoMYGKMC+KtDjwGqvEM2XUpCrUDHIYjEjMHP4reyDHKw8TY bLaJY+l0 uHfNtCVQNQdxI6hesrJuisu4RqkpXrCh60b/IOguqROfOQMvWBfbyvvsQaXkE/0FVxouYLPAQAqRVVu78TbPtEa7qNNEM8tgfRVT0lSOgEhK8f0/W3NfpaiCbSSXZkDq4EBljUuu+JZ2EyzNonUNUiX3jFkyxlIl4z8yqM1wLR9ioS2iorHbMd3yGqJIWkTIqFmKbPmM55D5qqsqegOtAvtsZAZkajbQXXfN+0gFobAKKWqQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similarly to how we SRCU-protect uprobe instance (and avoid refcounting it unnecessarily) when waiting for return probe hit, use hprobe approach to do the same with single-stepped uprobe. Same hprobe_* primitives are used. We also reuse ri_timer() callback to expire both pending single-step uprobe and return instances. Signed-off-by: Andrii Nakryiko --- include/linux/uprobes.h | 4 +-- kernel/events/uprobes.c | 56 ++++++++++++++++++++++++----------------- 2 files changed, 35 insertions(+), 25 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 9a0aa0b2a5fe..cc9d3cb055b5 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -107,7 +107,7 @@ struct uprobe_task { }; }; - struct uprobe *active_uprobe; + struct hprobe active_hprobe; struct timer_list ri_timer; unsigned long xol_vaddr; @@ -115,7 +115,7 @@ struct uprobe_task { struct return_instance *return_instances; unsigned int depth; -}; +} ____cacheline_aligned; struct return_instance { unsigned long func; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 26acd06871e6..713824c8ca77 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1860,11 +1860,16 @@ unsigned long __weak uprobe_get_swbp_addr(struct pt_regs *regs) return instruction_pointer(regs) - UPROBE_SWBP_INSN_SIZE; } +static bool utask_has_pending_sstep_uprobe(struct uprobe_task *utask) +{ + return utask->active_hprobe.stable != NULL; +} + unsigned long uprobe_get_trap_addr(struct pt_regs *regs) { struct uprobe_task *utask = current->utask; - if (unlikely(utask && utask->active_uprobe)) + if (unlikely(utask && utask_has_pending_sstep_uprobe(utask))) return utask->vaddr; return instruction_pointer(regs); @@ -1893,14 +1898,17 @@ void uprobe_free_utask(struct task_struct *t) { struct uprobe_task *utask = t->utask; struct return_instance *ri; + struct uprobe *uprobe; + bool under_rcu; if (!utask) return; timer_delete_sync(&utask->ri_timer); - if (utask->active_uprobe) - put_uprobe(utask->active_uprobe); + /* clean up pending single-stepped uprobe */ + uprobe = hprobe_consume(&utask->active_hprobe, &under_rcu); + hprobe_finalize(&utask->active_hprobe, uprobe, under_rcu); ri = utask->return_instances; while (ri) @@ -1924,6 +1932,8 @@ static void ri_timer(struct timer_list *timer) /* RCU protects return_instance from freeing. */ guard(rcu)(); + hprobe_expire(&utask->active_hprobe); + for_each_ret_instance_rcu(ri, utask->return_instances) { hprobe_expire(&ri->hprobe); } @@ -2166,20 +2176,15 @@ pre_ssout(struct uprobe *uprobe, struct pt_regs *regs, unsigned long bp_vaddr) { struct uprobe_task *utask; unsigned long xol_vaddr; - int err; + int err, srcu_idx; utask = get_utask(); if (!utask) return -ENOMEM; - if (!try_get_uprobe(uprobe)) - return -EINVAL; - xol_vaddr = xol_get_insn_slot(uprobe); - if (!xol_vaddr) { - err = -ENOMEM; - goto err_out; - } + if (!xol_vaddr) + return -ENOMEM; utask->xol_vaddr = xol_vaddr; utask->vaddr = bp_vaddr; @@ -2187,15 +2192,18 @@ pre_ssout(struct uprobe *uprobe, struct pt_regs *regs, unsigned long bp_vaddr) err = arch_uprobe_pre_xol(&uprobe->arch, regs); if (unlikely(err)) { xol_free_insn_slot(current); - goto err_out; + return err; } - utask->active_uprobe = uprobe; + srcu_idx = __srcu_read_lock(&uretprobes_srcu); + + hprobe_init_leased(&utask->active_hprobe, uprobe, srcu_idx); utask->state = UTASK_SSTEP; + + if (!timer_pending(&utask->ri_timer)) + mod_timer(&utask->ri_timer, jiffies + RI_TIMER_PERIOD); + return 0; -err_out: - put_uprobe(uprobe); - return err; } /* @@ -2212,7 +2220,7 @@ bool uprobe_deny_signal(void) struct task_struct *t = current; struct uprobe_task *utask = t->utask; - if (likely(!utask || !utask->active_uprobe)) + if (likely(!utask || !utask_has_pending_sstep_uprobe(utask))) return false; WARN_ON_ONCE(utask->state != UTASK_SSTEP); @@ -2528,8 +2536,10 @@ static void handle_singlestep(struct uprobe_task *utask, struct pt_regs *regs) { struct uprobe *uprobe; int err = 0; + bool under_rcu; + + uprobe = hprobe_consume(&utask->active_hprobe, &under_rcu); - uprobe = utask->active_uprobe; if (utask->state == UTASK_SSTEP_ACK) err = arch_uprobe_post_xol(&uprobe->arch, regs); else if (utask->state == UTASK_SSTEP_TRAPPED) @@ -2537,8 +2547,8 @@ static void handle_singlestep(struct uprobe_task *utask, struct pt_regs *regs) else WARN_ON_ONCE(1); - put_uprobe(uprobe); - utask->active_uprobe = NULL; + hprobe_finalize(&utask->active_hprobe, uprobe, under_rcu); + utask->state = UTASK_RUNNING; xol_free_insn_slot(current); @@ -2556,7 +2566,7 @@ static void handle_singlestep(struct uprobe_task *utask, struct pt_regs *regs) /* * On breakpoint hit, breakpoint notifier sets the TIF_UPROBE flag and * allows the thread to return from interrupt. After that handle_swbp() - * sets utask->active_uprobe. + * sets utask->active_hprobe. * * On singlestep exception, singlestep notifier sets the TIF_UPROBE flag * and allows the thread to return from interrupt. @@ -2571,7 +2581,7 @@ void uprobe_notify_resume(struct pt_regs *regs) clear_thread_flag(TIF_UPROBE); utask = current->utask; - if (utask && utask->active_uprobe) + if (utask && utask_has_pending_sstep_uprobe(utask)) handle_singlestep(utask, regs); else handle_swbp(regs); @@ -2602,7 +2612,7 @@ int uprobe_post_sstep_notifier(struct pt_regs *regs) { struct uprobe_task *utask = current->utask; - if (!current->mm || !utask || !utask->active_uprobe) + if (!current->mm || !utask || !utask_has_pending_sstep_uprobe(utask)) /* task is currently not uprobed */ return 0;