From patchwork Fri Nov 22 03:59:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13882753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06010E65D1D for ; Fri, 22 Nov 2024 03:59:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 791796B00BD; Thu, 21 Nov 2024 22:59:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F25A6B00C0; Thu, 21 Nov 2024 22:59:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56C9D6B00C1; Thu, 21 Nov 2024 22:59:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 257F26B00BD for ; Thu, 21 Nov 2024 22:59:41 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 928A3A16A2 for ; Fri, 22 Nov 2024 03:59:40 +0000 (UTC) X-FDA: 82812374526.27.4A2FD1E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf25.hostedemail.com (Postfix) with ESMTP id 3DDA3A000B for ; Fri, 22 Nov 2024 03:59:00 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QteQ7s1o; spf=pass (imf25.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732247731; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=kzCimHulDNaQ2vzaIOItQzUOv5NE9HMEM1/iDPORkqY=; b=o/NENvDFrZqqat5kygh4TxlTWuErvvsp50vkD4becrOR4eOFJFTTRZf9Xox3G+54kCPJ0t vL9vt1hmM56geKBqKcmQWHTAd8mM2DQvE0RUMgzEgOcwp9Fj3b16o9h1RU+hOhxnWPcRKI NRmo7XTRJW9WAVwPiaFgeBVxiuaZ/kE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QteQ7s1o; spf=pass (imf25.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732247731; a=rsa-sha256; cv=none; b=zNpUlkNMZOoRlIoU5ZBhXesLc8pHGvTqDhylZWMp24a6UONt2VrcA601leeEsqjEto62Cs YIC8EbNnkPZfsIbrVofQuMQ3DyJ9S14BFAnOAmpRqe54plp8SAcSmjQEHNrq8uh26EnRkC u8suvUjQQa7bzArR/un8tt2znX+tJxU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id DD26C5C0113; Fri, 22 Nov 2024 03:58:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 46D8AC4CECE; Fri, 22 Nov 2024 03:59:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732247977; bh=OLi4SqzR5SzIfTiMCGYx8+GjgOw02FBSpOAD4lu+xHM=; h=From:To:Cc:Subject:Date:From; b=QteQ7s1oRJNNSxaDT28yrtw6ZpTIDCC3Bnz7cwpKqEy47yp7lrHgGUVkrzUHFzTCf yvRZEQlWkXcYX4SjdEzXWsvfM9JmXPGmFO4dn3GN2Rqg3fTQXH9qrFOVA4AytSD66q 4Vm/K7EEQRQl2ME75jZPp5q3P3CJaHXe1VoEz84SCRFPuU41rEwayAKmo4+TouIgA8 JZT6oPYb0ZQE4YJMMuyAtEjjUARRzrT3GsVNIaubiSm4LTjCZGy80V1xEaNlQZY5In RFgqpA7CPI0ampH4G5c4cCAKOfZYjkG2i+sLwonjLAXvUMsFnnpZwA1wlVRexesHmP NhdxKXkqo/d8w== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, peterz@infradead.org, mingo@kernel.org, torvalds@linux-foundation.org Cc: oleg@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, shakeel.butt@linux.dev, hannes@cmpxchg.org, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, david@redhat.com, arnd@arndb.de, viro@zeniv.linux.org.uk, hca@linux.ibm.com, Andrii Nakryiko Subject: [PATCH v5 tip/perf/core 0/2] uprobes: speculative lockless VMA-to-uprobe lookup Date: Thu, 21 Nov 2024 19:59:20 -0800 Message-ID: <20241122035922.3321100-1-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3DDA3A000B X-Stat-Signature: 656r179z6domjkpo41oauu5o99jqjf8f X-Rspam-User: X-HE-Tag: 1732247940-158572 X-HE-Meta: U2FsdGVkX1+VikQNNJDW38MpuJhFZ8cV5z1a/a4Ikd+dhsUUXEVhi9u4CTpq5vpLDpJvkrUKNwDa71RW0Hpx1apOQ64pn7SKiLBo8x8fDx0Rm1bebYvEu0t4VTnBbo9LDEE21ycyXZ8Nt3+LwcHA39st+k4sY56nzCaAwPJkUCAFcZ3xKE6l6+LhZ2/Y8jGvsCVxmpRA5caqCp87uU56URmec7q47rmt7GTf6NiT0xdp6fUX79Jb3HeQk3ihUTpOWb3qZvlaRwHH8ETRXD38lJSfGRZjfWEhrAnAyd2hISj+SvVJf0hp/7xEEslwnBQdBTjM/sB5MycJEhe+qP/L56KNfX7ggJ+wrTcYyMRHEdca8q133I1urCoYPohMX9Dotv/dDr3Ss3BT5AlefIayQB4CRdHm4jn84lmOv2xEZ3ejoKL7O2KECzrzzOuv6hUQ4WVWzze3yfev8OYlquHLjrNQjlU7Nh4gB6v7J23p2UQJDH/WdJJJj1qKuveieQzCNjCLCYNFSRmp45cZcstjc6z0lCjeX+Di7qVy6eDsBe2j6STCV6F8fOeDRS9sY90e6WK+Bmmgl4qCeOD28wk4o5YeNVw9ebnoV9cgXakakXB8bfm+e4RlaeEkvGY2bisd3YKZetHahBUfVgXBMi26OK7J2JnMcRatEbT74rrBGXR03EQzyCzZMbhPxYxCe5e2OJlbcI+1e+K0nyEPFJZJc2NzlcgAsRbMg2xvaSIrZNKGBmwrYmbifZD9jSEZu++VLeWUEM9mZJUvqvvL5AwVtwTLQ1Wt6XahShigNh3AvCQyjjgE++VScGxRpvIXGzTcg96p8tZf3hnlmqobFeDc3UHq4R24+l25CvhmP/bxrOJ2xh4fqOiYHRQ+SurVT1bYfxM60o11GnuN/gjUSiG3KGvd87k80XMOnqilCL35+7fXCyZAh3cI1GclVh8ENsmYHKMiHSVmh2fzulYBpmy TGoYVVk4 GuPWT45fBvxN22SAiK6eb9mdjZ1eExYHY02KBTcqAQErRKBZHNUxuWndWjRb7/oboNEsebA4uxiE+zvP6p7Ozknd5wDPIAbJe+a1oFDfBnJVyOj3fCtYr/ejeIsFwgTdBltYRNMK9aB4Bopi5wyUrpe0R3hbvnfcDy0wIwS8z9jc3S1p0hfIVw134ULtg7pMQALThnWcCKlj5MeW2UrYBvoaJOXZ7yHjOa0jyBTaEDvuWgowbe3zNprn/f1FTyYeMhQI3vws5jdmtcFI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement speculative (lockless) resolution of VMA to inode to uprobe, bypassing the need to take mmap_lock for reads, if possible. This series is based on Suren's patch set [2], which adds mm_struct helpers that help detect whether mm_struct was changed, which is used by uprobe logic to validate that speculative results can be trusted after all the lookup logic results in a valid uprobe instance. Patch #1 is a simplification to uprobe VMA flag checking, suggested by Oleg. Patch #2 is the speculative VMA-to-uprobe resolution logic itself, and is the focal point of this patch set. It makes entry uprobes in common case scale very well with number of CPUs, as we avoid any locking or cache line bouncing between CPUs. See corresponding patch for details and benchmarking results. Note, this patch set assumes that FMODE_BACKING files were switched to have SLAB_TYPE_SAFE_BY_RCU semantics, which was recently done by Christian Brauner in [0]. This change can be pulled into perf/core through stable tags/vfs-6.13.for-bpf.file tag from [1]. [0] https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs-6.13.for-bpf.file&id=8b1bc2590af61129b82a189e9dc7c2804c34400e [1] git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git [2] https://lore.kernel.org/linux-mm/20241121162826.987947-1-surenb@google.com/ v4->v5: - rebase on top of Suren's latest version of mm patches addressing Peter's comment and API renaming request; v3->v4: - rebased and dropped data_race(), given mm_struct uses real seqcount (Peter); v2->v3: - dropped kfree_rcu() patch (Christian); - added data_race() annotations for fields of vma and vma->vm_file which could be modified during speculative lookup (Oleg); - fixed int->long problem in stubs for mmap_lock_speculation_{start,end}(), caught by Kernel test robot; v1->v2: - adjusted vma_end_write_all() comment to point out it should never be called manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be reworded (previously requested by Jann), so I'd appreciate some help there (Jann); - int -> long change for mm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam); - kfree_rcu_mightsleep() for FMODE_BACKING (Suren, Christian); - vm_flags simplification in find_active_uprobe_rcu() and find_active_uprobe_speculative() (Oleg); - guard(rcu)() simplified find_active_uprobe_speculative() implementation. Andrii Nakryiko (2): uprobes: simplify find_active_uprobe_rcu() VMA checks uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution kernel/events/uprobes.c | 47 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)