From patchwork Thu Oct 10 20:56:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13831109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EC0DD2444F for ; Thu, 10 Oct 2024 20:56:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 38A506B0082; Thu, 10 Oct 2024 16:56:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 33AAB6B0083; Thu, 10 Oct 2024 16:56:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2027C6B0085; Thu, 10 Oct 2024 16:56:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 01D346B0082 for ; Thu, 10 Oct 2024 16:56:50 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 991C016040E for ; Thu, 10 Oct 2024 20:56:46 +0000 (UTC) X-FDA: 82658901738.14.5363725 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf23.hostedemail.com (Postfix) with ESMTP id E22F4140003 for ; Thu, 10 Oct 2024 20:56:47 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=jwEBw6JN; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf23.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728593696; a=rsa-sha256; cv=none; b=lvqR5XrqO9EtEui/capIBSKMvgz6Fy6u1uzRsf8umpWKH/Ruy1Zc4IfeAkG0cZIe0H+LFb UgqriBjG6x72XqZtvIEWq09zzPFBVjFI1o1lh8XzQ/aDikFdg2BK29OPQzABKT1pfQS9WH ZICfpgUe7IzsGRd6Zw/eRPcf4BqF/gU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=jwEBw6JN; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf23.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728593696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=81E0doWYlOnZzoa+2c0i3PbUDFwWzhhnktASNx8QCIs=; b=o4DwEakwGQ6MDNXdNEGY22I/nTiMuM2ZA5UO8OYej/ooWgaFjkGRWvP79Q5PrtFk55dCkw e03gBok3BKVi4VCyGHvRwafsLiMNhrRoT9dLggjWup2kqHqFSs5y6VuYRPTLwbtv6xZviN HX7yuHYUO61YMETOxK6mgFWT9QJdjhk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 7E8905C5DDF; Thu, 10 Oct 2024 20:56:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4E578C4CEC5; Thu, 10 Oct 2024 20:56:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728593807; bh=tH2HcY6rQCU/6gQa+L+AuOeDayO0u2RWtUN85rJcHWQ=; h=From:To:Cc:Subject:Date:From; b=jwEBw6JNnDAvo+Nn7aTkh4T4g/t7dFkecxwENqt8tBspsVmoIQTPH3Rj5ZaGeD2ot 4lAzUUe+VjltpHrOZU71opy9pNO5sNyn+b0u3HoQwz/RQfipK/pTVW6tfk5Jqns/Kx ZdJFic+C1Nk0/Mr+xyWEeeHVC9M5Rt7PnkxRsl2ISKWs12ze4cbX4UuzP7XQsz9LJ+ dXIpCOMOmFendm65S9btudipga7Azw1ppGlX8zAjhS8SDwBV2wy8PLGlQoIGraiFQZ o5Sqovdcm72UElzAyvgM7DaE2c3Nc/YOi2LyTGNLI8NzKCTc4YdsbDQr1rOkYnf/Bl IDN3rQK9H5Ogg== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, peterz@infradead.org Cc: oleg@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, shakeel.butt@linux.dev, hannes@cmpxchg.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, Andrii Nakryiko Subject: [PATCH v3 tip/perf/core 0/4] uprobes,mm: speculative lockless VMA-to-uprobe lookup Date: Thu, 10 Oct 2024 13:56:40 -0700 Message-ID: <20241010205644.3831427-1-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E22F4140003 X-Stat-Signature: gxthxuzjyig8p1pyh3n4q9psjipec9zg X-Rspam-User: X-HE-Tag: 1728593807-70458 X-HE-Meta: U2FsdGVkX1+MUozpI4uXgs4EgfzG6ayUhECxlXX+KBnzs8uJ8rCFguBjZUig9loSlrFjyB5+0lqhdrg51mWctRc+hXrsrBOHgvC89pRehmts/ZrhrhY0H5NkqB8BLJ71HsQbB8gguGto4P02SXp+NolCDIDgqwpygJ8ayNShh34aLkLM8TJrUQ5invoXLeboFT9uQITGYUy7HNf02FRrRPG077SWUMkNkNujDKxPk5kSo7PMwzbgaShCrwm+uVBT5MDic8u9/cKic++Mj0kzRlXzCvyuREmgCo0hj5JUDvAefFlviS2BBUYOpXgIqI7tHBzuN4F/Qs0xifn2qsmGbVj4ATC6SBRwUMNrUnXBS6SU5T2xtLcGdwZzP426QS5ouMgecuzSTV70SU72+3AExZIEgsPRUK/Z3sQHRSTDAqN72hoKtUv4XSDJrzOpapP3fSbfKfCvcRULyNLOYCKgI04kVmS1O7HczMTcpABhNsguiy56wqk2r9OGWmp46TbB5/MJhhbVF6xxybzy3qG8RF80rF9bSGGcnncsAvdne/GMJVRmdYX9AN+9fdnZOwznMXU3bMQONpyojxPFv9ZgEnqIvsEDrxi7Y2+FeuAfBXW+bkeROW2NKqDjB/ji+PwTYqxrdJHQFaWwnoSI64xg/gNPFwJZAaC48CUtjIk/K31DafVj00N6XHGesn7a9cpX+d4lCIz4B1wATcKoUtomxphaDAFMTVEFxAEXNf9njsZb7KnHS1z6+XHeY1rTnhKeDrgpDXd9ZDxriRdbuoNacsk+X1WdiUiOOYiQWSMwCqFrn1K79zsicO0xdjtjmHvsVCWQ1x8avmeiBRdHnYcXv2gGsyBowwmHtNyc73oCWFbH6E7Q4jbh4fK5urEiyDQhR25LpfOY4QTZVilbEfq3uAUHscFrpmKUqrjBOZ0TiyvlNA4ilDdohrSAfY5Lvq7GuTNInTmUyWOm3jhm4xb 0OGNpeUv 5uk1gcRp9VIOAAbKb7XTQqqLXD5nLpby2KRgVfqagylgE663b7uMH9jrolM1o2rDjZrGJXF0FTQcNqtc6xZhYz4Voz0oW8I7jLIzkvyX3gpueSmrn+5NPuA/CYYxwF3cmyzLQUTcIf84DiVFZBBlDXGnI5yVVChc/B5nddtSYDTOawO206zXHjaH5QUp71w6/00JmGlLLr0/sn6hYMkuOzyCx7cBFRt8op3f2dyw9qvquf52x8teb/GsMTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement speculative (lockless) resolution of VMA to inode to uprobe, bypassing the need to take mmap_lock for reads, if possible. Patch #1 by Suren adds mm_struct helpers that help detect whether mm_struct was changed, which is used by uprobe logic to validate that speculative results can be trusted after all the lookup logic results in a valid uprobe instance. Patch #2 follows to make mm_lock_seq into 64-bit counter (on 64-bit architectures), as requested by Jann Horn. Patch #3 is a simplification to uprobe VMA flag checking, suggested by Oleg. And, finally, patch #4 is the speculative VMA-to-uprobe resolution logic itself, and is the focal point of this patch set. It makes entry uprobes in common case scale very well with number of CPUs, as we avoid any locking or cache line bouncing between CPUs. See corresponding patch for details and benchmarking results. Note, this patch set assumes that FMODE_BACKING files were switched to have SLAB_TYPE_SAFE_BY_RCU semantics, which was recently done by Christian Brauner in [0]. This change can be pulled into perf/core through stable tags/vfs-6.13.for-bpf.file tag from [1]. [0] https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs-6.13.for-bpf.file&id=8b1bc2590af61129b82a189e9dc7c2804c34400e [1] git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git v2->v3: - dropped kfree_rcu() patch (Christian); - added data_race() annotations for fields of vma and vma->vm_file which could be modified during speculative lookup (Oleg); - fixed int->long problem in stubs for mmap_lock_speculation_{start,end}(), caught by Kernel test robot; v1->v2: - adjusted vma_end_write_all() comment to point out it should never be called manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be reworded (previously requested by Jann), so I'd appreciate some help there (Jann); - int -> long change for mm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam); - kfree_rcu_mightsleep() for FMODE_BACKING (Suren, Christian); - vm_flags simplification in find_active_uprobe_rcu() and find_active_uprobe_speculative() (Oleg); - guard(rcu)() simplified find_active_uprobe_speculative() implementation. Andrii Nakryiko (3): mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures uprobes: simplify find_active_uprobe_rcu() VMA checks uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution Suren Baghdasaryan (1): mm: introduce mmap_lock_speculation_{start|end} include/linux/mm.h | 6 ++-- include/linux/mm_types.h | 7 ++-- include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/events/uprobes.c | 52 +++++++++++++++++++++++++++- kernel/fork.c | 3 -- 5 files changed, 119 insertions(+), 21 deletions(-)