From patchwork Tue Oct 1 22:52:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 768FCCF318A for ; Tue, 1 Oct 2024 22:52:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A05E06B00D4; Tue, 1 Oct 2024 18:52:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B6796B00E9; Tue, 1 Oct 2024 18:52:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8559C6B00EA; Tue, 1 Oct 2024 18:52:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 64E5A6B00D4 for ; Tue, 1 Oct 2024 18:52:19 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BE11D16114F for ; Tue, 1 Oct 2024 22:52:18 +0000 (UTC) X-FDA: 82626533556.05.DAE5621 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 2D39020002 for ; Tue, 1 Oct 2024 22:52:16 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=MOoGW4fU; spf=pass (imf13.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=1r/5WEzbkiCYg/pll4syUE9iL9cXDbI4KqPiyKkGTQ0=; b=6FWisAvoOdioF7+pChp84SsxkNKTaYytTndA2Cx7MPzOZrkqR1/4PpowIj8cW1vqzASFtl irwiT6aZKyy7LQkDDFMpJAwuRUDMYZdmMOp/IeBiBu8Z67jKbPNBfOgE8jJYdDtyP1CMVZ Q/VFqrdmwIGNCYzzMb2fyHXFtx3p5hQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823010; a=rsa-sha256; cv=none; b=Qfdu68T1L54GeWhJ6qwCL9hFlNYtHfrhOLqE03S8Tx9dRGknekh82Ij3qZQOmT4ouFakPn yxdUn0RcUvno5Y8SVBBFCzMAXyXlJPM/ZCd/msD04HdMLsRzPqmpjj3WUloKul1iLJxwaL BUgK9w3emXgAvdxUGb226UnNcOck0O0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=MOoGW4fU; spf=pass (imf13.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id F1AA95C056E; Tue, 1 Oct 2024 22:52:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8E59CC4CEC6; Tue, 1 Oct 2024 22:52:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823135; bh=jVm5JoE6ytGpFpN+XW5rkiBwjgERB/1SvKPY0MEK07o=; h=From:To:Cc:Subject:Date:From; b=MOoGW4fUK1mub8zHKlSLKYINOQ3VpxTjIMf4tzRRP3TT1K0s42P8RvQ0Fi0yhllvN kMNpF47RJK/HsMqGSjRfkYItmcku72FabjdxS5ANVw9pwcsUeA9qD6YW0vy+Tg/D9R +IygtOcPh4W5LcquRJBX0DmwbyqF4pqvE5M9rzTY2GryBcPRwHapOQcUh0AUs+ykfw MOJC6jQc5+9Hign1iq/qhrJWKt512KxSbf9k9y790+F3rM6jQJxtbczcPoZ73pV7Sz hIF/DEGFIpHMBfY73MUkSs7Jr5cQO6KLpaffkLwEtNy2gr5HPliSO2HYh7rFJkx8Rq ufdC3IZtKYzmg== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 0/5] uprobes,mm: speculative lockless VMA-to-uprobe lookup Date: Tue, 1 Oct 2024 15:52:02 -0700 Message-ID: <20241001225207.2215639-1-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 MIME-Version: 1.0 X-Rspamd-Queue-Id: 2D39020002 X-Stat-Signature: sak69r5imwhwq5uizaoqjoyprcqwme8i X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1727823136-392522 X-HE-Meta: U2FsdGVkX181HqHDoXB7d+0+uwrNcS5nL4VRR2pFVsS1MUbjlkZa6eQhWrQ3h+3HKNnYPbh46905omcdJI5wdMd4MDFg6xrgZ6uTqKUpTVb4RsdErVbxAKt2SVlgqPjdWHoGszp7YUeM7bBSecE5d+yzNY4ZXaS73oMQwbMSgadPyVnrn0oV4gFPYDkHH7+FJMuI5c3PPjCBr8Zg0/asefOTFUaFjUfad34MXFrUpKphKu4ad3Hzn+ajVOj4AXuZ5vqsZ6SA9L8fP/2uD8isZtMyzFb98ISfsJDBRYzYfUrJMO/nu11trAHtRsxWQ1ej28Ctsk+ck7bOM6S/czrOqzvDKJ1VwTdXs0feoYNPRn84Y3pVKzXhX+Lf7rIg38Se2ZDeq5hQWrK/6LPRg1wC/SxdOHhBXQqFOk+66b5XcdUa3QOPbFJ/qGFPsiG+JBIcnbp49f/dAGsvvW40sYfhUzkQoAvp05I1WDYpbJkbWS9ly1u/4AnP+WBqYsTJQbWsCGJ0r+NUX+XuWh3+IGztMLr5HVmAvIhBQQ3/xN+mwAhr5RD1NKElNS0zoQcB6DUEfVVSaqhwg6U9F72vkeW9wchZVpsA4+L6CBv2/Ypc/+1GOZQ0a1Cnyzo3rDAkpFE7uz21k/6r/GNNGukx1F1G4Xmxij/9ZpWifY/3xe9bBKBl4hwhyd/dYo81SmjnHHJiLB1KFSoWe3cuktf+rsTpalHrbTKVhEYX4F4RypRBzld/y0DB3aNZo3q8UWBfvheB6oKKToupq6Fw0khXcDHugBHrzhEELCvbFDcI/3+o1CYhrfvR3yMghbtqnP9Q8DvnxCiLunbVjwhJ49dhUvNd6WrjjDczJc2yR7q31oqC64zBo2aRXQDSOlSxuRCAk1hbuaKkwCmRr/zMCbggy6GjFn7bZZW9zDMGE+wk6wlXe2XMXNYgtjT+NrnU5Zr/a+mlSP3KzGOnm+mUNWL3Tc2 5hx8w5Fd sqTi/PZ+56+zPxhtb9RWMBo6mtOUPBcBBlL6wc7lNa+1IFVSGJvFfa3GmQIwDGLrsA0bwB6NvgnXm01w6An8Au//hZtKDkJykYc3ui2aSNfK8RvaVJ0524OWqwmKFV6Bl6HoAF3K5jXlW4wOhQDm0UqCNgYsXhMzsmOPM0u9Zev2LTBjXKBfTPa1lxyNiw0X0JMeQaYna2wiEWwPsQzW+hoKEFmIUu1zBJl+d X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement speculative (lockless) resolution of VMA to inode to uprobe, bypassing the need to take mmap_lock for reads, if possible. Patch #1 by Suren adds mm_struct helpers that help detect whether mm_struct were changed, which is used by uprobe logic to validate that speculative results can be trusted after all the lookup logic results in a valid uprobe instance. Patch #2 follows to make mm_lock_seq into 64-bit counter (on 64-bit architectures). Patch #3 adds back RCU-delayed freeing for FMODE_BACKING files, which is necessary to make speculation safe to access struct file's memory in any possible situation. Patch #4 is a simplification to uprobe VMA flag checking, suggested by Oleg. And, finally, patch #5 is the speculative VMA-to-uprobe resolution logic. See corresponding patch for details and benchmarking results. v1->v2: - adjusted vma_end_write_all() comment to point out it should never be called manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be reworded (previously requested by Jann), so I'd appreciate some help there (Jann); - int -> long change for mm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam); - kfree_rcu_mightsleep() for FMODE_BACKING (Suren, Christian); - vm_flags simplification in find_active_uprobe_rcu() and find_active_uprobe_speculative() (Oleg); - guard(rcu)() simplified find_active_uprobe_speculative() implementation. Andrii Nakryiko (4): mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures fs: add back RCU-delayed freeing of FMODE_BACKING file uprobes: simplify find_active_uprobe_rcu() VMA checks uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution Suren Baghdasaryan (1): mm: introduce mmap_lock_speculation_{start|end} fs/file_table.c | 2 +- include/linux/mm.h | 6 ++-- include/linux/mm_types.h | 7 ++-- include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/events/uprobes.c | 46 ++++++++++++++++++++++++- kernel/fork.c | 3 -- 6 files changed, 114 insertions(+), 22 deletions(-)