From patchwork Tue Oct 1 22:52:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818830 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB687CF318A for ; Tue, 1 Oct 2024 22:52:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 611116B00F7; Tue, 1 Oct 2024 18:52:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BE35680023; Tue, 1 Oct 2024 18:52:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 412FF6B0108; Tue, 1 Oct 2024 18:52:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1DEFA6B00F7 for ; Tue, 1 Oct 2024 18:52:23 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9A7A116114F for ; Tue, 1 Oct 2024 22:52:22 +0000 (UTC) X-FDA: 82626533724.18.D34C034 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id E69E8140003 for ; Tue, 1 Oct 2024 22:52:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dfDuVI4A; spf=pass (imf09.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E0UMJijSGhyPeMd9wA20oMGeQINhZ0MT7UqcI2QQ+RQ=; b=k0nXGVefx1VVEYGHtiK34SRNMTW22ffErEkBOI8twWFq6yVNQdWndQxf8E2uapSTY8aA9y bpD7UnHLDQfi7GcAPSJ/5dAWE6IDFTtDM/cbDa1wuWao88PhF1oA034elZ5w4HeXJj0gJl rmjE2OG4w/7LjaK3rCEL+Me4bgRcXEI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823012; a=rsa-sha256; cv=none; b=qQpfVM7Ggicc3mhvtmXzkpQeXMxPaGI3Ilk+DcKkmmuLkOvvIVfdZhFEcp75M6HOF+yd0a dXjFqvKY1jx7HOVciHX1ZcHMKS1CnhQPXky9VYPRAe6IEe5L2wkF4uwqzAsUMWtxfX1d7b NN6casWNHuI2fydcnbDuDLbifbza3JE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dfDuVI4A; spf=pass (imf09.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1621B5C5524; Tue, 1 Oct 2024 22:52:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C73A1C4CEC6; Tue, 1 Oct 2024 22:52:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823139; bh=5eizI5x+ShSEQgV2sFgQYF+jiwqy/RpZnCYPy8CWlCY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dfDuVI4A1C9B6XefEXXmRXfyt7/s7dX48eXgg1McB+htkj6q+gilGOE4d61u22XCh UfxULXP1HekgDrUTsX2+Ln5wBjCakxnoD/gFrptDXm5zUyoSqPWUf30SbjqdXEbzQX 1Y7WBvvnXnURdWM7kDKwCuD1ZyLjVha0EIEcEHAPgeIvQbAIYIl3uMOjNn4Psmap5M QFFj8DAP9dAq9Mj7rOjscOJ9UFzHxizXyyksvbX1ftosYFrQe6GyfemxxqilV6KEBX bQr4Hj1fllwJKPcEpXl2xOQYQECHviW+3pSvnf592Nq2IwBGBWXnrjNhPa2grRrgbA zgC9Q7nB+8MNQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 1/5] mm: introduce mmap_lock_speculation_{start|end} Date: Tue, 1 Oct 2024 15:52:03 -0700 Message-ID: <20241001225207.2215639-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> MIME-Version: 1.0 X-Stat-Signature: y3b5mx73pn9jmibnsnp1bybjh9pk9qxg X-Rspamd-Queue-Id: E69E8140003 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1727823139-103016 X-HE-Meta: U2FsdGVkX18ynZDAkJ1Y/wjPBzB8fB/8qE2uh/9nDPQrYBZPDETbLbuRst7OT80GwLqyG7MK3bI4qobUcvsJI97d1QCx2vdEohjAyJo5r+i15PTlDfI6NPMYF0Gucej3E57QEUK5wCMTiAUfJN81lQnjUu2clH8sDwLg9SbrS/4N6xJv6LD1I0pku33bd1SW0bkXZoSICVJHMpKzssUF2k+t44i6/+EDlGnjXHjgQWI8HqLpV6Zl2ORnMioqFw1Onnfxcudo7y9q9mMXfEJ0pJttWW24tWBLWf4Sy0Hw1DXSIdM/LrjTjkrBWEx5cVn3HYMNdks6hnBkXrOOK10gZFV+eGPfy9msLG66nTPPE06s+QVcSWJtugrnEVGdF/62ivHopxt0OPOoOZiPfeZvv1Ek3m3UvBkj7ItgfhXpFhZftGzpWcJSua7VFkfbfcwucnFAnYrsbpUITi5h2ybTa4XHlNbtqfO+DXL6dw+qOkHaes5P9rRi7uAYJjp9rRRqYrPZyku8+s7ymeWL7zGV7i7vxeq2xttay/bQL9rdn6eWARMg5l9EmAB7SurhivlaLvwW5Ax55adHcWE1aNC0q7oawA3gjYLteCY2yxPSPc9mY1DuVNUZ7mcJKdiFeCZ9UdubYU5oDor5SL2HAi667EKwNc9D9XBLkshTyHlkvnu622w3NyPPTfk+g32mvFDLAkdSGiXO0trNjDHiD0Vg77MmPRgLdgcwliLW9rGRk2c1ZbacqRd3oUopdeHz/bxgIVBc4TL+Vh7KH/+BcHqLJrWIXEtR6TU9+ZB7xd1f5kxrhb3z2RynkhxkwqJ8vpJfiBib3fRAnwVgPHdP0ZAfNjSbIRopAJvLBbH0tH7YBiX0+kuf8fPjUKNglgmzy7uA2KsXlcj38c0nLDsSCib8g4RfP9v3VV7VT8sznCRQaBP3wCUB2hWjGecptsne1KgEjTB/kLQnX592UhNM5Ap WuCpA0Cn PAKH+DwCTYs+A2eUKWmMkQYvq5hek+b663qvVjn3ByPq4tVJscPdt3OeU3Ukva809w2JrZ6bBs4pRvHuNjU1yX1fMJlMEKID+gP/tjIsD7wW9hpjV2WNyzoZyk0Mqxm0ysHKTWcwVL4BDERskamisQOpbQaocxuDBB8WOI4prvcarj8tfLvYRMaIqMK9hUEeXhNNNTnJfOx9/hUlhgJiBOeUDOV+qRginOxbJPN3AKg4zoy6awc6ZDw3Fwm8rHvGrjPSDOLBD3pq9wDmgjqAJH4b8/LZ9KllLxXnDFNWPqMEahJG/GghL7l72s8DCQ/kFRjOd6Q7TxN7hIujD/QLgRctVX8jbvc1ObqYtXdk46E+89K8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Suren Baghdasaryan Add helper functions to speculatively perform operations without read-locking mmap_lock, expecting that mmap_lock will not be write-locked and mm is not modified from under us. Suggested-by: Peter Zijlstra Signed-off-by: Suren Baghdasaryan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20240912210222.186542-1-surenb@google.com --- include/linux/mm_types.h | 3 ++ include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/fork.c | 3 -- 3 files changed, 63 insertions(+), 15 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 485424979254..d5e3f907eea4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -876,6 +876,9 @@ struct mm_struct { * Roughly speaking, incrementing the sequence number is * equivalent to releasing locks on VMAs; reading the sequence * number can be part of taking a read lock on a VMA. + * Incremented every time mmap_lock is write-locked/unlocked. + * Initialized to 0, therefore odd values indicate mmap_lock + * is write-locked and even values that it's released. * * Can be modified under write mmap_lock using RELEASE * semantics. diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index de9dc20b01ba..9d23635bc701 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -71,39 +71,84 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) } #ifdef CONFIG_PER_VMA_LOCK +static inline void init_mm_lock_seq(struct mm_struct *mm) +{ + mm->mm_lock_seq = 0; +} + /* - * Drop all currently-held per-VMA locks. - * This is called from the mmap_lock implementation directly before releasing - * a write-locked mmap_lock (or downgrading it to read-locked). - * This should normally NOT be called manually from other places. - * If you want to call this manually anyway, keep in mind that this will release - * *all* VMA write locks, including ones from further up the stack. + * Increment mm->mm_lock_seq when mmap_lock is write-locked (ACQUIRE semantics) + * or write-unlocked (RELEASE semantics). */ -static inline void vma_end_write_all(struct mm_struct *mm) +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) { mmap_assert_write_locked(mm); /* * Nobody can concurrently modify mm->mm_lock_seq due to exclusive * mmap_lock being held. - * We need RELEASE semantics here to ensure that preceding stores into - * the VMA take effect before we unlock it with this store. - * Pairs with ACQUIRE semantics in vma_start_read(). */ - smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + + if (acquire) { + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); + /* + * For ACQUIRE semantics we should ensure no following stores are + * reordered to appear before the mm->mm_lock_seq modification. + */ + smp_wmb(); + } else { + /* + * We need RELEASE semantics here to ensure that preceding stores + * into the VMA take effect before we unlock it with this store. + * Pairs with ACQUIRE semantics in vma_start_read(). + */ + smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + } +} + +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) +{ + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ + *seq = smp_load_acquire(&mm->mm_lock_seq); + /* Allow speculation if mmap_lock is not write-locked */ + return (*seq & 1) == 0; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) +{ + /* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */ + smp_rmb(); + return seq == READ_ONCE(mm->mm_lock_seq); } + #else -static inline void vma_end_write_all(struct mm_struct *mm) {} +static inline void init_mm_lock_seq(struct mm_struct *mm) {} +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) {} +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) { return false; } +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) { return false; } #endif +/* + * Drop all currently-held per-VMA locks. + * This is called from the mmap_lock implementation directly before releasing + * a write-locked mmap_lock (or downgrading it to read-locked). + * This should NOT be called manually from other places. + */ +static inline void vma_end_write_all(struct mm_struct *mm) +{ + inc_mm_lock_seq(mm, false); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); + init_mm_lock_seq(mm); } static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -111,6 +156,7 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -120,6 +166,8 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } diff --git a/kernel/fork.c b/kernel/fork.c index 18bdc87209d0..c44b71d354ee 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1259,9 +1259,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); -#ifdef CONFIG_PER_VMA_LOCK - mm->mm_lock_seq = 0; -#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; From patchwork Tue Oct 1 22:52:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09FFBCF3189 for ; Tue, 1 Oct 2024 22:52:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96AEA680023; Tue, 1 Oct 2024 18:52:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9193A6B010B; Tue, 1 Oct 2024 18:52:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 793E3680023; Tue, 1 Oct 2024 18:52:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5910E6B0108 for ; Tue, 1 Oct 2024 18:52:25 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 101268018B for ; Tue, 1 Oct 2024 22:52:25 +0000 (UTC) X-FDA: 82626533850.05.F37ECFB Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf08.hostedemail.com (Postfix) with ESMTP id 6E1CE16000B for ; Tue, 1 Oct 2024 22:52:23 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=heHmzfSm; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823049; a=rsa-sha256; cv=none; b=HuNwklkU1sLwXq6jwYCovd6iwNgqPgGEEPEOl3LXK0B6zzdWTBMjF1BVjpFUD+8cS3ACsN w0kO4Kqo775a31309eX0wFElCb1zB51PGI19HLRctz+1rXYjUwyDqMa2TiotadrbvbRpqJ sOhxJy0fFyqdvd3Ew2pV+TES4+hf/BE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=heHmzfSm; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823049; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8ZILQ5jD8NdORilhH1I0h4IhZkeTehuNJh8ARrlia4=; b=k8kUc37cJe1O/8Uh7+F0hglxJlLtFAOoqiOvXhEJgWmcMJNLVOzE+2fnB4AL6L24FlW2Cv LMeluGX/JokH8PDUKilHFvvTIaDOKRpM+kcD/UlRyicxAktqHYKD/TYBqmgWRv4BS67mJo EiKL9l4Dl2KFMnCjkgJz9xH7AlY9XzE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id A28365C5533; Tue, 1 Oct 2024 22:52:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09495C4CECD; Tue, 1 Oct 2024 22:52:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823142; bh=Q4lsQ3VDioDbBQhpNH028nIzfuY7M4lO4nHV2LRxuL4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=heHmzfSm2yzI1iI9a6pWLMysWPAc4UEw5I9P9k+LCuudPU3BDHrwWOqPAvRSOEqTP Ld0jpPXOtCuEWw1kQnr1yJxZdwUK1sroXcx6QboO6bkoM27/YJj51x4ArDOxWR7e+N 6TroV0+9NlEv+Gi6hpXaHDl9AgN4jyT3NE6il/bVevt/+NknSG3iX/M2eRxc8xAc4x DmsnJVCgep1LiyQEN3aTVQ3r3ZIG1B2qky5sxKLak2lGcIuCEBKb7s+4J8x771Ipfs Y7l+2n+hzxF9xCO4R2/LEb/EF/kjGm7gDDPXvnpTHGN5iCRHxN/cwUM+nhZ1rDPCOl 4im8A0phpKNdQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 2/5] mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures Date: Tue, 1 Oct 2024 15:52:04 -0700 Message-ID: <20241001225207.2215639-3-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6E1CE16000B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gcgkcbi6szigoi5nmhur8hhu1r15x6h7 X-HE-Tag: 1727823143-521571 X-HE-Meta: U2FsdGVkX18xq2az5GIzkgy0VqiCJuf0/2R3uv5uCAp/4AsDUv/sKAmvCOQXWukk+/Qgg3I+ft3GTIafn0TXo7YwyUQbQ9j7zl9J4nbqRpQF4kFtCI+4jTw6PZMfsZS/WRtO8he0jBxYGUsgJW+RMVH/DHVlM8E9/lA4J3gVfHPKfLboukdDwOD2UuwEa+Jjnp0wgEifcyehOjsAzZdw5BZAK/qRFld6WeQpblsG5I5FsEShgY8KVVBqAyF+si1y97Wfjy7jl7KvgqiG2ma7WzGpfScDnMJrw9Sp1nifM29Gqlcjz45v2ISX460CiRux7FKRXnoP5H0AvvumNy63gcDwaRlLl4iXJCH3aMVXizsU3sddfgdWnusUpoVEZ40jGgdvgnAAX7icbtA1ue20QaTYgeJEt+sngUyLQpT64MPdOf7QN0xYXX/ipdhtm4HyfrIzEwJ+ihXeT3jH3N0mmwTJ7ZKA9CcR83kqtHN/pw68kw76TFeXwwwsmEktQAgKtyDdDZZwxRqibwdTmLbHT4EcLvj5P63wdQEEa+EbqieBXcbEXYSDfVxVylZNPCRwUqYWV/AiVVLiMXtkGECkXNQXem1VXDoCG0iftcyghxn5uv+WYE2XH4uTZyFmyjVufBngX/c3QAgnrZoc7MxBQYzoWNAsMZHAAls4LW/6Nk0dFE0HKRraU6+KSa3qapHy2l2RptQUzx+/Mi2jOD0sYe3kT4v3MAVsu9v8fX/lFqXeBDcC4QCKXkvC3Hfgzx46UdfeQJsJHm8gEGnCEFTHCe2ziY4W/I+1xbY6g4SXV3RP9BNUH4roRdE06Ejb6Je6yaMyoxSdLBGhflhHjMO/Mr5KgOhi8TtxMGWHZsR5VHZ++URkegk6T3IhC9Fi53m+PVoWPaLpfFuTXyxIOVcJJNrAcrBhNZgmORBw+v4ukwR3auU9fbWyWY9aB4jpcCoHsfkbtwQ5YDJ1EJaHLEh yJbb06m/ KB6zhCqbQEOQmh+6/N3btsjrDXOgKz+9HV/Rmqtlrl7MHfbReYU0411ebcPMh/Wkllp8b2SG6hR8f3BUVdl6gwvMvV0/iJXetqNzWfwW9NEP0qvu5VbaS7ojuqrE4sLxAgCQUhNgDTin7my3AWMylfzLgTXn9omt7bYgPoQLBObpxfQYHkqt/ihNQoNcyPIkDxYFm15ww9XNdywhzlso6yRxsthR2nKpoVXCXXfEDpEjpmcquMVzDFUJ5t4QBJaJaH41eU17bjoEsTyRRykufPxNAAPYXDE8ssoFUk7EjU98kjfdhg8mjIh0MUw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To increase mm->mm_lock_seq robustness, switch it from int to long, so that it's a 64-bit counter on 64-bit systems and we can stop worrying about it wrapping around in just ~4 billion iterations. Same goes for VMA's matching vm_lock_seq, which is derived from mm_lock_seq. I didn't use __u64 outright to keep 32-bit architectures unaffected, but if it seems important enough, I have nothing against using __u64. Suggested-by: Jann Horn Signed-off-by: Andrii Nakryiko --- include/linux/mm.h | 6 +++--- include/linux/mm_types.h | 4 ++-- include/linux/mmap_lock.h | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6549d0979b28..f8e75d0642a8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -716,7 +716,7 @@ static inline void vma_end_read(struct vm_area_struct *vma) } /* WARNING! Can only be used if mmap_lock is expected to be write-locked */ -static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) +static bool __is_vma_write_locked(struct vm_area_struct *vma, long *mm_lock_seq) { mmap_assert_write_locked(vma->vm_mm); @@ -735,7 +735,7 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) */ static inline void vma_start_write(struct vm_area_struct *vma) { - int mm_lock_seq; + long mm_lock_seq; if (__is_vma_write_locked(vma, &mm_lock_seq)) return; @@ -753,7 +753,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) static inline void vma_assert_write_locked(struct vm_area_struct *vma) { - int mm_lock_seq; + long mm_lock_seq; VM_BUG_ON_VMA(!__is_vma_write_locked(vma, &mm_lock_seq), vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d5e3f907eea4..c045543f43d9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -705,7 +705,7 @@ struct vm_area_struct { * counter reuse can only lead to occasional unnecessary use of the * slowpath. */ - int vm_lock_seq; + long vm_lock_seq; struct vma_lock *vm_lock; #endif @@ -887,7 +887,7 @@ struct mm_struct { * Can be read with ACQUIRE semantics if not holding write * mmap_lock. */ - int mm_lock_seq; + long mm_lock_seq; #endif diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 9d23635bc701..fca527dece63 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -105,7 +105,7 @@ static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) } } -static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, long *seq) { /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ *seq = smp_load_acquire(&mm->mm_lock_seq); @@ -113,7 +113,7 @@ static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) return (*seq & 1) == 0; } -static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, long seq) { /* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */ smp_rmb(); From patchwork Tue Oct 1 22:52:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818832 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39BC9CF3189 for ; Tue, 1 Oct 2024 22:52:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDC5D680029; Tue, 1 Oct 2024 18:52:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B8B6B6B012D; Tue, 1 Oct 2024 18:52:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0430680029; Tue, 1 Oct 2024 18:52:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7794F6B012B for ; Tue, 1 Oct 2024 18:52:29 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F0935A9CB1 for ; Tue, 1 Oct 2024 22:52:28 +0000 (UTC) X-FDA: 82626533976.13.FDC42BC Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf06.hostedemail.com (Postfix) with ESMTP id 69BDA18000F for ; Tue, 1 Oct 2024 22:52:26 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CpHdhX2M; spf=pass (imf06.hostedemail.com: domain of andrii@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823081; a=rsa-sha256; cv=none; b=vI7y1vkzgCrh7ySD2WoBcAkd7fZEIYeQHUiP8SnnZsWrTPVIxmoc8L9/Cf49hEWUatKJBy 125vAMSzhoraMQ+AQHFf4vZIfg3D0hdogbYKPqwcF8mqgxZNkQMVZu4ogLGYYsP4PIED1N mHOfbqi9tvyvXui83x16BTQgbijwA2k= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CpHdhX2M; spf=pass (imf06.hostedemail.com: domain of andrii@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823081; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kj0vkyEpwZweBGsjeOW0IkTQksU+6PEipyZ/5oRsvKs=; b=fMZzpocR26X+eEOKOgbUfRKmHG+ALS0N4SOWeAjFF+eu+ROFv1yzmQ67K/94txkzQAcAr1 YJsxib07+aU/Hgzft0GwzYSGra/sFyj3gcjm9fb/PPD1EUVEQvSrjY9pJl+0L1f/xrNxx+ y8HqnvxaQJ5b8jzzsb/cvqhhacuHEsY= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 713A4A4336F; Tue, 1 Oct 2024 22:52:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4533DC4CEC6; Tue, 1 Oct 2024 22:52:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823145; bh=Fe0W0B2s6ceCb71LS5xDyOWx4mAiBCLoZvDtud5NppM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CpHdhX2My8bEMExNjtkiCBGzgD/icJoKKBqYiIrUbPzCTfYszjoLtbMmq1dssYj3T Vrwe8b43Ff5XJWhkflBzoIkA4mbw1tW1bc+gdwJ5gHBcaUuGvSIZVtObiyY9gzB6fL 279sn7GDirJgGJM+LjoW+rAaFbP+0qgSw780RGYXJSzkxUeqFy2lbgEVIGwSB2x2EA yVmzNitaw7bjQ8Gxrp5it9ifxi0NkWH3rkKE5gjhUNhD8GF3nt6umR84JDzYdOvj6S 032wn+XUSq9IQVRDNOwPE/ysqQ+e7s8s7xRGN/F99dVabaYTJU5jAaQ7kjjhMvZGTm 9/45fjGcrlZ7g== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko , Amir Goldstein Subject: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file Date: Tue, 1 Oct 2024 15:52:05 -0700 Message-ID: <20241001225207.2215639-4-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> MIME-Version: 1.0 X-Stat-Signature: y3bxcehjhg897zgeejeeb3i6xrygin7y X-Rspamd-Queue-Id: 69BDA18000F X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1727823146-975454 X-HE-Meta: U2FsdGVkX198cdqoLp7gm9RY789LFhOAFCf+IwHVxrh+VotbJ+tD6+E5QV8FKV8pqT1gT0MiIGHpX5wf78moyWQTiFA2ofyh/2hVEpGZos3V+QtwWXnzSpin81IISzQ98IL1Fzz66nfdHY0b27OGNl3MJMp47uENoZ1B7eYg9gjqZzmQ9hN+glgNAod8g+2V4aWQ8jFsFU1WJOzUJGf3JClYQIHsly0FMJ7oUNJiX1FKixo9N4LRJC42Tqh0fDve5U6p4BQMzUgX8rC6mzt0+2tDwmfv7rG1AjPIrTXhog62Y8AbWY6+QP7+P8mD31Yu7sqNi91KfMR3fLqkkO0VY9GIXcx0Ef0ygYPUFz5Jua8IidhQYoIPbeul4fBBu5z8h6Beeb0IKaog7rphoH9S1gW5z0j4kC0e8kDs2VXI1s/19ioXqrjoV166xwEMEtADj0dSAqGc+cBfsmX3d4Ux1PJDlc054R81IBKIFcZYzxVw1Mo5ED1fs01R0Jud9grReUwaMbwhOaHwiM7DIJ1Qeguxgn+d1F8+rc3B5EsMhyZzXVFtVKtJTUwU9ytjQyzeQmAX7v6TgphC9KqfTdRr7EfLx4agr7gho8bdDAHoBHQSWal3/aCQRgygCNbfIPWDMdtGKjBxFj3qkTjaY+cQfYnRUodKEKxQLU/Bev69oY7FBc3t4fT9wgYIpRq2CEXnb6Js3SOVhORTMj6cLsKtFBMbN4YBF+20HbZ/9Da2WU6GcFrSZN5KEqJtBgWsh57eDW+E1pJ2GFpUszIDpqriJMl2K0UM5IoCCwNDRImlz+kf4vEYkY2zme0QjPTVEld1ZZXXIlBsdVTplmZD7ZllrHp3zI11J5CmpHihG+kIbP7XuM+JmHgcpm2owOJ0efAWbOejqmg1JDvuZKeZvgvr95GC2d0UIenRf8nvbGcn3aNEQ3euBA25gEBBHIn4jdeNcWal3hWBpHbFa/MmMow kzlRDJ3R ovbGUInHZaRu1cEyOdf3sc5yNSPrswQx44ATMzBK84LZN/1ZaBP4+dieu/tTZZlR/CCC17cbOZXTFAXIvxXhh/o1voVgLId2VJuXiGahRFJ+q4gIObXPhck65S+EvvkbaIAPPa1fugprsKaMeVed0k4LQRW+d3j48hlwjSIBH3BKNdPvbSP2A+XN8H0ZYG8QR+3gke7EbN7tw9Xcn6g3sH7AEz4II/rWf9cNc1gKedLLWCVuNFWw1iYWnr+qhC79j0KD0tz8ooPCL3aCLz3MqxM8lNk1i+AgbYzmiqMRvwxj5sgWZ//98Yo9IF5tMjPIRauDgIa1qwb2bZmc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 6cf41fcfe099 ("backing file: free directly") switched FMODE_BACKING files to direct freeing as back then there were no use cases requiring RCU protected access to such files. Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic. We use headless kfree_rcu_mightsleep() variant, as file_free() is only called for FMODE_BACKING files in might_sleep() context. Suggested-by: Suren Baghdasaryan Cc: Christian Brauner Cc: Amir Goldstein Signed-off-by: Andrii Nakryiko --- fs/file_table.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/file_table.c b/fs/file_table.c index ca7843dde56d..257691d358ee 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -68,7 +68,7 @@ static inline void file_free(struct file *f) put_cred(f->f_cred); if (unlikely(f->f_mode & FMODE_BACKING)) { path_put(backing_file_user_path(f)); - kfree(backing_file(f)); + kfree_rcu_mightsleep(backing_file(f)); } else { kmem_cache_free(filp_cachep, f); } From patchwork Tue Oct 1 22:52:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E718CF318A for ; Tue, 1 Oct 2024 22:52:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB5B868002A; Tue, 1 Oct 2024 18:52:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B63B36B0130; Tue, 1 Oct 2024 18:52:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B91D68002A; Tue, 1 Oct 2024 18:52:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 776AE6B012D for ; Tue, 1 Oct 2024 18:52:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 35E2116120F for ; Tue, 1 Oct 2024 22:52:31 +0000 (UTC) X-FDA: 82626534102.06.13BD777 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf11.hostedemail.com (Postfix) with ESMTP id 94E2E40014 for ; Tue, 1 Oct 2024 22:52:29 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="GsSJ/T3H"; spf=pass (imf11.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ILAT7gwFiJFrt7yxTMqqez2zhgflG77f62jYP3UDfPc=; b=1R+mlzwZ8+1rNe9LhO1YzI/TMODILug+hnystrIkCjMrdhYLJtb3EeCSg09FY4W+/j6AZz l7GEwSY7kWojr92ZoA3Zmh+4yoghrr8AfGiuye+HZe/+3FyTWAUJd/5pelc9pbBV3WvoEc 2V8pmZZV6WXQaGSqqi4FBv6QZF5X2yg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="GsSJ/T3H"; spf=pass (imf11.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823110; a=rsa-sha256; cv=none; b=Iqwum/3AP0cxY9wUgCfSgwrp+D6zrMnCcpfmk55StF1T9CsnfJDuYEzn9CPFODDu3ZmUIe 11aABtM3P6BY19UktF0SesugpHFbYA2eVCFtZrN1JB06IprNDzDYZImb1owHIyXXi1U2CN CKVnOpdjiGM//w+ViaEPOpTAhaop2GY= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C20F15C5543; Tue, 1 Oct 2024 22:52:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7EBE9C4CECF; Tue, 1 Oct 2024 22:52:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823148; bh=bBBk1ELWuu6s2YRaDtSMqIlSwNWabgjEjVeq3ZwLyBA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GsSJ/T3HGEBenk4eNetFCKTWXI2642nBJifJ82B76LRpDRtOh+ZzBzGw2Gb/PimJy bUZ2Lfc9akr6eHVKSP5YmLBmyqxkQX+pIAh5EfBsiEs5u39K+IlG9YW7tdVlkyBWL6 AyfUg43cU/BbCV1VC5JOMxMmjh/EZ0IxlkzCkC1hoTt0RZZDKQ81ohEp5PbbC6F8n0 +uxYFbtYUGU8lxV0rqy6wzypxN7UiNcOC2tiCjhKpiCxpUYB2B6LXIe1rR9vB7tc5C 4hQtfKOCD2wiUeMIDzUZwD9u9dkClabmwxiNtcMhBkfvmv9wsO+fRK1+AXpUo5AJlv cwmrHDrk2xpjg== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 4/5] uprobes: simplify find_active_uprobe_rcu() VMA checks Date: Tue, 1 Oct 2024 15:52:06 -0700 Message-ID: <20241001225207.2215639-5-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hrppakr19bnb98m9x1zmu9s47arr46fm X-Rspamd-Queue-Id: 94E2E40014 X-Rspamd-Server: rspam11 X-HE-Tag: 1727823149-847751 X-HE-Meta: U2FsdGVkX184ZVSrLb6LdLqteu4cwH1S/wW89rX9BOTPv05axo4RYDagD1aWq4NmgoLfeaMPDI3G40SwANz5fnA/nF2dmw4MlmVxhWJ1jZfhRhS3tdFri5OdbAH2t+z4GTBqFkjmXxledqVgtDPR/Jy2YeqERcRh/zg9zVKHRD/FlmcLge6rFa84SadiPD1xzmaDueSbEkAqFEAiv1e/j90M0uQz4yc7HF38YI+Y77TROX09Dy9/YoZ/w76RCiPtRR7ESWCrXP4eBQ1zBuinPWQ4EOLrvbQVnxnld34EGu/0eOkfI7fZHQFkqL7VgkoHzYWicjAQe3SyjFozsKWh5VHaiFsrdy5dYBuM6OoZFvp7IUt44/ghTIo136hlJ8H4OhKVlhiRl/L48aqFFKP+Sxh0GdUYAsv3gEPG4IaCVSpmlzOG5HPjf13T5SYk5Wt3jRvwRsmL19wwObbig6dhhdV/f6MRcoO+G7+xUR+Lt1qnxjqnrvtnkQcWSCqSMKUqlBA3U3b64bai3zkY5uVF4N6A/FI1H4+8OFiAD2zGEf1Ak85qFB/taZIkoMD8pOrUlJqw0MJRzHxbsnBTw2/paEeGlpcaEMEI9ZevL7J7qTN0fS7j0o6n5kwvwHBdXJHdF1kjNarbvufUvmf2QYovRSUocxk0WaOOaPd3yptSj2VBrdr3JGtgflSzz1TqjFITN/Sp1T6XAEkZ+QGOSlflQdoc/cbJYeC0H1tdnCnvHOSbLfaizJgSDJZnK6OmWvJb4OiGLlhWgPsHNbjmUa+1ZrwRuic2k+gg9+VorsdwChaN8PJk9gWcx1l0fNc1AdcWk7TkrHhV+wxt6/0bcgI73T/zVJQ+Y/lWjbZgMn86tvelq/PJI5JZp1uSJA0uRZJ1DDdHFg84NF/Q+0v+AOrrW9wJu2JaitbhssAr7IERGXsKuy0eVUShD87xILeCosZPIzWfddg/XFynupJjYhy Sl3IdMkP bwPG+sP5gYKjlOgc9YUDpQn/luHlTsTsCHmq0jnC9iLXcJALxLgibj3d5eMfCqPkUHksKr3psaOxnU1443Bm496lxbYvWOHiDAujbLcReUSR1hG2CHrYdGAMIz9sWQEf9HhfHbJvIZpR17b2LN4WgBrPC9oIUObzj5gG8zIVTJDK9Z+RAE8z3nOVW1LkC9vDc53yoPlbP2qx9UAQcoV33bjimxFakhCKsEqPnACZO6EYF9qlQlZWFdrDesMFo+/L3Px6UxD/0qv4rr7lhI1cfUbCrCgk7lopudO1dkXWxQH6ZJxcRB7gg+27jfg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At the point where find_active_uprobe_rcu() is used we know that VMA in question has triggered software breakpoint, so we don't need to validate vma->vm_flags. Keep only vma->vm_file NULL check. Suggested-by: Oleg Nesterov Signed-off-by: Andrii Nakryiko Acked-by: Oleg Nesterov --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index a2e6a57f79f2..7bd9111b4e8b 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2091,7 +2091,7 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swb mmap_read_lock(mm); vma = vma_lookup(mm, bp_vaddr); if (vma) { - if (valid_vma(vma, false)) { + if (vma->vm_file) { struct inode *inode = file_inode(vma->vm_file); loff_t offset = vaddr_to_offset(vma, bp_vaddr); From patchwork Tue Oct 1 22:52:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A33DCCF318A for ; Tue, 1 Oct 2024 22:52:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 379516B0139; Tue, 1 Oct 2024 18:52:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3269268002B; Tue, 1 Oct 2024 18:52:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19F5A6B013E; Tue, 1 Oct 2024 18:52:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F13926B0139 for ; Tue, 1 Oct 2024 18:52:34 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A19281A0611 for ; Tue, 1 Oct 2024 22:52:34 +0000 (UTC) X-FDA: 82626534228.14.B4A6A8B Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf29.hostedemail.com (Postfix) with ESMTP id EF883120019 for ; Tue, 1 Oct 2024 22:52:32 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=PFEgw2F6; spf=pass (imf29.hostedemail.com: domain of andrii@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zh6UGPAIpYil8r9c8znFv5+mMykPnj6qvuUJn221mNU=; b=1ENaMerGKQQOjY8Io8yYe3/PzTLpPDNthhkpjOrPhOBYm9RRzEaYNnxa4tM32Me5kTOv1G t3Jhb7/jtPsXuqw3hVPV1wY85Uv88SJBKfr2o1zezulvGeiwefrthEfe9L+R372Twht1nm 5dzj3SVM2Ox89RHI7WEu5LOoJPs52x4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823026; a=rsa-sha256; cv=none; b=Yosw/B31XUmChYkSVun4bFV4lMYXNLYm732KRZ4/0a/bQh9EljRdyedLm0i8ngr0JvcGjz S67qnaRrVqQurmp7dpGhZD4WV6O6HWM309M3Go9EvxFCDy5GnxG4Ur2oo02BTF/NJ35GTY EhgNOcSO/wj6YPZdv9rZjIaEvBUKX5M= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=PFEgw2F6; spf=pass (imf29.hostedemail.com: domain of andrii@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 17B16A43316; Tue, 1 Oct 2024 22:52:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B8137C4CEC6; Tue, 1 Oct 2024 22:52:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823151; bh=SCu4Evjr8HmMRl0UxC9lAbHfcHKC1A3FSblyNAiFGvA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PFEgw2F6J4Xe2BwH8mam8pcz1GB5lffSYNDNPEEEt3HlEnC0Qbkq2oTQyDSIMLiWj DSmiP6dU+c8qyjU8OVM6piYG2rCtLmkETgbiTqvCoFJoNkzPfCM9WUr13Wc4fOGmxx pVljwJGWFCLV4CGYelIjB8GVPXZmSGOs0L65Ytw5qAco2UcKI34w3q0XS2tH7fr0BP boAMJLHgBYFItKgyeclKrBcBkdmEVmXM5zZO8b/wiZIPKMa9G4TIoh0qWgO9cmjxRb hxd0LFCdgGurtO0LPY24vdhvphyP3bAIMTVbaeO7gHFtPiSKqz2ap0TzSaXsxaE8C5 OGOaFD3hzJrrQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution Date: Tue, 1 Oct 2024 15:52:07 -0700 Message-ID: <20241001225207.2215639-6-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EF883120019 X-Stat-Signature: k5cuiaeicmibzkjfoqwanr4o34kokyo3 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1727823152-564326 X-HE-Meta: U2FsdGVkX1+LLt9sqmFCW+qWj43nEsvRaiohJ8OUsFaKukaX2CyrkM7OJnsnBjG1OolnLQ8nQnU2SrSS2tEP8HPsJuPvoubWWXlSWNqK9f/nW9/cobwSBasiqupSdhcwgTtSBL7woRjSnZ6r/si/hbUiW1GcjQs4sja7UgrSJQKYuuIdS1pRm8SheffbBSvBdxYYxQlzqZjs21t5nAfnG0s6Sva5SjsAVj6Kznl/Zg10gR7uWMPff0lVyXgNePpHSJSR32sAqCyRbGO6KrIJA+uMoPqH1Z8plcP63+39F/cdjRkm2I2iM/U0cZwRMAcFg6W8KJFR0sdZsRbRfKtvAxYUWo3uS396uiQFac5OlRyKVMovNuQVNzbzPdTyPnjwAXvac/ukEsnsOYRWajKSOJgfvtklOhYtocKx3Aabjj96J74bXWdSPxF3p9P86rEzjwgKKtpxkQuHTcJk2jMANEF8nRMC5/MBMeaoFF9wSMxQkYbh569JkbYOPFJ2WySxAEPrPcHZi92VReg1WfhfC8ejXhq89rMwuAoFMo8SCOaBmF2hoCkmsFFKsaUa5Yfb0X5u7EtI/BAcYdbG4u/rKVV7FaCuwdaijP+CgdrUh0u+t9XvPL06LKM81era5sctNHAmgbgH1MIInl1v1b1/iZf46PVrxRH2c/RYobqg1vG1xez8jGX31b3Ag+WRjETK8LF2mZ1ZXVYGxBk2ntW54Ge4msr8PCg//IMU91wtJxL+cdMWSklG6mzOKOHJn8tbyEHyiYzC8egRGcmqHoF527heTHCBRNEajPFEgd+71WwCA76X0vGguoaRymJc/as16GDw6cHfBH3KOL9nR7tAtG4wg/pJwsgIqLmz3lxUTFh0wwihR7AhKs9evXTg7lo2ifmnJeZHPwWjaYDQfSnPC0oMo+gMf/qnurtDRwq0lgjgZ3DE88ps1U34VA9Yh3anz85mYDs6bsL6CqtBhpd LU8O+HbR IQ6q8x7IQ4u/qxv8R2CO64LIfwa54XFJfZoiLGJ33fhZvYwxTjkUVpFHr0aL/qjNjv7WTuMZJ/Kv71/fjbRBFoupRXlaGMfVvhuB5402qq34+H1qaxWzXT7lFvqvAzeYzpeuOd5rRU7WRl/oVpCV1BJ8nXMxWZpnZ7QXe28DkBfMoQG4/XHTsXt2g5FrlFfM2w65+TzmzMwuFAKCqXlmrN9I/8ks/zipin7SHfpIFAY6hhqwWQGD1nE+lxKCPtMEUcKWD5rTKSjXx53DghVjPDnpz9Ml9yNfqIAX21SqAep62G8p/aECwzhdllZn/vEJvE/7VWrL8oDjSyFA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Given filp_cachep is marked SLAB_TYPESAFE_BY_RCU (and FMODE_BACKING files, a special case, now goes through RCU-delated freeing), we can safely access vma->vm_file->f_inode field locklessly under just rcu_read_lock() protection, which enables looking up uprobe from uprobes_tree completely locklessly and speculatively without the need to acquire mmap_lock for reads. In most cases, anyway, assuming that there are no parallel mm and/or VMA modifications. The underlying struct file's memory won't go away from under us (even if struct file can be reused in the meantime). We rely on newly added mmap_lock_speculation_{start,end}() helpers to validate that mm_struct stays intact for entire duration of this speculation. If not, we fall back to mmap_lock-protected lookup. The speculative logic is written in such a way that it will safely handle any garbage values that might be read from vma or file structs. Benchmarking results speak for themselves. BEFORE (latest tip/perf/core) ============================= uprobe-nop ( 1 cpus): 3.384 ± 0.004M/s ( 3.384M/s/cpu) uprobe-nop ( 2 cpus): 5.456 ± 0.005M/s ( 2.728M/s/cpu) uprobe-nop ( 3 cpus): 7.863 ± 0.015M/s ( 2.621M/s/cpu) uprobe-nop ( 4 cpus): 9.442 ± 0.008M/s ( 2.360M/s/cpu) uprobe-nop ( 5 cpus): 11.036 ± 0.013M/s ( 2.207M/s/cpu) uprobe-nop ( 6 cpus): 10.884 ± 0.019M/s ( 1.814M/s/cpu) uprobe-nop ( 7 cpus): 7.897 ± 0.145M/s ( 1.128M/s/cpu) uprobe-nop ( 8 cpus): 10.021 ± 0.128M/s ( 1.253M/s/cpu) uprobe-nop (10 cpus): 9.932 ± 0.170M/s ( 0.993M/s/cpu) uprobe-nop (12 cpus): 8.369 ± 0.056M/s ( 0.697M/s/cpu) uprobe-nop (14 cpus): 8.678 ± 0.017M/s ( 0.620M/s/cpu) uprobe-nop (16 cpus): 7.392 ± 0.003M/s ( 0.462M/s/cpu) uprobe-nop (24 cpus): 5.326 ± 0.178M/s ( 0.222M/s/cpu) uprobe-nop (32 cpus): 5.426 ± 0.059M/s ( 0.170M/s/cpu) uprobe-nop (40 cpus): 5.262 ± 0.070M/s ( 0.132M/s/cpu) uprobe-nop (48 cpus): 6.121 ± 0.010M/s ( 0.128M/s/cpu) uprobe-nop (56 cpus): 6.252 ± 0.035M/s ( 0.112M/s/cpu) uprobe-nop (64 cpus): 7.644 ± 0.023M/s ( 0.119M/s/cpu) uprobe-nop (72 cpus): 7.781 ± 0.001M/s ( 0.108M/s/cpu) uprobe-nop (80 cpus): 8.992 ± 0.048M/s ( 0.112M/s/cpu) AFTER ===== uprobe-nop ( 1 cpus): 3.534 ± 0.033M/s ( 3.534M/s/cpu) uprobe-nop ( 2 cpus): 6.701 ± 0.007M/s ( 3.351M/s/cpu) uprobe-nop ( 3 cpus): 10.031 ± 0.007M/s ( 3.344M/s/cpu) uprobe-nop ( 4 cpus): 13.003 ± 0.012M/s ( 3.251M/s/cpu) uprobe-nop ( 5 cpus): 16.274 ± 0.006M/s ( 3.255M/s/cpu) uprobe-nop ( 6 cpus): 19.563 ± 0.024M/s ( 3.261M/s/cpu) uprobe-nop ( 7 cpus): 22.696 ± 0.054M/s ( 3.242M/s/cpu) uprobe-nop ( 8 cpus): 24.534 ± 0.010M/s ( 3.067M/s/cpu) uprobe-nop (10 cpus): 30.475 ± 0.117M/s ( 3.047M/s/cpu) uprobe-nop (12 cpus): 33.371 ± 0.017M/s ( 2.781M/s/cpu) uprobe-nop (14 cpus): 38.864 ± 0.004M/s ( 2.776M/s/cpu) uprobe-nop (16 cpus): 41.476 ± 0.020M/s ( 2.592M/s/cpu) uprobe-nop (24 cpus): 64.696 ± 0.021M/s ( 2.696M/s/cpu) uprobe-nop (32 cpus): 85.054 ± 0.027M/s ( 2.658M/s/cpu) uprobe-nop (40 cpus): 101.979 ± 0.032M/s ( 2.549M/s/cpu) uprobe-nop (48 cpus): 110.518 ± 0.056M/s ( 2.302M/s/cpu) uprobe-nop (56 cpus): 117.737 ± 0.020M/s ( 2.102M/s/cpu) uprobe-nop (64 cpus): 124.613 ± 0.079M/s ( 1.947M/s/cpu) uprobe-nop (72 cpus): 133.239 ± 0.032M/s ( 1.851M/s/cpu) uprobe-nop (80 cpus): 142.037 ± 0.138M/s ( 1.775M/s/cpu) Previously total throughput was maxing out at 11mln/s, and gradually declining past 8 cores. With this change, it now keeps growing with each added CPU, reaching 142mln/s at 80 CPUs (this was measured on a 80-core Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz). Note, results above assume that commit [0] from linux-trace tree is applied as well, otherwise it will limit scalability to about 10mln/s total throughput. [0] 10cdb82aa77f ("uprobes: turn trace_uprobe's nhit counter to be per-CPU one") Suggested-by: Matthew Wilcox Signed-off-by: Andrii Nakryiko --- kernel/events/uprobes.c | 44 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 7bd9111b4e8b..960130275656 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2081,6 +2081,46 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) return is_trap_insn(&opcode); } +static struct uprobe *find_active_uprobe_speculative(unsigned long bp_vaddr) +{ + struct mm_struct *mm = current->mm; + struct uprobe *uprobe = NULL; + struct vm_area_struct *vma; + struct file *vm_file; + loff_t offset; + long seq; + + guard(rcu)(); + + if (!mmap_lock_speculation_start(mm, &seq)) + return NULL; + + vma = vma_lookup(mm, bp_vaddr); + if (!vma) + return NULL; + + /* vm_file memory can be reused for another instance of struct file, + * but can't be freed from under us, so it's safe to read fields from + * it, even if the values are some garbage values; ultimately + * find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure + * that whatever we speculatively found is correct + */ + vm_file = READ_ONCE(vma->vm_file); + if (!vm_file) + return NULL; + + offset = (loff_t)(vma->vm_pgoff << PAGE_SHIFT) + (bp_vaddr - vma->vm_start); + uprobe = find_uprobe_rcu(vm_file->f_inode, offset); + if (!uprobe) + return NULL; + + /* now double check that nothing about MM changed */ + if (!mmap_lock_speculation_end(mm, seq)) + return NULL; + + return uprobe; +} + /* assumes being inside RCU protected region */ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swbp) { @@ -2088,6 +2128,10 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swb struct uprobe *uprobe = NULL; struct vm_area_struct *vma; + uprobe = find_active_uprobe_speculative(bp_vaddr); + if (uprobe) + return uprobe; + mmap_read_lock(mm); vma = vma_lookup(mm, bp_vaddr); if (vma) {