From patchwork Tue Oct 1 22:52:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818824 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66EFB1CEEA7; Tue, 1 Oct 2024 22:52:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823139; cv=none; b=SBr1IF2vn8JA46ctX3RFnw+/Vrk6D3rh16pcEjPxbg59LgnjBFU7yXidaKSGxUQrPxdNZh4fnbQFnBlo/w/9ZJLcB5NZpTAvjkwio1J2iPu8M6n7JVHoSCel/qEKtHHMfgaqxP7+7rWFqdmCq8V6FB3uvkt+Zmj6I+2jJQYvNug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823139; c=relaxed/simple; bh=5eizI5x+ShSEQgV2sFgQYF+jiwqy/RpZnCYPy8CWlCY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MxVhcg84QHBQisXaY/nfF+rXUU6+cqouoBCeECilKqL2yvbpX1+OHiNsvJ2MaFKFAQLX5YHP9S6HzJPFRtrOuKOzt6iwGEtG67R9+/8nnvoyP0EMVQKzTbuMLCa/k0XXIi781y/jvZUQJRZ30Wpc56dlrTds+gss/bSdUQ4GD4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dfDuVI4A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dfDuVI4A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C73A1C4CEC6; Tue, 1 Oct 2024 22:52:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823139; bh=5eizI5x+ShSEQgV2sFgQYF+jiwqy/RpZnCYPy8CWlCY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dfDuVI4A1C9B6XefEXXmRXfyt7/s7dX48eXgg1McB+htkj6q+gilGOE4d61u22XCh UfxULXP1HekgDrUTsX2+Ln5wBjCakxnoD/gFrptDXm5zUyoSqPWUf30SbjqdXEbzQX 1Y7WBvvnXnURdWM7kDKwCuD1ZyLjVha0EIEcEHAPgeIvQbAIYIl3uMOjNn4Psmap5M QFFj8DAP9dAq9Mj7rOjscOJ9UFzHxizXyyksvbX1ftosYFrQe6GyfemxxqilV6KEBX bQr4Hj1fllwJKPcEpXl2xOQYQECHviW+3pSvnf592Nq2IwBGBWXnrjNhPa2grRrgbA zgC9Q7nB+8MNQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 1/5] mm: introduce mmap_lock_speculation_{start|end} Date: Tue, 1 Oct 2024 15:52:03 -0700 Message-ID: <20241001225207.2215639-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Suren Baghdasaryan Add helper functions to speculatively perform operations without read-locking mmap_lock, expecting that mmap_lock will not be write-locked and mm is not modified from under us. Suggested-by: Peter Zijlstra Signed-off-by: Suren Baghdasaryan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20240912210222.186542-1-surenb@google.com --- include/linux/mm_types.h | 3 ++ include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/fork.c | 3 -- 3 files changed, 63 insertions(+), 15 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 485424979254..d5e3f907eea4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -876,6 +876,9 @@ struct mm_struct { * Roughly speaking, incrementing the sequence number is * equivalent to releasing locks on VMAs; reading the sequence * number can be part of taking a read lock on a VMA. + * Incremented every time mmap_lock is write-locked/unlocked. + * Initialized to 0, therefore odd values indicate mmap_lock + * is write-locked and even values that it's released. * * Can be modified under write mmap_lock using RELEASE * semantics. diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index de9dc20b01ba..9d23635bc701 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -71,39 +71,84 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) } #ifdef CONFIG_PER_VMA_LOCK +static inline void init_mm_lock_seq(struct mm_struct *mm) +{ + mm->mm_lock_seq = 0; +} + /* - * Drop all currently-held per-VMA locks. - * This is called from the mmap_lock implementation directly before releasing - * a write-locked mmap_lock (or downgrading it to read-locked). - * This should normally NOT be called manually from other places. - * If you want to call this manually anyway, keep in mind that this will release - * *all* VMA write locks, including ones from further up the stack. + * Increment mm->mm_lock_seq when mmap_lock is write-locked (ACQUIRE semantics) + * or write-unlocked (RELEASE semantics). */ -static inline void vma_end_write_all(struct mm_struct *mm) +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) { mmap_assert_write_locked(mm); /* * Nobody can concurrently modify mm->mm_lock_seq due to exclusive * mmap_lock being held. - * We need RELEASE semantics here to ensure that preceding stores into - * the VMA take effect before we unlock it with this store. - * Pairs with ACQUIRE semantics in vma_start_read(). */ - smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + + if (acquire) { + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); + /* + * For ACQUIRE semantics we should ensure no following stores are + * reordered to appear before the mm->mm_lock_seq modification. + */ + smp_wmb(); + } else { + /* + * We need RELEASE semantics here to ensure that preceding stores + * into the VMA take effect before we unlock it with this store. + * Pairs with ACQUIRE semantics in vma_start_read(). + */ + smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + } +} + +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) +{ + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ + *seq = smp_load_acquire(&mm->mm_lock_seq); + /* Allow speculation if mmap_lock is not write-locked */ + return (*seq & 1) == 0; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) +{ + /* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */ + smp_rmb(); + return seq == READ_ONCE(mm->mm_lock_seq); } + #else -static inline void vma_end_write_all(struct mm_struct *mm) {} +static inline void init_mm_lock_seq(struct mm_struct *mm) {} +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) {} +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) { return false; } +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) { return false; } #endif +/* + * Drop all currently-held per-VMA locks. + * This is called from the mmap_lock implementation directly before releasing + * a write-locked mmap_lock (or downgrading it to read-locked). + * This should NOT be called manually from other places. + */ +static inline void vma_end_write_all(struct mm_struct *mm) +{ + inc_mm_lock_seq(mm, false); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); + init_mm_lock_seq(mm); } static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -111,6 +156,7 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -120,6 +166,8 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } diff --git a/kernel/fork.c b/kernel/fork.c index 18bdc87209d0..c44b71d354ee 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1259,9 +1259,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); -#ifdef CONFIG_PER_VMA_LOCK - mm->mm_lock_seq = 0; -#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; From patchwork Tue Oct 1 22:52:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818825 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0E721CF5D1; Tue, 1 Oct 2024 22:52:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823142; cv=none; b=J8Tx2s8vzTw6+FV37fePMlf/KJBlOs5dN43d0qRUjVR4s3elTexWVoFiXwAnRIE0y/VhayJlNa4ytl9hOjDtiLDXW0a6Gd2GETdq5u2rfDThhlTBVC0a8u/EurPDC6ksAuNK6ekmZLxgup10fJx6vRrb1rpA+8b1B+jE1EmZqqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823142; c=relaxed/simple; bh=Q4lsQ3VDioDbBQhpNH028nIzfuY7M4lO4nHV2LRxuL4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Cic6QzfYFVXNlUWfA+29s3cMa6KcxS0SerYvzOL8rXZyuctL8jJY3HgUOerXFERmTNqEyhGc6Qeri4MzRcxlwHKtZfq6r6QXb7kj9YYOHFAOVnnzG6Nv4bCtxKKwbvHwB1b9q/gp04TB5lJZw1vTMUvS2xf409AxUhlkgC98g2c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=heHmzfSm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="heHmzfSm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09495C4CECD; Tue, 1 Oct 2024 22:52:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823142; bh=Q4lsQ3VDioDbBQhpNH028nIzfuY7M4lO4nHV2LRxuL4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=heHmzfSm2yzI1iI9a6pWLMysWPAc4UEw5I9P9k+LCuudPU3BDHrwWOqPAvRSOEqTP Ld0jpPXOtCuEWw1kQnr1yJxZdwUK1sroXcx6QboO6bkoM27/YJj51x4ArDOxWR7e+N 6TroV0+9NlEv+Gi6hpXaHDl9AgN4jyT3NE6il/bVevt/+NknSG3iX/M2eRxc8xAc4x DmsnJVCgep1LiyQEN3aTVQ3r3ZIG1B2qky5sxKLak2lGcIuCEBKb7s+4J8x771Ipfs Y7l+2n+hzxF9xCO4R2/LEb/EF/kjGm7gDDPXvnpTHGN5iCRHxN/cwUM+nhZ1rDPCOl 4im8A0phpKNdQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 2/5] mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures Date: Tue, 1 Oct 2024 15:52:04 -0700 Message-ID: <20241001225207.2215639-3-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To increase mm->mm_lock_seq robustness, switch it from int to long, so that it's a 64-bit counter on 64-bit systems and we can stop worrying about it wrapping around in just ~4 billion iterations. Same goes for VMA's matching vm_lock_seq, which is derived from mm_lock_seq. I didn't use __u64 outright to keep 32-bit architectures unaffected, but if it seems important enough, I have nothing against using __u64. Suggested-by: Jann Horn Signed-off-by: Andrii Nakryiko --- include/linux/mm.h | 6 +++--- include/linux/mm_types.h | 4 ++-- include/linux/mmap_lock.h | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6549d0979b28..f8e75d0642a8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -716,7 +716,7 @@ static inline void vma_end_read(struct vm_area_struct *vma) } /* WARNING! Can only be used if mmap_lock is expected to be write-locked */ -static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) +static bool __is_vma_write_locked(struct vm_area_struct *vma, long *mm_lock_seq) { mmap_assert_write_locked(vma->vm_mm); @@ -735,7 +735,7 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) */ static inline void vma_start_write(struct vm_area_struct *vma) { - int mm_lock_seq; + long mm_lock_seq; if (__is_vma_write_locked(vma, &mm_lock_seq)) return; @@ -753,7 +753,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) static inline void vma_assert_write_locked(struct vm_area_struct *vma) { - int mm_lock_seq; + long mm_lock_seq; VM_BUG_ON_VMA(!__is_vma_write_locked(vma, &mm_lock_seq), vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d5e3f907eea4..c045543f43d9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -705,7 +705,7 @@ struct vm_area_struct { * counter reuse can only lead to occasional unnecessary use of the * slowpath. */ - int vm_lock_seq; + long vm_lock_seq; struct vma_lock *vm_lock; #endif @@ -887,7 +887,7 @@ struct mm_struct { * Can be read with ACQUIRE semantics if not holding write * mmap_lock. */ - int mm_lock_seq; + long mm_lock_seq; #endif diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 9d23635bc701..fca527dece63 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -105,7 +105,7 @@ static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) } } -static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, long *seq) { /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ *seq = smp_load_acquire(&mm->mm_lock_seq); @@ -113,7 +113,7 @@ static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) return (*seq & 1) == 0; } -static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, long seq) { /* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */ smp_rmb(); From patchwork Tue Oct 1 22:52:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818826 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9902D1CF5FE; Tue, 1 Oct 2024 22:52:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823145; cv=none; b=T15Jjza1kzE2iIhW1rgFyJhDsNxgWvoRaH2C1hd4wbNWvT5qu/Kh9IG+ew4uQYn5JP7+eEZaNUwYpHyntlcoGgaFYPIEdUIQTuaLrFxWx/17VtVGNCHBMhfag0r0Mek4URR0Fd7t/feyzKRTIY76p3+Rx8as29yizKQp5nsr+Hw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823145; c=relaxed/simple; bh=Fe0W0B2s6ceCb71LS5xDyOWx4mAiBCLoZvDtud5NppM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OTYqxHOMkGj5C5o4JIvvWEpHaSDr6M4L7CQu0xoMXLf1rjJW2kMd0iAfNxY8SD5YdabnNQs+ixmQC4LcR+kYQ1hvOizke9zQ1ry4l8KOmD7OKgqYW8NKgRjdAt+flodBjZaYVp4CCxmjtIrkARBkgElLdreRuRE76uegWWU99o0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CpHdhX2M; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CpHdhX2M" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4533DC4CEC6; Tue, 1 Oct 2024 22:52:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823145; bh=Fe0W0B2s6ceCb71LS5xDyOWx4mAiBCLoZvDtud5NppM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CpHdhX2My8bEMExNjtkiCBGzgD/icJoKKBqYiIrUbPzCTfYszjoLtbMmq1dssYj3T Vrwe8b43Ff5XJWhkflBzoIkA4mbw1tW1bc+gdwJ5gHBcaUuGvSIZVtObiyY9gzB6fL 279sn7GDirJgGJM+LjoW+rAaFbP+0qgSw780RGYXJSzkxUeqFy2lbgEVIGwSB2x2EA yVmzNitaw7bjQ8Gxrp5it9ifxi0NkWH3rkKE5gjhUNhD8GF3nt6umR84JDzYdOvj6S 032wn+XUSq9IQVRDNOwPE/ysqQ+e7s8s7xRGN/F99dVabaYTJU5jAaQ7kjjhMvZGTm 9/45fjGcrlZ7g== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko , Amir Goldstein Subject: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file Date: Tue, 1 Oct 2024 15:52:05 -0700 Message-ID: <20241001225207.2215639-4-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 6cf41fcfe099 ("backing file: free directly") switched FMODE_BACKING files to direct freeing as back then there were no use cases requiring RCU protected access to such files. Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic. We use headless kfree_rcu_mightsleep() variant, as file_free() is only called for FMODE_BACKING files in might_sleep() context. Suggested-by: Suren Baghdasaryan Cc: Christian Brauner Cc: Amir Goldstein Signed-off-by: Andrii Nakryiko Reviewed-by: Christian Brauner --- fs/file_table.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/file_table.c b/fs/file_table.c index ca7843dde56d..257691d358ee 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -68,7 +68,7 @@ static inline void file_free(struct file *f) put_cred(f->f_cred); if (unlikely(f->f_mode & FMODE_BACKING)) { path_put(backing_file_user_path(f)); - kfree(backing_file(f)); + kfree_rcu_mightsleep(backing_file(f)); } else { kmem_cache_free(filp_cachep, f); } From patchwork Tue Oct 1 22:52:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818827 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BF861CF7CB; Tue, 1 Oct 2024 22:52:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823149; cv=none; b=ioBhYR/DFEf85+HaaHO79QfAu322RN4ntjjxNFL3BxQ2+sdiGXBsZv6d8w1IxjT87RGpA15O8Wy8IFhYwmOWyHFqktz0K0GpEG2rwjPIWV9zPPvAYvczLbXXcxFkJ94nSDghpX2N10gtDwixcPFdSMRivXQ5Hxn6s7SbVTaJI74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823149; c=relaxed/simple; bh=bBBk1ELWuu6s2YRaDtSMqIlSwNWabgjEjVeq3ZwLyBA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gsCfGf6fn9GJt3y2zIKTA02ESEmzhX5XblxVDAwVcBb1QHfMjIdPsR93t2bDKIKp4g/1/9HF+nmzSgSDo9lZS8kuGPPZW5y3sRhWfw46Bv5tJxYcKHaf++AQR3H8Mpn5v7GR6GRgAy2Tf57XG8saDvjRh06romb7ezF2B5bpxYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GsSJ/T3H; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GsSJ/T3H" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7EBE9C4CECF; Tue, 1 Oct 2024 22:52:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823148; bh=bBBk1ELWuu6s2YRaDtSMqIlSwNWabgjEjVeq3ZwLyBA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GsSJ/T3HGEBenk4eNetFCKTWXI2642nBJifJ82B76LRpDRtOh+ZzBzGw2Gb/PimJy bUZ2Lfc9akr6eHVKSP5YmLBmyqxkQX+pIAh5EfBsiEs5u39K+IlG9YW7tdVlkyBWL6 AyfUg43cU/BbCV1VC5JOMxMmjh/EZ0IxlkzCkC1hoTt0RZZDKQ81ohEp5PbbC6F8n0 +uxYFbtYUGU8lxV0rqy6wzypxN7UiNcOC2tiCjhKpiCxpUYB2B6LXIe1rR9vB7tc5C 4hQtfKOCD2wiUeMIDzUZwD9u9dkClabmwxiNtcMhBkfvmv9wsO+fRK1+AXpUo5AJlv cwmrHDrk2xpjg== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 4/5] uprobes: simplify find_active_uprobe_rcu() VMA checks Date: Tue, 1 Oct 2024 15:52:06 -0700 Message-ID: <20241001225207.2215639-5-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 At the point where find_active_uprobe_rcu() is used we know that VMA in question has triggered software breakpoint, so we don't need to validate vma->vm_flags. Keep only vma->vm_file NULL check. Suggested-by: Oleg Nesterov Signed-off-by: Andrii Nakryiko Acked-by: Oleg Nesterov --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index a2e6a57f79f2..7bd9111b4e8b 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2091,7 +2091,7 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swb mmap_read_lock(mm); vma = vma_lookup(mm, bp_vaddr); if (vma) { - if (valid_vma(vma, false)) { + if (vma->vm_file) { struct inode *inode = file_inode(vma->vm_file); loff_t offset = vaddr_to_offset(vma, bp_vaddr); From patchwork Tue Oct 1 22:52:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13818828 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70B231CF29F; Tue, 1 Oct 2024 22:52:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823153; cv=none; b=my8ACXey4gZt2QdCdd2iHaenHNk3nYllLlkR+ucQs9YCnwuUpjaNQbvsynN1sOnJ92+52NZKhHzNGdMKnMU3APFezLrqE8o8dME21Hpy4iVHrnAlPEd/cqSbx154F0ysAcAm6Sd5K/9pWjWU1d21D4MoszQ5aNxyzrq+LaDLZ9s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727823153; c=relaxed/simple; bh=SCu4Evjr8HmMRl0UxC9lAbHfcHKC1A3FSblyNAiFGvA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YwJ8nqxyjwD3CcBTzS5sLpMMmQTO/U/CMeTtZGtt6DjDfOagQ7TTVymogKnMS7bPhmiid0/B1hhsQrQwwhw9xzqOJY6YwZZJcP1EH5ey/3KnCrJom3dtdFY4/hKoScTTrSJUeldJXwoH8jb2ctAe2eLN2/UV6DYKWmm/twP4m0U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PFEgw2F6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PFEgw2F6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B8137C4CEC6; Tue, 1 Oct 2024 22:52:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823151; bh=SCu4Evjr8HmMRl0UxC9lAbHfcHKC1A3FSblyNAiFGvA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PFEgw2F6J4Xe2BwH8mam8pcz1GB5lffSYNDNPEEEt3HlEnC0Qbkq2oTQyDSIMLiWj DSmiP6dU+c8qyjU8OVM6piYG2rCtLmkETgbiTqvCoFJoNkzPfCM9WUr13Wc4fOGmxx pVljwJGWFCLV4CGYelIjB8GVPXZmSGOs0L65Ytw5qAco2UcKI34w3q0XS2tH7fr0BP boAMJLHgBYFItKgyeclKrBcBkdmEVmXM5zZO8b/wiZIPKMa9G4TIoh0qWgO9cmjxRb hxd0LFCdgGurtO0LPY24vdhvphyP3bAIMTVbaeO7gHFtPiSKqz2ap0TzSaXsxaE8C5 OGOaFD3hzJrrQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, mingo@kernel.org, Andrii Nakryiko Subject: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution Date: Tue, 1 Oct 2024 15:52:07 -0700 Message-ID: <20241001225207.2215639-6-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241001225207.2215639-1-andrii@kernel.org> References: <20241001225207.2215639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Given filp_cachep is marked SLAB_TYPESAFE_BY_RCU (and FMODE_BACKING files, a special case, now goes through RCU-delated freeing), we can safely access vma->vm_file->f_inode field locklessly under just rcu_read_lock() protection, which enables looking up uprobe from uprobes_tree completely locklessly and speculatively without the need to acquire mmap_lock for reads. In most cases, anyway, assuming that there are no parallel mm and/or VMA modifications. The underlying struct file's memory won't go away from under us (even if struct file can be reused in the meantime). We rely on newly added mmap_lock_speculation_{start,end}() helpers to validate that mm_struct stays intact for entire duration of this speculation. If not, we fall back to mmap_lock-protected lookup. The speculative logic is written in such a way that it will safely handle any garbage values that might be read from vma or file structs. Benchmarking results speak for themselves. BEFORE (latest tip/perf/core) ============================= uprobe-nop ( 1 cpus): 3.384 ± 0.004M/s ( 3.384M/s/cpu) uprobe-nop ( 2 cpus): 5.456 ± 0.005M/s ( 2.728M/s/cpu) uprobe-nop ( 3 cpus): 7.863 ± 0.015M/s ( 2.621M/s/cpu) uprobe-nop ( 4 cpus): 9.442 ± 0.008M/s ( 2.360M/s/cpu) uprobe-nop ( 5 cpus): 11.036 ± 0.013M/s ( 2.207M/s/cpu) uprobe-nop ( 6 cpus): 10.884 ± 0.019M/s ( 1.814M/s/cpu) uprobe-nop ( 7 cpus): 7.897 ± 0.145M/s ( 1.128M/s/cpu) uprobe-nop ( 8 cpus): 10.021 ± 0.128M/s ( 1.253M/s/cpu) uprobe-nop (10 cpus): 9.932 ± 0.170M/s ( 0.993M/s/cpu) uprobe-nop (12 cpus): 8.369 ± 0.056M/s ( 0.697M/s/cpu) uprobe-nop (14 cpus): 8.678 ± 0.017M/s ( 0.620M/s/cpu) uprobe-nop (16 cpus): 7.392 ± 0.003M/s ( 0.462M/s/cpu) uprobe-nop (24 cpus): 5.326 ± 0.178M/s ( 0.222M/s/cpu) uprobe-nop (32 cpus): 5.426 ± 0.059M/s ( 0.170M/s/cpu) uprobe-nop (40 cpus): 5.262 ± 0.070M/s ( 0.132M/s/cpu) uprobe-nop (48 cpus): 6.121 ± 0.010M/s ( 0.128M/s/cpu) uprobe-nop (56 cpus): 6.252 ± 0.035M/s ( 0.112M/s/cpu) uprobe-nop (64 cpus): 7.644 ± 0.023M/s ( 0.119M/s/cpu) uprobe-nop (72 cpus): 7.781 ± 0.001M/s ( 0.108M/s/cpu) uprobe-nop (80 cpus): 8.992 ± 0.048M/s ( 0.112M/s/cpu) AFTER ===== uprobe-nop ( 1 cpus): 3.534 ± 0.033M/s ( 3.534M/s/cpu) uprobe-nop ( 2 cpus): 6.701 ± 0.007M/s ( 3.351M/s/cpu) uprobe-nop ( 3 cpus): 10.031 ± 0.007M/s ( 3.344M/s/cpu) uprobe-nop ( 4 cpus): 13.003 ± 0.012M/s ( 3.251M/s/cpu) uprobe-nop ( 5 cpus): 16.274 ± 0.006M/s ( 3.255M/s/cpu) uprobe-nop ( 6 cpus): 19.563 ± 0.024M/s ( 3.261M/s/cpu) uprobe-nop ( 7 cpus): 22.696 ± 0.054M/s ( 3.242M/s/cpu) uprobe-nop ( 8 cpus): 24.534 ± 0.010M/s ( 3.067M/s/cpu) uprobe-nop (10 cpus): 30.475 ± 0.117M/s ( 3.047M/s/cpu) uprobe-nop (12 cpus): 33.371 ± 0.017M/s ( 2.781M/s/cpu) uprobe-nop (14 cpus): 38.864 ± 0.004M/s ( 2.776M/s/cpu) uprobe-nop (16 cpus): 41.476 ± 0.020M/s ( 2.592M/s/cpu) uprobe-nop (24 cpus): 64.696 ± 0.021M/s ( 2.696M/s/cpu) uprobe-nop (32 cpus): 85.054 ± 0.027M/s ( 2.658M/s/cpu) uprobe-nop (40 cpus): 101.979 ± 0.032M/s ( 2.549M/s/cpu) uprobe-nop (48 cpus): 110.518 ± 0.056M/s ( 2.302M/s/cpu) uprobe-nop (56 cpus): 117.737 ± 0.020M/s ( 2.102M/s/cpu) uprobe-nop (64 cpus): 124.613 ± 0.079M/s ( 1.947M/s/cpu) uprobe-nop (72 cpus): 133.239 ± 0.032M/s ( 1.851M/s/cpu) uprobe-nop (80 cpus): 142.037 ± 0.138M/s ( 1.775M/s/cpu) Previously total throughput was maxing out at 11mln/s, and gradually declining past 8 cores. With this change, it now keeps growing with each added CPU, reaching 142mln/s at 80 CPUs (this was measured on a 80-core Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz). Note, results above assume that commit [0] from linux-trace tree is applied as well, otherwise it will limit scalability to about 10mln/s total throughput. [0] 10cdb82aa77f ("uprobes: turn trace_uprobe's nhit counter to be per-CPU one") Suggested-by: Matthew Wilcox Signed-off-by: Andrii Nakryiko --- kernel/events/uprobes.c | 44 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 7bd9111b4e8b..960130275656 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2081,6 +2081,46 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) return is_trap_insn(&opcode); } +static struct uprobe *find_active_uprobe_speculative(unsigned long bp_vaddr) +{ + struct mm_struct *mm = current->mm; + struct uprobe *uprobe = NULL; + struct vm_area_struct *vma; + struct file *vm_file; + loff_t offset; + long seq; + + guard(rcu)(); + + if (!mmap_lock_speculation_start(mm, &seq)) + return NULL; + + vma = vma_lookup(mm, bp_vaddr); + if (!vma) + return NULL; + + /* vm_file memory can be reused for another instance of struct file, + * but can't be freed from under us, so it's safe to read fields from + * it, even if the values are some garbage values; ultimately + * find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure + * that whatever we speculatively found is correct + */ + vm_file = READ_ONCE(vma->vm_file); + if (!vm_file) + return NULL; + + offset = (loff_t)(vma->vm_pgoff << PAGE_SHIFT) + (bp_vaddr - vma->vm_start); + uprobe = find_uprobe_rcu(vm_file->f_inode, offset); + if (!uprobe) + return NULL; + + /* now double check that nothing about MM changed */ + if (!mmap_lock_speculation_end(mm, seq)) + return NULL; + + return uprobe; +} + /* assumes being inside RCU protected region */ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swbp) { @@ -2088,6 +2128,10 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swb struct uprobe *uprobe = NULL; struct vm_area_struct *vma; + uprobe = find_active_uprobe_speculative(bp_vaddr); + if (uprobe) + return uprobe; + mmap_read_lock(mm); vma = vma_lookup(mm, bp_vaddr); if (vma) {