From patchwork Thu Sep 12 21:02:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13802686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A206EEE262 for ; Thu, 12 Sep 2024 21:02:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90AF36B0088; Thu, 12 Sep 2024 17:02:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BADF6B0089; Thu, 12 Sep 2024 17:02:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A9266B008A; Thu, 12 Sep 2024 17:02:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5D0016B0088 for ; Thu, 12 Sep 2024 17:02:30 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1BDE21405A6 for ; Thu, 12 Sep 2024 21:02:30 +0000 (UTC) X-FDA: 82557309660.12.8D92CD3 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf26.hostedemail.com (Postfix) with ESMTP id 61FEC140024 for ; Thu, 12 Sep 2024 21:02:27 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="1pi/tZR+"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 34lbjZgYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34lbjZgYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726174861; a=rsa-sha256; cv=none; b=5GiV9Lix+EDp3FLWvj+YWW7ruZHBJseQXZ7ahosrpw9ZpCeSaFWuueHBhOIqGc4Bvn2cCn +u+GKH9ysIJO9mJ7itS/0ndILH+JyhWEwJz/2RLMRDvlgGvFXAhEpMyCJgLN6ahk+1bZ9v ObcvjDaqveSmdOWYEXunIfAW1IJojjo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="1pi/tZR+"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 34lbjZgYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34lbjZgYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726174861; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qhdYG9qzYkU8fhMrG0yOrq2vgwt4qH3lwD7TAy4ybww=; b=XxQ+ZVK85H4Wfw5X3G+zA0JouRPNYlgVjW40mXi48LxA02vcd5OGbWShwU/l1LXq1cJ7TK SesJxeGJOxp8glbTUP46rXzuhy7I0vbXpUT5HwVv/cuxjDFDIkGQV8PM0Qepj5ZV7ygIsO kBzZF9X9kkqqVi/JOHsHHrQANUj/89w= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e1ce98dcafaso529847276.0 for ; Thu, 12 Sep 2024 14:02:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1726174946; x=1726779746; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qhdYG9qzYkU8fhMrG0yOrq2vgwt4qH3lwD7TAy4ybww=; b=1pi/tZR+qexLROep24Fw0yCP05BfPKlest2oQFaMTe02xXYMAm4OZVE2WJ03dfLaZc Lkvi8mIRUdPMG5E+3vpOtgoHkrUHIUciMFI+cP4W9rpoVqt0bU0X1mu/LKD5Kv+rIuNV TJ4X2J45v3p9QL4/Q9oiXtm6a1sI+rMdCrQzhZslYXDFZ3y2fZzGx63TEu7GErQqmTMj Z/mVHwg5gpHGxoUdR2sprrQC3mNM36UBYZ/IaU8GNC9zmegwhD0LAhpr59JcrXoo0YSD wn8FQsxltyq2/iyXeUOFMmsKgTVdh3uU2GtRi4Ta4iCBaCYyrsBc3YCdxVybiZRHTyO5 2Kew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726174946; x=1726779746; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qhdYG9qzYkU8fhMrG0yOrq2vgwt4qH3lwD7TAy4ybww=; b=LBdLcsnzeEhxzAFBkCn/XPQ6W591DMSIhqq9cbmyyIw2AbZbtNcUo7ONcRruIQOqaV 6+vQRiN74198vcNhSIjpbC2HykYdLsykmoDpUZEGRpWV8oMz+cHHOMqoIUIqQxqrdkDF MH5yMLxDcdWNYhez6G3BZB9xd3Go/lqVAPhX9gxsvt1OkYUJXdWrEFX9XxHUo+WKngUK Fw6nrbW683oKfGc6wkD7K5izkwbejANKdrqvj0r62xwL8NIWIGkLLiByYP0b4vgIh5Xn Lq3ADhXy9AjnZWV0oNCndpfnfdnoeOMfXcmZP15FjVOOeFG9Oc0jdQgxbcg7Cuuj17t1 C43w== X-Forwarded-Encrypted: i=1; AJvYcCVw1sDIU1t5IQziB2/7LGXwAtItZMHQcZugihRRZ/29VBn1QsqmLxTbHwLHKP9lqGRG4kkL07i43A==@kvack.org X-Gm-Message-State: AOJu0YwKSK5DB1UEPc8vLqpk4y7sfJowmyD+j6SKcbjG2x7ktE7rFepW 0bEY+qGKuYpan5tHSSd2NM44ShtX4uisFNLAZXONWYEfuUBd8NFFElnbOg4k+2WOd7nDo9p2w0W 54w== X-Google-Smtp-Source: AGHT+IEPfAUTVLLk7c4i9PufrTZIibPhbQ62xoH444q1VdGgK0TSyAVusflkXa93qX5xScpfTaJSu0i6+jw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:68a2:335d:594f:da63]) (user=surenb job=sendgmr) by 2002:a25:26cb:0:b0:e1a:8735:8390 with SMTP id 3f1490d57ef6-e1daff7cd07mr967276.4.1726174946099; Thu, 12 Sep 2024 14:02:26 -0700 (PDT) Date: Thu, 12 Sep 2024 14:02:22 -0700 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog Message-ID: <20240912210222.186542-1-surenb@google.com> Subject: [PATCH v2 1/1] mm: introduce mmap_lock_speculation_{start|end} From: Suren Baghdasaryan To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, andrii@kernel.org X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 61FEC140024 X-Stat-Signature: xq9urzgs66rztch7mscq8mm3drg4xfho X-Rspam-User: X-HE-Tag: 1726174947-605227 X-HE-Meta: U2FsdGVkX18rPIoGSyKYbuYg8ApeDUg0v10DA/RKj+DkedMArAgsfylY2zZTEu5dB8Y6FlhOQ4G+8wBSp96TB1LhbaT2Sp6JMeTM/QkONbceYNtXzodfO7R1SqEGTCJVvduhGdn9eFe3+EVXk90y2CHxqYrQ646MufO/V2IrwNSEdstxgEBmpa2Rwe9NxK9MFH4vFIAuFUc6eaxFBjhjQQKeqZ+lhlvRE8y7l4WPiH6NyFl1Xh1YrfBkJAGQ84MzjYJwAzcGcnUoCqb5O8i10y9dM5z2EdlZc2SMNAFmwVWzqpPQZiWhpNNIAzmbdiiPc1vyB4VskwVx+j2kqjrmgzpSB3jUefQJ3AV63FZOVIGVdQcyQR9Eiwbl34jv02o1x5AcpuQxPwdKmuBcYDtUhtdNYJ7dxWONlUygS7eKFPk3yPNL4GFm+DOOGivNwtQ3C6YkwzdzyZ/kic89WtkOIqugr5TfqRVvefx2/Aa8g0GdjJ2f+tYPegMF53Rlcy2yRi7NeT876F1yoPf/NDuKLTGulKW+96OzccZV8u3OGQGpjfdG0TPQofkv7ny2Gt0jISEl9BScGdXG7qRjQLdBm4FRxCfZhNvsYawzpa3S63hV7fmxTsrIOH310H7qJstAiVTTjmqyJKvJfN6s4N4lORXwCMLzfO5vaNbaaacC5SKyhZNxj2h5ac7tzMOvmGycERJLH/WTafwg3L/vPKrBdKzLwlU4ONTL8zlJ47EqAIcDJrwQNBCVIAEHWijQxbrnv/GPkhM1Yo7P72/XM1NMOYVptENnD3g38lwrl2CkoT1dSB+m8EQGQ5JfkkYBXdoKAXbXO0TvyvpIXlve2ra5LHThp8EoWy1dnEX/WWVgdryuzAL0C77hTjzFBRN4LWwtFJ1thV0gCnkGDunm8HhprXRidHKmXxr3VyZNLtRyDfJCU73Jd+vw0RnrKtBWBKJvnYRuKrq4T0FLSD08O9K cerXGxyR 871AoDqYYG1Tx5i5N5QFZvzy1BaZUJ60xmROSTdmnNPup02rfWAI8xRgitOhOc2FLT3spJcS/qLoeXJc53d2sTGMrXVqBTWL6xbwUS592+VsN5BHVKXUegFY5b3HHDE32+WVTRjJvCABAmhaMOekEiYscmWOLrrnAnO54CFqSy38PAUm/3wYn+fDJMSkJF8Bs08F0PDyOyO3TC9Z+z1gNNVAuvR3QxfJVS2btJq6iSQS5EsrKFCSiRswNHys4gFeI19fe642it0BmEHv1lbmSm9yowOEDhiKQVVMdxBozkkh2+CCEMN0n4Iv7eSIoM+spPcMb6b4Ds/PAw35LeBN4LLalZoNBY3qvgxbeBMLGIUgd/9ZZ/k5NZFEzpZlV+4Msbx7AxoClkKbB2Yh2K6uqUlFjjFOjYyKYT2MyR0yI4fVGtO1yUyA/zOyXaGZaKfn5qNDQs+KTrI3TJmfF5bsrqWzoD402VQt871n/zgXUcYLY6I10iv321xhOWxYAFB62EXhvn8Zq5Uywil1SZDbSsWs7zyexf0FiH+n69eg6YVDv2zwaGT3UilyfZV6WfqLjZhu/IMWJvjgW1dfMR9R5wPD7KocLB2kfn+Es03O2ecZcKWj8Rvhh9WOiCvx2z21rQ0Amfkx6I3PV5gaKIiL+3UZ7/AsLgUxL+oCdTHvBQx912YA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add helper functions to speculatively perform operations without read-locking mmap_lock, expecting that mmap_lock will not be write-locked and mm is not modified from under us. Suggested-by: Peter Zijlstra Signed-off-by: Suren Baghdasaryan Signed-off-by: Andrii Nakryiko --- Changes since v1 [1]: - Made memory barriers in inc_mm_lock_seq and mmap_lock_speculation_end more strict, per Jann Horn [1] https://lore.kernel.org/all/20240906051205.530219-2-andrii@kernel.org/ include/linux/mm_types.h | 3 ++ include/linux/mmap_lock.h | 74 ++++++++++++++++++++++++++++++++------- kernel/fork.c | 3 -- 3 files changed, 65 insertions(+), 15 deletions(-) base-commit: 015bdfcb183759674ba1bd732c3393014e35708b diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6e3bdf8e38bc..5d8cdebd42bc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -887,6 +887,9 @@ struct mm_struct { * Roughly speaking, incrementing the sequence number is * equivalent to releasing locks on VMAs; reading the sequence * number can be part of taking a read lock on a VMA. + * Incremented every time mmap_lock is write-locked/unlocked. + * Initialized to 0, therefore odd values indicate mmap_lock + * is write-locked and even values that it's released. * * Can be modified under write mmap_lock using RELEASE * semantics. diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index de9dc20b01ba..a281519d0c12 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -71,39 +71,86 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) } #ifdef CONFIG_PER_VMA_LOCK +static inline void init_mm_lock_seq(struct mm_struct *mm) +{ + mm->mm_lock_seq = 0; +} + /* - * Drop all currently-held per-VMA locks. - * This is called from the mmap_lock implementation directly before releasing - * a write-locked mmap_lock (or downgrading it to read-locked). - * This should normally NOT be called manually from other places. - * If you want to call this manually anyway, keep in mind that this will release - * *all* VMA write locks, including ones from further up the stack. + * Increment mm->mm_lock_seq when mmap_lock is write-locked (ACQUIRE semantics) + * or write-unlocked (RELEASE semantics). */ -static inline void vma_end_write_all(struct mm_struct *mm) +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) { mmap_assert_write_locked(mm); /* * Nobody can concurrently modify mm->mm_lock_seq due to exclusive * mmap_lock being held. - * We need RELEASE semantics here to ensure that preceding stores into - * the VMA take effect before we unlock it with this store. - * Pairs with ACQUIRE semantics in vma_start_read(). */ - smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + + if (acquire) { + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); + /* + * For ACQUIRE semantics we should ensure no following stores are + * reordered to appear before the mm->mm_lock_seq modification. + */ + smp_wmb(); + } else { + /* + * We need RELEASE semantics here to ensure that preceding stores + * into the VMA take effect before we unlock it with this store. + * Pairs with ACQUIRE semantics in vma_start_read(). + */ + smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + } +} + +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) +{ + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ + *seq = smp_load_acquire(&mm->mm_lock_seq); + /* Allow speculation if mmap_lock is not write-locked */ + return (*seq & 1) == 0; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) +{ + /* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */ + smp_rmb(); + return seq == READ_ONCE(mm->mm_lock_seq); } + #else -static inline void vma_end_write_all(struct mm_struct *mm) {} +static inline void init_mm_lock_seq(struct mm_struct *mm) {} +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) {} +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) { return false; } +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq) { return false; } #endif +/* + * Drop all currently-held per-VMA locks. + * This is called from the mmap_lock implementation directly before releasing + * a write-locked mmap_lock (or downgrading it to read-locked). + * This should normally NOT be called manually from other places. + * If you want to call this manually anyway, keep in mind that this will release + * *all* VMA write locks, including ones from further up the stack. + */ +static inline void vma_end_write_all(struct mm_struct *mm) +{ + inc_mm_lock_seq(mm, false); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); + init_mm_lock_seq(mm); } static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -111,6 +158,7 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -120,6 +168,8 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) + inc_mm_lock_seq(mm, true); __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } diff --git a/kernel/fork.c b/kernel/fork.c index 61070248a7d3..c86e87ed172b 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1259,9 +1259,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); -#ifdef CONFIG_PER_VMA_LOCK - mm->mm_lock_seq = 0; -#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0;