From patchwork Fri Dec 6 22:52:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B1FBE77173 for ; Fri, 6 Dec 2024 22:52:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96E866B030B; Fri, 6 Dec 2024 17:52:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91E8E6B030D; Fri, 6 Dec 2024 17:52:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 749DE6B030E; Fri, 6 Dec 2024 17:52:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 56ED86B030B for ; Fri, 6 Dec 2024 17:52:16 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1581A1C6B8D for ; Fri, 6 Dec 2024 22:52:16 +0000 (UTC) X-FDA: 82866033558.25.1FC64AE Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf07.hostedemail.com (Postfix) with ESMTP id 73B7740014 for ; Fri, 6 Dec 2024 22:51:56 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=akPkumXV; spf=pass (imf07.hostedemail.com: domain of 3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525518; a=rsa-sha256; cv=none; b=kL03I/AISMGJotQQK3JVrhUdsfGVXNmZM7T9EV4kE1VagIbgoO13zbpzxRjfRWN+c0r4Dn 8oAfaMRLfEufUeG+cYZU3vCL+u5GUP1AHB/37lriHBeqw52XyEPGtbXJIyPXFaRncPVdFJ UuE9r/tZcT0Zly0vVCyygS3+iWVbZDk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=akPkumXV; spf=pass (imf07.hostedemail.com: domain of 3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Y40baPUGWax4wYRQAtfppaknEh3um9zBwyBL9xV1Y4=; b=Qmp3r9nBJLEZKVBJdfg7Mqfw1L6MNkqwWtfqTyLqoZN5YM4KS9TCT66QJOu6zu0T9tEFZm 3gcb4I0+UnYMWNO3mOyGCOqw35XIMJc6NQd9pfZxFlMdCN2FwTj5qsuqK7FwFLuNC14EvK z5knoxfhOIU3Ttcyjqtt+Km478+5iIQ= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ef7fbd99a6so275887a91.1 for ; Fri, 06 Dec 2024 14:52:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525533; x=1734130333; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/Y40baPUGWax4wYRQAtfppaknEh3um9zBwyBL9xV1Y4=; b=akPkumXV3REPAU/q/XweZ8BeJPKRkue2SZjQDJEMduUWhIzLDMMGO1WlsUjDo20k47 OsOEXiYsHHYgRBAXChavJhmhTwI3kBvsRejnWuYAlcyMBo8Z5heDgKLDTjxa17M4BPpL fQjnr+hIP1ly9n0ZrEZsyDtGY+ZJntjbA3lflldBXDqOjFKcmmZ4aBsRiXqgUhMzJ20F Fydkus+thW/xWEBcT6zlamosIxJdOfF5trIRlUPdm0fYLHN+x0QFXBbOVzJEnNCXJIRi 2oVC9nO5YLeWV1K45DEfuLvKvVcUpHN/VCA2IkbBCQfr28L/IkJ7KwPsD4R4x/6ZGQ+v ZLJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525533; x=1734130333; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/Y40baPUGWax4wYRQAtfppaknEh3um9zBwyBL9xV1Y4=; b=VkEOq4s2+h8V17rS6HK88IJX5KLIZizA23JaD8fEETao3kiYK11/x1+fpINYnoEKKj p2xnnovUND672FsVwwTgRZ7dkCq5T5YuRembJz3I6EIv1+3iX7t3ZpjY4QnbVoc5s5eO QT6NFwnTqazT4Fbgw9NdV9cNvXuCiXtsJ3n2IcglB5JV6e6Fsnl+9fqr5XL/2E9k8AUZ zLDAexsXGbHaL314vydCNaZovpmxMTdtBrsAT8rBJZozTkyUGssRuHbpKUYAkFEfPqKU YFVi2tsZFMhzumnfsZSXWGwHg5Y2to3TrakwXy8nZv04h65DIH5IFaa2GLTCTOcStOAD EvWQ== X-Forwarded-Encrypted: i=1; AJvYcCXjnyUctx0XxpL9/2XvDsO8kTKzVLru+z12fcpzb2JAC8o17NdSmFDfrYOuoTlgJafTwhKcb583OA==@kvack.org X-Gm-Message-State: AOJu0Yzo12CaQpMUZkKXSaeX4ctGjQvnLszF/cGUnMzq58YaYo2rbfeK rbMWrvbHyjeozHQqgzFppFIt3fARhEZP/VAv+GdzSVrDYoVtjFO5xuxDGV+uQLDxfyXdhq3nBW8 49Q== X-Google-Smtp-Source: AGHT+IHCd2OHlcBdUGfCCxX6+VS5I4FF5b82AKepICjOdPKRrERfe6SXg3caG61pXASigp7AKy6oYeA6ikU= X-Received: from pjbmf7.prod.google.com ([2002:a17:90b:1847:b0:2e1:8750:2b46]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2d43:b0:2ee:ba84:5cac with SMTP id 98e67ed59e1d1-2ef6955f863mr7322434a91.7.1733525533068; Fri, 06 Dec 2024 14:52:13 -0800 (PST) Date: Fri, 6 Dec 2024 14:52:00 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-4-surenb@google.com> Subject: [PATCH v5 3/6] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 73B7740014 X-Stat-Signature: bownb4s4wpfobao7xz7iqcjeowwqdmaz X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733525516-350001 X-HE-Meta: U2FsdGVkX18lw5rQGmog/R1FR4ERq9pW9d6pmzglwgz0dfjd0CJ2PhC0kvPehurcwItEXQMYAyGW85TjdkWdBRvr7EOfUpmwbHpKZec0uJ+20rNodsat0oO9MX6UKyWfmW/vG0EtrtGN/+8RjgxbfN5iqWGp/h/oGJFY3TDfY7axnbfYmZ2LNUJ/Fj9HBaVOC4IHi3NIXw378fc7JoztOvS0MmBJesvtxJF8nfqH1wCBB1akuu5HhSfhU4NZa+kzx1CUu4+GQIM0f6kfaCkpHJjMiLN6FSlno9J/sd0ZdcVEOISZHqZ18RdIXPZfO7vzlJMZ5zcEXj8ujSClIAR142Bozx8aUS9nb3+kvpJjy8krITeX8c3TuK/dQa+wUxZ19+72gay888Y/Qsjop65DlxX96TsFa3gCY/DOReAo9Pp6csFnH0NG19YTZ2w3fDrNfxxplaajsUdQGxf61M2e2uksHmqeUb0ibLb7cXGavfoTWTvEhtN7v0N8yeBFFpvMWPe4exV4FwLIVq3CQl0ADvoJ9S7iORpATu57M3zdB2bv6sqQkC+zT+TFaI+qCjYU8PCum50X80g5AwuvlW6pKRsBFSO0y7c8m9mNe+dMMW2XZgFdvSA5jJejtP27lSnFUqn9jRIh43b+ss6teoOInlaHTeHDGzk9pO7c5xxe30GxUGwySBpXrci2FMssV+1CfVkKgCqsgnIFD58fho3X3xTxbSh40tIR7zVa4bXpEzWbY4H23QvcysessSf1crPi+qtzUMp9ceq3pvhRBWkZEV8rqZH4EL5wvKU27Zt2tm4TBZ09h2jt2ucbunkMKcW3gjqZnXWFpYkewgErTEwAmH031OiR5UWq4vkzqMtL4YqV3prmBJzZyVUud+w5BaNT6pLQ2Lz9fwMoS8D+j3AXnrAZQVyfkP+Qvc0FR+DaYyoNKB+Y+xsO4/wwV5hYrH3rePav/3JZk+LO+gA6T44 gYqk+WfB XX50UaPIAARtv2hWPEWF1+n73DP5GBWg/zUDN9fQYz17fcTZSOEOLAJmuu4soKy23afcSsZYhvSTR9Jj+91Lzmoxj3t1OyjZfxN6iv+kB4vtX7NP/iVSK/Moh7e8RvZLGN3ez9VExlR+Qkv/sGoReNNJpz8X4kSXjXyybvEW1aS102UmPl/Gmy7fG8V1CR5I3zUjuAcl+A1I6TiCY3nOPfSwGN66y9tQDImGWTMMmB+T/+vZlR4PXNzWQTMYQcag0+Z8sfMsM301bENz9xx6ROwghBecz/cMZSHtWH+bv1DyQ35brkZuroYcAoWV76/+xQRAvAHUFmbfAFPItZiiNCHMl6b4v9HC19iITC1gkPYEhbZyeIUEBvz+uv8wxL5U+NoNksRCeNgfS0FR9pefBwGguf+8TyJQ1K5RNfo2C8LUFeV0vNMd5wlxCxkn6Do0p/0j0z4OFSHD6yRZp/eGLkHwwAbSKqZWUhboDEPkk8ELFWelWpr2v9UiM11+RmeXnCgA4g/8anc7PRrMIoC6944fSg0e955KfgQEcy4J5FBf45iplThKYlwoq7AWxdku+0FwX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ee71a504ef88..2bf38c1e9cca 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -820,12 +820,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -856,8 +865,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -890,7 +899,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1085,6 +1097,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index 21660a9ad97a..71990f46aa4e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index f823906a4a0f..b252f19b28c9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6372,7 +6372,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index a06747845cac..cdc63728f47f 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1220,7 +1220,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1295,7 +1295,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 295d44ea54db..32d99b2963df 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -156,6 +156,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -385,6 +386,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 568c18d24d53..0cdc5f8c3d60 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; }