From patchwork Sat Jan 11 04:25:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13935769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B36DE7719C for ; Sat, 11 Jan 2025 04:26:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 170416B008C; Fri, 10 Jan 2025 23:26:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FBFD6B0092; Fri, 10 Jan 2025 23:26:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2EFC6B0093; Fri, 10 Jan 2025 23:26:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A99786B008C for ; Fri, 10 Jan 2025 23:26:17 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 646C71610B7 for ; Sat, 11 Jan 2025 04:26:17 +0000 (UTC) X-FDA: 82993883994.18.B25076C Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 9858714000D for ; Sat, 11 Jan 2025 04:26:15 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cA7O7rNL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 35vKBZwYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=35vKBZwYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736569575; a=rsa-sha256; cv=none; b=a3fwABWxNMf7yQ36oIvPJqleVAn1EO6Bm5Tu72E+FOAEY2wY1gvPsCfOx/iurMLQggxiQD 0MisCmiKC+oNY5e0ja0y55yq8+QFr1LCmiQhZC3Vx39DOl+psGmtGwYMdK9Bbde5v4LAvb bAJuYwnPsYNXNK2WFFEDyBJrKAPfemE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cA7O7rNL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 35vKBZwYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=35vKBZwYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736569575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SElJeL0WT8TUy5A74YFI2zLpX797Z27dhV0LPxWhKd0=; b=8X4hMdXN42y6UE061gymBoXeCokudrB5xDBaFjVGJgG+WmvV04W1hqs/rJgXABaYEWw/t+ jfmtdz9fFbzv4M8ZL2h61I4rjx0+yqrBCouIhzIMdCJOG8fMdE8X7X++y4Yf4+kXyWFpoR 5IfYTUaBbJLYbEOHIHW2+qxQCkZl+4Y= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-216717543b7so68519465ad.0 for ; Fri, 10 Jan 2025 20:26:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736569574; x=1737174374; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SElJeL0WT8TUy5A74YFI2zLpX797Z27dhV0LPxWhKd0=; b=cA7O7rNLNbVtbprMSWBkBlHYALbVgURmZGgWQtnq3I8ZalYtUpVjfVKHXEyDcwkBCE RnPZk3G1NyfHs3a0IMMaQhV7V2sQ7lzMu4ax0hQDohnFq88RZpWoj8f6ZsbIxcv3awt4 pOPXt7FVt22ZlQTiVfaZ5+SNl2/74HzcGcLwJpfl3AAEWsJuy3zAF2KvwrtW4l7ToZ7M 4cFrzrQGzJT7Qy+nxLNsZ26HIUqdR7xzVNr3W8jb6sHpucjefGvqkQ/nt4v3muHw13Oi wogM+hfP8uYUSeCp5gf6ghAsQ1slEYPxAM5AVsZmkK5MNOOask/kDRHjlcuUgyTIfVZd 327g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736569574; x=1737174374; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SElJeL0WT8TUy5A74YFI2zLpX797Z27dhV0LPxWhKd0=; b=A2Q0p5ZSSGBGN50JVOVwlQPobT2N930kbAr0tMEel5JxN//SGjqaC0DgAGyH4/dqf5 BiJqf08eaReOIV1qr0wws5bxhvZbwrttLxrRquol5rg7uDLxaQbYw5snN8L6M5JB7E/L X7J8iySDf13JOj8P5mohnaTwsta8fPa5DubpGtvhoVmZCtccQkW4HfV7YTTVHxglnEWl 1tl+jQHNJSRG9whDgjY0zaVRrcplx8irdB17HAdIg8Ppjx7rqLxvaVbL53w/rFfjwp0E cFKMNL8BVzj6JJattYZxQ7OQujRTcNc8956f+dmErZIKOe5OuOt9bgJAkXnyEbcjS1y2 pTSA== X-Forwarded-Encrypted: i=1; AJvYcCWjgIEnjPj+/+BrcECD5OL/WvihropE8YNctCztxJ7+NN3VejN70jfqGb+bAj4cfD4BFu0OJs82Yg==@kvack.org X-Gm-Message-State: AOJu0YwI+g5uTK7Xb6ucUawwR7hNsfYeZOQs3/x7cmt+47zCz0pB+kom 7K7HcAAHAI30TQnRhHqJMtm+DhsFv19yf4/VtGkVCzPoQLq7MLBCihMkhNac68YrMkOFsCQ6PK4 P+Q== X-Google-Smtp-Source: AGHT+IEQSBazEnMEVuc7MWzbILvMQe6z9E5ialwdpHDO4Dnoni0JWQpSkI+iskP8PjwM5g+e4yexHOB3pPo= X-Received: from pgbcv10.prod.google.com ([2002:a05:6a02:420a:b0:7fe:5385:5c99]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:dac8:b0:215:773a:c168 with SMTP id d9443c01a7336-21a83f48cf9mr212973855ad.1.1736569574261; Fri, 10 Jan 2025 20:26:14 -0800 (PST) Date: Fri, 10 Jan 2025 20:25:50 -0800 In-Reply-To: <20250111042604.3230628-1-surenb@google.com> Mime-Version: 1.0 References: <20250111042604.3230628-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250111042604.3230628-4-surenb@google.com> Subject: [PATCH v9 03/17] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9858714000D X-Stat-Signature: yormidto81ied6qoibpiza1moqi5d8xq X-Rspam-User: X-HE-Tag: 1736569575-822085 X-HE-Meta: U2FsdGVkX18Ro6LeXnQGP5DCmfjaatSLcrpI9g8WXl7zZ0mtwCII1R7vRCEk7pkmOUqAUQ6wLn1zAIFb5cwJCwN0N/w0qIV+/rEUyaKdnv7rnQs4+23NlZCcjbRnN3LusP6U5ROpNqYGw3alWEI1jcTPglVxNThx1jSih7ZVsgUpElJ7BX0iXDu4hYGECuoqppwwApHq6jJsYdCKOouvXl1cO335l/osQjuM98T7DPU6vSurpGme8kUOR8bijgA6nBLebX0Bvhu/Csq4wVrkhNeOLXXCMqrMnJVgrxbly0W+4+7gJ3X7cI0opys+S/SZnC7KvoywNVHqNc/7kbBD7BW42PounC3TkdUZmOHiCjfRRKAEM3MC48hDoTe8lt2h2/998cNX/17Z/oH+VjyJpgqeA+PaGz71Ey9YnDmdqMAFxv1cHbjt46pwZ5dOXjtrqyoIu6FETeR8366gxs6yAM5kGDf/PgGCflEaJVBB/HnSbS85ikzZZDD9RIH97wLtQsc3oTIS0FT1D7aO7YeAFb42KimVhU9sAb5ib3WRBp/ConftTIqeA4iEsLVqAJnPmsSAkubTow1cX/nraw0VY+2cFKjVe0DHUoeU4tUcwySm2xZGSYz4QX7/z4ngrAfJrejMCeHJYRMoZCWrLeLw+ZkWin8W26N3wVAn48ERqGwYWrDha2vpeN1wDohX6gf3pFO/b/DEj/tLGFEgAbqDoXgGvrcPCXLLZFkVUnKiFknxnp/ALdiBkDbmP9J+W4h0o0guM1YY6AbWMLp2yEYvDT/LzYbf7emB1CqKgAyjNRAqx6Y5ixBWxxTd5f8jfXdiYiIjPoTaa5012irv5RMacanUi5ZDF0nnO3g1LAcR+cAivq3MZ7FRbtLPGuyRSuVR/PuI0lZHkozptdipWC0sUvIbuOKkJtH1ZttoUOJTTdQVc7pnuWerH/0WnloR/CaIDU2/r4TU+t/6KvI7vZU HYKf0jKO lRkjsltYc2jhebe/mteG788YcoQrToIHc9XsM+4DXrPScL+poD5rBq2L3yJDegG0wdYFF+BnIMDt3rFQk6go4n2cLQM1/pqaHxMKacHyuIs/pYnzlGp0wz4mh51IvXc5WlQ8mq+i/odMOlRD63cLi34fVTuIn5Cx/gHnw0eRqLqeJSlEfiPYKP/es8MZTvWwhRij6kJ90ELGK7ZIj9OAeNtz44/mAzCdN1LFg81BjojnpOIeWl1ObdgBD2TUs+rhazZZvf0pPb3qy6CgTQrRtlt8WRb4szEnwXzvwQ8xYPazSZvBuCleiJ1Ow/a4jC+ZiGEX3BEF5G2tt6AYRe/i30sKd3HhOs0LdPKp79tqJ0CwyFvzF4X1qhxAQOAOUyPfJTEo+v5Sa8zQz/0FY7Rgde0/CyhAs6fdvWQnOe0PxUgLGu+Igx/ypUyGINZIYtJ4gX3LEZm/xkhE4KIeAUiAdSn1pRQlVDKG1meh/mmAvHpPpJ0AfrBDs7gEalV/ywmut8e+V5eNEos0c5TrRFzlhdAtta3XXZ7NePhF53FoxdJFXCqycWLGGaEdXnS7GCiDfsI6xYSeUbm+J7bJDGdnuvERtspjST7q54BhSCb4xEsLWaNht4qD4JnBLKaTT17KXIS03kbaD8Xdjacs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ed739406b0a7..2b322871da87 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index 40a8e615499f..f2f9e7b427ad 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index 2a20e3810534..d0dee2282325 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6349,7 +6349,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index af1d549b179c..d603494e69d7 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1221,7 +1221,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1296,7 +1296,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index a2e8710b8c47..2a2668de8d2c 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -157,6 +157,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -389,6 +390,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 96aeb28c81f9..47c8b03ffbbd 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -469,13 +469,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -488,7 +492,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -514,6 +519,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; }