From patchwork Wed Nov 20 00:08:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13880640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB778D6C2B9 for ; Wed, 20 Nov 2024 00:08:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 796FD6B008C; Tue, 19 Nov 2024 19:08:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 721486B0092; Tue, 19 Nov 2024 19:08:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59A0B6B0093; Tue, 19 Nov 2024 19:08:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3BD716B008C for ; Tue, 19 Nov 2024 19:08:41 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B038640356 for ; Wed, 20 Nov 2024 00:08:40 +0000 (UTC) X-FDA: 82804535646.26.BE96734 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf17.hostedemail.com (Postfix) with ESMTP id D743D40017 for ; Wed, 20 Nov 2024 00:07:59 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xf5vlbBA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3hSg9ZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3hSg9ZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732061227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b7bh6sSGESOsY9pdAptx272UcEa/Ygad7puDmCEq6Kg=; b=QAVl/GdYxueic8EKQSSnNOvDke1XOleLx85nFxD9YgX+cZ5qO1aavhHUQK+EwyE8XBZe7l ZOajZDQoA+xUT7Z+PkA1At0HKC26kv3z2xefLfmBl0tLIwQd+sEXYl7j7hC18MtH5ib0KA qlIls9jbXNPuWGVlS7l8JDC/oWhyFTc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xf5vlbBA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3hSg9ZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3hSg9ZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732061227; a=rsa-sha256; cv=none; b=XdLKlfGe6nWHWo7lvMKBo9xNNZdjUCBCgCwcPz3X++kgrTKoKC4gIzz6QP9FtEQ6OL0Ta3 KfKByHF/Z8z+fCk4XrKQuhrBnL5uLONXpv1cABDeWMTtQflY5H1bvd9lQkNW3LDdf8kvPl 64i/UJnDzJrDoQA0pichPnQJvRW5zEs= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6ea8794f354so56591017b3.2 for ; Tue, 19 Nov 2024 16:08:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732061318; x=1732666118; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=b7bh6sSGESOsY9pdAptx272UcEa/Ygad7puDmCEq6Kg=; b=xf5vlbBAU8JnTEDqnb0YWCZRqIBHh4X0ADPu5+IqWqU/fjxW0Lsh57kulnkamJXq+C PoaMqnVaP/dLEYh4zaWG5nN7hdtXjyA0+T2XcP9QvjUaSnOPRf7+luoJXM9WcQsEZLB8 whbvserj/hKcksA4KpzZLQ52+ad23+fRYSt8lvsEN00wVX4Py/N3oxdhUXvJsL4m9K9v Izrd5FjK1Kxq5EQkiYAEWH7ZTCM5uKRuVpnnb0BCG4lgloYZlIzDAaJniz4zgYFkCjGK ziqoWph56fY1vwJJdMpEp6kAhzaw35rwZrVthPPmTlEiLi54X677m+/F3WInE1Uw45ao Vh2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732061318; x=1732666118; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b7bh6sSGESOsY9pdAptx272UcEa/Ygad7puDmCEq6Kg=; b=qC5MdBsYs73esS8CaBQlaGsv7T4L2MJeCREKZTlwQoFbZEKALQOsYR298E8wCKWDNp RTo1JTxJsVSQUynm0RuC8PDqTCdUNFD3eBtYH1DeIAn9njNBWmYqU3v6O+LA4NAMHENE ICYbr1+N9VLPxUpyxtIjopmhUu0FzhftnzxUX99f+bTTjfevMqzMhnH3KkzIMijdFro9 7L895uPGPIPM7IuLp8UNNef1wB3UGiTkCmpjD+fPe+NQBM9P9o9hePIjfXTdJ2QIzyx6 hXFgBrbYrKoujKn3yg+X15unfXy6z9HuUY6dYbeZRm1qD43O176t11KipR/o4VVdZeVZ 0ROA== X-Forwarded-Encrypted: i=1; AJvYcCVn1/FAFOz4z4Gby3/2Bxc3/HqksMEWq1zu5rk8fx9O0VYrfZdE0g/KCPqOixrUDXc+RjsZhXYqtg==@kvack.org X-Gm-Message-State: AOJu0YxHBUJSTdz11z38UAnZC/s7YtjwgE43TWWohp1BOLIAIw9bvvXe xN7bPl3AZjZxMu+xKLF332byP67W6++CjVjh5j/M+CB68RVONa7gKDgodyREgPJ702JgnkXsMGW o1Q== X-Google-Smtp-Source: AGHT+IGtVg+pmzlMfUNmBABZ8fcc/vYjedj6yHOltiegFss4vfkD05ygP4e5H1ECoxnohN55I5GlusLW3iM= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:af45:bbc1:fc75:e695]) (user=surenb job=sendgmr) by 2002:a05:690c:5811:b0:6e3:8562:fbf with SMTP id 00721157ae682-6eebd2b241fmr6217b3.3.1732061317506; Tue, 19 Nov 2024 16:08:37 -0800 (PST) Date: Tue, 19 Nov 2024 16:08:24 -0800 In-Reply-To: <20241120000826.335387-1-surenb@google.com> Mime-Version: 1.0 References: <20241120000826.335387-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241120000826.335387-4-surenb@google.com> Subject: [PATCH v4 3/5] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D743D40017 X-Stat-Signature: aubphogtifwwrd4yt37w4nqbudk1ajf1 X-HE-Tag: 1732061279-374192 X-HE-Meta: U2FsdGVkX193hzCAhYCaUMV6OsLaECJdBheMxTPiveH2Suj0wQXUq0YQf/FfGxmSRiWDaP8D/47GKqIliLCc5Yms43Yj4pOp0tESpXR4tykNuZlhrQquMG85dEIAZaEfQKPJyUdx5wbLlAMZJF53lGzORvVsbf8pLwbc4ZNugGMOp9Qu0l2zpiNyrC5NozY8wK/BisPnuG5UZW3V5d4R7RzN7EwU3b3LtUlnnH7ASJAGzf9yDqyzScRNpZpwjMnODote9L+6R2xRIjlJJGFU8LNmceOB1wNciSDBy4dTRJazT3rZkhn0XyDrTOygjms9u3DH2dFvNPIr5hgYSo1KT5wf2+wb/Fo3MTGeQqtuOpJOcSXLjg0Yn9+jt3cReGY/ZyNYiUjipfrCPgmfhrDnCuGoJSsEQdyo37q9q38exz8TY3KS2VfkBY3+RBlvMVyY3+iXEwoP/OI+roA7fwpS5ps0jmFJGtWfRQFB1qvPM4EYbI1qWU38Zk6Q2ihgItqFE/Qzql8GGfA+elgvxczgC7JvHOaM7AMIlGkbIfoj6of/aNqrB20lqAFp+SyepsRyJ1kuaxv96PddokEobbtYIkTWMpslqe8KotnFVBuYIDxpju/VtlXwTqSfXtzWKHmdfS74TqYwExDPk829hcRIr/LyatcATNEcAVc4M/5puuvGXSFMCHYJ1DUnLPmiySxJJmsUCltIk+vNKGaRR8O4NiZvMSLLSCU2VR8ug+dCZRKL1qpp/Clxjb53lE85MptJ5S3F5WKmsXxN/TVk3n51jL3w31IzTOEUFfpOntAUb79ZZ1pE+UsnLYRLIsys20SBf2dNkKYjMsHCDoK/4/NVV1N0VPfmHYY9/2K5qI5N4OIBVFOWdOofgzvnXCQLF3vQoGPOy57VHb85miPjB6RRiuYQnHA2Vjg5bWOk2pIWsdBZR9tjHuNvgp9ttbQCVgqXZr41AgZHRiL+Jvqjdmz V2kP0rQ6 eYeN+gXVQ4jPFpXtZRT5ZGPa1v/PUg6j1mP4G7mZUkiPZxlV4EZPpk8atpBHkkpBtDZbx8tkDTh0yRqcEiZZu6k6U6KVKYC61sHG/kYB2PURyfZqKh4gcaXKLDJ14IAsIiFzSUAkydyNHESO8+5Nu/WjCLv5RAvBP1D33sgZx6RwT7S6EZBvvHsdjrfN9qCir3tZpR0YftndjUvcU7kx7W5Dj2yS0yo6tp2apghW782AzRtuyPZ1xvI3DctEMaKMRZqaikBcHL9SC3WtxPGVYP0ql06ln159wZZFnNtBIYStSxm48yX4hB1u0c4Is91bFHaRFOwmQo4uP1MtzuK1laSRtoyxgzBvjWK04KeFsBommPIa/cAi6OI0J+M1Be5SCVqpPEZ9jHtHnIhk6vKHkVKD9bCuf71vgDoyUaHKPbuCQqYBZ3FCUjIsyPpdncx7t/bAPnEai+9TJ2NglxwoTtZpDofQmGIAhu8oJO4viYhXRCPS++qZ911hxPp48/9Qv+XGPrR/sZUBr5l2DcLpPmtlTrMhD2Y3HMBPcwREqnjAbVY0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 737c003b0a1e..dd1b6190df28 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -808,12 +808,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -844,8 +853,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -878,7 +887,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1073,6 +1085,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index 7823797e31d2..f0cec673583c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index 209885a4134f..d0197a0c0996 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6279,7 +6279,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 8a454a7bbc80..73104d434567 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -295,7 +295,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1220,7 +1220,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1295,7 +1295,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 388d34748674..2e680f357ace 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -162,6 +162,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -385,6 +386,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 11c2c38ca4e8..2fed366d20ef 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -414,13 +414,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -431,7 +435,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -457,6 +462,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; }