From patchwork Thu Feb 13 22:46:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7423AC021A6 for ; Thu, 13 Feb 2025 22:47:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AB50280007; Thu, 13 Feb 2025 17:47:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 93184280001; Thu, 13 Feb 2025 17:47:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73613280007; Thu, 13 Feb 2025 17:47:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4D14A280001 for ; Thu, 13 Feb 2025 17:47:09 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E6A031416BF for ; Thu, 13 Feb 2025 22:47:08 +0000 (UTC) X-FDA: 83116408536.11.AD234C3 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 3442F20005 for ; Thu, 13 Feb 2025 22:47:07 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yoG7ZMC3; spf=pass (imf13.hostedemail.com: domain of 3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IpC5RSofcM0ZStaR556vDop/rdew408+n2xS+qUK8/0=; b=zEzcBKh1zbbLqAGL/Zi2xLNDpyB05I3u35WToNYEWl+InXXmtKyqRkfZNfQnIrvDoUEPpF 4DqZlEErRi4oq7HzG+hB8JaB+hiGn/gxZavF9LKxtXLizOkYCwUI7H0AinM9sqnnoOji66 gMnQMyZW1qxPOa3DPuLqwkqk74c58Vg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yoG7ZMC3; spf=pass (imf13.hostedemail.com: domain of 3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486827; a=rsa-sha256; cv=none; b=4ob+u/NjT34m/QfpxD0ZLBF6CaOlq0cJCR1kDhPIXDi8GEheu9YHM7oyYxXqAEBlJJB5HZ sGd9dKg1YZVS5U3k8u7XT8p9jRWJVmM/ExDBnjQU00r2nozZqHQBdc9dbUHhgwx85tAKHg yrV3zstKKsIYaTtN/4pXFDXzK0Qub9c= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220e1593b85so16022135ad.3 for ; Thu, 13 Feb 2025 14:47:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486826; x=1740091626; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IpC5RSofcM0ZStaR556vDop/rdew408+n2xS+qUK8/0=; b=yoG7ZMC3VuCXW4qRutJG48NrlfjrE0Jm0u/KfRu5fQGdgCSsZUXfvBEyAn1J6e7sb6 U6ESv2uGCCakTucQPjhotLrp7fyAJBJ7fkwIjziE8yLSSDlgUtTYSuwjflks6eUrXHgG CZHTC9uWAdkW8ozryqTj3E0kvCUw9Fy/IwqM72CCsNhfyx1NUtzkVA8rEFV8XiF3Defx 14uKWqPMFzn5Iyy/7obgI5rV7AOf3HM1Dqv6c/hCZIAdgKhRzqCUh44Js+6z2SY2eyJv MZdHDuHiOHo5XfjyxzQrQ7OdZbeS16ufoMEccMlg+BUY8W8kguYDuUijRr7vPVUSIFZo C/kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486826; x=1740091626; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IpC5RSofcM0ZStaR556vDop/rdew408+n2xS+qUK8/0=; b=UiAFoPbr5VXGImEjMqEfkcDOfi+D+O0P5yBNuWvATm8iG9cIu3lTQISCDNXXz6IyRo hUOuFhUG57aCS47uyrJzWlCCSRwd4VAeBLhvg53hI2qwUJKq+Kj6VGNySOVdq7aOAHRf GeHR/BI42nsJsqf8jFyHnPGRA72Cuba9Mmr9V/jReiPb6aWOoulXz4ZawIBewUqHhWC5 vpoNz2kqwqWvSJXMAkOQC0SaYhtxTRrtR7Cl2pEQsRe3I4qNdbk4CqLDN7x1LDhJEgIn mzkYdvbrn3J2U/o2koF2yTuQLH5Xb9XU5wE7Nr7xJEHTHPZgNXx1f9X0/JNldi45R7Fl /8hg== X-Forwarded-Encrypted: i=1; AJvYcCUUzNCFScIL3jZfudxtQ7DitnQvku5MAeb05w8wc2QzhAMtrtJ+s7gSYo4gANIt5e6Dmb7CrVZVRA==@kvack.org X-Gm-Message-State: AOJu0Yxys+n6g39JXT8uSQ9FAUZG1Qb1GtCZ8QfWyNIDK/xtSBu0QruY qhFj/9GkWEU/Eyt4ctGzfYO4C3sQRYEClD2KTcRstcURbqHJSliiBdK33ETk21OdNK8hTV/TI80 BZA== X-Google-Smtp-Source: AGHT+IFlXfVeKvLwc31ADPhBrs1WWy20ACeukrGHKQSKrbfZQq/alc0hc5JcyIHneAFIaVwEe3JXjLuYCE8= X-Received: from pfbbi29.prod.google.com ([2002:a05:6a00:311d:b0:730:83d2:d6a3]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:50a:b0:1ee:69cb:4b42 with SMTP id adf61e73a8af0-1ee6b399cd4mr12028188637.32.1739486826085; Thu, 13 Feb 2025 14:47:06 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:40 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-4-surenb@google.com> Subject: [PATCH v10 03/18] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Stat-Signature: difyctx5rs11imewmdcms5ddud3on656 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3442F20005 X-HE-Tag: 1739486827-35743 X-HE-Meta: U2FsdGVkX19VGumPUmtzS4pTUqnovoDRPq9L3f4hWlU2wit3aAhsLj1JsLFeI7w8aL8N2PTcSsYaKQXYX1WXLSBprvBI/shOYEm6bvzxNUqVYg+AG3KTG+RlPdhln0ecpymGRKFlDsqYBSBrUfaPfu/3lcIO1M4g3ghHkUT99x+X8ATKqCZI+gRFgdiKi8b+ZtFHUTwbD1iruoWtC78Yz2NztQwUtXopaAR0jP5fCwlACn3VtjKUTkq9Dc5fTth9amNjEQj2snmLo67Eub6h8lIqalw1m8vKJNEDBiDBOElMeNDHbUelsWno+r4KaxCfZF+1iy9FgigQJY2z9qTU9TTM1glYWzUyDy1RhuPBAgH+xg/9E6PnazwgNCk4UWZgm0KF9WqwYa+Tuh5MhsjupkIQyoDbhxSR/ovS37LyEA4WhPEE5TFBKazFaWyNbrxwrDQ4s3eCU5WtA59enkgI33AWgTpIDaURmKWfI29/pPbmXsW8JXtM0ffJ9Lfdxvy2Hb0LlZ6VZhXBo5gdjUuFPXzAMEmsVmvPFZjDNZQbqDwfqlBRR7QgfsjCllx96f2hM+ISGC+4lJzZrSYx8CL7Z1RN5oVRd+vxBQ4EHHAse9lzoFqIC+s+lmtDh6eZGfYt5wTJxnUsen4tTZe5ZfiNdvrUMqeXsmgHyb/tzMRSF3Ysr7karu7rZAXtgOm0/Ag/PJorBuAwnjCW1cz1eCnpjWd6qOQSwuBdC0+WvNlBVic0/KYQZY5JNeF6zkPb1Tg9QqIgFBQm48ijDGADKo4uzmcAqIEwx94buBL0k3fOX+DTIogCP0SJ8RyVtJndHEdUHsPScewH0QlFz++t+S8QQbiBTjM9x3WB8TcU+y5MdQs6xJ+XNkBno65rVdT1+IByxRjHW0Sopsd7fhLJmTlb4LYaSW71XXkqPejmSr6dPwiuI/+0ubsAHAZxfDOtJImn/xUshhFSRNxJszTqLP3 58qB4Ue4 TLZvhugFMAJ9lvmLzCL1h7BomTIN69My54AuD7x9Fap24OOk69Lpmtcin+Yajdy30JuqomdzqNsM+7fEVW58g7wGUsa0vwGbNuRTMTsnxEr1T51MdvLbVq7R3GKm7xPzTnpyPA4Ni2v7DVLrx2QEdS11wFbeMt3znDn+wjhJijz0QlQO0DEjaJWNiYSRKWMus6upSD1HuA4zaGTWIPuLzTHrLMyPdg7Iqe5BW285jZdAhBKZd27jGDnKBCSuEk/uTDQJ0cELtXIDx/ehkjRYTZ9DeoWKRSFua3ghXDOSpZrriDlxo6uH6OqcwgP3iQaeiDoqUzI/lMtXVV2jkb2Lw+uB6hJG1VEsxkb9uNDkbhrIJxJnoqRMobNTZ8ECPsqzIL0fswYczKKnoJwJpEuU0me6FoHuVEvyuhbWnHiY7kv6bUdTJg434/zV7U/VMwh0Ad8IEG9I6HYakHvP3vx9eGgpjOat9y0r8ocOHp+v3fMhUPMprSHmE3qgoON5J9eiJjXMPxC6CLxy81FmoGolhouet/24eDUBVqrwJa/sv1q54B5HWkdcuzmtdUt+R90bVMV/uSkWdFfBirloNJhgTQ3MS05cqPWMvMn7xV9xkruFOwv2wYBNE4ELOgynACreHJ3sqI0vymEZmjsU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e75fae95b48d..cd5ee61e98f2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index bdbabe73fb29..5bf3e407c795 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index a8d6dbd03668..e600a5ff3c7a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6414,7 +6414,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 39146c19f316..498507d8a262 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -341,7 +341,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1238,7 +1238,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1313,7 +1313,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index e55e68abfbe3..bffb56afce5f 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -205,6 +205,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -437,6 +438,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 4506e6fb3c6f..f93f7f74f97b 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -471,12 +471,16 @@ static inline void vma_lock_init(struct vm_area_struct *vma) } static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -489,7 +493,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -515,6 +520,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; }