From patchwork Mon Feb 27 17:36:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5E08C64ED8 for ; Mon, 27 Feb 2023 17:37:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 503E8280006; Mon, 27 Feb 2023 12:37:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48C81280001; Mon, 27 Feb 2023 12:37:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 353DE280006; Mon, 27 Feb 2023 12:37:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 25641280001 for ; Mon, 27 Feb 2023 12:37:32 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 06327A09C4 for ; Mon, 27 Feb 2023 17:37:32 +0000 (UTC) X-FDA: 80513778744.07.EDDBFA3 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf28.hostedemail.com (Postfix) with ESMTP id EF54EC001A for ; Mon, 27 Feb 2023 17:37:29 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FEvdPkue; spf=pass (imf28.hostedemail.com: domain of 3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519450; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=arZZ82aLoZGzTODI4V3Ds+XnXRSv5CvpGtFg8O7kSHU=; b=WAJlDBXKeqQxXFZZIfE1OvYQnyGA71jg5GUYH+MqMlvgHbGjhNCOzal0RPTJuJYx3GTNgq QovPevyIeRmUAMIdLHDZ0gR9leb9BgFLKsrrspQHOL3z9pBj57mieNpy5sUE8wXhUsvtQN OpVUTSSLA8Jt8CStIL/rYdKSoCVh+Ho= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FEvdPkue; spf=pass (imf28.hostedemail.com: domain of 3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Wer8YwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519450; a=rsa-sha256; cv=none; b=VOq7RmfR8eLDnhsx+eZXZ2a5rJO2XrtrG6OBn4yYCjIuecrVNR+a4xf1axuVS10YaHcrWp KyB7wZjbx8wF9X2KMElu4NJSkMy8Sp7kkBNWmQHXzyvo+ad9U0DGy9HpoxuQCcNDth1cOT hid7Yp3uJ/r6cAfgS5XQSLCB05rkNGk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5376fa4106eso153568777b3.7 for ; Mon, 27 Feb 2023 09:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=arZZ82aLoZGzTODI4V3Ds+XnXRSv5CvpGtFg8O7kSHU=; b=FEvdPkuedKMzzGsmheGWPizHKb0nGjYHVNnx+9kAk6JWaGUBiX99d1Bv/O2GM6XSSl gB8+ecHQ4IjEhzwJPfml0bM5L6LYZ2SwUORgwjGkuiwqlRjsYJT4KPxwzw1UlC60m4aZ A90w9fKe9n+yvmErALp7FrbdpDMG/bmb9rYVdsVvHwUyt0atMOUCv41ig6x8R+tdHGZS c/nc1AnKLbHyXNdSxCybAre4lyR5J1UNPqZOtvyqt/5yJckeBvBSk6OPvMXjo6UK2C1R s7BFEuIEED1IHyyqkGYjy7EtZvl+QTa1y4yJouRGAzmHG8rSxuivFT0pWoYd4Spinq1k BQ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=arZZ82aLoZGzTODI4V3Ds+XnXRSv5CvpGtFg8O7kSHU=; b=amIogAhpEmwXwy527EkGnyAhTeLhIdxE8lFXoFEEW4zYUwWezUW3JuThy5sOYv6zVP czJCStRHqm8gGYaH46G171IKYwBnHDA6F1vbQCXdakpLnRFMX3RPaz117S47oHN13yqG 6di/sgoujAQf+ArxVv0+pZreG8YUlYR2UlNQ61080yCfBB2o2mxqpDJJnE5qpFd+J5fn CCBFkZOqATom02KL/9vEu5W1BipuqlS201maa4r1F70dQ0K/1ea2C9frQAX8iYmTaOKq /Wjg1XQwXOq6uFCV/ovVe16OnrVQgxnFkpEFqEE6E6MkFbmg2i02OJLjHb2OMY4tObLr +Asw== X-Gm-Message-State: AO0yUKVDjzL5xG0bi4sMDLmKD6p+aAd+ujjz61jKMZU+NKdIjKttqeZE RnMCaRmpnZNAmbY16qfHKAnLioDTkpo= X-Google-Smtp-Source: AK7set/UVPuqJo669/ldHphDhViQWe/utENdYlZ/T7NeJi51KLkDK1GElSVGwaBDs6B97rQ0tmz39GqO0ZQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a81:ac68:0:b0:535:18be:4126 with SMTP id z40-20020a81ac68000000b0053518be4126mr10693031ywj.6.1677519449031; Mon, 27 Feb 2023 09:37:29 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:21 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-23-surenb@google.com> Subject: [PATCH v4 22/33] mm: introduce vma detached flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: urcc3tskid8fmoyr59dzhwtd3coxxs8j X-Rspamd-Queue-Id: EF54EC001A X-HE-Tag: 1677519449-376355 X-HE-Meta: U2FsdGVkX18P2KPGWT9RUTirivBo35UbGIQyrU6/OPYPtaizODdPWMBRzehcXq8KDnbKmGSSray5/clyxxcM+MO2Jucbo6SAlXYcZkCjOPq+ZkazE0RjeDSUgUEJOt6uQnlC8OtuGy5/cWj2DB4LeSVUB4nMEqjsPC6N05l77pCKJPluDm9qqCU3WhpAqj/tm0PZz9jzY2xxrvhVvSrt3xgbYB4dz1lguxaKHRGNbesr/GF+EOnd0W/havGzLE1tg6rMk7tOoq9Z9Vxq8FPSChtJImQr86+fBvV20xQif9Al+Y8OzpUB+06FBpVa/nAZ42K78MN/PgbWJK/19UPoApWQhx3CmQLNxbP92SSGxAgEILF3AMcKdNGylV/xCC7KGuIVltCfnSrsLDFDU8exRvfb9hszydIlmSwrxawTUItdNqK0zZIWgEBZw577fgN1h6D9sgO1xBoFvXk4pX6fYULwyp6fb2AZCjshn+pNavO8hayZODXntACr01LT+5vYP01zrayoXqXE4ExnvMfo1zotSUyYiU1eh+N5Mti4KpEzR7ikL26veO2jQlNzNP4ipJnuAfleowkZM2hssXJh6fX03y0zTdIADGg2bW75BWaNVp/nahCXVEnrC5pQB2JUtGkHQC9mZr9uzi9NIfDcrzu7ZJm9pKkRr19LfV5dD5Z0w2RexONwMRraTgWQP2KXATCqIPZK8j9zceTGlK14lq3KWwVFhu7wzot24yIePeeAAGX72mWMnaP8lOceNWtzMMJ8R/htGKCF4Epv1yZF/6yRN5YPNsuQuSyUOj8tQprLBn9qWd89aZYnQYlxMYgXIc7wGeLE4yitjPIHMB25Qv8ps7ZWotKvR5bweZ0INGFEblKAnCtrUe+fKuMyUPzqkfxfUSatTPLUtZUWTDSdGhxYQyLzU1PPPkarvb5eqpZjmBVx33yVhXqrbtkyTH2x3dGSImk2LjdBYp48E0b Kxqlbiby EhT7yGvm8yTmQU9BIDE9f7+BPV4Km/5T8Uu4exevbvkuwX1ZufWvMQI/40KqRE8fSUVfxQCrdYrAXgGXu+f3/8O0w3+fd8P2x6D/2mo/La3ZU8wqPm2SDA+gYQX+fvc3HoxZMHpvCioEP/hOjKYmm1Y7OpZsz/WwLmSl7fkNe4NtP6YbYcRbMTB4oO/ciWzsweRM4uTaKuTa41PFY/BvgCdYn6yDVVU8mJl9/iuWfyRszQqpaWH6Nn/MIHy2GSExEVhKlALEydTQEXtXCmMpnCwILo4f5+6RHk/Hachg8j8DuyKOKPRoNtP6jbYin3oLhr9pIFU3at//aRwYW4hNfoEn7B9oe3Gofk4j0J9dIMyok+u6gGeeWPMnLt63PlmjS4ctKB+6Cy5R2PBM6aVkSo0w2X69UUOeHKnEsOMrUuob0qI94n2TdroMUwYpKM7t7IPeP/9nK/0yA8bXXLMgojtCjRAiNeN65Qo3uY2zfzZtmMUTriz2my44YB5m+uV9EicnwstdBNzFjpXtIRHegWhvH+9WroZ+/iFmGZFcJfCbYGZB14AKVpScG1Z4p6VM/HtnUOctvZrZwpIpFoDEQL43q1+rT1qPbEWgNPi6QQ0hRCgUgt41Ok/AZI30mnUa5tmW4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 11 +++++++++++ include/linux/mm_types.h | 3 +++ mm/mmap.c | 2 ++ 3 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3d5e8666892d..895bb3950e8a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -693,6 +693,14 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +{ + /* When detaching vma should be write-locked */ + if (detached) + vma_assert_write_locked(vma); + vma->detached = detached; +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -701,6 +709,8 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma, + bool detached) {} #endif /* CONFIG_PER_VMA_LOCK */ @@ -712,6 +722,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_mark_detached(vma, false); vma_init_lock(vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a4e7493bacd7..45a219d33c6b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -511,6 +511,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; struct rw_semaphore lock; + + /* Flag to indicate areas detached from the mm->mm_mt tree */ + bool detached; #endif /* diff --git a/mm/mmap.c b/mm/mmap.c index b947d82e8522..df13c33498db 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -600,6 +600,7 @@ static inline void vma_complete(struct vma_prepare *vp, if (vp->remove) { again: + vma_mark_detached(vp->remove, true); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -2261,6 +2262,7 @@ static inline int munmap_sidetree(struct vm_area_struct *vma, if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; + vma_mark_detached(vma, true); if (vma->vm_flags & VM_LOCKED) vma->vm_mm->locked_vm -= vma_pages(vma);