From patchwork Sun Nov 17 08:09:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 576E4D68BDD for ; Sun, 17 Nov 2024 08:09:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B78F99C001C; Sun, 17 Nov 2024 03:09:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AFF986B00C4; Sun, 17 Nov 2024 03:09:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92BFA9C001C; Sun, 17 Nov 2024 03:09:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6C5816B00C3 for ; Sun, 17 Nov 2024 03:09:54 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E9CBF4018B for ; Sun, 17 Nov 2024 08:09:53 +0000 (UTC) X-FDA: 82794862122.12.1C0F0F2 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 235DDC0006 for ; Sun, 17 Nov 2024 08:09:30 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LiVq5fsD; spf=pass (imf10.hostedemail.com: domain of 3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ydc3xnWflSOvT3mKyRecGyeDU6jHNwbSKwasb1j3Pls=; b=xcChxZY9ZEot3dN4Q23zj3pxDcLZ3w+CxCoxftSrSfz1lU+8bypxpRdWapoP43XfPhlMSa r35pHyJ7IUjtM5UA44UJjo0R8RYLOeuWifIoI41gAVJN0WPYOniLe+qXmuqLE+L8v0UGem lhgP+OcwOv5pTnZfyfHiO0guPsbDIZk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830901; a=rsa-sha256; cv=none; b=SI9AOrbVf5hwVRMteN+Ni2jRKQFgMeBOibDyTkJfATvUIkrq+70mAVzlNBlNg8unq6didH 678O9EKZmMitms4+5QQEKumZmC3CPIxykt0HzD7hrb9S+t2lc2F7htrYptJSbcWFXD2tkK l66mEirPCi2lsLMo2nSXs2kUhy9HOnE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LiVq5fsD; spf=pass (imf10.hostedemail.com: domain of 3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e02fff66a83so2666007276.0 for ; Sun, 17 Nov 2024 00:09:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830991; x=1732435791; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ydc3xnWflSOvT3mKyRecGyeDU6jHNwbSKwasb1j3Pls=; b=LiVq5fsD4ohrthLdvb0x1BEF5qQefZ+3b2urJ8B2MT90Vsp9DaScEEWURH7hlp6on0 lPPqMWKQjFOxFxWQNqiGdYDZ6RGystjr1jkPHB03hbxjcVGMFLSGCZwStRA0aqP/1pem P68tkAbMneXHyuepf8uLGDhCFAnooCAsXHoRWJvjAOPjL8SF2ijFf/JXtAPkfPZnaMtO Oi2qDWpIrFED5CCWVKgfbzaIwh3wo5sKFX50HByQ51h8VDr6EoQqVaX6oXq6uOuPZR7L ZU7Pn5ZTWZJWckLKTRyFboQbrxFFntWuQl6ozXArj1HFg0skE1S8US571PHfip6c3w/s 8T0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830991; x=1732435791; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ydc3xnWflSOvT3mKyRecGyeDU6jHNwbSKwasb1j3Pls=; b=JKGpSAVtWAZ7VCSyk7USmjw47LTOtZiWhbPt0/DJJPdVfB5YW3H3bJbuFP44wP1DLP jnwybGLYK0RKXs85rx1NyIKx+rHQl3Yf8E7CxaYAgMY5rImgcveYQyekJtAA+FnvpgML PQXCtQzM+/risfvp8vXKFb3nAa0fIHUC52DYokUf98HdjIgN9YIHh3VaeDbhUUN1PDc2 +yrMhUM+uP1sSEIs5i8+dGvMvbXPxdIy3rjkEzRHR0SfqBZZjg3m8um0JOBPwALLzPT9 JON5aa9Z6qEtG8ZOq3mCH7Ej/8VEIsSjVmxspNurxnF9Rh/wxQ8cQmsMQL5ju7RAJpoo bb0A== X-Forwarded-Encrypted: i=1; AJvYcCV94Vv/1eUSTwR22xWJVfHqs1q1Vo0gjzBzCE+DeE94dC2rPnH5x6k8YCfnBvYE6ni6p0oFhNGWcA==@kvack.org X-Gm-Message-State: AOJu0YzzRuHb/lP2reM33+z3VtvwbHa0hKM4IW8XrF45xMM0RyAuqqHJ YYt8ioIOZAr6WT6z5A0hsC1h45Avg8syQxIZXwIUiSAz0CQn3ngGVTkD2cRyw96dRLh86Vdkw3N Zuw== X-Google-Smtp-Source: AGHT+IFBFQn/cqW/Nbk+DrZLqER7IMLE2PxGHl++p5as9o7fv/NkZldv72m8b+Y3E+tx/FPb9DpLbddXQ/Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a25:e097:0:b0:e28:eae7:f838 with SMTP id 3f1490d57ef6-e3825bc27f4mr8116276.0.1731830990609; Sun, 17 Nov 2024 00:09:50 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:30 -0800 In-Reply-To: <20241117080931.600731-1-surenb@google.com> Mime-Version: 1.0 References: <20241117080931.600731-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-5-surenb@google.com> Subject: [PATCH v3 4/5] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam10 X-Stat-Signature: ijdkj53ika8ry3afqjd8z97e8xgwc8sy X-Rspamd-Queue-Id: 235DDC0006 X-Rspam-User: X-HE-Tag: 1731830970-255536 X-HE-Meta: U2FsdGVkX1/WKegCnS+OM+1vN6l8OQR9sXx2MUiZ9lAN/mdJWuceC6AtdorfGlECA4f4E/9XfJpwnZUY9sebV68XgICt8PTqllgZ5O5MM611oSj1WDs1wiRylvSRf2FOjbzHN0IfIhXQ2038X2a3A6j9jXR29Bb3ZW2qsxa3uFsK2mMrUb6A/4X+3eaZv8xb1bL5YVcOINsjD3gz2iD9LR8dWsvkRBFlQVGbTeUUp2O4T+OF8oWLP+xwhLD4GBDy8N8x+2p6DhQrJKUZJ0guCSjYwd7tEj3XtiqGkbxbRRwLC2muSuOstc7dOmBBWa5gkjKraQ9TiwwSfkorSa9Ey2SnpLPeiUdP/h9tW37q9JV8wQITp/FHGiQyukTO/Pbc8DS4BKlsrUSJjpJlYrnlESN51MjxmAcGivW01yb/XpS2gimHhm/UgsGE+1DPun5dAq/lvmKBtr688Y526DgA19DflccT2AUn2hBjL9dpPXyQq/Wy1jIbPp2Z4eWk7+gIDEvgLZMd8RCpCP8q0tpbdWgs0GFIcyYFwy6T7uX2yQSmtHQjm4RCsDz5UXXxNC1x1cwM8t8a8NsTV20PsUdETJP8ihWz6pwqm0OBOjmCJ7yPF/2cMQCUTCMJLJUWF/gFdnToGpelHXJn6TZmT9mDTR4d7sywKpBfcBhkPuVDGMRGBJkh3CaCxpcn7BluLxrGBxx2cPbDgV5OLWkgYKXxuDjszfNr9KipKcz109pp3fw03irH4SbBCWxAHOraqZ5UuVQ2Sd7RWaEYnJOsXL3A4YgD/ykC2DeTsyDdQKHSyFtAF0bhjL8N9KGToBH/1xZetavgpGM3B4TpcRMBRXEwzZRKeSiLIUk0zOXdCKsDBFwWSQm7AMGy09w07b4UnDZWrf+jFuoRD1J7fNED7uPSsM+4MSL4oBVM/wOv7nipB1nAQDZXOqdpQb3El4EvtUKfGqg9gKzljx0TDGEXMKP SQz4cODT 1TKQatM6tFDi4msY6hruTwTUGpm2PDvf0P+3jlK0RYEp3EmWp++fKBpbt/PVSG1uKzRwlE63el7E7OOb1+KF0co2gTo0EfS3SblVMpeJThxRRtrgkMo/KruY08qCwAbiS8m52UKZAa+1/3neOi+5YfzljH4BbV0HSO64mjPDgr2vfbldXl/b0919IGoXgp+KplreIwCY3pLL3lS1abUHgCWQGchMQq08zIy8vNAQojmIr5pt8htQH0ywBo2b9kH2KOwkcbzjf6p9KSrCK0zQA1yIuaMWxQwYV6PL8QOnTfnQwCuwl/KXtFMBYWCrt5LPPtREr+0u3F6zOsA2cyyHDD7AgJaIc9j79JEW9/Z9vQ7ny1yTB3L8XZ3aovAAvvjNUpRsFlqWk46qlUN2q07SUYRV4Kow+6eWN9gGhXjR20w8KMNPGoKHSImHLRn7blmmX0F0I11WW0hI7P7eGTQhe6gRUpYV/qvbXfLjfrSMnIEfZbrUg/I+JnB5r2kbVqjANuwW+cPIrSQBAD834Spg82w4zo2omKQS1Tj4EdoojGtxsQWPOA2RU3RUa7yZ6/Ba66JxR8zEGHRZOhThD30ipm7kG5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu(). lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it. vma reuse introduces several new possibilities: 1. vma can be reused after it was found but before it is locked; 2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read(); 3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while being read-locked; For case #1 current checks will help detecting cases when: - vma was reused but not yet added into the tree (detached check) - vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. For case #3, we write-lock the vma in vma_mark_attached(), ensuring that vma does not get re-attached while read-locked by a user of the vma before it was recycled. This write-locking should not cause performance issues because contention during vma_mark_attached() can happen only in the rare vma reuse case. Even when this happens, it's the slowpath (write-lock) which will be waiting, not the page fault path. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Adding a freeptr_t into vm_area_struct (unioned with vm_start/vm_end) could be used to avoids bloating the structure, however currently custom free pointers are not supported in combination with a ctor (see the comment for kmem_cache_args.freeptr_offset). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 48 ++++++++++++++++++++++++----- include/linux/mm_types.h | 13 +++----- kernel/fork.c | 53 +++++++++++++++++++------------- mm/memory.c | 7 +++-- mm/vma.c | 2 +- tools/testing/vma/vma_internal.h | 7 +++-- 6 files changed, 86 insertions(+), 44 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd1b6190df28..d8e10e1e34ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); /* Use only if VMA has no other users */ -void __vm_area_free(struct vm_area_struct *vma); +void vm_area_free_unreachable(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; @@ -690,12 +690,32 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +#define VMA_BEFORE_LOCK offsetof(struct vm_area_struct, vm_lock) +#define VMA_LOCK_END(vma) \ + (((void *)(vma)) + offsetofend(struct vm_area_struct, vm_lock)) +#define VMA_AFTER_LOCK \ + (sizeof(struct vm_area_struct) - offsetofend(struct vm_area_struct, vm_lock)) + +static inline void vma_clear(struct vm_area_struct *vma) +{ + /* Preserve vma->vm_lock */ + memset(vma, 0, VMA_BEFORE_LOCK); + memset(VMA_LOCK_END(vma), 0, VMA_AFTER_LOCK); +} + +static inline void vma_copy(struct vm_area_struct *new, struct vm_area_struct *orig) +{ + /* Preserve vma->vm_lock */ + data_race(memcpy(new, orig, VMA_BEFORE_LOCK)); + data_race(memcpy(VMA_LOCK_END(new), VMA_LOCK_END(orig), VMA_AFTER_LOCK)); +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { /* * Check before locking. A race might cause false locked result. @@ -704,7 +724,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) @@ -721,7 +741,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { up_read(&vma->vm_lock.lock); return false; } @@ -810,7 +830,18 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) static inline void vma_mark_attached(struct vm_area_struct *vma) { + /* vma shoudn't be already attached */ + VM_BUG_ON_VMA(!vma->detached, vma); + + /* + * Lock here can be contended only if the vma got reused after + * lock_vma_under_rcu() found it but before it had a chance to + * read-lock it. Write-locking the vma guarantees that the vma + * won't be attached until all its old users are out. + */ + down_write(&vma->vm_lock.lock); vma->detached = false; + up_write(&vma->vm_lock.lock); } static inline void vma_mark_detached(struct vm_area_struct *vma) @@ -847,7 +878,11 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline void vma_clear(struct vm_area_struct *vma) + { memset(vma, 0, sizeof(*vma)); } +static inline void vma_copy(struct vm_area_struct *new, struct vm_area_struct *orig) + { data_race(memcpy(new, orig, sizeof(*new))); } +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} @@ -883,7 +918,7 @@ extern const struct vm_operations_struct vma_dummy_vm_ops; static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { - memset(vma, 0, sizeof(*vma)); + vma_clear(vma); vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); @@ -892,7 +927,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->detached = true; #endif vma_numab_state_init(vma); - vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5c4bfdcfac72..8f6b0c935c2b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -667,15 +667,10 @@ struct vma_numab_state { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - union { - struct { - /* VMA covers [vm_start; vm_end) addresses within mm */ - unsigned long vm_start; - unsigned long vm_end; - }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; }; /* diff --git a/kernel/fork.c b/kernel/fork.c index f0cec673583c..76c68b041f8a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,6 +436,11 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +static void vm_area_ctor(void *data) +{ + vma_lock_init(data); +} + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -462,8 +467,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * orig->shared.rb may be modified concurrently, but the clone * will be reinitialized. */ - data_race(memcpy(new, orig, sizeof(*new))); - vma_lock_init(new); + vma_copy(new, orig); INIT_LIST_HEAD(&new->anon_vma_chain); #ifdef CONFIG_PER_VMA_LOCK /* vma is not locked, can't use vma_mark_detached() */ @@ -475,32 +479,37 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void __vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma, bool unreachable) { +#ifdef CONFIG_PER_VMA_LOCK + /* + * With SLAB_TYPESAFE_BY_RCU, vma can be reused and we need + * vma->detached to be set before vma is returned into the cache. + * This way reused object won't be used by readers until it's + * initialized and reattached. + * If vma is unreachable, there can be no other users and we + * can set vma->detached directly with no risk of a race. + * If vma is reachable, then it should have been already detached + * under vma write-lock or it was never attached. + */ + if (unreachable) + vma->detached = true; + else + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); +#endif vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) +void vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - - /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); - __vm_area_free(vma); + __vm_area_free(vma, false); } -#endif -void vm_area_free(struct vm_area_struct *vma) +void vm_area_free_unreachable(struct vm_area_struct *vma) { -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else - __vm_area_free(vma); -#endif + __vm_area_free(vma, true); } static void account_kernel_stack(struct task_struct *tsk, int account) @@ -3135,9 +3144,11 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| - SLAB_ACCOUNT); + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), 0, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| + SLAB_ACCOUNT, vm_area_ctor); + mmap_init(); nsproxy_cache_init(); } diff --git a/mm/memory.c b/mm/memory.c index d0197a0c0996..c8a3e820ed66 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6275,7 +6275,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* Check if the VMA got isolated after we found it */ @@ -6292,8 +6292,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check since vm_mm/vm_start/vm_end might change before we lock the VMA */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock(); diff --git a/mm/vma.c b/mm/vma.c index 73104d434567..050b83df3df2 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -382,7 +382,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable) fput(vma->vm_file); mpol_put(vma_policy(vma)); if (unreachable) - __vm_area_free(vma); + vm_area_free_unreachable(vma); else vm_area_free(vma); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 2fed366d20ef..fd668d6cafc0 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -632,14 +632,15 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void __vm_area_free(struct vm_area_struct *vma) +static inline void vm_area_free(struct vm_area_struct *vma) { free(vma); } -static inline void vm_area_free(struct vm_area_struct *vma) +static inline void vm_area_free_unreachable(struct vm_area_struct *vma) { - __vm_area_free(vma); + vma->detached = true; + vm_area_free(vma); } static inline void lru_add_drain(void)