From patchwork Thu Dec 26 17:07:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13921396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D60A4E7718F for ; Thu, 26 Dec 2024 17:07:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACD2E6B00A7; Thu, 26 Dec 2024 12:07:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A7C8D6B00A8; Thu, 26 Dec 2024 12:07:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F5936B00A9; Thu, 26 Dec 2024 12:07:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6CE8E6B00A7 for ; Thu, 26 Dec 2024 12:07:47 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2FCF11409CB for ; Thu, 26 Dec 2024 17:07:47 +0000 (UTC) X-FDA: 82937740074.03.34C9343 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf14.hostedemail.com (Postfix) with ESMTP id F380B10001E for ; Thu, 26 Dec 2024 17:06:58 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3eYaom9e; spf=pass (imf14.hostedemail.com: domain of 3YI1tZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3YI1tZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735232826; a=rsa-sha256; cv=none; b=iGe18AGrfdeDunURH894iLKPbNFBWm66/0KcEtVAWoBa4IUjf0NaAGQaL4FMLD+T+hqyge u1yWO9xJ3/FKLHATXP/IpP8UGB/emw/tcScqQKIxrUoGquiEnqlit8aHMB1bnPf+xNMwkK MVzDFZhtOqOwIHhLa4SwK+dTxEQ3akY= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3eYaom9e; spf=pass (imf14.hostedemail.com: domain of 3YI1tZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3YI1tZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735232826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wJt30IjlBNVAevAu7cS4hzA6cm3MdBwy61U0C7q4UdM=; b=LnWZCc+Y5Q5q+duFvtgrd3UZol2ngmLCMi4hq+guiViwJmR5nzAW1M5khVQwcNtwFlQAQp DZEwfjOeAuHhbeSIM41XsA5zjSVzXZpXNKPGTt5Wh3XGsGLtXxl1Yg/yABeCFvhQ/tOBNn el/xaB/UqbfP40g+OvruOairQkbi0rQ= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2162259a5dcso144838925ad.3 for ; Thu, 26 Dec 2024 09:07:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735232864; x=1735837664; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wJt30IjlBNVAevAu7cS4hzA6cm3MdBwy61U0C7q4UdM=; b=3eYaom9eDhuolfy6DgsA8QO+1+UaiGPmD9//+pA0CwxtQACewjKFUxA6xJW9d+igCz WTYlF7pwHCLBi9TCSfV6cdlv0H69IqkLGalu8SfjgyNUpkKhIYzrGXkvb4uZ8ZlgwYkX kpuUi8S4nXmuTu2cTjTTSn9HfZXNHmAyJ7Z8cc6ukdWaYmq+AY0TMTqEuX0DzJdngqXi J9nlkCN8Yz9fSfLv9W3XJcQ+rlFqql9uU7QVF2xWwbef446BRv3XyqF6KEWhc2jd/8h/ 3uUsx7B+SHssj6VDvgZzzwDASPBRBRnyVRS5UKaPaOAlVOTqONlzdKJacoRrXJ63/DCK WdLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735232864; x=1735837664; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wJt30IjlBNVAevAu7cS4hzA6cm3MdBwy61U0C7q4UdM=; b=a18g/JxrM68Wi/BWt7A+kV8uFUV9TI4AIbwyPZu8kxaD5r8oM8ItTvk+4oumVkIQ4V PcxNlfp7kFrAM6SPgqCCs+HtU8yoUtgx7cQgivnNlt1ASX171G8ucVK64XToZEHyL3MU X2fR5l4gGO678s8ycf/7PrdTw56DBbyE9WMUNUU9RrwCnfec5Gm6S7Z96S3K/vSjloO8 tMy+L9xwEuYvegzNAPB2aGfcam5O3D9TsYPjaVBJcYw4Y3Vf7NIuj01DG2jIpwZY67GD q8Lc7BtJVLf8M3wx/UN4++8xeVvjoKZtQtE8TG0L8N0YPSbRqM+xezvrBrX2zM32bSvv Lf0g== X-Forwarded-Encrypted: i=1; AJvYcCWIL+N3/H7KTe+xQ36D/Qk1ben3nJYnP6SYTznecntg9ub4nf7uctZypOLzcfd+mKDXuUbzMuWbMQ==@kvack.org X-Gm-Message-State: AOJu0YwAoQ1o5GA8VyvTyKwBH8vYXA/oUGBXMK6kZggNE9rMn6aw6phc 7eEYN6PiaZK8yWUwU8wXmZfxH2ouVWMaZb8EYg6zH1mxr71HrU0tRf98kxWDdbUma1yVSfzKkvE B9Q== X-Google-Smtp-Source: AGHT+IFXJnhfwZOHdM4geNls1CRjrBe2ovOlMU/gzO5O0nW/gc0UsQXMJYqFrpyCsqUXFXBoYnj2R8jomuo= X-Received: from pgjg7.prod.google.com ([2002:a63:dd47:0:b0:801:9858:ef95]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:339e:b0:1e1:aba4:209c with SMTP id adf61e73a8af0-1e5e07ffc0amr38092469637.29.1735232864140; Thu, 26 Dec 2024 09:07:44 -0800 (PST) Date: Thu, 26 Dec 2024 09:07:07 -0800 In-Reply-To: <20241226170710.1159679-1-surenb@google.com> Mime-Version: 1.0 References: <20241226170710.1159679-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241226170710.1159679-16-surenb@google.com> Subject: [PATCH v7 15/17] mm: prepare lock_vma_under_rcu() for vma reuse possibility From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: F380B10001E X-Stat-Signature: da8emjcjg37ukrjj4esjcm9dgothbbj7 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735232818-842059 X-HE-Meta: U2FsdGVkX1+IKGgM6DiWJj++jykE05V13DnC9qrBKHxCn2rVIHAnITi+NyR9m3Jm516cKEyI8n1Oghb85tHE9ApP7+YIwgQ5MMeC7HaDudGOeKp2vDOnJ12nqx7Ccutl1dfRZzU0mObqURPgSq5TdUD3FRjdz8mfdRRN+ab66gWwjGv8+EZO4zuTC1zUPQwj7zYG7r7jAwmxMly1QU3Czn+RSSITaAUY+04L/QemiThwcbGI0CmRBpL4KO3qYh52DGPHJMGfY3r8QZjb+s92fPWNTW+boJQEyWMopvjzeLDB0oqquhbaJwkMwvReuZaqLIl5buWomhEFgTHjxv0uUmmh1wS2OKunPR3Yd5zaDe+Vl+LEjod8bNvAph+m/XRKVbx0kZKoleLojFoE02+xuCXPkVVL5Af+M+Ll9cIK+Yuk9/Z4OkPtsc1zIvcZIGvPYGXbkIC8kZe7eIMQ21N5AVngUSVs13jH3rBJ96JREyb6K+76NHTfImRL0rEpES9A4VaQwgKN/1gYRg4vpqeOshL84Uo6IYuspuVwRXeTzkVrIUhMFL7S388A9jlvQdt5yEQkmqDfEhSoi58roErPoNUrI292I8F7qerd8ykbA5tWu1qeL/jeAer14ZXIUAUn92pZLfvuFjOzx2nrWI3TTioHnv4Z0x/a/V3gujZCcoEpMP6pAmacmrMlFEotN3LQNneEago06Yz0VcyqGVx8zFGkL2YjGGozDj7clK8sh73uatVJF2MUOBqiX0Todar2JM4+p/O/oKo7kiq8jjlWn2b3OMSm1wnCHmgFMxaWCMW3fCtgGZvmQwavhIQ0n1+um76Sn9mXakSWoq7thceLqqC5hP0tNLpo5w1MTgOBoiNJjhKtaq0ebBWPnjEF2AeJbj/PKIIRoOX3Ct4Qn+5kgg0To1KQ1GuTH1dpddDEIOtMdYI21IpmCci2QJ6nFttNg7vgwfh1AGizb8IuC0Y aqMC8Djl YFx3JFCQai23isQQB2JxLwL94vZ2pfQ6RRqT4rMBw0MTj6s71FyU5/0FAqXVoDA1EYLe66QnoRaVJCxOkAxMiu3Ifmuz5ooRDUM7tXQ+Eroh1fH7e8BAQN5hl/A8nHDnyvus3vCvlL/NxaKhTYtBJ7zp3i5fdo85FtBspJdKQFHN4Jruh+MEM8FoNTYdnmhYsQSi05W8DPNA4VjOzVLpJ0QuPkZ2AaRr1Ww8rnkss2VF/hgo8bnlvxT6xvqOyh1qXzIBk304DRQcNQ+/zxhtjo2Io2QyPXYsjFhvLs4d5qdIBqydAVgrC0tBJndsuLq77zytgMUOHjXDtC3mOn2ezXj8Y22nZRtZAp9Rc6qC3n/0aoQ5NICPpV2UwwlVqn6DYoiVsZVgCvkPIpAZWkXkfsektE9406T8YmscOYbAPd9slBE+octogO0h/Mzwm+wm0IXxbOkrEchKesqvRHCkVlCd9IDEBYuunZSZGtvUBElPMz9u4bqmTaiKba08IWO8i3nx2My+E9agI8LWTq7jSGgij6D8tpvRWUY7Lg3z+mxPf1mhmcraCqYXAIFcwsIpsDPxCAhr6cMlRyvUFlr0hbBlp/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 ++++++---- mm/memory.c | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 40bbe815df11..56a7d70ca5bd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -730,8 +730,10 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { int oldcnt; @@ -742,7 +744,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; @@ -767,7 +769,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(oldcnt & VMA_LOCK_OFFSET || - vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { vma_refcount_put(vma); return false; } @@ -905,7 +907,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, static inline void vma_lockdep_init(struct vm_area_struct *vma) {} static inline void vma_init_lock(struct vm_area_struct *vma, bool reset_refcnt) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 2def47b5dff0..9cc93c2f79f3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6414,7 +6414,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* @@ -6424,8 +6424,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock();