From patchwork Thu Jan 9 02:30:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13931821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C01FE77188 for ; Thu, 9 Jan 2025 02:31:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55EF06B00A6; Wed, 8 Jan 2025 21:31:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 50D2B6B00A7; Wed, 8 Jan 2025 21:31:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3618B6B00A8; Wed, 8 Jan 2025 21:31:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 103DF6B00A6 for ; Wed, 8 Jan 2025 21:31:02 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BEC8C1615A1 for ; Thu, 9 Jan 2025 02:31:01 +0000 (UTC) X-FDA: 82986335922.06.362029B Received: from mail-oa1-f73.google.com (mail-oa1-f73.google.com [209.85.160.73]) by imf17.hostedemail.com (Postfix) with ESMTP id 058344000A for ; Thu, 9 Jan 2025 02:30:59 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Wh4HgmeO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 34jR_ZwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=34jR_ZwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736389860; a=rsa-sha256; cv=none; b=N7Ss9cpDTyA601l9DHmcS1lrm1MwTuBL6VoTVbiEHfeA3Pq0bcCFwOh3quRCwvwq5khceU 558JSY4GRg1xQ4UrtiAJ610eEdxUpDPx+LThXgEvbccitRt1CZ+iDRlglfwscviqgBxLML w5HmrBH3sJlkP2B8oMbGFJwoXGybaZ8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Wh4HgmeO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 34jR_ZwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=34jR_ZwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736389860; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7tJeInLbN8BEQlNwxZIs7XzIeMe2D6U4SFT5BJyi6RU=; b=jQOf9hzfKKv4eVYsKCBOI9yLsN8lXvjKmq8Fuf8GliffTxQcZvPMd8V+VaBTdXXCso3Wks +4anPh5epRs1zRObQz0iIDaQntMj8F3H2n0hGz3gtYtdH5h1c/axgQGH9w25Ir7AtkTJoR P7W86+HPwu1z0WkGTZdPjCNlOCIxMH0= Received: by mail-oa1-f73.google.com with SMTP id 586e51a60fabf-29fbc131cadso1005484fac.0 for ; Wed, 08 Jan 2025 18:30:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736389859; x=1736994659; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7tJeInLbN8BEQlNwxZIs7XzIeMe2D6U4SFT5BJyi6RU=; b=Wh4HgmeOOtFV1rTpou4jhVVgNKYfgoUqH8DUe83L9tfutTIjrp4t+PLWN6fzQV+CWM 6SBcXTM6Bya7+FC/USQY9X7QYBm8czE4dnf5JhOls4cPw3at9ANYSjLs5vfHsFS9xC1k kQz3qE/x70UeSomxxB2YkorgElgQzp9/a9GXmHtNb1HqCtetBsG1ZieEAfgY9ba64wGh 4WICFC3ZuccbDPuqVUF4sunqj2VFtnGbWyFvZ3/4/heyJIk+uXspMUuLnL4O1B0gFlMV kiPG7wCnWwVuQZVoAXM+AnkgIckSFvBlbSlPYc5UVxq6Pg4tG2iJcxUa9s4zAxhKlE+U Jmcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736389859; x=1736994659; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7tJeInLbN8BEQlNwxZIs7XzIeMe2D6U4SFT5BJyi6RU=; b=KTlsyUTZ1LOFMCTqvD20q36hCvYvyhJaxEUQIU+cFMighHJiWGNOzfpm48XA1h4V8F ftJPfz46YmiPqDp31IpnDhqPZ052iuYUGhvf9jubP/dFRBag541klY0f1e0fAw8A2phb uMNoqYeagW0HUhn6NaIuakG8fN3175vrbhw5KVTNFK1axTNdZhf5GTSZaugPhREc92z4 ZqqWfRggdGu448O9uzkPBLXBli/8TipiCNLIV7cznJyti+ZibXLByjUnHaX9xYFrzh2S 5r2o/GBhRnsdh98G8yXVV6+UeZ53A9dkaStUlcMXqzqL+RppUzwp2/S4eOHrjsjn8ZFA /wQg== X-Forwarded-Encrypted: i=1; AJvYcCXY3f5ZpxX4hMwtqSO4jmxxJ4egePW65sLsV5vhS+eTXmnLTmdmzZTj2KcWUH8sXZ5Gjv4ExNzKww==@kvack.org X-Gm-Message-State: AOJu0YyIbn6Q/67+yzRdNgdRBP4Hyf6nbUU43Hj289hIQmMHNPaN0nwn SRSfz56KTa7iiWa1TgHaPgENAMy0JigJsRhJHYJ3bxGIdsp6A0OQzX49X9cKv0xsgHTuDvyd0pP o3A== X-Google-Smtp-Source: AGHT+IHGMXCQEV8FC6jUm6LzXHjwAdfbc9NVyk/P0Er0Y7oYPXQ/fIgh5u42bhXHdFVsWWVl9KMqX04Ngxs= X-Received: from oabqt18.prod.google.com ([2002:a05:6870:6e12:b0:29f:e638:5c2c]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:7d8e:b0:29a:ea3b:a68e with SMTP id 586e51a60fabf-2aabe8aa92bmr945059fac.0.1736389858958; Wed, 08 Jan 2025 18:30:58 -0800 (PST) Date: Wed, 8 Jan 2025 18:30:23 -0800 In-Reply-To: <20250109023025.2242447-1-surenb@google.com> Mime-Version: 1.0 References: <20250109023025.2242447-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250109023025.2242447-15-surenb@google.com> Subject: [PATCH v8 14/16] mm: prepare lock_vma_under_rcu() for vma reuse possibility From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 058344000A X-Stat-Signature: 1zrrbimpe1nea9t6nyk7kznuat7hh1ky X-HE-Tag: 1736389859-42691 X-HE-Meta: U2FsdGVkX1/2BKEfsW90jaOswILqWd9S9q9jag8xjSRHzkEfrukA1SoShraiIUGiDAjWl+KK3u8/bLJCy57KQwz1voV9nPuL9amdEa8bgij1aWclqJwtJokO8R2jy77wxnauO8XIwHxudVBQVvuFfk17PS6q9SHnrVVQZx2Y9b80wCSAtn2cH40Te4lRd6ho2BIwYfnR85zF21IKkdiITgfsVAsoZR9pPk7cIpaKsZRpJKFzh9DxNhioSILj0x00Lumw2iRw/qf3IYYHT/PVgZ/hCcKmyQlC7JScqWX20NHiyp/SdHh/nLxAFTJs5fyEDXfNU8t20iMuRCzwnCh07cRbEs/pcSvZ0E/57/m2H5oRr5em7bR8yOx/u48LZ3LrxWTLzobt5BRipod39Obc4FVSkI7hOx/zKV1NWOC4XxkHRXFhqOSTahUvs0oT3vUpxd0nEnahgWTLwzaFiem4+51gAZud0BJoLNRHdoewTjjLMhaC9NctaYb/uifHjFyYd21Yp37w3zCpe7gWJM9+17AgA9p20RT67lQBKxV8E8u2RuieVfU5pL7PziIm3gbQVOz6XENg8a+mJGeRugTiWzyKZTsr03mPUXMxB0QFWtWfRyiTMmPbvZXdZ9sd7/V+HOmhvxWaU8ZVb8IAA7HIO6unwOiifF7BfwbKk2zcZzqtiUsv0d4XF9OSbEXkc+4O966nhmshbWYZzGa142Qs7toB/oV1mwbxqQ2QhgDMxtcqm63UuBIh4u6BLwLQOj/nM+ACmnVUvlhyRacB0CA0MX8Pi+5IbBE9k7KZ09n6ifFVrzp4b9LZGHDLM5s4wf96FRWefIucy3yedS6MsQD8G4XzLgZKyw51bRtRWnLlNRm4eECfnzavLEyFtD9ZXufm7RBWbNp/mxtNxePAEt/vt56GONWgPf/cyIQdzvoRPfGmotB8fqKe07z/MgUKhiukT24U0YM0jI6S/cXSog6 GMNE/aUO rMvxucteJAg/r2pDH8SptyQNoDJUX7e6SD7DK2f9P0X1fUyMhU9xoyx6515bCFJGSKl8ULWxqD28jpplXat+haACETpUIDQZaKcqFgryyQjNC8rbatfEDsB7J1Kc052/WNBT9gUt7DAHM2+0q/7UjlFNVM7eOHSrtjmu9Rnq0Aec8WKoJIVC1LfPljUqXFkxIqi/unhXHRr/+bcKvKLu5EowuZMbNOyy4dwoWfrJ2H8qXb3I/wkOO1OOtfVOSoa06QxJGRO4TrzlQmFaMmXg8akrLb1IpNVjuHhV8lFb4yozMMFe9exyxB6LoA8kLN5NTfwFli2PqsRpS8gNZSwZirfO+L6o7251kxI70cX+nJq8E85sOAPHnkfdMkwI07v0CcoOcQvOcvbLcVQBVp6p0KkXATZ+E/ByYWnWKQS97CoiIvcdJop9ThcZtTf99D150FuH7dtl22IIIHzxjbue7emh+xtEMLkH5cDPjowA+cxot7UObShNoo1quxTvTF6R2/zXC4uS+9NPHMnP4Se42eRcMH8Fvxo2ulo1oDAYU+Rnugyn/FRdKdDE9N66L8IOubIGpj7bX6RruKAZhhqp+SPOVv4lBMZe3vcfQQ1RFwMdhgPsDwS292TE9sTq46TKW318UXH/tkhk6qEhTt1lGhvnokU6byoman7Q1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 10 ++++++---- mm/memory.c | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index aca65cc0a26e..1d6b1563b956 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -737,8 +737,10 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { int oldcnt; @@ -749,7 +751,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; /* @@ -772,7 +774,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { vma_refcount_put(vma); return false; } @@ -906,7 +908,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index fe1b47c34052..a8e7e794178e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6465,7 +6465,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* @@ -6475,8 +6475,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock();