From patchwork Mon Feb 27 17:36:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32862C7EE23 for ; Mon, 27 Feb 2023 17:37:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA812280007; Mon, 27 Feb 2023 12:37:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A0E50280001; Mon, 27 Feb 2023 12:37:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88512280007; Mon, 27 Feb 2023 12:37:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 78F76280001 for ; Mon, 27 Feb 2023 12:37:34 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2177C80A2C for ; Mon, 27 Feb 2023 17:37:34 +0000 (UTC) X-FDA: 80513778828.13.C65C844 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 44A6618001D for ; Mon, 27 Feb 2023 17:37:32 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M2xu6PJm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=zjMrW5reaThVuk2dhZY4+w8TctGplRAeDE53t8jwuL7tJQQwPVfxYm6SDhUQopZ+XH/+/6 SYjE4QQUG9r0QMV3dR6eAXa/d566Ndwq4yvfLAj8K8VwW9ZGZ5LVvWym4/cHLPGkP58XVx 6tEawo4EdV/wJdhatDolA/4mAILTS98= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M2xu6PJm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W-r8YwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519452; a=rsa-sha256; cv=none; b=m0ejVuPJtDDdgkkAr1TNoJZp0RoniaesRwVo4qmLiSoXc5kESCCbRAN6GyLXDiadHW7mRy uRuYRSfUDJTOvQRzL1DMDs6zFrDKxtSh4YcwAseDZDMiacCQqTh7bfDD5OeVMzqycX0UKB CmT1Y9LZ2D4obePsbSQoYzevgI9ABNc= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-53700262a47so150876027b3.4 for ; Mon, 27 Feb 2023 09:37:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=M2xu6PJmgtYFCFjZtLFUAhDJwsRj1PNvn9tpCCynDkmhHKgKWgNMPKAbFeMB0d7vzQ aKVeG6bXcHl5yxZ6JEXjbkVB2BvokKvbI4LCTQCQPJySbeEhyr0HdzjFd7UV7XC52Nqd IM/Y4Y2oz4P4pwkohu9Bqul5qghmRgidkrq66aJWcYZe5Ah2frrlViyJFqWif5WyIuln lwhBindA8HTOjteANHg0VaUlc+wXF2i0bjESG2yKoDnx3/qzuFoEw7K8TYty79uXoeAT DG0JzJNCfRj4E5rbsNquxz9O5F/H/zOUYZHgnCU0K2ioTwSn7zpOFzVjAn3t3qu4Rbvy YYMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=WZEP7LhTOrS/NYuK8cQSqPtTYnhNjfDTBAC0G6HZvX0B3doLQV8BmwjrPqRvg2Z+Rr xu239Xvd3vOpxzg45DffpMFA+IKTxIB8zrwgseE5mOjlAAT5diEglV0v0iWtUW9K6JXS mDw9aCeSdckV8SLSzwJB9fzJZ8U60vlD5j0EM9JKB/SRHrGv5M1eGV1GdJZLSyg6td0c XKu//xCuSs4dufnial8AzXqp0gC1/ijgF5oSWoS4ZiNCbleo2Rfmvn9mvD7IL8FZX8Yu NzEP1nH3I5O5r/5k3Glar0l75aWvDB4Vb8quA+OACaMkr0prQWt2omEFmQpV9TKH+kIC 05lA== X-Gm-Message-State: AO0yUKU7M3Y0IM94v0tIJY8GnfFxamvX6LNU70vt/JkhP0Nl5Y8wpULe meVW3FeTELNSnALaqAk8Tv2OQ1sS3k4= X-Google-Smtp-Source: AK7set8ux+VAfJlkMlZWU+S8XCIwu1HZ1wVSlOGqhGsiL0aWWxFXwk2KW/a7dOnnyUwvPtqegO11CuQKxxM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:6cf:0:b0:a03:da3f:3e65 with SMTP id r15-20020a5b06cf000000b00a03da3f3e65mr9154608ybq.6.1677519451327; Mon, 27 Feb 2023 09:37:31 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:22 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-24-surenb@google.com> Subject: [PATCH v4 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 44A6618001D X-Stat-Signature: nk94po8o33958is6wwy54t3pxb9ofrd6 X-HE-Tag: 1677519452-589118 X-HE-Meta: U2FsdGVkX19GYoRBdAKUBfX8bBnpXN1kwHs3Ao5bbHHFP6rbCd8m3Sr0ce3gFe+zcdnbOJUmwdmsV3ybG+K13Aj7+07jla3TjLamOS/npJSCeRSiA6Bv9OSlLZYtOkJHopd3J/z76CmnDS7orEtB0TaeD27C1P1CXZiet9nzkl5l/1BpwHSFdMIRlZpil17k75whhQLl0FxYARLCsvyBO12TCxUTUoGR2xt8OmzgT4rDVsak82vS9OKNYte5BZE6+jLI38eF5MBkHsNQq5puXJs0KvFs/kLB3bJl0oOfmwtBKxkJE4kFpV6rgJFw4SwD4XW6GdDLYdPfqRlYd/hLms4508xXTExyzSeMCgGFKyUeFQr0U15AFsfLmTd9KBAgeDZuTzujKns4MzU5pMI7t+TFM3SAkj9ktqOkCXlMsYLFGGaV0Znl0JuwS1pXzfLfasVlIVoJHopQxwd4N5k9ljLzEG0mN8E2ctNtqwA+JzGfdToYamWW0aF8NMZ8O3yZr15Gb8UApTWITEhdNzWdni3ixaK2VoziYo1hxkbGQu01dUW2xPQ0d+FqGqQaQ8F08or9U7Uhy2tnzeklZH1oTCnkqxI02BZlPNjdiR+ZXUukTJSob+nTG0QXRwGS7CRTQCXqQP0+QKRpj1i28f7ynblyDS3gV+aMGy61JmoYoGNMpYD8Y4DiU4nah4AynhTsOFsyvzbNK+7rd7w5F7wlJb4jaOmBM2GjLUjve+vkNVNQV5JE3S6TKbjlHHgRXm8c0VI4rBXEPbWwBu2CAufAYkUVcP3usQF/arwWFtX2Cfk8jxik988RNwkyuaku6Pbc+/HjgiXlqYWHjT4SaqE8+LKZxO+mSKosbe2AfTB6i3ot8sHschs9lt0deOCx0mU8JzByIzTFs1xpSBtBP7nDBOAvKEi72IUCDwbnfXZfHlhRQzfyPei/ntbHudeg81A1ZZIEbTmXtvAcfDaPRmu 9vZMWlZx 3LTsQJUj9XIMroMYgJQmZYJREQFxhhVraBVHNmxbzwqNetLtKLBqe+5EfKzUySkSU7ERTIo+WOAUEqWG5G9ZJt7/oqEmBa7j+T0obSImXkJ+kHMkwKlt+pCbsYUmOYYjXxcW67n38l+5s5Av7b/AL4lVDC7wHpU31+njKI9RhME0elGQP5Q/lEzLQzJG1lyaLaLrMTrizGwAr/33BimGUL0ZrnKWNSX3vqzO0JfOp0sdKKCLyfjqyG7HPEvOr7+ufDM+BzbX7cFpxYTenPHoKGnKPLdFsCnks1d8zn69e6RIw/yD0Wzkx1HY+9sgBpT13fHGecIpFYFWN0Qp64gnustOMqK9PWsbQTuKc7SlgsO+UfeS9AT/nTZX7kmFgif+xOfI94s3TZAMruYs4KKbiByK50eACvk24vyITETrBWj1UQfwiM24YODimWVjoS8VbhRAmlC/03dvsIGVx7vfR6ansdPuB0+T4iIPmWdm58W2MKxgOtqikYEnvxG5Rc28F4URX50OuJpLFAUnY+VS1bhzPu9aIx7OhuX2MZSkzzf10a3ubGFM5kHvVfgS6OOXS4MN633Al1q/EQtIm+a1lqZjfW/RZmjnZSLS9vUedMTKCFo3HomfxNJOVsdKzKJeGHXj/oe7UsL1IeOt+SAWYaV0ZK1evMnGL1FtQWdq87HbyYWE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 895bb3950e8a..46d2db743b1a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -701,6 +701,9 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) vma->detached = detached; } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index f7f412833e42..bda4c1a991f0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5221,6 +5221,52 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + + rcu_read_lock(); +retry: + vma = mas_walk(&mas); + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table.