From patchwork Thu Sep 1 17:34:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06A4EECAAD3 for ; Thu, 1 Sep 2022 17:35:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A87580017; Thu, 1 Sep 2022 13:35:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 758F68000D; Thu, 1 Sep 2022 13:35:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D21D80017; Thu, 1 Sep 2022 13:35:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4FBF28000D for ; Thu, 1 Sep 2022 13:35:25 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 131411A104B for ; Thu, 1 Sep 2022 17:35:25 +0000 (UTC) X-FDA: 79864218210.18.51F4D99 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id AEE8B14005A for ; Thu, 1 Sep 2022 17:35:24 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-3449870f0e4so11662767b3.6 for ; Thu, 01 Sep 2022 10:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=SI5DHC7dpiRJVHhhWYV+lwBh2iwV9A0Ou3FEYYZKTIo=; b=ouDtFH7Jsoj5d0amZ6w++vn1JIcEBsAg0+Jc3jSu1CI/YwtCjIjWCXa3h1daFKHXKc cShsEzXCG6pl7Qo8JUwfzrGY3LljVOQU7WP+kQRk9jvRvgn5CYGLdVxDx9mvrs6u6z8K /gtKC5pqelI7Ja6YPAH2266yNgoWGngIuBTy9e4Ky4U5csa2GQgfjUrkug/tPpsGFles yx5i2EqiywQtBnfaWAJzlm/RHBLABr6GNuXXMOHeNq811kfvPtT71YKWoN7/bLzrrmPy z0tKwDEKWW16qZuqOrYHGA9KvA/VUBzIFMBe0AdL5oZh7HL1RXGViqdJIuZcrmP9S3mv xPFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=SI5DHC7dpiRJVHhhWYV+lwBh2iwV9A0Ou3FEYYZKTIo=; b=yCxUZZYzVamRrTLypWxh7Mub7Vhxfq2/Eku/RqtGUI2IuzkG3CAbkEvfw1Ci3POBJP 7EoNPou6rZVPIvDLcSwR1dknNQab4kA3XEpJI08umrwK/JdT4wajwpFmBbyivf6u4cvs VkOdt1HoBst3Muyw3tOUjpCMRzNora6ucqVwIpZ0NbEImsmt5x/kctZpUzuBGhuYeKor SKH4lX/BmN4E4kTXvd5ZzUqsBeFthXwxuKbsY5YpaNi6hw3uzCjhxkJsjAhRm69duLGC ZHBK0MR27LEW41ORB5aToKSW2+s0/iu8wnHxwA3XUoeyS4E18gBAmPcOLKrewU1rFxmA r1fA== X-Gm-Message-State: ACgBeo0odHlR0BpWRWkpO8VNS7HJgL6egPZFmgMuijXwD30jtJokuha5 NvIcfj/3+Lj+R+KI48fjIKe/2B/kPgo= X-Google-Smtp-Source: AA6agR4TWv71WCFs9L88lUEt8sy6dDZVYBPPgP4/10a2Bx/Ool+t8WQWT1a/Z5ySDpeCnHjOLWwcwNW6snI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:8e8f:0:b0:691:9579:d3c5 with SMTP id q15-20020a258e8f000000b006919579d3c5mr19748036ybl.249.1662053723940; Thu, 01 Sep 2022 10:35:23 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:49 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-2-surenb@google.com> Subject: [RFC PATCH RESEND 01/28] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053724; a=rsa-sha256; cv=none; b=ON615n7ywFqgmqffUxTFd3RORlDRe/NEmU9K2NwzO9GeEf/0t+o5lerPvBxUCtHzIaZFWx Gsmw0zZ6gwCO2bdUJXuRqC0zbvBHzCRpGJ3vLHNKOikfqvqGAjgyvxXAIhBZlZxXFK2riH fs0I0eMo5uLi9b2iHhPtuMAqxxcN1KY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ouDtFH7J; spf=pass (imf23.hostedemail.com: domain of 3W-0QYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W-0QYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SI5DHC7dpiRJVHhhWYV+lwBh2iwV9A0Ou3FEYYZKTIo=; b=zoAut5Z1gsT4Yk/l7bcZSqFFsUhDWunZvjyiTRmlpHetJrXZT/A3bxbMt59DmO49w5WQAO druUoj6rUk0j88dW/zW+BbYWACmmLV89HkPbf96RSMRX871F6nP9CUIJVyYn1s87Q2cp06 BXtWhcBjqSTA7XbbR3zWLu9Yg/B9JpA= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ouDtFH7J; spf=pass (imf23.hostedemail.com: domain of 3W-0QYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W-0QYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: AEE8B14005A X-Stat-Signature: iet8185u66ymjqeob9pku9y94mp85e7c X-HE-Tag: 1662053724-202857 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..58c20fad9cf9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1124,6 +1124,19 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Thu Sep 1 17:34:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B84A1ECAAD3 for ; Thu, 1 Sep 2022 17:35:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5382580018; Thu, 1 Sep 2022 13:35:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E8938000D; Thu, 1 Sep 2022 13:35:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3885780018; Thu, 1 Sep 2022 13:35:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2B6AC8000D for ; Thu, 1 Sep 2022 13:35:28 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F193B40FD6 for ; Thu, 1 Sep 2022 17:35:27 +0000 (UTC) X-FDA: 79864218294.18.E808012 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 7846E10003F for ; Thu, 1 Sep 2022 17:35:27 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id s15-20020a5b044f000000b00680c4eb89f1so4878703ybp.7 for ; Thu, 01 Sep 2022 10:35:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=p6utfGSGU2ZWrUa6c0eUAGSz6ILgySPKVkHsaiIDl8M=; b=LbkT/AkX1WcT/2fyJlqSXegH2trsBNyd6QrbYfZ/YY10o0bOfV+Ix+uJLegE6lSF35 ne85BxCuyx7lBpnSiZ42TY86RYtZR3AAc5vYrFsx4gBxgTwamIONmdTxtR/0iKqdUJEh gCGpqjCaOQV7LzbqDjbUJKDXfAR5/j45YmmGmhxu+HXk+Hwd00xyXauy+PEzZWtLyw4U rqIn+41hVfJNUiIovZy6pL0FSV03X1G3rIrczW6PRcfgw8lXvG9gGaQmr8gUZrra+vXa CCuSzq9IAYSPtY4Wqdt70146omIRNbSeU/rl39Xw+3e0A38BgKuGXtDi12EMKnqTbGy3 VnZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=p6utfGSGU2ZWrUa6c0eUAGSz6ILgySPKVkHsaiIDl8M=; b=ny1HITUU/AP6bYOiwE2jRGBF+yWD+2ER3D2qEIYrnNYc52Lm2P6Rs4xW/1Exznz6hp uaeSwjMWag6pw4PZseS9Ggyb2ONNmbsVTTAv52Oxmb0b39cI9FCHOFRniBMU2xRFROlx N2R1G7Rbwuaq3B4jm13rnBu4zGbtqxcYpF8JHAvb/Hee/NAh/gvboxe4MH96+jwOZHIa d89Uo7ZVuE7Y21l0swa0bEptMu66yiA8c09C1LmT0WRObXmSwD5Un5oh4Q90tSbOrUQa Xm55G4j/K+dQzTj5OQxujrxN3XLXS4e7izsVpwvkcC0jxFw3NOc7+Lpq9iQ9uBGO4HNU Wq0g== X-Gm-Message-State: ACgBeo3uH94nXdFzaQR33eX9IB3SEiFT+4zonNVWOlWakPA86WAz0ikU wRZkQtBKXPG/7U8iTxNgqzSyOEhEBmXToYA5YX+BMdSNITlMLN9+uAyYQRmf6exkohbuuuf6ls4 cPsCXG1U= X-Google-Smtp-Source: AA6agR4JzlVLX4tUxHHNIbaNOZW1dz2wXadaTbKwNizSq+Zw9LDNKeWUkMtSnrbieYwXe2AgTTBR2su3VSU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:5cc4:0:b0:327:59f2:d673 with SMTP id q187-20020a815cc4000000b0032759f2d673mr22886370ywb.272.1662053726626; Thu, 01 Sep 2022 10:35:26 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:50 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-3-surenb@google.com> Subject: [RFC PATCH RESEND 02/28] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053727; a=rsa-sha256; cv=none; b=yBCYY0MRSiVaaygYIc0kP2GyjGuC0qWGhZXphHhmSnCRRUmQbQayLHXasYl9PP9sQqgWWO ZWaQh4GW3n2kfX6RkkGxo5Hzwj9mqFKJp7DIOM9P8Zbieg99rCnqwn+m/jVTuhA9Ek0QEX 4Jjn56ScXdKdRqSPTWUmeoxL+r9zfZg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="LbkT/AkX"; spf=pass (imf14.hostedemail.com: domain of 3Xu0QYwYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Xu0QYwYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053727; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p6utfGSGU2ZWrUa6c0eUAGSz6ILgySPKVkHsaiIDl8M=; b=E4R8FQGEU4SMykgrGjsnFNDkuA4XQGn/tbMpT6sTcL62O7fGjODrqgdlSF9x4L3Dv9HJ82 Nk63IexFhYcvDg6Cms+q4tgiJ/7v1oPuDE2WQpzGsTWvKSa9Nlt3RAZKni0FPhR11y48Ln 8U/sccmkFo+atGup3zVtLzUGMfsCzD4= X-Stat-Signature: dxxnef53kcfecpjf7p666qhj5oz6bf9e X-Rspam-User: X-Rspamd-Queue-Id: 7846E10003F X-Rspamd-Server: rspam07 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="LbkT/AkX"; spf=pass (imf14.hostedemail.com: domain of 3Xu0QYwYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Xu0QYwYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1662053727-135976 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..bed25ef7c994 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -403,12 +403,18 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct rb_node vm_rb; diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..614872438393 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,10 +481,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Thu Sep 1 17:34:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EF8DECAAD5 for ; Thu, 1 Sep 2022 17:35:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97E2D80019; Thu, 1 Sep 2022 13:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 905C78000D; Thu, 1 Sep 2022 13:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CDF180019; Thu, 1 Sep 2022 13:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6E8A78000D for ; Thu, 1 Sep 2022 13:35:30 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4F3DA121073 for ; Thu, 1 Sep 2022 17:35:30 +0000 (UTC) X-FDA: 79864218420.25.C81923B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf01.hostedemail.com (Postfix) with ESMTP id EE6AD40053 for ; Thu, 1 Sep 2022 17:35:29 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id k13-20020a056902024d00b0066fa7f50b97so4870775ybs.6 for ; Thu, 01 Sep 2022 10:35:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=LVrV0vnZTvQ/veouCVIJ2VKTA1Jw28r4MCl6DiRcaDY=; b=l1V4yVONtNOUfesG20Bnq72dRtkCeRcz+SXGtADVXwnLsewU/bV2Wj+5K0ek8rbJL5 /VnIb4yjvzQAA64ppyeiv2tN7SgqNypFCNDrFPFTIYKrt4K6WTB+yC3yHFvMB+ausBHt SSGrmorbtqBk+v9sXJa+cknHYK7SdsAdZfIlHKdNs08dvIfwUr5y5G+bCOuk9GDx3jZe o3Fb422GusB/mLwU2fWqp5J3gMNLaE1B/UaEPdikTdaz6ywBZryiSPL/0bdT3F8UE3iA 1RBsfylmTB6McidRxeHwpl546i9VITPOIoqZ/5tnno4M6ww922Mvm4adk4fXxTHJ359z mtIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=LVrV0vnZTvQ/veouCVIJ2VKTA1Jw28r4MCl6DiRcaDY=; b=rMd2IwEPYK2htuSUoIX3j1UceG0YHJbJ4KvE7J9fn1geotT2D+SZIIPkbKqyZxd2B0 QwaVuRIY3p1gcl94IzGoF9wwVfeNWHesdxx7HhdVrDvRwUkHB0yHUyGWq5ZaZCuUXMGD v5SdRoYuAX61VcCeI2RGjneo747DPt/dde2iHLcmEm78Iwcq/PmLLkZ1ghdD9hStns+1 qS0N4j3kZxMVQ01mfYiq5sLpetKrU/V50321zEftUBrhZIQGlywVigUBQCFDHnAU01hB ensAx4FUlbI0/Ajr1xWAkdhIMoNqx+Vgr565IgWEsPKUJQnkHMONkwwXOtNFm5glCB3q 1FNw== X-Gm-Message-State: ACgBeo3eOTK8NWOSsdJUMRpxBBwkItt43EGh8SsSNweTPKFDVxdAm9ny 3tGiRkHsy9/MCwx9a3kcrQyo8e0m8cA= X-Google-Smtp-Source: AA6agR5xL9cSC69c+1y/uWKTqLFokWxQZenGw3Ln10dS2nyve2acvx9iJCiS8dOE4h1muByL940MlpUFLCY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:25cf:0:b0:6a2:fe22:657 with SMTP id l198-20020a2525cf000000b006a2fe220657mr784117ybl.481.1662053729233; Thu, 01 Sep 2022 10:35:29 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:51 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-4-surenb@google.com> Subject: [RFC PATCH RESEND 03/28] mm: introduce __find_vma to be used without mmap_lock protection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053730; a=rsa-sha256; cv=none; b=1JOqvHZXSLQeXL6ANZ6oRxF3d9Znn4nXnB0KBThN6YoJiYVKf316qdloZZZg6FK2usdYI9 FUksZj6GujvKBYVRO5ewvk9uIl+NS/cvWkYhny+ONgqSnxHGetSsr5leL6ea8Pbg/u7zfv JMGVlCqPKJK7dA8mN3bfZAjAJprASp4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=l1V4yVON; spf=pass (imf01.hostedemail.com: domain of 3Ye0QYwYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Ye0QYwYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053730; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LVrV0vnZTvQ/veouCVIJ2VKTA1Jw28r4MCl6DiRcaDY=; b=r2hHoisNhci9iM4FQOkGalRoq80zidgWBF98cxQr8fGoS9u1E9isnwfRuJcGiTUkk+jvms WcuLycuUclWGXWpRzIiOJ95D+9wiaKPgzHRaae7ZqlKRQatP5LRVlb85+FG8U/ozhRAXti ouiHRgbzzVy0ju3gbWnNALS7oZ76EQs= X-Rspamd-Queue-Id: EE6AD40053 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=l1V4yVON; spf=pass (imf01.hostedemail.com: domain of 3Ye0QYwYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Ye0QYwYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Stat-Signature: 7sxb74qhbhhk1zmrzpff7km6npmpkjam X-HE-Tag: 1662053729-726417 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add __find_vma function to be used for VMA lookup under rcu protection. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kent Overstreet --- drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++-- include/linux/mm.h | 9 ++++++++- mm/mmap.c | 6 ++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c index 32e92651ef7c..fc94985c95c8 100644 --- a/drivers/gpu/drm/i915/i915_gpu_error.c +++ b/drivers/gpu/drm/i915/i915_gpu_error.c @@ -507,7 +507,7 @@ static void error_print_context(struct drm_i915_error_state_buf *m, } static struct i915_vma_coredump * -__find_vma(struct i915_vma_coredump *vma, const char *name) +__i915_find_vma(struct i915_vma_coredump *vma, const char *name) { while (vma) { if (strcmp(vma->name, name) == 0) @@ -521,7 +521,7 @@ __find_vma(struct i915_vma_coredump *vma, const char *name) struct i915_vma_coredump * intel_gpu_error_find_batch(const struct intel_engine_coredump *ee) { - return __find_vma(ee->vma, "batch"); + return __i915_find_vma(ee->vma, "batch"); } static void error_print_engine(struct drm_i915_error_state_buf *m, diff --git a/include/linux/mm.h b/include/linux/mm.h index 21f8b27bd9fd..7d322a979455 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2712,7 +2712,14 @@ extern int expand_upwards(struct vm_area_struct *vma, unsigned long address); #endif /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long addr); +extern struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr); +static inline +struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +{ + mmap_assert_locked(mm); + return __find_vma(mm, addr); +} + extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr, struct vm_area_struct **pprev); diff --git a/mm/mmap.c b/mm/mmap.c index 9d780f415be3..693e6776be39 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2250,12 +2250,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr) { struct rb_node *rb_node; struct vm_area_struct *vma; - mmap_assert_locked(mm); /* Check the cache first. */ vma = vmacache_find(mm, addr); if (likely(vma)) @@ -2281,8 +2280,7 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) vmacache_update(addr, vma); return vma; } - -EXPORT_SYMBOL(find_vma); +EXPORT_SYMBOL(__find_vma); /* * Same as find_vma, but also return a pointer to the previous VMA in *pprev. From patchwork Thu Sep 1 17:34:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A07F9ECAAD3 for ; Thu, 1 Sep 2022 17:35:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4263E8001A; Thu, 1 Sep 2022 13:35:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D58D8000D; Thu, 1 Sep 2022 13:35:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29E178001A; Thu, 1 Sep 2022 13:35:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 19BAC8000D for ; Thu, 1 Sep 2022 13:35:33 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DE47D14105A for ; Thu, 1 Sep 2022 17:35:32 +0000 (UTC) X-FDA: 79864218504.29.D1A2CD1 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 9D19920032 for ; Thu, 1 Sep 2022 17:35:32 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-340e618b145so166693217b3.2 for ; Thu, 01 Sep 2022 10:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=SmLlv5S95s1SYD/hq46yY1KSJ7CCwWbvNekA+AzNinI=; b=IEnDpG3YZTPCaLyxObjk7amCPTURBOnqU7bJ+HMdi+ub5B49wgcvI5S5rtmD90rv2Q AZvqZR25+q4mnRP5jUchvAffVc/FzpkmhNDkv4FRnlMM4OlY6itcjxw3UZXpfpnem++C EJWIMKaiqyYuCH4R4X9b15IgDB9xODniSPpGeOsiXfRmLSll0Bi1jUHU15lpXlXRSc5J +CLYoEdfwTQrLCZejly8sqYsr7A+RsOmJlAwK6fhbCZlwscLc4WTaIF87eXTXBY8BoJB i633PKqUr/jul5ZT9UiyTCl07dJq2r9+I6Xxo/FEUAli95MNIWyATvVTZ7NKsjxM0gu/ 3dZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=SmLlv5S95s1SYD/hq46yY1KSJ7CCwWbvNekA+AzNinI=; b=Hi0ikPnumKqK62fAwdG0q9Em1jpfAX47jQqMM78PTUMs5ISygx/vb2MDtPaDt297a6 fssPt1yUVxQZf2f7o6X/2M/ysmjBjtUcCWXgYhs7pyERUKDmihCJRU0aQL2bf+G2Lvca f5b4JFAz3qJPMs6ZdA7KsxVgUVDqCLlio4Wb1RyAMAxBzlPspG9gNnAFD3/lbOqTBX6m hhWuFz7Gj5ThS3wKMl/Xp9Jsqgce5XBZWQAniwtwouwfOPAahPHWJ2xWFBxSZXOQ+yRb IuF38glKtnVdZ3ycZTsswUPWbwa+mjPgOtvDSKVnOQc7wdS0VIVKPyfigwcsYRC7yuR2 331g== X-Gm-Message-State: ACgBeo3aOB3x0790r2JpdJVP3E9gRv9oijbTM0Dfe2Do0wV64Qjv0JAR is0jOaACRFoM8S4FB/Hu1OLpJbhiGZI= X-Google-Smtp-Source: AA6agR7LHtl/RnUFMM8Ba/UDLwcr+KKZqP3s3F3o0L7ZSX4nHIA8V0azyARB9tbw+yLexNRTedxLJOpzAXk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:1366:b0:691:4335:455b with SMTP id bt6-20020a056902136600b006914335455bmr19454643ybb.282.1662053731874; Thu, 01 Sep 2022 10:35:31 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:52 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-5-surenb@google.com> Subject: [RFC PATCH RESEND 04/28] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053732; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SmLlv5S95s1SYD/hq46yY1KSJ7CCwWbvNekA+AzNinI=; b=3N0nRr7swV83WzdEiEKw4m42/OOWgQbjv/iaSNmMkvA3gx1TUGyduG7gGH73Vg0sbUL4CD 56YLEQnkl1DnqPB17kauT3ysMlvC40u76EEfgccqeR1pbs7cgqYNuFPBSYUwuLGCqXVPgI OsE+FPm0Hph0dn6pykY+pUXoKCrCSTw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IEnDpG3Y; spf=pass (imf13.hostedemail.com: domain of 3Y-0QYwYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Y-0QYwYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053732; a=rsa-sha256; cv=none; b=JMhlQYOsw19ka8V/dj1qBjjyAaA0nHMmx6U0T17pYOlcu/zM+skKD80V7YIWVE96JHl/wb ZsLC7F5IrEjZ7O4DgROTx3J/sSXwON4lmNttHXspOTZqkbmkDNlxu2gL1nOQg4fzlXeogA 8zrS9nVC3w1Yeksd6pDpOryLzDSFbTg= Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IEnDpG3Y; spf=pass (imf13.hostedemail.com: domain of 3Y-0QYwYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Y-0QYwYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: 5sjkjqz5remowqwadmqdtkmrmsrr553x X-Rspamd-Queue-Id: 9D19920032 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053732-619380 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Thu Sep 1 17:34:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C4D9ECAAD1 for ; Thu, 1 Sep 2022 17:35:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEB5A8001B; Thu, 1 Sep 2022 13:35:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9C898000D; Thu, 1 Sep 2022 13:35:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C88D68001B; Thu, 1 Sep 2022 13:35:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B97318000D for ; Thu, 1 Sep 2022 13:35:35 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8BC9EAB75E for ; Thu, 1 Sep 2022 17:35:35 +0000 (UTC) X-FDA: 79864218630.14.976367C Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 3EEF4140059 for ; Thu, 1 Sep 2022 17:35:35 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33ef3e5faeeso228801227b3.0 for ; Thu, 01 Sep 2022 10:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=eIMt0K4Dq/+E4lj9SQ/fnMIDQQYvNY/8MPu6+maRPr0=; b=OULzLK//lObTRMGAKryDz/AAItdyiK5yqhZqIIxu+sL1Dieokt2jidmnz3C7/IkvFZ m9XlR/LP/dWBo2UxnXyYn7qdVCzxgLbSFD2CG9EOYY1gruRvre30gd8M27oZrFHurKvO KqLi6QjNq7f2EcLrXhJZo+HJMeLkP9RVRxlBnBeo9kXpz7L5PtGPdJWGZy8ptDnDibsD BU++Zg9dqZVio0ctwQpkNxICzBLOlMkeBzfVJBfO5ajkBVunOWr6F+myD0p5P9YWiss6 hO75UDowCTjrDDdzDEi22CFFlp+R1gx4mTcBiehvp3OGDbyQbHz6/QfMRzJJk1Rc6B1v GPRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=eIMt0K4Dq/+E4lj9SQ/fnMIDQQYvNY/8MPu6+maRPr0=; b=uATto1ia5wqlpHmEI5rTYo3Raz4yZm5NH3GaeRMiVem2gwLTgGkUzDSLPIDHuOohNk HfbbsSwblpMavkJ6koYH48jC+sWEcBHLo4gqDJjT4XjvWHKQamHV0Hk801DiFxCdzEyT QCIu1iTQim0VM4ZC+EjjELh44Y5JPb4AHUDnj048RYAfvZhOI0lod+R9UdXkXRCmV4ua /qmcX9IWYKNtxsgqeLNZqI9LVhYtxgdkki+2Gtz3+tP6XJ7m69shqrpgCxfXYhcA7+RH ClBxCbk+eA2bcebcdJeZfgGZ8Z62YM7FyJ33kJOOQUKRIfYsVFroIFaRocLb3gfZ8Hfb 6/Rw== X-Gm-Message-State: ACgBeo38Vni8E0Hj9x7/0w+dMJIjnV0zirfQkiGRgygi+QPycbtDIsy/ Mq03tbQ+YyHAEjBCfShBgAa1Z6VQcpzwEc22bq5Q3XJ2unVq29ZKZEr3Jk/j4xZDZXjxpxVYIWW LHydlEEo= X-Google-Smtp-Source: AA6agR57KkNIgUFHWPdFiQWyoY7oyjG1ZvVgnOFqYQ9kJCo94JBJEo43qz2wa5ev7XB45jgLE1aOSNe2TR4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a0d:ff86:0:b0:341:5844:5527 with SMTP id p128-20020a0dff86000000b0034158445527mr14770867ywf.504.1662053734514; Thu, 01 Sep 2022 10:35:34 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:53 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-6-surenb@google.com> Subject: [RFC PATCH RESEND 05/28] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eIMt0K4Dq/+E4lj9SQ/fnMIDQQYvNY/8MPu6+maRPr0=; b=K+sUrYK19S7fOJa36TFxUjJI35DhbbYC1IJJ/k6EdxyFrYw6qtqNaOvYsy09JJ1c9jWItk OEoSm8M0RrIyHrFccDitWebRgkHp4ei0gkiDyETSugCRz6ZVLNzhoel6QdZbmohaPDDpp4 elM+U/zPgTZDhiJHctSVuZZbZDerFlk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="OULzLK//"; spf=pass (imf23.hostedemail.com: domain of 3Zu0QYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Zu0QYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053735; a=rsa-sha256; cv=none; b=yrSiW071zBOBRsV6RT8A59g2ePH6POm9QMI+AlsFQqrTUdgckhQlTNGc6UkRgFxpxJlO9Q wWONZUzfbfR+8ZNpRHN+eQuA0wq8auR3j3kXMIXd+RI/x8cmE7J9v7Mjbz1LcPd+XoHByq G/u8wCODLY6WeJ1Rmnf9Ydq3dbyNZT4= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="OULzLK//"; spf=pass (imf23.hostedemail.com: domain of 3Zu0QYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Zu0QYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: 5s7f9hthahnttkcy65d8dejziwdos6i7 X-Rspamd-Queue-Id: 3EEF4140059 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053735-748439 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 78 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 7 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 105 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d322a979455..476bf936c5f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,83 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +static inline void vma_mark_locked(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result but it's not critical. + * False unlocked result is critical but is impossible because we + * modify and check vma->vm_lock_seq under vma->lock protection and + * mm->mm_lock_seq modification invalidates all existing locks. + */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_locked(struct vm_area_struct *vma) +{ + lockdep_assert_held(&vma->lock); + VM_BUG_ON_VMA(!rwsem_is_locked(&vma->lock), vma); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_mark_locked(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +696,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bed25ef7c994..6a03f59c1e78 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -486,6 +486,10 @@ struct vm_area_struct { struct mempolicy *vm_policy; /* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + struct rw_semaphore lock; + int vm_lock_seq; +#endif } __randomize_layout; struct kioctx_table; @@ -567,6 +571,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..a391ae226564 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_mark_unlocked_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_mark_unlocked_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_mark_unlocked_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_mark_unlocked_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 614872438393..bfab31ecd11e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -475,6 +475,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); new->vm_next = new->vm_prev = NULL; dup_anon_vma_name(orig, new); } @@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index fbe7844d0912..8399f90d631c 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Thu Sep 1 17:34:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16CFCECAAD1 for ; Thu, 1 Sep 2022 17:35:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8FD68001C; Thu, 1 Sep 2022 13:35:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3EF48000D; Thu, 1 Sep 2022 13:35:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 891A28001C; Thu, 1 Sep 2022 13:35:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7B6038000D for ; Thu, 1 Sep 2022 13:35:38 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4D18D1A1020 for ; Thu, 1 Sep 2022 17:35:38 +0000 (UTC) X-FDA: 79864218756.10.D11AA82 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 0373F14005E for ; Thu, 1 Sep 2022 17:35:37 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-340ae84fb7dso202783907b3.17 for ; Thu, 01 Sep 2022 10:35:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OSuAy3hdf+ZsQNxs3kxuMbhy9QBzPPm6GtoQo3aB3NA=; b=KO0CCoeJjfr6uQrGAGyvNgiswHns8WtW+vNCGbmb+v3lgrTdGqCfv4Q0htUhCqi7L8 NjudqvxWt3vJ1U4f4TaUrpiWDGDkYHTslXKwJR5AqzCzKMPyrwODeBVbDXy2W3OJ+0AJ 32abPt60ZQu7g2dOidue0bSh6Xh6XjE19T0FxDLe4AuFdEJaS9z9AXmzzaZHqpBfL4WP CC3bKhkCnU66eINFmY1rA50iR0WfFedon1RB2Q6DxPGeP1j32wzMpXKoaqT2OAwWSdUJ 2Dr+2H3hwJpx9s6tPPyXHohy9BSbHgL+aey4lgZu/S9fAtQBjK/jZiqIlpp1Ch7Jebxy Blpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OSuAy3hdf+ZsQNxs3kxuMbhy9QBzPPm6GtoQo3aB3NA=; b=7c9YP9cfMFw+7dgxRAFWjj7AJevOioUieZHEV1MUhsoKJb0UOjYIu9D2j5NSoFZMSC 7lCg4waVfIKlOtITF8gKyzlCEUe/n6yHe9+HiJtwHm1CyvGkFqIpt2LukLztXS4ONIUG soc0G/dVdReGv82VLRMcceYE9HSRbJDZ3P8xcz+mrRvbFQwxYoRSm811I6lmuwCX+B3R dCn6rRjpfX/NLgaumpRKsAoBIpUB8Gus31WiQIVFs1a3Rt5NrezZ+HaOK5/oSlqAlq5N MOBVhtISRR2A35UXrRxU2zu/iQl8jHzI+PrGH/TmAYx6W/0/fZiR9OUOYF+hkTG7I9Va +ocw== X-Gm-Message-State: ACgBeo3B/7Ta5JNgjondAQRp4pDRum0KxbbI7votfKHU6HNXmfjWxyHk iguZvg7rnNZOY8YI3qiZV92Ukh701hI= X-Google-Smtp-Source: AA6agR5mOhnOZ5TXGA7QbMyhBcTFpfMYAPy+7Q1nZkdORl1Ru4vF2e+A26u5Wht4tIYfxlBv6SIkkjkdNGw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a0d:edc2:0:b0:338:957:a719 with SMTP id w185-20020a0dedc2000000b003380957a719mr24916063ywe.132.1662053737185; Thu, 01 Sep 2022 10:35:37 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:54 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-7-surenb@google.com> Subject: [RFC PATCH RESEND 06/28] mm: mark VMA as locked whenever vma->vm_flags are modified From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OSuAy3hdf+ZsQNxs3kxuMbhy9QBzPPm6GtoQo3aB3NA=; b=kYDDGlDqPV7W9pXCV/L46XmcNF63rm5yOd5ThAUXD04/qy1s/DvqqPfW4GYLLY8Pdnoei+ 8btLlWDxnqIFLrUAKhmsZcLc8rVZkVHFvPOMynv83S2DoRugWhnB4tNamZ43KximsNicxq wVh2+qed3xwI0YdcG8Au+8A7gsXnfIA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KO0CCoeJ; spf=pass (imf09.hostedemail.com: domain of 3ae0QYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ae0QYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053738; a=rsa-sha256; cv=none; b=R/BK4JWAoKJ9LUlNVza8qkuM4+tLSaSMwguU9xQbSKTlY+fTEx0hrubfFj25hB2vGFiBZs yh6cvwXa9A7MAu5/sfrU293r6rwtEVpqAaoJMyA5g9wqRrxJvny5/pZleNtydPyIkWAPcM MgJRm08pEK2HIMgWEIyGquehNeAxgrk= X-Stat-Signature: m8bjnq6b4thdm5w5j1dpbpbzth5atgzw X-Rspamd-Queue-Id: 0373F14005E Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KO0CCoeJ; spf=pass (imf09.hostedemail.com: domain of 3ae0QYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ae0QYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1662053737-847018 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: VMA flag modifications should be done under VMA lock to prevent concurrent page fault handling in that area. Signed-off-by: Suren Baghdasaryan --- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++++++ mm/madvise.c | 1 + mm/mlock.c | 2 ++ mm/mmap.c | 1 + mm/mprotect.c | 1 + 6 files changed, 12 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 4e0023643f8b..ceffa5c2c650 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1285,6 +1285,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, for (vma = mm->mmap; vma; vma = vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; + vma_mark_locked(vma); vma->vm_flags &= ~VM_SOFTDIRTY; vma_set_page_prot(vma); } diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 175de70e3adf..fe557b3d1c07 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -620,6 +620,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -653,6 +654,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; return 0; @@ -734,6 +736,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -891,6 +894,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma = prev; else prev = vma; + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1449,6 +1453,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1630,6 +1635,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..a173f0025abd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -181,6 +181,7 @@ static int madvise_update_vma(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_lock held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; if (!vma->vm_file) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index b14e929084cc..f62e1a4d05f2 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,6 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_mark_locked(vma); WRITE_ONCE(vma->vm_flags, newflags); lru_add_drain(); @@ -456,6 +457,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_mark_locked(vma); vma->vm_flags = newflags; } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mmap.c b/mm/mmap.c index 693e6776be39..f89c9b058105 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1818,6 +1818,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, out: perf_event_mmap(vma); + vma_mark_locked(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); if (vm_flags & VM_LOCKED) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || diff --git a/mm/mprotect.c b/mm/mprotect.c index bc6bddd156ca..df47fc21b0e4 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -621,6 +621,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = newflags; /* * We want to check manually if we can change individual PTEs writable From patchwork Thu Sep 1 17:34:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F278AECAAD3 for ; Thu, 1 Sep 2022 17:35:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 956B98001D; Thu, 1 Sep 2022 13:35:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9062A8000D; Thu, 1 Sep 2022 13:35:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CEA88001D; Thu, 1 Sep 2022 13:35:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6E1CB8000D for ; Thu, 1 Sep 2022 13:35:41 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4C0DA121089 for ; Thu, 1 Sep 2022 17:35:41 +0000 (UTC) X-FDA: 79864218882.21.019579E Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id EBBC24003C for ; Thu, 1 Sep 2022 17:35:40 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33daeaa6b8eso233290647b3.7 for ; Thu, 01 Sep 2022 10:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=byYdzgQ2nb1BwMzAK8Db5EPtebwswE17va5vsTNHDvI=; b=U5hxVmDFnvg3Zh8mYi2DwtWs5BxWMTUQESfFDYoN6r6izeiWA5yaRxl1nHFUTbwk// Dnd8HHqghiMd97Zw1zIzmeB4ars9SXmdp/qLpi9efptTbpKgnXouiU6jw117QN+WP9qs 3UWSwrzjzVFJ688YmwTVx+XNrjpewHxiYcGpgCH/HREdk2crWCVRYVVfKgVwMdBm3Bmp 0cmyfTq+1Is86HcJ8rQc8vT+aBrCdpHLLyCE4j/1ArHGf12SJ516YZIyRc8DbZTCjB20 ojn6KPPAcVM9EzDOR735/i5/Hcf8Q/BBi2k53VcG0tY24s/K8Lqz5/8Ca9tGedcRR97p PE0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=byYdzgQ2nb1BwMzAK8Db5EPtebwswE17va5vsTNHDvI=; b=P6cvAQAn3D1IFOwnkT9CyvBeF7mkUZfhI+2n0CUd4e4lOs45j0oO6Kwwnu+okAskJI klUJXTbbyudCsaVCbpPSzgvMIU2Y0YfKsRdxN8bEZ5tuS3Xiwxm4v6E97BeQ3s+mS0wT vgAiJgdoOELbYSBS8E6qniP0RjYcArdozGERSbjK2C9wri1a1b8ozo0tEVpLxccFXKzE eMWHgH10diwTiTwoLfuI6vZ5n6q/OdWq/5hybVJc5b8TY8O/Sxr97tdHGJx+wHGEAgRP LknD/FNBK55MJIjL6PmpN5Vabk1LJ0YwzZn6NBErDPDBSTGUnSR+sqMK5jxq/xzw9tP2 lUpw== X-Gm-Message-State: ACgBeo3GT8aKtQPaVf/xAdEcIkPZVY4X6onf8sWMtJRIjjD/Eu9rWbOg i5gXQ6ORpakpiFvfzOsQ7s7jxwMkxEI= X-Google-Smtp-Source: AA6agR7fnwkRjroGVHJxi1RhhEsQJMH0R75zEmK00qUWtCXpP69RbgB6GRXVWsVMb26fPpccLe4aPhca22w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:40c6:0:b0:67a:5c8a:21d2 with SMTP id n189-20020a2540c6000000b0067a5c8a21d2mr20567843yba.158.1662053739927; Thu, 01 Sep 2022 10:35:39 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:55 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-8-surenb@google.com> Subject: [RFC PATCH RESEND 07/28] kernel/fork: mark VMAs as locked before copying pages during fork From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053741; a=rsa-sha256; cv=none; b=BaozHNcwY725v8xSrs8KsN55+ZMza5WMXVDupmdDDPYg+yBj6NZqqVs60lavWORImv/1a2 uEecCiiLgQ8VonqEQ43L46p3BFxRTNsBTUZY4aKadv8WKkVpirKACWLgpkjs5RbLoRhdcw V6QJzfVvVh3znpzN7d9c7uEympZL6e8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U5hxVmDF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3a-0QYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3a-0QYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=byYdzgQ2nb1BwMzAK8Db5EPtebwswE17va5vsTNHDvI=; b=nbBxXAUb9rLmGRqoGq99vBpst3ajiHo/AkyAt1kyowCPTYHZXzOOr/qa8HRzRCbE+a6E7M FQHBxz8pp1odbNLpL5LwUfUPllV/91wtsV6oGYMI5pFoGxIwgXK6CZr2uuZc5D4lB3KQbY ERbXwSyMsj7UE+zVN2fVTRjntfwSkco= X-Stat-Signature: ts4unyat6fd4kykittm3d85hrzhnh1uf X-Rspamd-Queue-Id: EBBC24003C X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U5hxVmDF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3a-0QYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3a-0QYwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com X-Rspamd-Server: rspam08 X-HE-Tag: 1662053740-498935 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000132, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMAs from concurrent page fault handler while performing copy_page_range for VMAs having VM_WIPEONFORK flag set. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/fork.c b/kernel/fork.c index bfab31ecd11e..1872ad549fed 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -709,8 +709,10 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; - if (!(tmp->vm_flags & VM_WIPEONFORK)) + if (!(tmp->vm_flags & VM_WIPEONFORK)) { + vma_mark_locked(mpnt); retval = copy_page_range(tmp, mpnt); + } if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); From patchwork Thu Sep 1 17:34:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E463ECAAD1 for ; Thu, 1 Sep 2022 17:35:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E12E18001E; Thu, 1 Sep 2022 13:35:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC0978000D; Thu, 1 Sep 2022 13:35:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB02D8001E; Thu, 1 Sep 2022 13:35:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BCD0A8000D for ; Thu, 1 Sep 2022 13:35:43 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9684612103A for ; Thu, 1 Sep 2022 17:35:43 +0000 (UTC) X-FDA: 79864218966.27.CD09115 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 4E24A10003D for ; Thu, 1 Sep 2022 17:35:43 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33dc888dc62so234876897b3.4 for ; Thu, 01 Sep 2022 10:35:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=sXPRs4un8Hv1KgYmIkiOTpDZHlwje1UN18vDRt/znXA=; b=LBJTSzoCXGb4iiWKyO0OEdlY4M2wMr4mJ5fdr9ss+SWylmfoHLBZ3bDcjUEYCuu5eS feBNbsez+RUGQT+LSEIL3jhNfGQ3mzrdU70gDdYtaUSlqG4c3zRWe4jgoZRdVjvHQEvZ SPCTeif98/8TgsTaO2lykBXQO/Ydnr0/XtxYChsH6bMYMfoCTEgHM9QVPOJS0Mu1k2UR t6jxn7uXNz4Ix/YY107csqUZ1FzpL1YUPLT0wcSHa5tJ7jOlfkB6DfkgR6ONkj0xZI4Q CdDjSBazMRaOu6i4p9xpWVSdGu4/KGHF+Yxu3Dy4GMEd28lc2vp55tWuelmW724hNsiA vxOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=sXPRs4un8Hv1KgYmIkiOTpDZHlwje1UN18vDRt/znXA=; b=06eSvHg/4Uk2JcgzGO6kwWQcW85pj/OFnCEq+F5SNyaL0sOmUqeAKbfjqxteMWfgIl 6nuazlP+tNneLYmpIVXaMpxkoR4GuJgzMt3RsNtRx627BUwffXE63lhU/WhT0/GN3GkY 5lfJcnq/sE+UKfsNvJoMuJDZqYObT2B+EuIIW01X+TU/GGdJmuREncJlBTExOwZv3ZpK r3MK8dUFwAlDTFHDqHlyxOw1t7l3FdOekrhPrAkOczRpEmozMRe0Dg1rZqBywKxXSg/V kvjieRf4EJptDNy9VvDnhicvdWs653Tz5OUgVhYnLUpfoayuYWh6UYEqOoMIIjtnC8Tz BbwQ== X-Gm-Message-State: ACgBeo0GpbzKmIpjD3bGjPDRZqEhEiaWjp7dzHPwGG/YCJxdgb4NYJ4O W8L3oclFeH6EwrnMvTVkwcw0EwJYdaE= X-Google-Smtp-Source: AA6agR4Ppy4TlAx5Lt/45OnUgJ5yxX2999apq2x6zNaCId6+AMBNoDmW6LxkNrA1Ax704mp9y4OtGqUy/dg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:120f:b0:676:aaa4:8434 with SMTP id s15-20020a056902120f00b00676aaa48434mr20939193ybu.218.1662053742565; Thu, 01 Sep 2022 10:35:42 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:56 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-9-surenb@google.com> Subject: [RFC PATCH RESEND 08/28] mm/khugepaged: mark VMA as locked while collapsing a hugepage From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053743; a=rsa-sha256; cv=none; b=cIzIH+SehEA8+60xhoT2ZaKsUCqQaVkO+uwav14gLmY15elmo3f+ggk2zKQEBwB8ejuIDw BdVzVOOWBURVyTy/6e27/GYZMl9zdk99e/6OSLJKugLzNjmhm+sxLkSadX6DE8OOWpy5VE oc8Aav7zEtmd53cWSQeHbik/YBdA3oo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LBJTSzoC; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3bu0QYwYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3bu0QYwYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053743; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sXPRs4un8Hv1KgYmIkiOTpDZHlwje1UN18vDRt/znXA=; b=CkyXcQW0fC0einm4XjmNl7n/7Px0KmGvjXzSViSHsFd454KkUQnqVJNjz7zLTbJObGmSiW xzKCkJW1TETq9CmSHWD8UisV68n7CMplaqHFjnLhAhVewOlGPuTGLtEAt/hhWPqM3TH6ZF e84MYPoW4vvjzrnTYZckch96ra6Wge0= X-Stat-Signature: rena8hy3ofwcabz8nzygzs964qtuqc35 X-Rspamd-Queue-Id: 4E24A10003D X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LBJTSzoC; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3bu0QYwYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3bu0QYwYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com X-Rspamd-Server: rspam08 X-HE-Tag: 1662053743-126315 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000043, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while modifying it in collapse_huge_page. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..030680633989 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1072,6 +1072,7 @@ static void collapse_huge_page(struct mm_struct *mm, if (mm_find_pmd(mm, address) != pmd) goto out_up_write; + vma_mark_locked(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, From patchwork Thu Sep 1 17:34:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBF6BECAAD5 for ; Thu, 1 Sep 2022 17:35:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 586BC8001F; Thu, 1 Sep 2022 13:35:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 535A88000D; Thu, 1 Sep 2022 13:35:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D6FF8001F; Thu, 1 Sep 2022 13:35:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2FEF68000D for ; Thu, 1 Sep 2022 13:35:47 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 08D06140FBC for ; Thu, 1 Sep 2022 17:35:47 +0000 (UTC) X-FDA: 79864219134.16.C508695 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 9E9908005F for ; Thu, 1 Sep 2022 17:35:46 +0000 (UTC) Received: by mail-pl1-f201.google.com with SMTP id i1-20020a170902cf0100b001730caeec78so12089805plg.7 for ; Thu, 01 Sep 2022 10:35:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=f3HBd//8MTEjGtUKjIJ6VlUao66B8ATA87dRitXKN6o=; b=osN5P5u2gNPNqCUBvTG+Zi66behXZjVMeiTYran173d3bK7x/C4pNezlQHCFvkYQk7 qOQqzQeCFbXQDX9e+QRzeVBdZKr22OFya12CKF/BNucIJ15gJeM6CJ6xJtVhkJ86JMIW z0BrEcSFGRuuGzZ7AIa/XsKdDh+JkQT9DWUmeCCWJ5VsR1VGSIBZmgZx0ZCJYBHl+XPj GoCI/RFibFsonWSqgiZwplWva8pXUgT5urz+18AucAe/seeiQYlfr5LWvUC2qg96tyno slrVqaBJlR1Fm38ZmASWvylf1J3rbhYM8n7OsiK4uazMHDS7+qFeKWZULOPVqfmGOKH5 n3ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=f3HBd//8MTEjGtUKjIJ6VlUao66B8ATA87dRitXKN6o=; b=s0aTz9jpN6x0BIXX30bj91iVGXEgpjf2MbtsI/oHGAPtMiheXGtuqYjHkGp1HjnT98 6tlgxny3xa9KuSXnL41mBuhDA+cDoV+EwjWCGOpAyprGeCauZI6+qCmaoXKclIZdphlS mkM4rsBDhMBhunTQpA8JssYcTDMJrQIq+IE6qImSJc7ksTztM/uVfp+n0rwZjMvYkR3i k/hlMZzYp8C73YaJnqI5s5Klrk5fL10VUDdUmHayyX9hdv/gcaqOVa1f0jIRhaOIcj9c yuuqUTGa4pKORNSZYACe7Pny/40/WCmLVkdfi7O63qBtvu+iDSq+bDCBFIQV5l8LXbwI ZflA== X-Gm-Message-State: ACgBeo0NQhDxbHq/UYTDtOddOJcrM+JIy8WzXuiV2jSaCrsqo29zIwYU AL1e83hGbXumLWJ+9YiPZwLlp0ZuNkA= X-Google-Smtp-Source: AA6agR6Q3wPydZYjoco/mRjMkhmNWxJo3Xdi/ebZ3ovzfa+ebF5iWZWFhqHdfMvubIqW+qGVzpwNqTYAd9U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a17:903:2351:b0:175:58b8:cdef with SMTP id c17-20020a170903235100b0017558b8cdefmr4890777plh.30.1662053745409; Thu, 01 Sep 2022 10:35:45 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:57 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-10-surenb@google.com> Subject: [RFC PATCH RESEND 09/28] mm/mempolicy: mark VMA as locked when changing protection policy From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f3HBd//8MTEjGtUKjIJ6VlUao66B8ATA87dRitXKN6o=; b=BK47kD+N1z8eR4bz60BztQJVnHBJ6lkY0trcCt/XaqFahQ+2vSVZo9W0Irfm72wmvNEQGP CbhojOtXhgzmFqK6z7Otzoj+6tbU5BsTXjpiQp7Z9+9/yZFj3d+dcV0gdBTiuuUe0dlaPX oOc04I0haujpWSaYI9YPdPQDK0hK/jk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=osN5P5u2; spf=pass (imf02.hostedemail.com: domain of 3ce0QYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3ce0QYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053746; a=rsa-sha256; cv=none; b=20hePyLWll4U0G8b/zrL4rOCYyW7ZE0whFw9vFR/GVSDWc9udson1kIyxRxPbiR6fbmFiR ba3zRnljZ6UgVFP2kgXRw5JnHH0Y0lyrXnd8twnP4OqUVcwjl91rDqzsyGPRTY4idYsmv2 jNspC+fwX6HPM/CiPp4tX/hJA8aZfcU= X-Stat-Signature: y5acm9xxtqsx4i9iec4cn7emh6fm1que X-Rspamd-Queue-Id: 9E9908005F Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=osN5P5u2; spf=pass (imf02.hostedemail.com: domain of 3ce0QYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3ce0QYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1662053746-387590 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while performing VMA protection policy changes. Signed-off-by: Suren Baghdasaryan --- mm/mempolicy.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index b73d3248d976..6be1e5c75556 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -383,8 +383,10 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) struct vm_area_struct *vma; mmap_write_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for (vma = mm->mmap; vma; vma = vma->vm_next) { + vma_mark_locked(vma); mpol_rebind_policy(vma->vm_policy, new); + } mmap_write_unlock(mm); } @@ -632,6 +634,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, struct mmu_gather tlb; int nr_updated; + vma_mark_locked(vma); tlb_gather_mmu(&tlb, vma->vm_mm); nr_updated = change_protection(&tlb, vma, addr, end, PAGE_NONE, @@ -765,6 +768,7 @@ static int vma_replace_policy(struct vm_area_struct *vma, if (IS_ERR(new)) return PTR_ERR(new); + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->set_policy) { err = vma->vm_ops->set_policy(vma, new); if (err) From patchwork Thu Sep 1 17:34:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E193FECAAD1 for ; Thu, 1 Sep 2022 17:35:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64A3D80020; Thu, 1 Sep 2022 13:35:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F9838000D; Thu, 1 Sep 2022 13:35:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C16680020; Thu, 1 Sep 2022 13:35:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3F2548000D for ; Thu, 1 Sep 2022 13:35:49 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1D1AA1A06C9 for ; Thu, 1 Sep 2022 17:35:49 +0000 (UTC) X-FDA: 79864219218.27.1D2DF47 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id C45A82003F for ; Thu, 1 Sep 2022 17:35:48 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id v5-20020a2583c5000000b006964324be8cso4905834ybm.14 for ; Thu, 01 Sep 2022 10:35:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=DiVLSVAVO410fRJcvWWuX+r6LjGy2i2DyacaHp6fzuc=; b=kNx38Jhqjx9XlmKxOVsRKukj17KeQAoUkvOgezSjJcI7qDa8s6siy6+ZCSn18LnVHi n4QLcTPTYnWQedOp2NOA065emr+I7/FhDV/20btA2UJprNGZCfJvY2fzU6JS3rffkIEf 2FurYAnF0+QmVLyL1XlOy/6eS9/B08Rzb+EHT4ihXLuAM0YGViLuLR9Pls5xzIb0RA++ peqVg5K5+yJOyniQSXXLxLw8N9e7jk4WJ8bD01w4ADM9IUXJZsw/A2NPWpCJiNE61osh 1/9xE4X1j0QNkz0GiHjK4YBUmosqgJkbBEuUlP/cGGRjR6eOLhrxcNhT+yoJZX8wCM9W ZvhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=DiVLSVAVO410fRJcvWWuX+r6LjGy2i2DyacaHp6fzuc=; b=TVaCbgEcPf59eiMo5kzAitzYWeG7oqTRmvqzPjB75OGumIWwI713wBbl3eu/ccplnI u33YoXlih1Ux1/Se9gqjZZDnIyF4rM6AYUvX8N8UpOB/Ou1FvsYAx0VOZi/ZWkKjmClb UeESbqJ2+ghZO9Y6nPqqjIQGxuqCpzUaWOZ/sdqAxzPmiUEgQ6CEcob55ueH5H+EOXpS qCv6BzHGw4rugnuhqZuYMzpthPs/k9ZKUNt996Pp+an7YcL+cZdUYPoNT7c1bK7vIOo8 ouK85YGRnnLYSWXk7j4XDwcf2REpKgq52O18YrUKbRaNTZLfG8R1UdcCxQr9tpuGE+4l BCqA== X-Gm-Message-State: ACgBeo37p0Mib08frzwzasJWX0UkkPYcp1hdtXD8gS6Y/a2IPGNFbzky I7XDFKBZxo/fcmgc+afr9W02cFafqOM= X-Google-Smtp-Source: AA6agR4qMG/7UuuiqhHhvohWNal3nFjDJKXC1X/Q7rlLCDCvl4ZyXaMMKRMmizmDLQF7LDsGfEt+/G6N4D4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:bf92:0:b0:695:8d8f:2af4 with SMTP id l18-20020a25bf92000000b006958d8f2af4mr20427159ybk.200.1662053748060; Thu, 01 Sep 2022 10:35:48 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:58 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-11-surenb@google.com> Subject: [RFC PATCH RESEND 10/28] mm/mmap: mark VMAs as locked in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053748; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DiVLSVAVO410fRJcvWWuX+r6LjGy2i2DyacaHp6fzuc=; b=0lMSBduz7p8CL0AsZe2HYvGa5D0dXhFAnlswkIAG7pbxs4JdFROdLmbz/JodgGJ2W1Usrc sCUZf/M+QB6MG0EwwK1a/nWK5go1ymOKoXIm5DOpr1IbVrEVnbBABnDzP7PNX0QhTdSMSz inN2zhDbGG9OYcrKL+BEgqRGoENXLnc= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kNx38Jhq; spf=pass (imf13.hostedemail.com: domain of 3dO0QYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3dO0QYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053748; a=rsa-sha256; cv=none; b=vw0lTgrQYD5qxp5CU7Z9CkztKMtA7OLzmODvthe4WNdg1sLEooKpNlW524cNslSfI+lPKW Hs5Mm+b7BWQdsNF50DrLv+zcyeBeEfT6kwX6ygraOsay0OUjT7ZL9ALgEk2mEUiVfmAa3L KT2oZnADb9x4bVkEgnZ7GNGu/FhDIvQ= Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kNx38Jhq; spf=pass (imf13.hostedemail.com: domain of 3dO0QYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3dO0QYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam11 X-Stat-Signature: g1hr48bqwz85dn3dpg65okuspdcc6fnh X-Rspamd-Queue-Id: C45A82003F X-Rspam-User: X-HE-Tag: 1662053748-604543 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_adjust modifies a VMA and possibly its neighbors. Mark them as locked before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f89c9b058105..ed58cf0689b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -710,6 +710,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, long adjust_next = 0; int remove_next = 0; + vma_mark_locked(vma); + if (next) + vma_mark_locked(next); + if (next && !insert) { struct vm_area_struct *exporter = NULL, *importer = NULL; @@ -754,8 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next == 2 && !next->anon_vma) + if (remove_next == 2 && !next->anon_vma) { exporter = next->vm_next; + if (exporter) + vma_mark_locked(exporter); + } } else if (end > next->vm_start) { /* @@ -931,6 +938,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * "vma->vm_next" gap must be updated. */ next = vma->vm_next; + if (next) + vma_mark_locked(next); } else { /* * For the scope of the comment "next" and From patchwork Thu Sep 1 17:34:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C0E0ECAAD1 for ; Thu, 1 Sep 2022 17:35:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C158A80021; Thu, 1 Sep 2022 13:35:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC6548000D; Thu, 1 Sep 2022 13:35:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A67D480021; Thu, 1 Sep 2022 13:35:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 960198000D for ; Thu, 1 Sep 2022 13:35:51 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 80345140FBC for ; Thu, 1 Sep 2022 17:35:51 +0000 (UTC) X-FDA: 79864219302.08.A0830CC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf26.hostedemail.com (Postfix) with ESMTP id 39B87140058 for ; Thu, 1 Sep 2022 17:35:51 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id v8-20020a258488000000b00695847496a4so4934772ybk.19 for ; Thu, 01 Sep 2022 10:35:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=IUrOZBQLzZPUHaSf75EcaCRmzwc/1Uln8Bhr3CCRW40=; b=RqTxEDKmKz8lQ9e4qoEc4uYxV2l6ZaYMJyNDEM2PccdKcjIk6CnRCwBsEF2cj3JFvG j1vM9+I5MFJ8Y28kW9pfnzwXL/sLYGBlEyABKHcApM/RmMkSzapOCNpQMv7qth5lMe9Z r4q/GOZemPIxyE1pZ9TWp0QmyjLs2nHHEkPOAyS+5ktrUUhx4EO8hL8MjHB9ieBIjUIm Yg7XSf+rAL+v2NSP5NDXERvFs20Phv4/bhur7fMLyFA6L2AN5XBQFKbc4pQ3AuRWkJ+n hhBQmROSc08GaGFWWbM/WsNha74+z1T1INvujKYoiSdnjlDI7QzZLxl1fThOa82IPdUF 4lpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=IUrOZBQLzZPUHaSf75EcaCRmzwc/1Uln8Bhr3CCRW40=; b=gvOXRf3P6ovF1LoXl2ieQPmZ0VawpDJw7NFAusC05jveK6pZHXU2x962efX1/MGjHO sNdwzJt/crAm8a722/6xLyCUeKEmF66QwgsVha51lCTwJSMSrOMLr2GhCdwlmS4HRODn iRwf2Bh5mU5h0KBpTNeUztRY/07KDbhBHzO8E0i8+mGXbbCr5rlWTFXBdcTMqlnTKh8i g369SLXSrJP+noA57dsHoS3pQTva28YbhXLrL2luiWxBQgexY/HK71/r+f2hE9fn6kFG w94d/gg+BZhjhZFzXTg3KZ+Dj53JY/Ftei0AEGdE/UGsQaSGN4MOCkxSMYCaq3Dc6fq5 +wBw== X-Gm-Message-State: ACgBeo0HiEM2UAx2sTrTNsEGOVs64Wb7DiqnRC+tbdIl87UgBa/ZIc/V sqF2pk/qnentpFfisFsFjHmipEevdnk= X-Google-Smtp-Source: AA6agR7hyHeb8khg+/3ZoJp3oFVauMFyBSatbGnIGl1CpSfGDYZylUwLzf/LZXvggze+7Aght5uln8kJbB0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a0d:e083:0:b0:343:2928:867f with SMTP id j125-20020a0de083000000b003432928867fmr6902464ywe.467.1662053750534; Thu, 01 Sep 2022 10:35:50 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:59 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-12-surenb@google.com> Subject: [RFC PATCH RESEND 11/28] mm/mmap: mark VMAs as locked before merging or splitting them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053751; a=rsa-sha256; cv=none; b=rQpYiDFunTPWGyFzdKFjGgqkwtHLeVpWAeuWhrErFsyNUDsTrBSMDtRolFZL8zbOgmGyLn Ck0fV3AcLS02FqSOlSvT4Wn3ZI7ZA+qQVLivbVLC2VCa8aeULy56xj92yoAM9/XTkw54n6 MWvVG7/ZuujMKKeFSrjLae6jpAmDkt8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RqTxEDKm; spf=pass (imf26.hostedemail.com: domain of 3du0QYwYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3du0QYwYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IUrOZBQLzZPUHaSf75EcaCRmzwc/1Uln8Bhr3CCRW40=; b=HTSb6fyBStGY0Ga+IkuUVKRdHLeG01LQgZ/etBvdGJEQT7x8wxWmfC5M+gVmQpFylaz8iH yvT1gbJfsdwNtH7jpjfkC6xiRk+etF/kaVp2TbnJc0MlTsusl4qtGWy7oinrsPW2pb7SaZ RUQ4wd2/TZn33S7OnXAmSAiPERaXiW0= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RqTxEDKm; spf=pass (imf26.hostedemail.com: domain of 3du0QYwYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3du0QYwYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 39B87140058 X-Stat-Signature: x8iqfte877twxp9rsy3ipfdwxnxtzigg X-HE-Tag: 1662053751-392780 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000027, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Decisions about whether VMAs can be merged or split must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Mark all VMAs which might be affected by a merge or split operation as locked before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index ed58cf0689b2..ade3909c89b4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1147,10 +1147,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_mark_locked(prev); next = vma_next(mm, prev); area = next; - if (area && area->vm_end == end) /* cases 6, 7, 8 */ + if (area) + vma_mark_locked(area); + if (area && area->vm_end == end) { /* cases 6, 7, 8 */ next = next->vm_next; + if (next) + vma_mark_locked(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2687,6 +2694,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *new; int err; + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) From patchwork Thu Sep 1 17:35:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB8ECECAAD1 for ; Thu, 1 Sep 2022 17:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C33880022; Thu, 1 Sep 2022 13:35:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 472AE8000D; Thu, 1 Sep 2022 13:35:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3617180022; Thu, 1 Sep 2022 13:35:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 286798000D for ; Thu, 1 Sep 2022 13:35:54 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0A71AC105A for ; Thu, 1 Sep 2022 17:35:54 +0000 (UTC) X-FDA: 79864219428.04.7C5C447 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id ACA2910003F for ; Thu, 1 Sep 2022 17:35:53 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33dc888dc62so234880237b3.4 for ; Thu, 01 Sep 2022 10:35:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=SOzt98PYJdog9tZtODUdaF02HJMaRj5G7nUIc+/7MKg=; b=kTPvKd0Pj/2ICi2JaJbj4mvw83rGSI8D/7SpSALxOaZxZxYULx54dkMykzJllj/oUS acc8cAwohzpv9TG/117+M7kDarZ4pbcDRJw9kbqISJMHllJVQJpNPTnmGZHaFtdhW7jV te5Y1eTVCGFOLeMRtGhakQSv/C95dQTatvR3jNPFepOpi9BM4A2IDo4sHrk/Ld/IusAA C91XyLoa1Kj543iPIZk7cC0y8cEUyAun26iVaV/jukaSDh0nNNy3USNMwmiwPC/u6rrR uYxoBw75emQZr0TAylDwoEOSkfchxb9rhKIYoM9w58cv+ILGWIFbRnVoqe89UuzvgGW7 /7mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=SOzt98PYJdog9tZtODUdaF02HJMaRj5G7nUIc+/7MKg=; b=jQu+ld/KFOrNOGb20BFT4YXnvN9XjMiW5yI3BV4mZBYpGFGLG02KN0XJrzPHQtQLay bo48YS+V9hJ4tiqjXBrmWso1GmA2WwKCC5XwaAiDJT49ovBT+xezUDqpg9nGjQHo0SzE Or6Sxutt3MTg/fF2WEV2nzE63LNcz8QVHCW6V/Ujcw3YAqvzXlB3/jCaQWZX82jAmBfg ZmuoGcrKVyEhg/qZkycdT712kxw1u+xiskc3WMFDj5joEK8ib/a27jFcCaZoLdSjpavT k2BbBEv03x/aYscZjkZryzBC4tRgQ1Y6qhXEpRSoto/38EOZ1yaVSEc5b4hDrNg+EQD9 d6FQ== X-Gm-Message-State: ACgBeo2r29aN4I2aY5alHpUFza82upx78OqsjaKXVU4msC+qwTrGPhWb hEtrKZJxHJunT/trISF70QzZ96Sost4= X-Google-Smtp-Source: AA6agR6YY4L9oqUniRiEloavxF9b7QDLSzaw7mgfeu4Gq//fIM/XBjFxGsDu3HVa5YMmDCpCvrS70nu4RUs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:3746:0:b0:69b:a100:521f with SMTP id e67-20020a253746000000b0069ba100521fmr16650279yba.135.1662053753176; Thu, 01 Sep 2022 10:35:53 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:00 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-13-surenb@google.com> Subject: [RFC PATCH RESEND 12/28] mm/mremap: mark VMA as locked while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053753; a=rsa-sha256; cv=none; b=Ej0gWGwMVfLezhXw2e1DqRk/qnRu+5/QU2NaipDDncVgic+Qgs36IBTucFCuOa6bSh8m+L dVGl7No0Jwm8NhPjL2TrJ1r4o3jMY/btJvVA0UNrRHp4Or2RtWr1CrpcUrA2CHUv0AvrxB qpkkTbdIvPTTvIoVg3ZqSk8t0ZXJ9ts= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kTPvKd0P; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3ee0QYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ee0QYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SOzt98PYJdog9tZtODUdaF02HJMaRj5G7nUIc+/7MKg=; b=fiW1Zgn+m0pRKNV0J9XUjryXmcClH7b2L3J8fBbZFvd+TBk2Ze2HhooERzcVQ3B3SFB9gq jxoXf6dykbhjxwck00QDvJJLz2nFtLTLTRvIjUQkhiwJ1zdNujlxT6vuFhhNyBIP7kiA7p JjvCXdRrdDH37lFQzEpd7iddmmnijAk= X-Stat-Signature: jb5rscw4spceadqwymc75itwnr357jr7 X-Rspamd-Queue-Id: ACA2910003F X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kTPvKd0P; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3ee0QYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ee0QYwYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com X-Rspamd-Server: rspam08 X-HE-Tag: 1662053753-976812 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000012, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mark VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ade3909c89b4..121544fd90de 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3248,6 +3248,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_mark_locked(new_vma); vma_link(mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; } diff --git a/mm/mremap.c b/mm/mremap.c index b522cd0259a0..bdbf96254e43 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -620,6 +620,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_mark_locked(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Thu Sep 1 17:35:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88696ECAAD3 for ; Thu, 1 Sep 2022 17:35:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A17F80023; Thu, 1 Sep 2022 13:35:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DB828000D; Thu, 1 Sep 2022 13:35:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 054D680023; Thu, 1 Sep 2022 13:35:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E6E938000D for ; Thu, 1 Sep 2022 13:35:56 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CBF8E1A1048 for ; Thu, 1 Sep 2022 17:35:56 +0000 (UTC) X-FDA: 79864219512.07.3F6A51E Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 7BA691A0040 for ; Thu, 1 Sep 2022 17:35:56 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33daeaa6b8eso233295317b3.7 for ; Thu, 01 Sep 2022 10:35:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=r773C9YRjCBNvMT4CzLpPbmrmyQxRNwpck2TtfCck1k=; b=ObA20LITrip87VziaHHJxhsQdt90Sartw38u0QjInvigVBBYp9UQt6FfSRSeXBXEZU PLrSrOLMVOwI3/sz3dUO/NCHhkXbIW3GF8M34MWk3xttvYsGitw637zJhtS5J3KbjVTq FLk9jwJNLpWEznbWA1xdOdydu+93KP79dvrPgyK7xujYPxVhRBYXk7sgB7YC6n2PoUnC kIiWKC2Oq5VXot4jMCKtmju4UEkl0k8b2etD7U9K2bdu4kKMwZb5udL9M4YbR2lJrZYA TRfi2Vr1zk+DfsN0D7bvgFUbxdpPUkljp0W3vF7qidlJ9uVGD7Pq+2IeCcnvt21qYUIk bD3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=r773C9YRjCBNvMT4CzLpPbmrmyQxRNwpck2TtfCck1k=; b=dQha4rBdcD1KnryQOplMJ840yTP11L+3oEnd1yh55cuKqjWlvDs+w9MH8nEuWBGK+F tm361EhJF6EH/82KEZ0LR9CwH/Ndx4DN/J9Gm7DVtri/iYr9fEkGj3kq6v6iBBIw8PiZ nm28aUb85ef8f/JbKzkxyQdVDxOYU+z3d6Q8z3LNcg15Qge3Q6X5E5XWdY9ZY0/0otwv 0JvFcz32wUv55prTtnEoDuarn69lx0ctOjnTc0dz8/flB1k3cu2CfdQy0zsqpfSGHha0 wbJT0HunHG2Cq+gWL7XzgT3vRD5nsseeGioT55R3u9H8LkNfw+zwtAWg5RMCrsk2a5FX jQMQ== X-Gm-Message-State: ACgBeo0fz9hmG8rwpN68yRLxNQoGJJyrckzMKcBnNGevLV4rVQE9FCdo Dehf4Zed30Qk1vlREsgnjDJG/mD8sQM= X-Google-Smtp-Source: AA6agR5CArj0XA3uZUba700jvv45FjGtdD6CkgMKCuV3RkIgq+Sm8gt1hyIVkRx9I0S41UWPyjD3FlSg3pw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:dd05:0:b0:69c:97aa:f08e with SMTP id u5-20020a25dd05000000b0069c97aaf08emr9069882ybg.583.1662053755821; Thu, 01 Sep 2022 10:35:55 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:01 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-14-surenb@google.com> Subject: [RFC PATCH RESEND 13/28] mm: conditionally mark VMA as locked in free_pgtables and unmap_page_range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053756; a=rsa-sha256; cv=none; b=ETvp4Hav2rJDP0gXSIi7hfLtH9flJBiBj2cncAXtevvhPOmOKO5m6eWJ53BGEyL23QWK1V 150DY3A+dG3r3Tv+lb3Vz2x4f65KtJCLO0B9D++/NGCVnz2S2Y2BYTBFSBmV0EXuBvN2Qx ZwfWnosXfC6hukYl4BeIXkIyQPoUkYQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ObA20LIT; spf=pass (imf19.hostedemail.com: domain of 3e-0QYwYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3e-0QYwYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053756; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r773C9YRjCBNvMT4CzLpPbmrmyQxRNwpck2TtfCck1k=; b=3UectlnCLWvAHh+f06yZAp7LBsRCXpQnEDX5fd+6YdCzedBBQoyGEC7v3YLTmlTep5DNpw PsOBVRjWqQ1/xNsNUvUZ4+bxvu572eVU134zRnoz8CxY68MIorzNBM4VBCtf2PsIEgdy6C 3xoBsYiauygYVZUzI8RiaPPg/tqZJ7U= X-Stat-Signature: e9b7beztx8jasi4zk6dxsbd8rcu9gfjo X-Rspam-User: X-Rspamd-Queue-Id: 7BA691A0040 X-Rspamd-Server: rspam07 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ObA20LIT; spf=pass (imf19.hostedemail.com: domain of 3e-0QYwYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3e-0QYwYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1662053756-247455 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pgtables and unmap_page_range functions can be called with mmap_lock held for write (e.g. in mmap_region), held for read (e.g in madvise_pageout) or not held at all (e.g in madvise_remove might drop mmap_lock before calling vfs_fallocate, which ends up calling unmap_page_range). Provide free_pgtables and unmap_page_range with additional argument indicating whether to mark the VMA as locked or not based on the usage. The parameter is set based on whether mmap_lock is held in write mode during the call. This ensures no change in behavior between mmap_lock and per-vma locks. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 +- mm/internal.h | 4 ++-- mm/memory.c | 32 +++++++++++++++++++++----------- mm/mmap.c | 17 +++++++++-------- mm/oom_kill.c | 3 ++- 5 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 476bf936c5f0..dc72be923e5b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1874,7 +1874,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, void zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size); void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); struct mmu_notifier_range; diff --git a/mm/internal.h b/mm/internal.h index 785409805ed7..e6c0f999e0cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -85,14 +85,14 @@ bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long floor, unsigned long ceiling); + unsigned long floor, unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details); + struct zap_details *details, bool lock_vma); void page_cache_ra_order(struct readahead_control *, struct file_ra_state *, unsigned int order); diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..9ac9944e8c62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -403,7 +403,7 @@ void free_pgd_range(struct mmu_gather *tlb, } void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long floor, unsigned long ceiling) + unsigned long floor, unsigned long ceiling, bool lock_vma) { while (vma) { struct vm_area_struct *next = vma->vm_next; @@ -413,6 +413,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -427,6 +429,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, && !is_vm_hugetlb_page(next)) { vma = next; next = vma->vm_next; + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } @@ -1631,12 +1635,16 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb, void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details) + struct zap_details *details, + bool lock_vma) { pgd_t *pgd; unsigned long next; BUG_ON(addr >= end); + if (lock_vma) + vma_mark_locked(vma); + tlb_start_vma(tlb, vma); pgd = pgd_offset(vma->vm_mm, addr); do { @@ -1652,7 +1660,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1691,7 +1699,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, i_mmap_unlock_write(vma->vm_file->f_mapping); } } else - unmap_page_range(tlb, vma, start, end, details); + unmap_page_range(tlb, vma, start, end, details, lock_vma); } } @@ -1715,7 +1723,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details = { @@ -1728,7 +1736,8 @@ void unmap_vmas(struct mmu_gather *tlb, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); mmu_notifier_invalidate_range_end(&range); } @@ -1753,7 +1762,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < range.end; vma = vma->vm_next) - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1768,7 +1777,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, * The range must fit into one VMA. */ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, - unsigned long size, struct zap_details *details) + unsigned long size, struct zap_details *details, bool lock_vma) { struct mmu_notifier_range range; struct mmu_gather tlb; @@ -1779,7 +1788,7 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addr tlb_gather_mmu(&tlb, vma->vm_mm); update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); - unmap_single_vma(&tlb, vma, address, range.end, details); + unmap_single_vma(&tlb, vma, address, range.end, details, lock_vma); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1802,7 +1811,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, !(vma->vm_flags & VM_PFNMAP)) return; - zap_page_range_single(vma, address, size, NULL); + zap_page_range_single(vma, address, size, NULL, true); } EXPORT_SYMBOL_GPL(zap_vma_ptes); @@ -3483,7 +3492,8 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) { - zap_page_range_single(vma, start_addr, end_addr - start_addr, details); + zap_page_range_single(vma, start_addr, end_addr - start_addr, details, + false); } static inline void unmap_mapping_range_tree(struct rb_root_cached *root, diff --git a/mm/mmap.c b/mm/mmap.c index 121544fd90de..094678b4434b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -79,7 +79,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644); static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -1866,7 +1866,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file = NULL; /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end); + unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end, true); if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); free_vma: @@ -2626,7 +2626,7 @@ static void remove_vma_list(struct mm_struct *mm, struct vm_area_struct *vma) */ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct vm_area_struct *next = vma_next(mm, prev); struct mmu_gather tlb; @@ -2634,9 +2634,10 @@ static void unmap_region(struct mm_struct *mm, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, vma, start, end); + unmap_vmas(&tlb, vma, start, end, lock_vma); free_pgtables(&tlb, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } @@ -2849,7 +2850,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len, if (downgrade) mmap_write_downgrade(mm); - unmap_region(mm, vma, prev, start, end); + unmap_region(mm, vma, prev, start, end, !downgrade); /* Fix up all other VM information */ remove_vma_list(mm, vma); @@ -3129,8 +3130,8 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, vma, 0, -1); - free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); + unmap_vmas(&tlb, vma, 0, -1, true); + free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* Walk the list again, actually closing and freeing it. */ diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 3c6cf9e3cd66..6ffa7c511aa3 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -549,7 +549,8 @@ bool __oom_reap_task_mm(struct mm_struct *mm) ret = false; continue; } - unmap_page_range(&tlb, vma, range.start, range.end, NULL); + unmap_page_range(&tlb, vma, range.start, range.end, + NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } From patchwork Thu Sep 1 17:35:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3927DECAAD3 for ; Thu, 1 Sep 2022 17:36:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC9A580024; Thu, 1 Sep 2022 13:35:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B04F68000D; Thu, 1 Sep 2022 13:35:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A35980024; Thu, 1 Sep 2022 13:35:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7ACF38000D for ; Thu, 1 Sep 2022 13:35:59 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5C8E71604CC for ; Thu, 1 Sep 2022 17:35:59 +0000 (UTC) X-FDA: 79864219638.08.7830033 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 14A3116005E for ; Thu, 1 Sep 2022 17:35:58 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id k126-20020a253d84000000b0068bb342010dso4889937yba.1 for ; Thu, 01 Sep 2022 10:35:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=CVmSMjmLYwB9FBtpYSNLJ0EV++HpteWYtmAikuEnyj8=; b=RQU4Av3GuaGnYpE9oeVZ2Qc2D9oyD+ip4sYivAtlCkyS2/hNn9rv0v8p2M0NIGYJND Yf8x5cX18al4RtSO4Q1PhkVmfw1aU1Ab/Cll55dcKrRpQKauqrB7xyNC80pmZ6bW9bAC e95UH0jr+QtsvW6E0Yi+7Q29f7LTJtJy2OM2cQAlA/mMmKe3iC7olC3Rgqr7kYyQ1CJQ Co86J6JNei7V+ykfhNyFAPymL77cyA2ZQroFaHENAEs99CXDu6wa2X/Gpfr/HBQQdv4H QNCI3+GHYcOaSG+P0vp/Qcr7DGv7/3qUpIL4nQqbN3ksWTquAWAHtimY4d22gaOesNyT C0Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=CVmSMjmLYwB9FBtpYSNLJ0EV++HpteWYtmAikuEnyj8=; b=3FnWpkedYmONfOGuaYlfl2EUkvFLTteedV46tX3XWUUwYDL3TnPvzLN7I/NGaQ8NXJ SbWn27k5aR4EO7QPKIfvmE53x7DJ6cbCA2aeeZ6mXzTbizUnSMUSXDRyjc+XqO325JIS WQLmMbLaJlHm1TTX/QK3Lyn39e8mc9JSSHwRmDSpzQ/Izxdj5CkOrwczWsrfWgFsJUqD Xj/0+Vb4PPkWrg3mj95H/kxcwrqXn+4ColtYKYqEzNuwpFWE00L9OXQGPt+wU2aQ1Rif PCZgCmA46/cP8SjJ9F4ipcT63yY9l4qX65/dgOWdRir+CT3Z4LKmzFRC5CLpkfZdp5lh LvDA== X-Gm-Message-State: ACgBeo02z8nvOzpDdtS0o89d6m893Nd4g6GZWWHmgNyDju19FOiscQNK kR6jdDhF8w+gkYDX+9ZQUvmLZqkQShs= X-Google-Smtp-Source: AA6agR571otBJ73SV6fz6quBhFyvEDhG9tsbvxbf3EGsugmeaIr00wfh/qpDUZrPhQkoYyC6wjg4lFM6a0U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:1c3:0:b0:6a2:2d4e:6bf8 with SMTP id 186-20020a2501c3000000b006a22d4e6bf8mr1787307ybb.564.1662053758377; Thu, 01 Sep 2022 10:35:58 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:02 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-15-surenb@google.com> Subject: [RFC PATCH RESEND 14/28] mm: mark VMAs as locked before isolating them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CVmSMjmLYwB9FBtpYSNLJ0EV++HpteWYtmAikuEnyj8=; b=CoqvNL2nm9VQpkV5IQDtLEZX8d9MggPr4kBbmfZdCj2WtB1jQoCD90etxm5RkswCIxqUCB yOsCgv/uFOzVAsdS5do3N34Xf0G7Da3bwiVWsjgvm5iMESCnnViGTae9Lx1asYKPzUuvkB pxed5hRMFLMW/MPnDNT3rBHSELxekxY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RQU4Av3G; spf=pass (imf08.hostedemail.com: domain of 3fu0QYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fu0QYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053759; a=rsa-sha256; cv=none; b=gRqRWw8uuaAtURbTi6OM+la0ui8zXZnbq3knhuqq+KarMCLewwIFUqXr4gh7rxuwWMtH/L aj34MpgZWchbCdFnoq0onZGmVT3bmKSW/UY94h+nXSvZ4OAo0SrqWd0w82C6QTapPT1Z5F +x+X/FRPFI/mjxrWgUp9AU3KeP0XlJY= Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RQU4Av3G; spf=pass (imf08.hostedemail.com: domain of 3fu0QYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fu0QYwYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam11 X-Stat-Signature: r5rbkeiuo3jzxortr1oqoruae6ujammj X-Rspamd-Queue-Id: 14A3116005E X-Rspam-User: X-HE-Tag: 1662053758-341302 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mark VMAs as locked before isolating them and clear their tree node so that isolated VMAs are easily identifiable. In the later patches page fault handlers will try locking the found VMA and will check whether the VMA was isolated. Locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 094678b4434b..b0d78bdc0de0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -421,12 +421,14 @@ static inline void vma_rb_insert(struct vm_area_struct *vma, static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root) { + vma_mark_locked(vma); /* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); + RB_CLEAR_NODE(&vma->vm_rb); } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, diff --git a/mm/nommu.c b/mm/nommu.c index e819cbc21b39..ff9933e57501 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -622,6 +622,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) struct mm_struct *mm = vma->vm_mm; struct task_struct *curr = current; + vma_mark_locked(vma); mm->map_count--; for (i = 0; i < VMACACHE_SIZE; i++) { /* if the vma is cached, invalidate the entire cache */ @@ -644,6 +645,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) /* remove from the MM's tree and list */ rb_erase(&vma->vm_rb, &mm->mm_rb); + RB_CLEAR_NODE(&vma->vm_rb); __vma_unlink_list(mm, vma); } From patchwork Thu Sep 1 17:35:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DABFECAAD3 for ; Thu, 1 Sep 2022 17:36:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B04BD80025; Thu, 1 Sep 2022 13:36:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8D168000D; Thu, 1 Sep 2022 13:36:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9070380025; Thu, 1 Sep 2022 13:36:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7B5C98000D for ; Thu, 1 Sep 2022 13:36:02 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5A3E6C105F for ; Thu, 1 Sep 2022 17:36:02 +0000 (UTC) X-FDA: 79864219764.27.53E24B4 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 17A9140051 for ; Thu, 1 Sep 2022 17:36:01 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id b9-20020a170903228900b001730a0e11e5so12193459plh.19 for ; Thu, 01 Sep 2022 10:36:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=jyHdLFPHMbjXGLnwUzIxKnvlA/GbLLDhxKEqR7mLfqg=; b=TCOmQuZ0ealmr2rj6cwh5S5dr2tHdE6kJbl5yIAByz1pqanqNTRiefLGtYBY7Bfwis HCiC64ttZOFWEgwG30yQXn+Yd1NgQ1aDGCJSNFZhazZq9lt7mplm3W+aibX2RLK/lLN0 uAX/ZCVxAoOvxpbxeS81kOPFHzixGFCJRTo1WJXrfNWNtHfKZEdnfDSt0EcuBcHbmrUl KJFv/JkMarGz2fGZyZ6GmZqZcaDNiv8ls/HHTwoDoMD28NmijZHaa+6d5o2vJqrlXSFO lKUsBOqEBkv1O0j454aqLOLj6gi+Hg3Qlj1nIC3dhAGDngGkgQrOdGW8W2dVGnwocfag do3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=jyHdLFPHMbjXGLnwUzIxKnvlA/GbLLDhxKEqR7mLfqg=; b=ndHy1pOVaWAg9Zud8KZ/eQoriM3XQynyS8uUnXkmMycoG0kYVaUfnDh9am45jC68yX 2DHzkMXHEYcfwmhfl60acAehXFDXz8dSCzSH2vxdQH8QLsXLyeUxwpyL4JINbApS9Arp RT17LiazkKz1hGRmxkgLjXfmWHsB0wvQiwQzAyP2b6ivQ1UiCjJX0nOdIiyfTGlUk3o+ SpOdD/oP7aNKjPXwje4JvS7y5OFxNDF1uyeM+mmK47QQqfJh4+wATiB5MHbkpPI6hSnx nbqoLYRENEo1DL0V2elp9j5sA67ph+RN4A0OHN0IbKRAWOq2JQHOvwRHoFG3Ua4VKlhb 0cGA== X-Gm-Message-State: ACgBeo2PFy2Efi4Dc9bctyzqFtPmSU7u0WEc5hmV364q+Hnuf6syyIT7 u7TybM1NUH7EO9dW1QD4/QLUapMlckM= X-Google-Smtp-Source: AA6agR4mIrBI+VqdmK9XxquJrzUivZu8ctMQy2Mh5DPiMdzbr+5VMUvdiKSOqW1XRdAe5w+IQuXOVrbXRFY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a63:2cc2:0:b0:41c:681d:60d2 with SMTP id s185-20020a632cc2000000b0041c681d60d2mr26204273pgs.502.1662053761168; Thu, 01 Sep 2022 10:36:01 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:03 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-16-surenb@google.com> Subject: [RFC PATCH RESEND 15/28] mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jyHdLFPHMbjXGLnwUzIxKnvlA/GbLLDhxKEqR7mLfqg=; b=X1TRKADt4gPkm6VjTXUvllXntrjNY6no7Xm5WM6sToWLy1NMBTnUUM4bm7OnA1mLHYQB+B CE3lNfcpSa9J+av3JBJeq9gcXzGS6pqUxFkefXOqXAgVH8QQvKgyTvI1XIc+WF5iCJtUF2 q781v2G43fcG794utA13cSmW7oKFprY= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TCOmQuZ0; spf=pass (imf04.hostedemail.com: domain of 3ge0QYwYKCL4wyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3ge0QYwYKCL4wyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053762; a=rsa-sha256; cv=none; b=k50UVkIWIhEUARMjkwil773pFQlIii/hFk9UHDKBcB3MxpahWE9COai7EghdONznEeIQGN OOF3nBJef4W+rBvNHC5MnqIYZS4eHimJJq3pYyVNxBgrXp+Mi2+la226BLi4YzZ8q6okKM fFKS1GovPO61GI/ohr7EbkOW+1YEfBs= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TCOmQuZ0; spf=pass (imf04.hostedemail.com: domain of 3ge0QYwYKCL4wyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3ge0QYwYKCL4wyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: ggkepajni9ef9rs6jm6uxmswz93s9uh3 X-Rspamd-Queue-Id: 17A9140051 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053761-113732 X-Bogosity: Ham, tests=bogofilter, spamicity=0.013779, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases mark adjacent VMAs as locked to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index b0d78bdc0de0..b31cc97c2803 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2680,10 +2680,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma, * VM_GROWSUP VMA. Such VMAs can change their size under * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ - if (vma && (vma->vm_flags & VM_GROWSDOWN)) + if (vma && (vma->vm_flags & VM_GROWSDOWN)) { + vma_mark_locked(vma); return false; - if (prev && (prev->vm_flags & VM_GROWSUP)) + } + if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_mark_locked(prev); return false; + } return true; } From patchwork Thu Sep 1 17:35:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAACEECAAD1 for ; Thu, 1 Sep 2022 17:36:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86D2380026; Thu, 1 Sep 2022 13:36:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CDB28000D; Thu, 1 Sep 2022 13:36:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FA7C80026; Thu, 1 Sep 2022 13:36:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 484BD8000D for ; Thu, 1 Sep 2022 13:36:05 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 269FE40FBB for ; Thu, 1 Sep 2022 17:36:05 +0000 (UTC) X-FDA: 79864219890.11.B0F1A31 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf29.hostedemail.com (Postfix) with ESMTP id C5A0F120082 for ; Thu, 1 Sep 2022 17:36:04 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id c17-20020a17090ad91100b001fdb29e1943so5759495pjv.0 for ; Thu, 01 Sep 2022 10:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=VpPGH26B3JaNkIfQAA8lk0M2gmgFVNCTIjkTHyeGjq0=; b=pbMRwoSFwQraZzGGZr7x5dEEGoBSSezJwWCkFmGpQHOK/gJ9jZl7heInRLvxfZubEx Wj7mEqLcXRU9xxWHf3t+7qSl6CnhQuuJSVxfC2zOp1biHU7QYf19J34qbstjhGHRGexS HGkcltAazUrmNYZt5EcV0prllxXaFrscwVZ4MeMfR5YBnXhADmsl05WrPEXFggi7Gjas iHFv8bsKjSzXGdd4TTCY9kFoj2NTn7nIEkBcfDuEl9YbNXUKLbiljFvk9VYo3diziCwb KLzO8tBeRwrd5fwSMEAfN7AuoAZXQe3a89hFtIYEbzmUJ/R7GnSkDta2dkqtwJ/Rto29 WT6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=VpPGH26B3JaNkIfQAA8lk0M2gmgFVNCTIjkTHyeGjq0=; b=iiHuqMaUVRdKOJ16fbJz50Zm0yCbvwu5o5KDYiZb4DbAex1hEuHH7pvxfP1ihTiNtG 4y14T0TFmlaHFROG/QG/LWlyXbP/utLhp9+RtDJLte2mgwzUVytWmE5IBTtQhNGg+Kw0 q2ZE2E70J3uNl/5w3UPL87l+H4SJgK6ZSGPFO7+8qKJOGmU+L9S0v15YM7RD0gtRJzI3 +C9w/Ta9T0zZMVwy8BrBBzD8UmTuD/J/ilT6oChiii4mLYHt+z5/YuvWYWk1AnnWoABN tiAV+NWoxw1CMsXLbtOaeVsw+t1arQqr9+DBOeZHV50z23Zra9cq/MuZLfaTscWRBmDx 4fsA== X-Gm-Message-State: ACgBeo2fvgV6TpDGj1cO39SbHyU3hjU7PM5nyZ9OYkBwfK5q5aOUHrsY zf1/J5+oJ44/3tKltNoleyYjCO1XvnE= X-Google-Smtp-Source: AA6agR5wOS2QTAVmPmwy5O8cxRkpTpms/e+UYL0OyAuyLY0hN8kJNMUzvy3oLdSFlDJRhvwngVhT3DguUUM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a63:b250:0:b0:422:e3a6:47f2 with SMTP id t16-20020a63b250000000b00422e3a647f2mr4539554pgo.478.1662053763882; Thu, 01 Sep 2022 10:36:03 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:04 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-17-surenb@google.com> Subject: [RFC PATCH RESEND 16/28] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053764; a=rsa-sha256; cv=none; b=nIB8ymQJZoT0WytrTX5BC5mMDbVXuwx7oZf+C0yApEr2etzCcAVq9Wrt8qjPREdDtnUb10 QAKhNoVjVfr1UCqHuh5UE1hClv1I3aRsDkn7h0Tf2fO9ylMjg0rOT6t51mTlAcYVm7sMmv PM7Chg77zBDmOxrtv2eo1XDHIDE2aA8= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pbMRwoSF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3g-0QYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3g-0QYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053764; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VpPGH26B3JaNkIfQAA8lk0M2gmgFVNCTIjkTHyeGjq0=; b=f7bxi0+hNmcp1qBiU4VU6ztstgQHMBxcWpjF8l2MpmmlOdn3lcYzIGiSOpjD2NK8E+MuhK tF/Rvl95kPAaAFyE9TWFKXJjmGIsqQiBW8IB/YYtZJ2VorkWhCh0WcSe3yPwseHZnBhloT lWARmDaWN46Vk47fUShi/sS4lz5P9Fw= X-Stat-Signature: j56imgoofup4eewi9fm71td7odewqbgz X-Rspamd-Queue-Id: C5A0F120082 X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pbMRwoSF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3g-0QYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3g-0QYwYKCMAy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com X-Rspamd-Server: rspam08 X-HE-Tag: 1662053764-524444 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dc72be923e5b..0d9c1563c354 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -676,6 +676,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -685,6 +692,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_locked(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/kernel/fork.c b/kernel/fork.c index 1872ad549fed..b443ba3a247a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -487,6 +487,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif From patchwork Thu Sep 1 17:35:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CF00ECAAD1 for ; Thu, 1 Sep 2022 17:36:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED4DE80027; Thu, 1 Sep 2022 13:36:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5E018000D; Thu, 1 Sep 2022 13:36:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD9D780027; Thu, 1 Sep 2022 13:36:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B6CA68000D for ; Thu, 1 Sep 2022 13:36:07 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 91BE340A32 for ; Thu, 1 Sep 2022 17:36:07 +0000 (UTC) X-FDA: 79864219974.09.1085554 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 5405940060 for ; Thu, 1 Sep 2022 17:36:07 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-340a4dcb403so215172167b3.22 for ; Thu, 01 Sep 2022 10:36:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=b/hqij6YMtUKn07CMMZGWuY0xy+IpIQzVd6D2LweKqw=; b=KRAo4osvkXfti83dlr7pDy0khew3B+/GxQjWEAl6TnacUwl+mtLybx0YLCFQUFV0lg sqmNuGzCNSRuL2SViXVJ2r0wrXOlXyxs3u9yAmqqg3AvKB9F3JJSO7rq3ZQJpqtxodOF i7G/Yua6+zv1pQayP4MuTvdM1yGvpgFJMwTj6dIMUAQKUS2ErYsxnukeEQ3tr0wanncl yMd9bQqlR0P+NMVZ/l+rGeUj3MMbqQRN0a5ORe8qDMpkAqtMCeKLfNSS6669H06w2E3e HhGcmv1HGm3kJoo9dZc+gF0PSVEl7OpLVCRJfkfc6aws3/7f1ZLX+WRY47T74JB1rzMb smKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=b/hqij6YMtUKn07CMMZGWuY0xy+IpIQzVd6D2LweKqw=; b=qpn1l1omQzq65sccHQTjhEjdbMT6tikNkNIAUJ9aIVPf/r/iHtX7zc4QveZ4VS/uLq QTMoGfwsVFIsluP2QJkKirP0nF88qrtZMDC91JOm0D5EO2Tl/yhbWcyzFJKCc4HGKkFA jkDJ+d900NGJo2cDCoRV/W/fv6OJ6w3wYnKUdimcHcLPGcqVv2QbdHzMtX9TfuzmQH/D h0XK41mybyitNsVve/aPIFEXqDiCMoxsu/I/HBnulCYb7hFHIvFTjx5uBAPy/xaEE0nf w7Kjb2HtErlBzKLIkOaryeNkeKlqkteSzHGkgCNUxPQifXQC6ThHtKnEZHYK+01xgOem weLg== X-Gm-Message-State: ACgBeo0WYawlHLr0QDjBZJjizDUI7qij014mAbVitoq9rOziyQolirRD 9LGpCNLAs7P2bjne9KbNUjBaUJa/Yzc= X-Google-Smtp-Source: AA6agR5jEvG/zgKOeD87JHuYvOxB9Vp8GPj5NQSbzUKBEagOp3ZLZ7YArxKySukyQ3HnfRTZ0d7o3HLJ11s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:91d:b0:672:e583:9ed9 with SMTP id bu29-20020a056902091d00b00672e5839ed9mr20010197ybb.225.1662053766633; Thu, 01 Sep 2022 10:36:06 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:05 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-18-surenb@google.com> Subject: [RFC PATCH RESEND 17/28] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053767; a=rsa-sha256; cv=none; b=fCAHflJ7r0aw42OHO7/CTZeZED+LPCw9RndY+hpFtwdFyEtgb9gT3TUBrgEY8A+B1y8e/2 9zdCU92NBLTFxEru7eRJIOrKxs3pymBsC3WqiNmLxb9kGsoNiwdLmd/xya3vRRLX8TNslV Mn6K9IKIb4bJF1D5tFffxVH2IyWEK1o= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KRAo4osv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3hu0QYwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3hu0QYwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053767; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b/hqij6YMtUKn07CMMZGWuY0xy+IpIQzVd6D2LweKqw=; b=IVjdD4TAxYZdeOGzyXGqTkApEw57HhRqX5pbvd3p+9vdPaSV2sqYQ4b4w0H3kuGZHYUYLS jdqdZmbUqg83bDHSn84IEaNRpK/8CPha3CoQSaVVHhy/wHxfdRMLqeii687whszeKsI72w IETNcGg3hc4SbKKHFs5nP0FwiGnqDk8= X-Stat-Signature: nphdb9gys13fn15du4htwkbsf5ox8d65 X-Rspamd-Queue-Id: 5405940060 X-Rspam-User: Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KRAo4osv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3hu0QYwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3hu0QYwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com X-Rspamd-Server: rspam08 X-HE-Tag: 1662053767-620833 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pagefault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to mark all VMAs as locked and prevent this race with fault handlers that would hold VMA locks. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b31cc97c2803..1edfcd384f5e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3538,6 +3538,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * hugetlb mapping); * - all i_mmap_rwsem locks; * - all anon_vma->rwseml + * - all vmas marked locked * * We can take all locks within these types randomly because the VM code * doesn't nest them and we protected from parallel mm_take_all_locks() by @@ -3579,6 +3580,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); + vma_mark_locked(vma); } return 0; @@ -3636,6 +3638,7 @@ void mm_drop_all_locks(struct mm_struct *mm) mmap_assert_write_locked(mm); BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); + vma_mark_unlocked_all(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) { if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) From patchwork Thu Sep 1 17:35:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7611ECAAD3 for ; Thu, 1 Sep 2022 17:36:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FB8480028; Thu, 1 Sep 2022 13:36:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5854C8000D; Thu, 1 Sep 2022 13:36:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B0E380028; Thu, 1 Sep 2022 13:36:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 26A0D8000D for ; Thu, 1 Sep 2022 13:36:10 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F2A18A109A for ; Thu, 1 Sep 2022 17:36:09 +0000 (UTC) X-FDA: 79864220058.12.CB1AFAC Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf27.hostedemail.com (Postfix) with ESMTP id AFD5A40086 for ; Thu, 1 Sep 2022 17:36:09 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id k126-20020a253d84000000b0068bb342010dso4890186yba.1 for ; Thu, 01 Sep 2022 10:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=CCIJQkAZ/MO9EmT200FCEP3Pgl0ISbihE0u87lNvTtM=; b=TDQRkBpQdXfWEYFjLGi7wwliNZ+aP0KfPsImVIZlXY8UjoGX+0qAFXnFV7XGW8IS6l y3VXrf+I2v9M7dPJOYegMdSPcyR+aiyAikKkPyRgRG+NjsroPpw9/4EU6cvNjVuOFM2u OAT4KFsNKXknM9gpAanaNp2m/7ee5CO2zFw2akG6RhT79MTZBIfrWOjgSBLIXLwhTrek 2rRbzNpBpA/qCfbKpI1w2r4GCLJvz7pk5sgu4I7rxGww2tvEjHalildILdcL14mqDmo6 PZNZoe3Kcvw46z9n2HY12LGtvHVlCAmPyMAI6EOC1YQSkgyXm1pQjGEvvxRKe2fezN+L aw6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=CCIJQkAZ/MO9EmT200FCEP3Pgl0ISbihE0u87lNvTtM=; b=qIh+JjdFGqBPCS64TKs/myiEXwyuCIBcxyfJjCITwuKSgA+V10SV1qHA8gWZkKYH2a fl1Uy+ujqc2FteNxs5F1eK5UyPdHOXT2kWZjLR2BmfIMKbBZH5OsDaWiLiUaaYDXy2uu Y4T5nVpj4zClTdApFMsszb0KJJtrcQe29+DaVTxBqFvsFRIxCZBGxkpCHcEdVQTUD5Ci eTSdfQikRZC8jvDBvqv4b5Q/Uov1/tLy6TvKdXbAnnaacTNfqFpmxJs9r971whaZ7z/I Deyn5bN32W5CjQqIMIjhJjBpsZ3wZDuez5KvStoscIDgxtSZjhByudNtLKoZ31fiNTqA p3CA== X-Gm-Message-State: ACgBeo25o4Jjsm8Vp8ARm6Acm4/IpJPSE8Aoex/xtNxnFjy5OxKD9SAP +vmUbBhRLznOXIF5rLNRGFnQ7FthvKw= X-Google-Smtp-Source: AA6agR4qr6moVIHZv2YyIMkU4NbM3gllPWtVCICGpkk6fJrsf7LoBe2Q5GLKhj1E2SNlSJehbfJ330C873A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:94e:0:b0:33d:bfb0:ff55 with SMTP id 75-20020a81094e000000b0033dbfb0ff55mr23236112ywj.322.1662053769056; Thu, 01 Sep 2022 10:36:09 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:06 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-19-surenb@google.com> Subject: [RFC PATCH RESEND 18/28] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053769; a=rsa-sha256; cv=none; b=Ui94+rOYMhcfmS6l0B7NrtOk07bLAyPbBV8PofPeEkMoQ11Ks2/IlYILtfFWOAA7bYZ9zT vsLct8feXUVFzbN4YvxFasPlxV2yM6a8mfyE2VECmPOumtC69aziGS+KdjFu8XFY/bFKRO QpwX8c14mINVOMpKXg2D8CZDYjSwLGA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TDQRkBpQ; spf=pass (imf27.hostedemail.com: domain of 3ie0QYwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ie0QYwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053769; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CCIJQkAZ/MO9EmT200FCEP3Pgl0ISbihE0u87lNvTtM=; b=X6H80T/hTQy8lRlfu5F6F3F3fceKp2p4rvurQeeXGiHaAcGykHAyaMtD4Zonl40h3WIO6O lCM593yYAHmQYJjZgckGKoohkTq3ErdYNQoHm/2u+DfZTjSKfdFXKaz6CL6SlvQW43a/sG 0xKQtY2ahaYJiTkv+huQa4ZOj+JwrkY= Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TDQRkBpQ; spf=pass (imf27.hostedemail.com: domain of 3ie0QYwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ie0QYwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: AFD5A40086 X-Stat-Signature: 9zizcyphbabyqcpkftghoiqyywn5q9r6 X-HE-Tag: 1662053769-49060 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0d9c1563c354..7c3190eaabd7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a03f59c1e78..36562e702baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -886,6 +886,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Thu Sep 1 17:35:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5439AECAAD1 for ; Thu, 1 Sep 2022 17:36:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E87AE80029; Thu, 1 Sep 2022 13:36:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E10398000D; Thu, 1 Sep 2022 13:36:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C89E380029; Thu, 1 Sep 2022 13:36:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B130B8000D for ; Thu, 1 Sep 2022 13:36:12 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8F9EE160A01 for ; Thu, 1 Sep 2022 17:36:12 +0000 (UTC) X-FDA: 79864220184.10.98D2BDA Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf31.hostedemail.com (Postfix) with ESMTP id 3BF2920060 for ; Thu, 1 Sep 2022 17:36:12 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id z4-20020a5b0a44000000b006a1c47c36beso1426604ybq.15 for ; Thu, 01 Sep 2022 10:36:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OeaHSvnl3LkKr4l31U2Volg5xaVk+RMFLZSi6dgjePI=; b=YWMUacgHyaw45xClfelut3B8D4opaNKe1Jn1EvQGqzRnq9IAzY1SbxxDOw3yyYq+BA clLIFH+vCJyTvZqkcuk+nPAAVvk+cIcjjJaBgcqIDZI820XVBHBVS4HV09O0+jx02gz/ GF82/kOz5NOA2SL5LGwa69qRF4uUvf5PcNaP24Dp6cprq2xiDwA1y9+n/C6LME/0C0u+ LXVRqnwXAQFfakRcmYHnZMbYxKQr/xy2mJHdmdiugDAKPQMxfZlI1xF8c8WSAaPgs+Sf 8cODiZAHBJrDioo32O0YL9A9GYzXrzP7oLTKiCYFarAedegbhpTrR5084qGw0ahgqNnD Ah8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OeaHSvnl3LkKr4l31U2Volg5xaVk+RMFLZSi6dgjePI=; b=XLGlLRlG+6fA+0La4DqKNIBVfj2z0pn9Ast6zKxkQdZThTzbrTjF3nYEFAGvGXjMcB C4AKEfyDBUhRg4SanTBKqX29wVXDyVml0YBgMVCjDkhz5xPe88Bwqbox2tZbKvvzuBQa YcSwA8cvoIN5xO1aIn4tPa3YM4Yuf5oXka5DkvIrQ33P/OnAL7I3zawryLcAj7afoDYM IFgbIP9EfF8CySLa+o4Lr0C88MtDLU0ISY5wxd+JI5iBli5pucXHFhatScfHiJJLAnPy E5CaYyaAtNryKJqwYu+tCu1BRA/mlqezKcY2Q4YpVTEw39mUqf9ZoBct3V/bOIkdVauj In0Q== X-Gm-Message-State: ACgBeo1U0+ORHK7J66dD0BJtyVE5wK/wPh3lDIUe7Rg+5IoZRAuT9A13 ZvyATKnr9XUPNtMvpwE4RWumY+he8Eo= X-Google-Smtp-Source: AA6agR5WRy5fZ7FOeYwPAz+MoQjsxloN5BfCZpV2SaYdV4girws1lZ0WYC4kcT46D4arwsdXcJqkq+Ln5bw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:6744:0:b0:696:52a9:2d84 with SMTP id b65-20020a256744000000b0069652a92d84mr20473184ybc.256.1662053771488; Thu, 01 Sep 2022 10:36:11 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:07 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-20-surenb@google.com> Subject: [RFC PATCH RESEND 19/28] mm: disallow do_swap_page to handle page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OeaHSvnl3LkKr4l31U2Volg5xaVk+RMFLZSi6dgjePI=; b=ag5Ieo7KRj89JzaNJ5mJRg0wXee7nbfpoKeqH7w3SPMCO+rd1j4olhLR9j74xVImzlY7yQ fyj/VyVGXx+G1IyBqv9cgEMO/ysC40yk083lB2bSAZ9ng4nMCQPKHcJFmLhKZE3Dp8rozX xhVXFh2ne2Q8IGlnEeBQIsVihQ3E6ps= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YWMUacgH; spf=pass (imf31.hostedemail.com: domain of 3i-0QYwYKCMg685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3i-0QYwYKCMg685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053772; a=rsa-sha256; cv=none; b=kKSYyxb9bpc9+hXqJEgAY5rRL7X9vdm2IZm/Qy2YbWAS/r691Dk9JIzaI0B/yd+YkqXZJP SE+sVCspLJOxrRHn2jwlsYVv/pc8zKxmfIhbFGmL2IFb+Z+Sx2i9Y5CwnNCbpJj3Qmyd2e ALi5WF1HCngPIG4pdDWofNjFYygKe84= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YWMUacgH; spf=pass (imf31.hostedemail.com: domain of 3i-0QYwYKCMg685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3i-0QYwYKCMg685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: ooshmzjiy9qggkkwetkt75cxqd1raxfs X-Rspamd-Queue-Id: 3BF2920060 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053772-243238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 9ac9944e8c62..29d2f49f922a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3738,6 +3738,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + if (!pte_unmap_same(vmf)) goto out; From patchwork Thu Sep 1 17:35:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6E9FECAAD5 for ; Thu, 1 Sep 2022 17:36:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44EB78002A; Thu, 1 Sep 2022 13:36:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AFA28000D; Thu, 1 Sep 2022 13:36:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22BA38002A; Thu, 1 Sep 2022 13:36:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 057CB8000D for ; Thu, 1 Sep 2022 13:36:15 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D81D21A1056 for ; Thu, 1 Sep 2022 17:36:14 +0000 (UTC) X-FDA: 79864220268.04.A5CC0AB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 89D1614006C for ; Thu, 1 Sep 2022 17:36:14 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-343a77f2129so37862587b3.15 for ; Thu, 01 Sep 2022 10:36:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=r+vF5Lo2i1OjVhetiEIJ/9q6C9lfd71c3zrmx7NgTrs=; b=H4jFJ8Dt7ZH5cKrXQpN0z8PEygipTnxhDURDwP2r/L6ST1buj0wC6b62XAFXTWTuHE TRectVq93AO6tGT7OX1/aI7LFkcRGjFnJwRUud+1aazBY2txe7GqBy/qCAKx7DzAzI1F 08tbsOZGFue1vB/xCd7zDPTLBCjfhCGGnp6kgmw7nw7+o4krMVIGwx+wiyWeS1/K9dS3 MY2rAdmwbAo0HfcpKpUrJZgpdcga0Ob80OnRbMBFu5K9+oe2e9P/nHO4ZkzgpYkHj1ef kx644FtyUEEJInyDO+nYaup+HokzUzXjSdh7on7VQkqKFKRY8gzUfUlLXLq38fi2PNgi LbOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=r+vF5Lo2i1OjVhetiEIJ/9q6C9lfd71c3zrmx7NgTrs=; b=U06W7hdV/ESCdLCKgRJAc1qyVlNvrPMBdZ84QFjMPQoKJlwDYw3OB3oZHSscp/SuSz NvzHMEBT4HsX5yuQeGlKmgirCMz53zXyNyKhoCcGb+k3hQAykbzbIA81fLVducFYOAYd 17lq2eoy93LhLuaQ1+rYogMivO5VvL5DzHJJp2b2bm0Or8LdFWA515Y+81jCzFtG7gJ5 lDM4OrqYebsDkaqeQ6cXEw1dqist4eSFHVpjc94DXnVlAPPPPSzGEmsVPn4wuXjgK3z9 BOjuE+JJvvRxBDLfAxk/WmSw4AF7eIX/OvSGsDRqLjzu1x7D8oPYIExW7if7NVGQ9Fg4 1aiA== X-Gm-Message-State: ACgBeo2aDYThOKHpInAT6eJzadecDvT/k5jnZQWBd9hgKsQrZuAY9fbJ byT081M490IKtKBSNKNovQSq+c6TLmM= X-Google-Smtp-Source: AA6agR4BDUj9pfWqCFb+xsqUI8F9TUy99Q1kXWNSZ/lAe4Y/IzT7ZfeAPxwUhMVzFJvC8KeIfmdI79dhOFA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:8d05:0:b0:68e:c838:c24a with SMTP id n5-20020a258d05000000b0068ec838c24amr19294531ybl.45.1662053773853; Thu, 01 Sep 2022 10:36:13 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:08 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-21-surenb@google.com> Subject: [RFC PATCH RESEND 20/28] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053774; a=rsa-sha256; cv=none; b=Lr0pVRvl6kVe5c4ZUbxKbE4bxYB4HfVSX7dxU6DcWtz8YJrRHgO/d96qqAlkwPLWq6K/ox dWK3RcKUoJNPxaIz3CUUT3BgenMXzsaukMxlbiUUmCpt3MwPJztOabCJ7LS+g2fiEJ0KEa 3n0z5uqTJcf1Tef+qQWzTTT/404bal0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=H4jFJ8Dt; spf=pass (imf26.hostedemail.com: domain of 3je0QYwYKCMo8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3je0QYwYKCMo8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r+vF5Lo2i1OjVhetiEIJ/9q6C9lfd71c3zrmx7NgTrs=; b=kPch6ZbXb0aNMXkqizTZ7xwXG9E21NA+/qtxQMqFilvq3CRwTRKeQBNSlsA8qZQVF0KjnH u1rMH+aJwbVClngWlmLEjdVLNkRYtKYgg6FYMIRq+rSjCqUiCEvRhv4GXgBy4C4nPMjdro /3f03QD60J63PFcQgqAkeHSJU4LTHUo= X-Rspamd-Queue-Id: 89D1614006C Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=H4jFJ8Dt; spf=pass (imf26.hostedemail.com: domain of 3je0QYwYKCMo8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3je0QYwYKCMo8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Stat-Signature: m637wc8d8sf5dtr5f11zzzgqkqbsm6rr X-HE-Tag: 1662053774-51297 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f3fc36cd2276..a325783ed05d 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -150,6 +150,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index bfe38869498d..0c2611899cfc 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -131,6 +131,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_vmacache_event(x) do {} while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index ce8dded36de9..075642763a03 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + help + Statistics for per-vma locks. + If in doubt, say N. diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..3f3804c846a6 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1411,6 +1411,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Thu Sep 1 17:35:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66247ECAAD3 for ; Thu, 1 Sep 2022 17:36:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFCEE8002B; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E85808000D; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD7C78002B; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B64338000D for ; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 80C7EAABF2 for ; Thu, 1 Sep 2022 17:36:17 +0000 (UTC) X-FDA: 79864220394.14.EE159C0 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 343D3C0055 for ; Thu, 1 Sep 2022 17:36:17 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id s15-20020a5b044f000000b00680c4eb89f1so4880039ybp.7 for ; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=eEZ4BqXkAmPCIUFHBPa2tHlcApvv3HMidc3eCYWOXuAzHrpAN0HS0lhYeZxby9RzBR 0w6QoRpV+iKQz37VUGml6Y4ETbNA7OPStJ8QApy5HYv7GzeLDO7+EvEq5dUOIcaM6UVC 7rLgM8Qnok81T8RLLnPTDqNW3fzyiOCIgOk9erIOBMCvX1CoI/RbBp7yoBH88B/Pqkwt H8ojE/1F974lnPVAUpGISzyJjuOBuorgNT9VfkfOyqVqOEVMNKxpaqT01PsyXu4rsbpt stRow1K+x3Cpkks4NKMTKIZbGi9T2v1db4rNZMcbp4CaDTkAzPc4JsqrDHDLe5K5oO1C a6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=Shuuzb1OefFNIm2DGEK32sMQkCT3xrZoo3q/+H11bIE0tSCJOrJyj9Shr0LNUBzsa6 HXqWoKz8aYiF25kJN9fxDtTX0WMJ5oavVkXLPq8czzwEmM+D6ctnPrqxt8uuGg6s/anD exCPG2qYI5b/uR2St9X8ZaUWOnHQOCfEzWh/Gs8AojfDKArCi/rkSnMz6fQiLAhIDDvy 6R4nhIfNG2dCgfWUm0hnGzzTEnKvFzH3ttN+vtfuT8ooAaP5BfSobtghRSlUX2U7z6XB QSElQt9Vb8P9s9A6VOOpE0YWSgUnr1c4rLgwOzYFl/MWIsvBuhUP1sYeuvYj5kBsEuun jIzw== X-Gm-Message-State: ACgBeo38gaxfTQOs+x3EmqebATwEAD0pBnTZLdfhDwjdHPMfj6vexKDH LBVG/Kpv0uDsKmwCVn4S9H/Z6DdWvW8= X-Google-Smtp-Source: AA6agR7n//3fxGZjvfyML08bNSpeGfau+aTgkZu5QxUJzLJc9oUWAiawZA1UWbNFrnWa2xxdQVaE06YdPF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:18c:b0:695:9d04:c3e with SMTP id t12-20020a056902018c00b006959d040c3emr19907526ybh.58.1662053776523; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:09 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-22-surenb@google.com> Subject: [RFC PATCH RESEND 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=wInUPXcMW90PaWqfO2Xhd/HUF3Vi9mskVH7u9Dydu070zEU1/s3jNv/LdAmak9hmHE26Tp vrzC0YwhEWQyH1849dKIE0ZY22U0iyF6Xx2GvlCFzGQJXO8xrrl6PTg5HUdW/AbgYqCMoS 2kDeZngvQGHVjHCJJylk9zjO2xa+39s= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eEZ4BqXk; spf=pass (imf22.hostedemail.com: domain of 3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053777; a=rsa-sha256; cv=none; b=oP4dL7CPIJuCoeeOFt4oX9tPRemrSqGZsCqt9wrRvrF/ubdczZYoziw1urrArhseX2hd7I Z2AJ6SnlLTc3hKtuoaY4OGmZTRtmv7hfHmXlMUKcrRc6Bn5SIS9jxv5acOIUvGCtIM0x6x wW/vfnGIWfwVSSypafNuBnY+m89/Fvo= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eEZ4BqXk; spf=pass (imf22.hostedemail.com: domain of 3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: b4oc38kuuswsgrpjdf545nen13xxbiby X-Rspamd-Queue-Id: 343D3C0055 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053777-458351 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Thu Sep 1 17:35:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCF45ECAAD5 for ; Thu, 1 Sep 2022 17:36:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D0AC8002C; Thu, 1 Sep 2022 13:36:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 731BB8000D; Thu, 1 Sep 2022 13:36:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 584A28002C; Thu, 1 Sep 2022 13:36:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 40DBE8000D for ; Thu, 1 Sep 2022 13:36:20 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1DCE612107D for ; Thu, 1 Sep 2022 17:36:20 +0000 (UTC) X-FDA: 79864220520.14.C772AED Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf29.hostedemail.com (Postfix) with ESMTP id AFB0112007D for ; Thu, 1 Sep 2022 17:36:19 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id j11-20020a05690212cb00b006454988d225so4967422ybu.10 for ; Thu, 01 Sep 2022 10:36:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=TUk3GnQXnzjX+dOXwqqLb3j5iouD0aWKTRx7OYILWmo=; b=YQqRrP1cOBMmwt/q5otvkRLq5UlRi5nXamV0kPAgq/Llb+F9rrwouZsmTwkjFMW1zs mWLW2S110jPh4eYPUbnJWz9aEFHyDND9nIDA90HkjT5HQ91YMufxWCmYs0WU2ooUk8Du wzdtu/YDYEQeIpF7CzLVtbR6weBauuj1l4a8qgmBNwRm70UISBCLpd93jjiqNIdE0vgs 06VcH4KM5paek23OhAbPGd46WTDb6+CqL4Pyf7E1NmlU70RTzjmiFKgXWz0xeqjkJr9W NG4J3tGGklDETuONG3YWLZrzk72gi/1u5qstFzrzMP1bg8GyXe5bAXne0/Fv+EFg0xsd ouLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=TUk3GnQXnzjX+dOXwqqLb3j5iouD0aWKTRx7OYILWmo=; b=Ab+dVtG9vmMo2Zk0w+zaarLGQr/lmbtNfL7o5l3csySMhXsixWBdXi3hx4vtqBms7n gVJVvjZEKJRb0lI0DgKJ7JDur89SHota7+1PM0Ei9UijSO/13voG9M5aJabssB2SdFhE mKwd2mI99jgY4wVI+PpDfLoQ0F9w2GNQPSC0eakV3Bj8VR3dfP6VHqQcSs6CHBmuPcd4 l0werkA5h+U3ooMaTNS07vDBcYSzeyJcCJE5Ur9bvvU7/WibwJ1D9vr3OGSvzxKSYVLW L+b9+y3z/WznyRx6fajvqUb0aJ6MfiK8LtmurGr5TZTPiSegLsCyHjRCfStMxEFunbeG +1wQ== X-Gm-Message-State: ACgBeo0YD6wYhteTEX3GfUhwOK6pQyRWjMm105QRXF3U4EAUnBSpP6Ob L0t8Wm+lxULZpQrqVJ8PyvqEfiwLyF0= X-Google-Smtp-Source: AA6agR5HFIn9x2Nfihxx+X7rvP9mQrh+0fzYESFNay7KqUAMxXqLyyvr7HgjmPtUD/1VpOxvT+Ik33q1JvQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:5f10:0:b0:6a3:7eb:762 with SMTP id t16-20020a255f10000000b006a307eb0762mr718810ybb.85.1662053778908; Thu, 01 Sep 2022 10:36:18 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:10 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-23-surenb@google.com> Subject: [RFC PATCH RESEND 22/28] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053779; a=rsa-sha256; cv=none; b=4hEApxwowk1Gyjrgcj5Lbe4OZ053n1rF83raI5CXHn/uGXyh1VBhldRdzjXBE78+sT1wiE YRPVqmzabBciKyVpDKJJ3pHMXZ9BvR118uCFzsnnnZ+4FJ/6UerQG13VbcjO16rDSl6el0 GOGlCe5alY1nWnpXbbhlo9vzgKehkqE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YQqRrP1c; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3ku0QYwYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ku0QYwYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053779; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TUk3GnQXnzjX+dOXwqqLb3j5iouD0aWKTRx7OYILWmo=; b=Semmu8Alaem9MGYienPGRwrmA9+QzJhvfwoiCJHhgu/TKWSzj75iRk87+MBd3WR0g3UWNZ +wSE9GovIGRWBvyTm5FbEpztvUmaBDRmAE7jeimhN7HvIk1PpP+oLuonzmw16nJxcwwQy5 GPB0XwVy0VrhgUDBRUr8wFaZsGhXUAE= X-Rspamd-Server: rspam09 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YQqRrP1c; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3ku0QYwYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ku0QYwYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com X-Rspam-User: X-Stat-Signature: r4ucbj7woejesnm75c4shy45hfhqau13 X-Rspamd-Queue-Id: AFB0112007D X-HE-Tag: 1662053779-64045 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index fa71a5d12e87..35e74e3dc2c1 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1323,6 +1324,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1423,6 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Thu Sep 1 17:35:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B9EECAAD3 for ; Thu, 1 Sep 2022 17:36:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB3FF8002D; Thu, 1 Sep 2022 13:36:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C14818000D; Thu, 1 Sep 2022 13:36:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8E088002D; Thu, 1 Sep 2022 13:36:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 928F08000D for ; Thu, 1 Sep 2022 13:36:22 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 75B8CC1045 for ; Thu, 1 Sep 2022 17:36:22 +0000 (UTC) X-FDA: 79864220604.09.740B95E Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 1C0ACC0072 for ; Thu, 1 Sep 2022 17:36:21 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-33dc390f26cso233052117b3.9 for ; Thu, 01 Sep 2022 10:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OXbMCMaGP4Ta1BmgGlOZJjCZdxKE/BRmpYL5lrPVZHE=; b=pWHztwJetxx0qowvevLvkxkCmwM1DSAQ7wjfJnXzpjijYQSBqPMfI0rG6nxi/Vrs1X CRVPAVdxEFSJIvF4wmbUwj7aCZVm5fVfcrhjUQbBf8n+p0SayMBlb1is6WVkqvLBgMjP AxVJ3sKIAz8Kn28I4Sanz2jyBLjhA5Sgmh51jOnVumpasjSH/FjDxpS14KsDzSHBeJ/z Yb37UvWIKHVN5xc1I6JjnIWO9Obyji3HzdX43E3p6Zl4/DUSGGEIR7W2aNpSc1VwX28m zHCfb00UkmpquMPwx5yKTUZ6c6Sedty+81FiCoVN909m08MOGYjyTskfX4dvSl+gNSYm xxfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OXbMCMaGP4Ta1BmgGlOZJjCZdxKE/BRmpYL5lrPVZHE=; b=Uo/wh6iQQeqH4pljmI0TMaO2p73lyJ+fYj7uIuiqfEzsYrCnjbjirO7cLOMXWGoyvk b6+aJXrdhc/qkgD4Nnse9yFb7nVU6nJgXhXrDD1+0+/RuoGjachOX58KRUsjow434ZSI y7ENDLb7Vjfi6Wl81AVsx++Y5Of1FIYMY8TzXUqyHHpv6YNd5GXLfszBpSAKafJxiLoT +U/arqAwdC1fa3E0wYvrCZl9Zlm9EbtZ8VIpj+JXZCbfcdRc6iZIB1WeFIGZG8RD+Hf7 VmKxyyWrgcdEKeYKabZ4jF99wzhZTpsrwl6T3HhprXfmCpakEeK7JTRGHgi3IJo3R3TG W5fw== X-Gm-Message-State: ACgBeo0RnC9h7QHCVBKYG+Ao1j65mElk5vMTkbruUcIeqAWf7nh2Uji7 2QeUZGMnVz5eZ1f+OvX4X9lua+M1ieo= X-Google-Smtp-Source: AA6agR61BxBFn9qdHU9nkuJxxLaKayaG/YdyWtcsF9HYohujUkmqtvR1t3mL9v3jybeuPbaW7lxRg4vyZUk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:340d:0:b0:696:4dbf:614a with SMTP id b13-20020a25340d000000b006964dbf614amr21859755yba.470.1662053781470; Thu, 01 Sep 2022 10:36:21 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:11 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-24-surenb@google.com> Subject: [RFC PATCH RESEND 23/28] x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OXbMCMaGP4Ta1BmgGlOZJjCZdxKE/BRmpYL5lrPVZHE=; b=u03BRhvZe0WeYGIWtjub1ukr/Ob9XAhwGr5ZxqxcxudJ/dy0IShFyzZ1Ln7fiwvrcelFiY TcQ+4E+jLDUSBs8i3uUZgdL2lnUmW7+0LyY5DFBeFT77FBYkA57nVe05JsxJngMYvrjXJV jSdywW5gyFfoaf6zdaqeGm5q1AQ1GeE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pWHztwJe; spf=pass (imf28.hostedemail.com: domain of 3le0QYwYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3le0QYwYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053782; a=rsa-sha256; cv=none; b=QSycFokzoP8Z1XNtNiuVLsO+G1G0J1CaxijVSrtRov8lz0LlXn9/YdW0Ybkmw5VQUTwaVZ gRl+psPWHhxjKoV4BxmjV5QbFYqHm5iBRYd6XdO9ltnf/HLWOVhiaIPo9szgjeKO8vdatR OrqjHTwoPsN0Bm4HTpIIqK9U8IE+kkI= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pWHztwJe; spf=pass (imf28.hostedemail.com: domain of 3le0QYwYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3le0QYwYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: hzm3eyx8nuo7ahpxc83sx88ef9xjor5k X-Rspamd-Queue-Id: 1C0ACC0072 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053781-361352 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..ee19de020b27 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA From patchwork Thu Sep 1 17:35:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27333ECAAD3 for ; Thu, 1 Sep 2022 17:36:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC8428002E; Thu, 1 Sep 2022 13:36:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B50E88000D; Thu, 1 Sep 2022 13:36:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A3658002E; Thu, 1 Sep 2022 13:36:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 82C6D8000D for ; Thu, 1 Sep 2022 13:36:25 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 508BF141068 for ; Thu, 1 Sep 2022 17:36:25 +0000 (UTC) X-FDA: 79864220730.01.EA0CFB6 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id D8EBA4003F for ; Thu, 1 Sep 2022 17:36:24 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33ef3e5faeeso228815927b3.0 for ; Thu, 01 Sep 2022 10:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=2BdbpYN+cfrtXte3gt5P0NOm/DHEryE/xIPhm7Rz1xg=; b=VLuZCO0l2heZzbazUktWTHDEraaU4YdHq4KUbShg6HpER9jaDaB5f4y1Bzq/1FTL0k Hef4re0Ge4vhRYArZVPVcwZDWlez9FLwBw1PsG7bcW/Kbgpuv9Pp9OJGNf6JnrGh3HSa qO0dYIzHK0WtoIREDnu1aZZK8mbeTv1bHWQjDQ3XJuqrg6Mb+eT6a9R2PACl9oYgExkp BYTnrJsd86ZN2nyjWVo4xWQ85kSe7EiDolkrvd8eM4uZtObKMsI19IUrWn7JY9Y8YYi6 3keq2T13Rmwv4h2gxcKkEIlsATye3QMfjNN6OckMSJWRV6GvnHlPAwoTd+EEwFOjiqHw hSeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=2BdbpYN+cfrtXte3gt5P0NOm/DHEryE/xIPhm7Rz1xg=; b=DqemTmdjsWyrY4AyXAdU8DSUQB4nHdF+Q+LRWlLmVnDirkFQSFj1x3MSC5sa3yPY5I T2HVovWZ896sC/82hxMj3v3+B41zpqLenDlIXolAHRkr83ay6D3DeK8OEyo4sMtzF161 2yt5Zqn7x8IzW7z7Cxj/DZeFg2m4hoiCXC3twC+7HVbNNTo6BPoeDtwAwFfZVGnazoGb uZcMKepT83KcjreTwfjXpStH/5M3szSHzCLrS4neuBUplaDagY81LIGR5MppZ+BhUAVX 901LSkLdQbGQqNO5DoJgC9hIrQkB56Bn0heqYqbmyJdxKF3FNSiLOQcYrmbjzc8R5EXg LBag== X-Gm-Message-State: ACgBeo2/q5qAZmEvWvKTGLQTpP8mkxhWDbuI32/eOCxZm467gAnwxqKU oQhUuA6z8VprJwfXj6V1U+5Gn6oMF9w= X-Google-Smtp-Source: AA6agR4CvVkCfuV2stbCR2feIvgwEhmkbLBkcVFIjpWyBvtco7HBu4vzJWRMfUlZ+fh/HitarNYnWQuKjvQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:5a56:0:b0:33b:52a8:c360 with SMTP id o83-20020a815a56000000b0033b52a8c360mr22687651ywb.329.1662053784110; Thu, 01 Sep 2022 10:36:24 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:12 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-25-surenb@google.com> Subject: [RFC PATCH RESEND 24/28] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053784; a=rsa-sha256; cv=none; b=IPbJNrHOYZ6gKJA8PvrcoeLsxdN82hao/CqdKIlOivI6wvrf2Y1ZE0ycngTLbRTjIm3HZD uO+qHBWph7ZVSvuDZIEXBeq5sEwaabEm54ndp/BGVTSJ/hkvBUjkrvfe2k+a2m9aGZSns0 RwbHzEU5MTnE6pHOm+luBxRg70jXbw4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VLuZCO0l; spf=pass (imf11.hostedemail.com: domain of 3mO0QYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mO0QYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2BdbpYN+cfrtXte3gt5P0NOm/DHEryE/xIPhm7Rz1xg=; b=vTrrVId7ELidLdcPcKkeTp5w6KHxMLeiQs4vcZ02GO7FgSl72H7TiSCi9LrPQnIqKdTGwH ou8HABcQsNJU5c/NDwweiSLnWlcu72EE9QgKe/JFeRsUzpgEmEGFBBTb02oGo2jPZmmkmR XMlEZX2RwaJ26Mw3ruSRYHkk0AR8Ua0= Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VLuZCO0l; spf=pass (imf11.hostedemail.com: domain of 3mO0QYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mO0QYwYKCNUJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D8EBA4003F X-Stat-Signature: h8jgbfd9trycg91jyxhofpen4ditu5rp X-HE-Tag: 1662053784-817870 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c33f1fad2745..f05ce40ff32b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -525,6 +525,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -575,6 +578,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -618,6 +651,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Thu Sep 1 17:35:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C52B0ECAAD1 for ; Thu, 1 Sep 2022 17:36:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FB118002F; Thu, 1 Sep 2022 13:36:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 584478000D; Thu, 1 Sep 2022 13:36:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 388F48002F; Thu, 1 Sep 2022 13:36:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1F0D08000D for ; Thu, 1 Sep 2022 13:36:28 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 032EE1C5AEA for ; Thu, 1 Sep 2022 17:36:27 +0000 (UTC) X-FDA: 79864220856.20.CE1B43D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 7246EC006E for ; Thu, 1 Sep 2022 17:36:27 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id f3-20020a056902038300b00696588a0e87so3564363ybs.3 for ; Thu, 01 Sep 2022 10:36:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=e91/L3Bkt1w9DT3T0Co0rdM+lRiTKLDE/Mz2x0qV8w8=; b=O6gzuYJNv3OY1tULNg+rSFd+sWkIjQSaV6yE8xAfnZPN1YTVFfR+wvj2NeC2hBY0+k GIVWJsEk2y8yIAo13lsMU4IUJIgKbBiJ1vv+1z7dNT8xT+lTOjC6URMO5pLiEVmaefML PewWj9g+Vcp5HQ91HsRQ365b/kw+OZnJbC/u2QqpxScDN6oJLilMCO5pf8M9kEkOXKy8 kK1hF6rXLpQy+lNimDC91Nhhjc7ZyM02kAhq+89cKu94J6/Mxp9HnTyvO1IKLxhNY5UV lVIZFG9WwP01TdBwwmdFuCPUOkErkzSY1WgEGRWny4u3kPuzbkYMq5itJGF6CAoS3las sKWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=e91/L3Bkt1w9DT3T0Co0rdM+lRiTKLDE/Mz2x0qV8w8=; b=Mr/SrBnzmUq+zt5B1TuR8f+s+jg7v8hS6L4nD6DLhPVF+xzJB7aQLgYg8ZPlbPo8o3 4BgkD5LWdcKw15nie4ygKE9IIJ5/CKFler91gXsMrHPBDhpa5vgKHAH0aq1UCT3Wk3jw wdn14ePCCL0HwtObnEgzaKPb1R0Mvo/2Ti6gTHK00DujgQHbFVbeWysSaSCOMTKtDAYE iGef2D2ewE6D8Spamr2z+GHnXdZSDRriu5NFswn3cuNt6K5kV3nzGbr2tFi1RNIIqDGd RzNJ23oH6d9Skj1aZvV5roGxuFPDN6lyHOmCek5C38lmF/yI8xg4756WcuJXgaewYoJH T5sQ== X-Gm-Message-State: ACgBeo1nQMyQMBdppOJdnVy6kIwB6kUFhJe8Fe2ptrb6kFrmQWmJo8wt p/lzPpoPtCTtVONaLDvkb1NCcjiVCgc= X-Google-Smtp-Source: AA6agR6JzhhY8hWMnjFDm/o9DPxBMt9dAFxOiR6uxCkipERPD2P+sojVgosImEy5Q2VS3ZGfPJeoFsIcTII= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:154d:b0:695:d8b4:36b2 with SMTP id r13-20020a056902154d00b00695d8b436b2mr21142474ybu.599.1662053786750; Thu, 01 Sep 2022 10:36:26 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:13 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-26-surenb@google.com> Subject: [RFC PATCH RESEND 25/28] arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053787; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e91/L3Bkt1w9DT3T0Co0rdM+lRiTKLDE/Mz2x0qV8w8=; b=0JYI8vWe0MBSEqeB07uONRZbxUkQrvisauxNTdhEaQ49eo3gE5K6NWLCHM8LA1Jq3U+HG0 AK8w5hjXq4pM//KC8/yTJcXNbE/9FiWsAyxsV88+UtplkZcA+qrlUXUp7Nb5gM2N4aMtbj HiB2ynfUHDR3Eo7r6mVhonWJB8ZHy/M= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=O6gzuYJN; spf=pass (imf28.hostedemail.com: domain of 3mu0QYwYKCNcLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3mu0QYwYKCNcLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053787; a=rsa-sha256; cv=none; b=Udx0D4HTXJJchLpWYYrcwa6SYE4hWiC5LcdsueDMN2EoYiKOJYYae4XNkRa007UgzCX4FE q2qHA+FaIMa/461C6kSIHiAy/6/lB9UPe4M2UfBYj5BCSTVXIOJnI8TbqCXKZP/P4MTTEa RLhHuWiPLyChSD5403+rHiCDAFTSb7M= X-Stat-Signature: m9sttygbfz1pidxswqc7wyh73nhn56w3 X-Rspamd-Queue-Id: 7246EC006E Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=O6gzuYJN; spf=pass (imf28.hostedemail.com: domain of 3mu0QYwYKCNcLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3mu0QYwYKCNcLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam12 X-Rspam-User: X-HE-Tag: 1662053787-536597 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9fb9fff08c94..0747ae1f3b39 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT From patchwork Thu Sep 1 17:35:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F2AECAAD3 for ; Thu, 1 Sep 2022 17:36:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC7E380030; Thu, 1 Sep 2022 13:36:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B50488000D; Thu, 1 Sep 2022 13:36:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A46580030; Thu, 1 Sep 2022 13:36:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8223F8000D for ; Thu, 1 Sep 2022 13:36:30 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5BD2E80F83 for ; Thu, 1 Sep 2022 17:36:30 +0000 (UTC) X-FDA: 79864220940.18.8AB0C1D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 1211A1C005C for ; Thu, 1 Sep 2022 17:36:29 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id k13-20020a056902024d00b0066fa7f50b97so4872442ybs.6 for ; Thu, 01 Sep 2022 10:36:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=GxTYNSy2f61J0vFNTG2PuheznwcSGYj4QW8UtE2cKCs=; b=kyDYAAajD8Ict5/thoW7CTIWmHJF9O2tLJRCa3mFAvG6icZFINT6cR9wp1PVCpk34d ZW6ubLRV3shjslvaFAJXlbBGTzPKjvgVDQeukMqdqtwYsjBpj2zHvD901sTZUMPhJtXT 2ltqU4ShN/OuGiFl+1aMyLRwy+OC1Eq/JzpPgbsdEFfqg/fH8TS912PoVzzOgyXwQY4c Neb0lZJutieMXKVcpmbRLxmWRs14AnetVBjrLGSATrWD8Sx1+eyAMSMAZjNqX2Pwkr4/ vSA7Eawe2WnSKfYYTXdrBr34gwdU8C4wAuI9C4koFQPreF/LzpOWpEs87ZTSq2ciWD6Q PcMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=GxTYNSy2f61J0vFNTG2PuheznwcSGYj4QW8UtE2cKCs=; b=v/GDvkIk+u7t0dLyXgBnFTjUVnJmsr53WBw+53crTedBaoqhJRr3ns6RyoNmmnDE+Q qrfhgujBiaXcqJt2bREhA9ITbvgQKVPfCTNi7zR+bseylWKSwFzyr9C1ySRFLh8cG/nk yACmWx4zr5AUfIGXoroBFcLLtURhBO8u7sEqlIMYh1SYh6FN3wp2DvWhSAngOz0PIBPO 9yjSFK7GbNSyjcZj/aDTcFMyelScQ3drh3O3YG9Vna5w4LDAN8+d2oQwkR4m1BxKuIqc 06ezgFVB7+RUuztdWj28S9oS3BoHjv6pA2v7S/7NV893On7feJjrdivKbVoyi5Tc58t4 wKcA== X-Gm-Message-State: ACgBeo1d30rmFdXit5NbE5LLjfGjr9EM/ifEMZScBpnLeHRCnaueC38m o4ECYQ3/5HGhRmatg/s9rHMkdvc3kmE= X-Google-Smtp-Source: AA6agR7h/HXbYJenfJy6WhKZaSvOltpWgwSfG3A/YQRl84q3AZw84By7flMTVszUCEhzYjvcGJ4xQ/dizwg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:650a:0:b0:33c:f030:7e9f with SMTP id z10-20020a81650a000000b0033cf0307e9fmr23813234ywb.16.1662053789340; Thu, 01 Sep 2022 10:36:29 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:14 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-27-surenb@google.com> Subject: [RFC PATCH RESEND 26/28] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053790; a=rsa-sha256; cv=none; b=Tl+Xn+LyamGOxZfk24diCYUtJWlvQuPPVhg3Bo4jBk6D2PmmoEX9v6lvXyirJF6xrEvpkw wjXOb4F/gTBBIL6UvowA1u8J2vi9dgFjLDD4Mf3dH4onIGWfTUXcUn4kuFF73G97YARDXw zfP1ncJkRkjVQ8+bQJMimo8z2Ink34Q= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kyDYAAaj; spf=pass (imf21.hostedemail.com: domain of 3ne0QYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ne0QYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053790; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GxTYNSy2f61J0vFNTG2PuheznwcSGYj4QW8UtE2cKCs=; b=S5x6kAsHpiTinXPsFAodSY1AJCnV5W0MH1HAxalCFmRIa5m6WdGBOX0Eg2yNbnzpXYgTw1 GJ3VclIQGPzjzM9xfXT7rpm9bnJdzwXB5zgaT/Qcp7xTLPyjiETpuxv5tdpb1KxmKS+KGl DKuiYQbAwbAzYxAYYrsiXcbMq0oWOK0= X-Rspamd-Queue-Id: 1211A1C005C Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kyDYAAaj; spf=pass (imf21.hostedemail.com: domain of 3ne0QYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ne0QYwYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Stat-Signature: oysjdggknxo51j9f1zafd5w17o4d6nom X-HE-Tag: 1662053789-660025 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 014005428687..c92bdfcd1796 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -450,6 +450,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -526,6 +564,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); From patchwork Thu Sep 1 17:35:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA44EECAAD5 for ; Thu, 1 Sep 2022 17:36:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D07F80031; Thu, 1 Sep 2022 13:36:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7590C8000D; Thu, 1 Sep 2022 13:36:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D34280031; Thu, 1 Sep 2022 13:36:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 449C28000D for ; Thu, 1 Sep 2022 13:36:33 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2565E80FF2 for ; Thu, 1 Sep 2022 17:36:33 +0000 (UTC) X-FDA: 79864221066.17.C9F99E3 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf15.hostedemail.com (Postfix) with ESMTP id C9278A0050 for ; Thu, 1 Sep 2022 17:36:32 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33dbfb6d2a3so234918887b3.11 for ; Thu, 01 Sep 2022 10:36:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=vKtrq3HIq0qVh4puVrk/99Ll6PM2V5XVbSUsmvDjj9I=; b=tivdYiD8ZAJK8kY23bdnMjSf6GqsaVRSwINiFtGtPiGjoWq5MFwTdXcs6F+jY7ZJCK /S9q6olR7R7v5bTdyGGJAkaj1esNAWRQw+6z/ncs3LyR7k1VQNGfhn+1/vYhCXSOHiL4 6opl/Yux1TzXEQKGzwy0i4oX47fV8vpc6BEr00D/YdbkLyZ9SHmRwW4kBhG1xNo5Bgh3 mtUhM/EXxGiBQ9J+E4RQUvuZwU4rl5n4anzWcV73TPe35S1Oe6kIZC4EGb3KKcT8isc0 o4+RnjpTFgDSTn6r4q7bAZvWDnDcTYKIxFWZmBACSU/yC2EIvY26FAMGgxLd2T0siGu4 jeUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=vKtrq3HIq0qVh4puVrk/99Ll6PM2V5XVbSUsmvDjj9I=; b=W+RICkDv2dNWIGWs24f7SwqiHg71dwDl71HknnLpmAHbaMMHNPGCr1cmipY1RNHHCf VgXe0Z3ri4Ege/tc2n5eY4Cb8OXEXcDUtBNkdlZoysB3L3vDbAiMy2Y72ueE7i3sEsVP woGYFVVnZi8j8IAI9YPmetnQjTOiSi2vBAAG1HTu0GTu6oTumIoj9ngxQjPTThtK2i8V pjVySE4KniBqSzLDoQC6aqopK8yzEt9NH9aMUAXWbdTW6dsQv6DH2KVt0NiQaaLtCGDo +UKHLZ0oiw3F8INc/YGGVQmqhOEXT23e9/X8cNwhTAoMUYaNJCMfAf+Q1l58Uy/PUEMH vm1Q== X-Gm-Message-State: ACgBeo0wtyLSPLhr4i+mx0TP6UQY5c6Qh+PVaHw8S5fH0Ry8TS+lu+YR IeuzT+eVz2WtQdTaHaJiQbqyfcuUDUg= X-Google-Smtp-Source: AA6agR6ETpVykmR2l2EiORnO/7/6aTPnOWA5nUbQjzShI0yNekZbMOoTb0Et/3IYHZBh1H/HSgpOo7WMY1M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:9a85:0:b0:69b:79d2:7f4e with SMTP id s5-20020a259a85000000b0069b79d27f4emr15568357ybo.379.1662053792100; Thu, 01 Sep 2022 10:36:32 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:15 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-28-surenb@google.com> Subject: [RFC PATCH RESEND 27/28] powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053792; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vKtrq3HIq0qVh4puVrk/99Ll6PM2V5XVbSUsmvDjj9I=; b=6tsexHYBK9khtX+BtlEJkm+/fLidrMIR0/B22OZwwrGZkscVC0vHIFg/BeURdDcrZ0iVvS LlelHuqPKpF2dcpg8M9Y7UyGuG7cciXtyi9lUEuSj1a0z3ssyl7Oj+qb+zmNNDGDP1E4ce eHA2N9lQ0hJlu3HDcuqenfznvp9slqA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=tivdYiD8; spf=pass (imf15.hostedemail.com: domain of 3oO0QYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3oO0QYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053792; a=rsa-sha256; cv=none; b=NlEp6aKjSP7Pj+WETjgb87LwxBj8LgBD4MsWAg5klf2mXNb696f1IQq2EAGE0zqWpQGZ8r 8symndvASlJwWy0c37hFJJDAdFYag48d6ljqExHnY0F1VAHxactTPsiKv39UE7RF45J4wK LTUc4/XdkR5HkPo27Kzn+kCk+mk0gkU= Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=tivdYiD8; spf=pass (imf15.hostedemail.com: domain of 3oO0QYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3oO0QYwYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam11 X-Stat-Signature: u1zhxppfkeygog3whautxm8yxdmb8rz5 X-Rspamd-Queue-Id: C9278A0050 X-Rspam-User: X-HE-Tag: 1662053792-214695 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000342, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on powernv and pseries. It may be use on the other platforms but I can't test that seriously. Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index fb6499977f99..7d13a2de3475 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT_SPINLOCKS From patchwork Thu Sep 1 17:35:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08F9BECAAD5 for ; Thu, 1 Sep 2022 17:36:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F33380032; Thu, 1 Sep 2022 13:36:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97BEC8000D; Thu, 1 Sep 2022 13:36:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CF5A80032; Thu, 1 Sep 2022 13:36:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6862A8000D for ; Thu, 1 Sep 2022 13:36:36 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4D2CC40FDE for ; Thu, 1 Sep 2022 17:36:36 +0000 (UTC) X-FDA: 79864221192.14.A2502C0 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf31.hostedemail.com (Postfix) with ESMTP id DBA1020064 for ; Thu, 1 Sep 2022 17:36:35 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id h13-20020a63530d000000b0043087bbf72dso872264pgb.17 for ; Thu, 01 Sep 2022 10:36:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=UrxjF4stvOlBFlzO5LCraBZUw6+ypd0ZdM4ea92X40o=; b=bRQh8Am0cRUFMzevPI1RLDrAVB56xe5Pyl618ML1xG5UbYShnlVJ+JUC6EbLQY52Qs CIQg2N84xHgD40zt1+Fs+lh7e1EeSPRAe2LH3ariXyDtYJfOUFwmaa2nyKSPAMv4C2LL ixfdMT3qEKwsQzU0zWx46k2Kgwbo6qL2ZMjk9kwAzxSkMtv3E+2VRRPySs58VX6WIHgc UmMCPxZ+/fBN36M7ZZRa1nwJWHDtOFVZGjkXHsv+iTsKxj9Rn4GNhqh8P+D/9RlG7Da5 L4TwcjQTaJ8DlcHg4rJEt6EYpCMy8N2APz0jZiSGuITg1YyaGh6hU+QCGH5FDg92dBPv Wubw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=UrxjF4stvOlBFlzO5LCraBZUw6+ypd0ZdM4ea92X40o=; b=YFHqFtZDGmjqigQeSo30uZPDJ0/uvwv9B/WCgB25PiYX/tGtwKL3oZY/DClegsRgmV 0Q4eH+GKV0q3xZ0bKJ/AcbAzEZq/u5ozI4zHTjksh9DIkwmdDqD6cE56Jdwuaz1JPJQ6 dZvDBn722PZVoHhhj+hfsSkvgKWQD/+v7yPQDVtfb9okDx0u/UEEbCPnosQZcMXSG7T3 sx2xBH2GvWfB24kY8BSGpx4NKEwcMp8A1oNrV2+W6v4nao8EJoqNqw4T96dppz84tUvm nlWNn8qHuPaAJJZxHHH+My9vnlbG4I7D4x5A+NmSzkfQvue1bbP0EVNDovFf/9dNMuzV 64zw== X-Gm-Message-State: ACgBeo2qvk1drbqLs7rI92Xiv1a5ZfuR6p+CfoWm59SUOt7G8Y8SmDUW jJ973lVkt7ux6YSXTo2KvuC7pyOt/9c= X-Google-Smtp-Source: AA6agR7mDmCeuzlvD1IqXjWBDc5GDtYGg6tZms9fC4JNMJ+6ZNkDA7fZXZfEW+TFSrB8Qt+rbpDI3NajKAM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:aa7:804c:0:b0:536:80bb:8270 with SMTP id y12-20020aa7804c000000b0053680bb8270mr31869167pfm.13.1662053794990; Thu, 01 Sep 2022 10:36:34 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:16 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-29-surenb@google.com> Subject: [RFC PATCH RESEND 28/28] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053795; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UrxjF4stvOlBFlzO5LCraBZUw6+ypd0ZdM4ea92X40o=; b=WkZJEe04cQnQqu7YWX3RDjtJNHB+wfQEAKRjbbYQdjpu9wZ9t5vwnsczp87uND71zJMte2 EaEK61FCL5bBx4woBFW56oXFpuYxWoplHa1m6w68bOMDl7sVEZFezjimedohuRV/ksnwHN 3k/2CF0Hytld7n6sXiTTmAUkz/LB6NI= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=bRQh8Am0; spf=pass (imf31.hostedemail.com: domain of 3ou0QYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3ou0QYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053795; a=rsa-sha256; cv=none; b=3JgFn53TqwnckhCsLHhMCxdw3aMvWg/HrfPvy4W8pDC3kMnzN7LQE7FWLAuy+zp/GFz6ow hf3/Yb87m0ehNldOqfXdkge9KuvDV3t5Vgu4Exutufe4kLE/3qimJL2awC9GiGhbevM77f DXzrHl/G0EI1yn5Mid5h1EuOQhNyQlU= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=bRQh8Am0; spf=pass (imf31.hostedemail.com: domain of 3ou0QYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3ou0QYwYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: mfcxbqdtsuc4xcofco6p3dbn7quh1muw X-Rspamd-Queue-Id: DBA1020064 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053795-824814 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 11 ++++++- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 75 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cbaa7b9119..81dff694ac14 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -249,6 +249,7 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 36562e702baf..6f3effc493b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -412,7 +412,11 @@ struct vm_area_struct { struct vm_area_struct *vm_next, *vm_prev; }; #ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ + struct { + struct list_head vm_free_list; + /* Used for deferred freeing. */ + struct rcu_head vm_rcu; + }; #endif }; @@ -573,6 +577,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif diff --git a/kernel/fork.c b/kernel/fork.c index b443ba3a247a..7c88710aed72 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -483,26 +483,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) } #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma = container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_list) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size = 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma = list_first_entry(&to_destroy, struct vm_area_struct, vm_free_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm = vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain = mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1137,6 +1186,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size = 0; #endif mm_pgtables_bytes_init(mm); mm->map_count = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 8399f90d631c..7b6d2460545f 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq = 0, + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 1edfcd384f5e..d61b7ef84ba6 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3149,6 +3149,7 @@ void exit_mmap(struct mm_struct *mm) } mm->mmap = NULL; mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); }