From patchwork Fri Aug 4 15:27:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13342032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 492B9C04A6A for ; Fri, 4 Aug 2023 15:27:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB9818D0009; Fri, 4 Aug 2023 11:27:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A90998D0006; Fri, 4 Aug 2023 11:27:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9313E8D0009; Fri, 4 Aug 2023 11:27:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7A49B8D0006 for ; Fri, 4 Aug 2023 11:27:33 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 34BCFA0924 for ; Fri, 4 Aug 2023 15:27:33 +0000 (UTC) X-FDA: 81086801586.16.DADCFD9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 4BDA8120020 for ; Fri, 4 Aug 2023 15:27:31 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OHf2SZob; spf=pass (imf29.hostedemail.com: domain of 34hjNZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=34hjNZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691162851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=71RLrK6Mb0aadLJnf8lNJkVJrOaKPnZ5THHnlI7+5Ec=; b=XaQN59Hz8WTtMUOFB2WxNec7XdFx2vtWQt+76UH6VW+qkt+Dfh97E098HGFJzlkOJJSqQm Qg+Zehkwgy7peYRZ82og7p/mLI2MmM75XFfeli9Yyoug4ZcK7V4PIh9rc34QnQDzsAgngU nVZfEeUwS5fo5s9jHq6PILG7lSZKWlE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691162851; a=rsa-sha256; cv=none; b=xObCI/k0JyfPvYfYkGoHY6muHR2e2NeJT+H3jSyA6YYUdQIQEJT610O5wMuiZVjLp/xcjD O4CGJxGBoyXfSTqvy8kJtEniHYl72Alj/TpBiKYmf+RdHnJot7yg3ZZmkMdGId0dKWQl3M Gslr3s0JZpNSSKCBJQ/nqRR1l1CvNMw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OHf2SZob; spf=pass (imf29.hostedemail.com: domain of 34hjNZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=34hjNZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-569e7aec37bso23205147b3.2 for ; Fri, 04 Aug 2023 08:27:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691162850; x=1691767650; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=71RLrK6Mb0aadLJnf8lNJkVJrOaKPnZ5THHnlI7+5Ec=; b=OHf2SZobpNzn3AGlr6KHtFV9ESL4KxTr1l448yH9mfHvQG6rFU7J2YifJS5llv2qPx PpgHeIlHU/6JTwXmumhaYwk1tXnlkjfVAFvV/PNzhQvXyGLkw0h2qL6sB8QTiwLXjuXe QdlIpLvwVjorrNCnM7IkC9Q1YluqlB2JrQO5HGkBispCXY7osl72PquRvgCbSYCjs0jf 8Q95VPHGibzWinw6bzA8chJFtlGe0RKXiavh8yMQCk1IFVfCD14UeIn7BThOkhTmJPVm P2JTYb8TjIzrhuRM6un7ppDVABmVHXJtK9xrW64xmqQjF3c4HViHrKlUAuSC+BXjxETY aA6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691162850; x=1691767650; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=71RLrK6Mb0aadLJnf8lNJkVJrOaKPnZ5THHnlI7+5Ec=; b=lLmrCZD3B4x9mFF9Uc+1DFh4ibmcttM0FVSNB577aYhr65bLU0KW3FJOf9Wj6Gm6G1 scs+j1bOldyAOGbfpUSJcMmPv1qO8Q2XR2g8hM+WRTLE6ndEPVMoHXb7EKAbhZlNNxhT BbqGFvlWO3ryJHJJfDX9PXx+RF2izF00FUz8SZdnsx7VzRaoy1Hg89DMGk7a7SZDXQ2Q /0E9cTOYhAdoYnZRzNJz7Xg628kP4UkGlEM1UaTzxtsvgenYwL/4UljgrqtbZ6/XOiIZ ldhDuyykX3cib5g45Z30n1ejZHbbZmffeOF9RTNgiqElAy2qAQrng2rQKBF0sJ5npMZN +HxQ== X-Gm-Message-State: AOJu0YwdG8IYKAMh6k+jmMIw0R4HCAPOLLtfmCSYinsWMdN+B1zSjg95 +9kMEvFZeDmhEjr5nzGzEuG0k18uJws= X-Google-Smtp-Source: AGHT+IFVEJGKTS/dGtb/l3tzkS7ijMf8ddxWQeKeC3bm0h05Pt8naF6mq9QSK0V52/6gQBVKwWbsDIHjIxs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:43a7:a50f:b0fd:a068]) (user=surenb job=sendgmr) by 2002:a81:ad65:0:b0:56c:e2da:f440 with SMTP id l37-20020a81ad65000000b0056ce2daf440mr16559ywk.0.1691162850307; Fri, 04 Aug 2023 08:27:30 -0700 (PDT) Date: Fri, 4 Aug 2023 08:27:19 -0700 In-Reply-To: <20230804152724.3090321-1-surenb@google.com> Mime-Version: 1.0 References: <20230804152724.3090321-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230804152724.3090321-2-surenb@google.com> Subject: [PATCH v4 1/6] mm: enable page walking API to lock vmas during the walk From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Linus Torvalds X-Rspamd-Queue-Id: 4BDA8120020 X-Rspam-User: X-Stat-Signature: 4eh1czznj9yndbfucjbbbmu6u6z6edok X-Rspamd-Server: rspam03 X-HE-Tag: 1691162851-348207 X-HE-Meta: U2FsdGVkX1/zjVX+z6O8TNZgwtnKxnaa8r4uhnp0C1Y1AfjC44w2IU8gE3XQMTzhOT4IiQfdO52SWDaNSIXGXiurfjkEUowZi8Fv2SLYOrGr9UZor6DPLaeGN90uZK3FR5afcauqWez25r1MQlZvmKLovAW9kxQ+xPirRe855IqXicK8UrkK5bJag/Y6BfPtLduKE9Jw/YvCrFQk1thoZ/rheGCfFTZxEGgspXHD2J62GqqfGwX7AbkCbe/t9fTIRtwMpQxOGyPUkY+7Sv6UZnqZH8oL8UegWj/DIP3wEKi1Z/b7HOdM31A9es186w9oR6mzgPQEFCo1HMz9WE4iQ6bhf1wRL8qmfABKYSqLBwQlylj128KC17LN+vheaxOtHEBIGRqwOu3yiUItm4ZrWaddq3uL7uXamNUGWHrKfMU+uaZNtN7/AEF1rInkqdVK5Muo8CZg+0NMxtj033nt9iWae2Ub9RrGe/5BzbpZEk+b3OT6MRyyg+KwFPl+aNGonx2DSKDOnLGyC88zun1l7ccbgZXtkSOcuf3IaG4uadmyf8drBxy43CbMZsm2RAPaAAwoN+sQTeiwPIXkV09zTy/alehugUyEJP82DHggjldsdEz1n/mvOi1YhurfjRT6dpD93aD8CGRA3nA6nCsKkfOZzutYX+npZLJrZHr6Sqre5wiZxYqFSWbdjPrqM5IdeYXduNqWRyuQlWkPNLvkv7VcG34HMEdkMm+W3Jq1YCtrAre51I12PQygnmsbQkAZL4/jxRo1LhOL24kOAnIaS6B3sRi2loSqOp5gqKlAzvf7ckNzZJK1J759cuiAaQYjzbgtwz4G5x4gSkr9ScVwzX1i/k/nbu7jDdWZKznAI4Tjftldl8K27Lv7fn2IMeE7QBxBAmistIA5viKYWqpKGe+h9VFFOOxAlxgpxCA6OmgGVdEadgxwDu/6EdiIF+nib4FTJN9cZTj5+3RnEdJ cRWDfbAP M3xp88TipxPD4YgqmnZc5JF2ngFItUvfSy4/zRd0IqBzOnOne/ET1LqCd6qWa1sk/4QJ+0Q5/F4aGWOF9G2ItEvwe0LOuH20nGy1UqOmrYY2RJiqI0UtPyg1fM51GHdYXhgnADzSnnTpk2aTXGFC0//mILC/g1TP28vEClJGUfuHYbe49TewPe+OFuFqJxt4D8thjz68LBLb4ryzjvIYntaLqwZUtlLuN/eLLZGkmrTYLrgEr9AzyQ6AwHno/AU9HUh5X8BTVeN4iGTAArqDTd4MAD1sWo8cFuRSr4asbd2AswPHzyRDJpn9hYELmAMU4WepUs2mqZaEPAjvrpg+IYNCFzHoKMKTAr64+5X+mZY0obvQ9uGddOamukEsnzagjeHm38xFv5cYq90oZPo8zcD/UYOJ16wTckMYIB4FI9DVNYNUQDBpVcgzMI0V1ahEIzNqWS9pkJFyB08afQBkC3kzEhTzPYTyqG6ODYZmfFxxGWJ2acvujkzSVc6y/bfs3AIcRS4AmiRe+cCwRsLxr6kNh34WZfbzTTBvbc4nRbLfwgrzy0iSAYpA53qUvAqQPkh5FzzVPbW18WZbb7Fbp+DYYE6CJ6SbTGElJCXWYD+7V3pJlnXhXKgAAhU6iR+mB/1aVa55csJ3aJGVy0Vs9ehMunQJYaCQQ21DvnmgkJrcNpU0m1m4baAmd1upoDPERgc6wj3drJQ8DQzTRzFIG8ZdkR66SvOkrUnp5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: walk_page_range() and friends often operate under write-locked mmap_lock. With introduction of vma locks, the vmas have to be locked as well during such walks to prevent concurrent page faults in these areas. Add an additional member to mm_walk_ops to indicate locking requirements for the walk. Cc: stable@vger.kernel.org # 6.4.x Suggested-by: Linus Torvalds Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/book3s64/subpage_prot.c | 1 + arch/riscv/mm/pageattr.c | 1 + arch/s390/mm/gmap.c | 5 ++++ fs/proc/task_mmu.c | 5 ++++ include/linux/pagewalk.h | 11 ++++++++ mm/damon/vaddr.c | 2 ++ mm/hmm.c | 1 + mm/ksm.c | 25 ++++++++++------- mm/madvise.c | 3 +++ mm/memcontrol.c | 2 ++ mm/memory-failure.c | 1 + mm/mempolicy.c | 22 +++++++++------ mm/migrate_device.c | 1 + mm/mincore.c | 1 + mm/mlock.c | 1 + mm/mprotect.c | 1 + mm/pagewalk.c | 36 ++++++++++++++++++++++--- mm/vmscan.c | 1 + 18 files changed, 100 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/mm/book3s64/subpage_prot.c b/arch/powerpc/mm/book3s64/subpage_prot.c index 0dc85556dec5..ec98e526167e 100644 --- a/arch/powerpc/mm/book3s64/subpage_prot.c +++ b/arch/powerpc/mm/book3s64/subpage_prot.c @@ -145,6 +145,7 @@ static int subpage_walk_pmd_entry(pmd_t *pmd, unsigned long addr, static const struct mm_walk_ops subpage_walk_ops = { .pmd_entry = subpage_walk_pmd_entry, + .walk_lock = PGWALK_WRLOCK_VERIFY, }; static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index ea3d61de065b..161d0b34c2cb 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -102,6 +102,7 @@ static const struct mm_walk_ops pageattr_ops = { .pmd_entry = pageattr_pmd_entry, .pte_entry = pageattr_pte_entry, .pte_hole = pageattr_pte_hole, + .walk_lock = PGWALK_RDLOCK, }; static int __set_memory(unsigned long addr, int numpages, pgprot_t set_mask, diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 9c8af31be970..906a7bfc2a78 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2514,6 +2514,7 @@ static int thp_split_walk_pmd_entry(pmd_t *pmd, unsigned long addr, static const struct mm_walk_ops thp_split_walk_ops = { .pmd_entry = thp_split_walk_pmd_entry, + .walk_lock = PGWALK_WRLOCK_VERIFY, }; static inline void thp_split_mm(struct mm_struct *mm) @@ -2565,6 +2566,7 @@ static int __zap_zero_pages(pmd_t *pmd, unsigned long start, static const struct mm_walk_ops zap_zero_walk_ops = { .pmd_entry = __zap_zero_pages, + .walk_lock = PGWALK_WRLOCK, }; /* @@ -2655,6 +2657,7 @@ static const struct mm_walk_ops enable_skey_walk_ops = { .hugetlb_entry = __s390_enable_skey_hugetlb, .pte_entry = __s390_enable_skey_pte, .pmd_entry = __s390_enable_skey_pmd, + .walk_lock = PGWALK_WRLOCK, }; int s390_enable_skey(void) @@ -2692,6 +2695,7 @@ static int __s390_reset_cmma(pte_t *pte, unsigned long addr, static const struct mm_walk_ops reset_cmma_walk_ops = { .pte_entry = __s390_reset_cmma, + .walk_lock = PGWALK_WRLOCK, }; void s390_reset_cmma(struct mm_struct *mm) @@ -2728,6 +2732,7 @@ static int s390_gather_pages(pte_t *ptep, unsigned long addr, static const struct mm_walk_ops gather_pages_ops = { .pte_entry = s390_gather_pages, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 507cd4e59d07..ef6ee330e3be 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -758,12 +758,14 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops smaps_walk_ops = { .pmd_entry = smaps_pte_range, .hugetlb_entry = smaps_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, }; static const struct mm_walk_ops smaps_shmem_walk_ops = { .pmd_entry = smaps_pte_range, .hugetlb_entry = smaps_hugetlb_range, .pte_hole = smaps_pte_hole, + .walk_lock = PGWALK_RDLOCK, }; /* @@ -1245,6 +1247,7 @@ static int clear_refs_test_walk(unsigned long start, unsigned long end, static const struct mm_walk_ops clear_refs_walk_ops = { .pmd_entry = clear_refs_pte_range, .test_walk = clear_refs_test_walk, + .walk_lock = PGWALK_WRLOCK, }; static ssize_t clear_refs_write(struct file *file, const char __user *buf, @@ -1622,6 +1625,7 @@ static const struct mm_walk_ops pagemap_ops = { .pmd_entry = pagemap_pmd_range, .pte_hole = pagemap_pte_hole, .hugetlb_entry = pagemap_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, }; /* @@ -1935,6 +1939,7 @@ static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops show_numa_ops = { .hugetlb_entry = gather_hugetlb_stats, .pmd_entry = gather_pte_stats, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 27a6df448ee5..27cd1e59ccf7 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -6,6 +6,16 @@ struct mm_walk; +/* Locking requirement during a page walk. */ +enum page_walk_lock { + /* mmap_lock should be locked for read to stabilize the vma tree */ + PGWALK_RDLOCK = 0, + /* vma will be write-locked during the walk */ + PGWALK_WRLOCK = 1, + /* vma is expected to be already write-locked during the walk */ + PGWALK_WRLOCK_VERIFY = 2, +}; + /** * struct mm_walk_ops - callbacks for walk_page_range * @pgd_entry: if set, called for each non-empty PGD (top-level) entry @@ -66,6 +76,7 @@ struct mm_walk_ops { int (*pre_vma)(unsigned long start, unsigned long end, struct mm_walk *walk); void (*post_vma)(struct mm_walk *walk); + enum page_walk_lock walk_lock; }; /* diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 2fcc9731528a..e0e59d420fca 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -386,6 +386,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops damon_mkold_ops = { .pmd_entry = damon_mkold_pmd_entry, .hugetlb_entry = damon_mkold_hugetlb_entry, + .walk_lock = PGWALK_RDLOCK, }; static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) @@ -525,6 +526,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops damon_young_ops = { .pmd_entry = damon_young_pmd_entry, .hugetlb_entry = damon_young_hugetlb_entry, + .walk_lock = PGWALK_RDLOCK, }; static bool damon_va_young(struct mm_struct *mm, unsigned long addr, diff --git a/mm/hmm.c b/mm/hmm.c index 855e25e59d8f..277ddcab4947 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -562,6 +562,7 @@ static const struct mm_walk_ops hmm_walk_ops = { .pte_hole = hmm_vma_walk_hole, .hugetlb_entry = hmm_vma_walk_hugetlb_entry, .test_walk = hmm_vma_walk_test, + .walk_lock = PGWALK_RDLOCK, }; /** diff --git a/mm/ksm.c b/mm/ksm.c index ba266359da55..00c21fb4d94e 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -455,6 +455,12 @@ static int break_ksm_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long nex static const struct mm_walk_ops break_ksm_ops = { .pmd_entry = break_ksm_pmd_entry, + .walk_lock = PGWALK_RDLOCK, +}; + +static const struct mm_walk_ops break_ksm_lock_vma_ops = { + .pmd_entry = break_ksm_pmd_entry, + .walk_lock = PGWALK_WRLOCK, }; /* @@ -470,16 +476,17 @@ static const struct mm_walk_ops break_ksm_ops = { * of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */ -static int break_ksm(struct vm_area_struct *vma, unsigned long addr) +static int break_ksm(struct vm_area_struct *vma, unsigned long addr, bool lock_vma) { vm_fault_t ret = 0; + const struct mm_walk_ops *ops = lock_vma ? + &break_ksm_lock_vma_ops : &break_ksm_ops; do { int ksm_page; cond_resched(); - ksm_page = walk_page_range_vma(vma, addr, addr + 1, - &break_ksm_ops, NULL); + ksm_page = walk_page_range_vma(vma, addr, addr + 1, ops, NULL); if (WARN_ON_ONCE(ksm_page < 0)) return ksm_page; if (!ksm_page) @@ -565,7 +572,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item) mmap_read_lock(mm); vma = find_mergeable_vma(mm, addr); if (vma) - break_ksm(vma, addr); + break_ksm(vma, addr, false); mmap_read_unlock(mm); } @@ -871,7 +878,7 @@ static void remove_trailing_rmap_items(struct ksm_rmap_item **rmap_list) * in cmp_and_merge_page on one of the rmap_items we would be removing. */ static int unmerge_ksm_pages(struct vm_area_struct *vma, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { unsigned long addr; int err = 0; @@ -882,7 +889,7 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma, if (signal_pending(current)) err = -ERESTARTSYS; else - err = break_ksm(vma, addr); + err = break_ksm(vma, addr, lock_vma); } return err; } @@ -1029,7 +1036,7 @@ static int unmerge_and_remove_all_rmap_items(void) if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) continue; err = unmerge_ksm_pages(vma, - vma->vm_start, vma->vm_end); + vma->vm_start, vma->vm_end, false); if (err) goto error; } @@ -2530,7 +2537,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma) return 0; if (vma->anon_vma) { - err = unmerge_ksm_pages(vma, vma->vm_start, vma->vm_end); + err = unmerge_ksm_pages(vma, vma->vm_start, vma->vm_end, true); if (err) return err; } @@ -2668,7 +2675,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, return 0; /* just ignore the advice */ if (vma->anon_vma) { - err = unmerge_ksm_pages(vma, start, end); + err = unmerge_ksm_pages(vma, start, end, true); if (err) return err; } diff --git a/mm/madvise.c b/mm/madvise.c index 886f06066622..bfe0e06427bd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -233,6 +233,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, static const struct mm_walk_ops swapin_walk_ops = { .pmd_entry = swapin_walk_pmd_entry, + .walk_lock = PGWALK_RDLOCK, }; static void shmem_swapin_range(struct vm_area_struct *vma, @@ -534,6 +535,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, static const struct mm_walk_ops cold_walk_ops = { .pmd_entry = madvise_cold_or_pageout_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static void madvise_cold_page_range(struct mmu_gather *tlb, @@ -757,6 +759,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, static const struct mm_walk_ops madvise_free_walk_ops = { .pmd_entry = madvise_free_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static int madvise_free_single_vma(struct vm_area_struct *vma, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..315fd5f45e3c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6024,6 +6024,7 @@ static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd, static const struct mm_walk_ops precharge_walk_ops = { .pmd_entry = mem_cgroup_count_precharge_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static unsigned long mem_cgroup_count_precharge(struct mm_struct *mm) @@ -6303,6 +6304,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, static const struct mm_walk_ops charge_walk_ops = { .pmd_entry = mem_cgroup_move_charge_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static void mem_cgroup_move_charge(void) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index ece5d481b5ff..6bfb762facab 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -831,6 +831,7 @@ static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask, static const struct mm_walk_ops hwp_walk_ops = { .pmd_entry = hwpoison_pte_range, .hugetlb_entry = hwpoison_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index c53f8beeb507..ec2eaceffd74 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -718,6 +718,14 @@ static const struct mm_walk_ops queue_pages_walk_ops = { .hugetlb_entry = queue_folios_hugetlb, .pmd_entry = queue_folios_pte_range, .test_walk = queue_pages_test_walk, + .walk_lock = PGWALK_RDLOCK, +}; + +static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { + .hugetlb_entry = queue_folios_hugetlb, + .pmd_entry = queue_folios_pte_range, + .test_walk = queue_pages_test_walk, + .walk_lock = PGWALK_WRLOCK, }; /* @@ -738,7 +746,7 @@ static const struct mm_walk_ops queue_pages_walk_ops = { static int queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, nodemask_t *nodes, unsigned long flags, - struct list_head *pagelist) + struct list_head *pagelist, bool lock_vma) { int err; struct queue_pages qp = { @@ -749,8 +757,10 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .end = end, .first = NULL, }; + const struct mm_walk_ops *ops = lock_vma ? + &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops; - err = walk_page_range(mm, start, end, &queue_pages_walk_ops, &qp); + err = walk_page_range(mm, start, end, ops, &qp); if (!qp.first) /* whole range in hole */ @@ -1078,7 +1088,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, vma = find_vma(mm, 0); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, - flags | MPOL_MF_DISCONTIG_OK, &pagelist); + flags | MPOL_MF_DISCONTIG_OK, &pagelist, false); if (!list_empty(&pagelist)) { err = migrate_pages(&pagelist, alloc_migration_target, NULL, @@ -1321,12 +1331,8 @@ static long do_mbind(unsigned long start, unsigned long len, * Lock the VMAs before scanning for pages to migrate, to ensure we don't * miss a concurrently inserted page. */ - vma_iter_init(&vmi, mm, start); - for_each_vma_range(vmi, vma, end) - vma_start_write(vma); - ret = queue_pages_range(mm, start, end, nmask, - flags | MPOL_MF_INVERT, &pagelist); + flags | MPOL_MF_INVERT, &pagelist, true); if (ret < 0) { err = ret; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8365158460ed..d5f492356e3e 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -279,6 +279,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, static const struct mm_walk_ops migrate_vma_walk_ops = { .pmd_entry = migrate_vma_collect_pmd, .pte_hole = migrate_vma_collect_hole, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/mm/mincore.c b/mm/mincore.c index b7f7a516b26c..dad3622cc963 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -176,6 +176,7 @@ static const struct mm_walk_ops mincore_walk_ops = { .pmd_entry = mincore_pte_range, .pte_hole = mincore_unmapped_range, .hugetlb_entry = mincore_hugetlb, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/mm/mlock.c b/mm/mlock.c index 0a0c996c5c21..479e09d0994c 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -371,6 +371,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, { static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range, + .walk_lock = PGWALK_WRLOCK_VERIFY, }; /* diff --git a/mm/mprotect.c b/mm/mprotect.c index 6f658d483704..3aef1340533a 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -568,6 +568,7 @@ static const struct mm_walk_ops prot_none_walk_ops = { .pte_entry = prot_none_pte_entry, .hugetlb_entry = prot_none_hugetlb_entry, .test_walk = prot_none_test, + .walk_lock = PGWALK_WRLOCK, }; int diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 2022333805d3..9b2d23fbf4d3 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -400,6 +400,33 @@ static int __walk_page_range(unsigned long start, unsigned long end, return err; } +static inline void process_mm_walk_lock(struct mm_struct *mm, + enum page_walk_lock walk_lock) +{ + if (walk_lock == PGWALK_RDLOCK) + mmap_assert_locked(mm); + else + mmap_assert_write_locked(mm); +} + +static inline void process_vma_walk_lock(struct vm_area_struct *vma, + enum page_walk_lock walk_lock) +{ +#ifdef CONFIG_PER_VMA_LOCK + switch (walk_lock) { + case PGWALK_WRLOCK: + vma_start_write(vma); + break; + case PGWALK_WRLOCK_VERIFY: + vma_assert_write_locked(vma); + break; + case PGWALK_RDLOCK: + /* PGWALK_RDLOCK is handled by process_mm_walk_lock */ + break; + } +#endif +} + /** * walk_page_range - walk page table with caller specific callbacks * @mm: mm_struct representing the target process of page table walk @@ -459,7 +486,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start, if (!walk.mm) return -EINVAL; - mmap_assert_locked(walk.mm); + process_mm_walk_lock(walk.mm, ops->walk_lock); vma = find_vma(walk.mm, start); do { @@ -474,6 +501,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start, if (ops->pte_hole) err = ops->pte_hole(start, next, -1, &walk); } else { /* inside vma */ + process_vma_walk_lock(vma, ops->walk_lock); walk.vma = vma; next = min(end, vma->vm_end); vma = find_vma(mm, vma->vm_end); @@ -549,7 +577,8 @@ int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start, if (start < vma->vm_start || end > vma->vm_end) return -EINVAL; - mmap_assert_locked(walk.mm); + process_mm_walk_lock(walk.mm, ops->walk_lock); + process_vma_walk_lock(vma, ops->walk_lock); return __walk_page_range(start, end, &walk); } @@ -566,7 +595,8 @@ int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops, if (!walk.mm) return -EINVAL; - mmap_assert_locked(walk.mm); + process_mm_walk_lock(walk.mm, ops->walk_lock); + process_vma_walk_lock(vma, ops->walk_lock); return __walk_page_range(vma->vm_start, vma->vm_end, &walk); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 1080209a568b..3555927df9b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4284,6 +4284,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ static const struct mm_walk_ops mm_walk_ops = { .test_walk = should_skip_vma, .p4d_entry = walk_pud_range, + .walk_lock = PGWALK_RDLOCK, }; int err; From patchwork Fri Aug 4 15:27:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13342033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E1D4C001DF for ; Fri, 4 Aug 2023 15:27:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 848868D0013; Fri, 4 Aug 2023 11:27:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F9BD8D0006; Fri, 4 Aug 2023 11:27:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BF168D0013; Fri, 4 Aug 2023 11:27:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5F6868D0006 for ; Fri, 4 Aug 2023 11:27:35 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3415CB30B9 for ; Fri, 4 Aug 2023 15:27:35 +0000 (UTC) X-FDA: 81086801670.28.7F8BF66 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 5F6A140021 for ; Fri, 4 Aug 2023 15:27:33 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="hVA3/ZpF"; spf=pass (imf12.hostedemail.com: domain of 35BjNZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35BjNZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691162853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EHpUmzrQnDN6BdHRPVO0KcfDXP0OZ5FbBAepEsSX9Ig=; b=VDyI3a27dFE9hbyNsad1N2clXeoOBWrWB4HRrwOoxSA4SwfJ1TU9f+h3jRk5rOPN8SHTyN k8VWdoUIsSBSJMYVgEvOBXuYY3qoa5zuqSrlebl8pXf8DeC3RwYD0hdEH0+uu25T7wKkhV Bc0znKkozC749MbHSlAuIz9bo+MZeck= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="hVA3/ZpF"; spf=pass (imf12.hostedemail.com: domain of 35BjNZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35BjNZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691162853; a=rsa-sha256; cv=none; b=tITEa0n84J4/orZlivgx/84378cAc6v5D9ICNCSiIFcoDF7PCZKYDGwcQG8rio5oycL9fC OY8jxUBGUNlCEp2LpkdKXUSq92/BV7IR14YvE39HfkYKZ4TS6YDTtwFEg7l86mSt5sagf1 8844FXwTykwbzuR4Y1Eok/G77lrphVU= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d11f35a0d5cso2412503276.1 for ; Fri, 04 Aug 2023 08:27:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691162852; x=1691767652; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EHpUmzrQnDN6BdHRPVO0KcfDXP0OZ5FbBAepEsSX9Ig=; b=hVA3/ZpFJEmsauiGAaXBbjIpfsmpWfR3fbsRgVp5rAEKSt/9A6Xd3kd/eO3AJKJZGn 3AC2ZtIRwv64xHd6WForq8/KFn/Rx8J0hO2y9EPElGCTjTZlFAkT/InmtwUwzE1eqnYK kJASoqtH6CN0ybpUSytgjtDaJ9liKxmHr1g1t4U6N9jwmAFqgwbXaE7cZYKwslngmO8u IcikEpTE0wWzJ4jIGpYVyW//mlltwrZAHvNVNKEreIx7RlfWw5qL77YBxevZNGlJfhg9 fpzNtyDU3iTuo29Qe3cFV5Vv2NkySasem1meuIOuN49ZkF8grQs850+nFjQFoo4gac+A WX/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691162852; x=1691767652; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EHpUmzrQnDN6BdHRPVO0KcfDXP0OZ5FbBAepEsSX9Ig=; b=coDdoLRgpq3WvcdSVTY/si4XPb2GTVO7A2cq2vq25P1CJI4BhWqkmFcK6zFHPYwRyN mkJN5uda+WpnHKykFwdYyQathQ3JCiMD+klYXVR24uQwcGVbkOohkIlT0+1YkClBEPWi 3sT15F+k5uuZX0BsrRJlyJWf59W/ztSIee4B6aboAXSqpVJafNjSbst5ltZE/QCs7YMw 4eXYpJSWhuXcrNWYmwb9z7jc4N5EM+/KEdixS0X4r0OHmCHExBQlLGlvOGF7Y+G/tIV7 6GVen+t4GfZIIZ1d1YeYVe5Hzj6l0+kWf/4UA/B5zJ3dDSxerjYAi1OWyE+yiLvDB951 awlg== X-Gm-Message-State: AOJu0Yxjo0Q8Gu6wqP3P6bbVSF2IsPKILs5bu9I31umubVsFVRcuk2bL 252gwP0E8AHEHvf+XPsy5tns0k5QmCg= X-Google-Smtp-Source: AGHT+IFqHpzaEQ49U8hAJkm1uL5RaOqEsE7NFQT8Ocv+VjqN2lk1j8DiU0ekVvKXC1nMsrd6uQfY/SIVGJo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:43a7:a50f:b0fd:a068]) (user=surenb job=sendgmr) by 2002:a25:aa4e:0:b0:d45:1b81:1154 with SMTP id s72-20020a25aa4e000000b00d451b811154mr7593ybi.2.1691162852582; Fri, 04 Aug 2023 08:27:32 -0700 (PDT) Date: Fri, 4 Aug 2023 08:27:20 -0700 In-Reply-To: <20230804152724.3090321-1-surenb@google.com> Mime-Version: 1.0 References: <20230804152724.3090321-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230804152724.3090321-3-surenb@google.com> Subject: [PATCH v4 2/6] mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , "Liam R . Howlett" X-Rspamd-Queue-Id: 5F6A140021 X-Rspam-User: X-Stat-Signature: aetxw1u5nw4au6etxbf6oc37kbmzedf9 X-Rspamd-Server: rspam01 X-HE-Tag: 1691162853-32297 X-HE-Meta: U2FsdGVkX19CJdxyGiaj0vjbEaVA9SlbDv+13GMxBzmSku4ZlHP5tM8eKk7DzzYrFly3Hg87GRpPzipGFrNJ2ItVZ2BtFekaddhWShC79Ivol8Wq6qYcVw50P+d6UOpi7lII6ntUZJxJoCLEVmPBcJzGo1xzRo1cZPDhX95PENJ3jGnPA/4qGr7yRZ9kSfBU3WLhrBCfxYWhMKB9aXTEfwn5rrPyixEDaIvXqvskyNO9WNpIik2vKlf+tQQNbE8IGh2BidP6qQ1PA4ERduD8dvuYBo2ubljhO4erdOZzwGJ68WXq0giv9lrOa40zS1/RC0yLd4YicU2kzL3/l6WghndqezCKwVvwK+e6OJ/S9PHsDn1gNmkU4J/cZ28QyS4DLgeFH/E8eUK39KVOim7MsytYBnBi0e/mDg/zlT4pDRYD2Wp/d4TXnvJnCfwxkvgGaOXFq+VGcUxUY7++30jfhtbvadNVfyPnaegq9jSUaRBzW2IxMCv8+M6r+ye8EDt1KeqdvOp/xdsG7v3CHX09/rgobkpYIjKUdsIkMcf+v/82AIWVJ+IrPgDycC3EY8go8/5VpAZMGdd3OvPTghMEIBMM8/Zjt+YIZ+aKCW0lHfBf8GAqVcvFyFdZxO3xY6K8lyFt3CdvjPS0bbBlUkJjwyvBwoODFgmPjEypUWfQRgFeGb7gKFpf0p6yPdu3wnupl3M9FJrtN3nIFfStany9r0xxc/xl88SzvY/QYjzhaKc6iBnSyIz538rR9Hmh8x5fDglI4ZBHSUXSePcge2dj8q9P7X0zPje2537dVp6fgMm+4RhjS+CEIMuk71UYmPcB2166+d++7738uuqQwIJyeekKlQ05TeJNXlZaz3rM5E6BWJmeyS60R7MpJT90mXnzttcOHzYRkeFRs8xKy5wFo89OXoxfIW71TMnyF9e1xePYxIAu+cc/vcDpM+nMZX9HhqdC9OHeR+L2Ro/rFZk +xZRGMYL lKaTWxROmpZutk6msA/mExpKSY3w9Uz/jDfVD+9W3fAJA8F9GLg6jq3UsifZGtdnsmlQI594rtqwabXzjolJFQbdxOyZEkm5EDCPNfBG/O9VuCbfewIH3/N+/3KThMeztc4uu0Y3uA6GTv03J6XvU7kZEARL0YmLjJVj1n9rinwg/tEghuslU0rhuZ3bK3YE6M317P1pT0DNkgJ4qvy5QSgWskam6YT8M3qvLB6SOoyXwDkDTWbq2dIAL/lZgYh/9SvINtZuvRN6pOhJ05j0cvfmEvLQXUiWA0acQJVq/SZui64CAXwKNLhjuhi7FXtEsmniCaqO6LYfHLHFgbl3bFQ21IAz+kfOQ4yTt8QT/gcwoz7pQ0H2mtX8HXjxDH/cqtE3nm/gUTLcGBU85QJqHH+i5lXLnYYgRtDu6CNGktvmP9ehtnR9LQWH1yGU+Iu8VkXIQsDcwlfJPD4dJ9fclzPlw49E/aCCMfCot9j4Oe+k4oqTyXC8hFebs6Lfkl3kSPse+GrEE2IzDfvXARalkkbtWm56AIIfg1D/DfjbFC2QGuXjtKxbKcgn4FQNwfrFCfoKm6ffOsRhAdRpp0JyNsi7KLy5ed6V/K+/rtV22dLTyCkwjKJwUGb20kTa6iK1BHnGkTQyOw1GmprI57ruwHUEL2fCGtYayHR6hR/Gr5lNDViN4Gnuj2Uk+FrPFIO63FPGJoG+ntKAHneG7t+T+1qJunw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When CONFIG_PER_VMA_LOCK=n, vma_assert_write_locked() should be equivalent to mmap_assert_write_locked(). Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 406ab9ea818f..262b5f44101d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -750,7 +750,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline bool vma_try_start_write(struct vm_area_struct *vma) { return true; } -static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) + { mmap_assert_write_locked(vma->vm_mm); } static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} From patchwork Fri Aug 4 15:27:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13342034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE58DC04A6A for ; Fri, 4 Aug 2023 15:27:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F264E8D0006; Fri, 4 Aug 2023 11:27:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED5668E0001; Fri, 4 Aug 2023 11:27:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D774A8D0014; Fri, 4 Aug 2023 11:27:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C97948D0006 for ; Fri, 4 Aug 2023 11:27:37 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9D2E9C1535 for ; Fri, 4 Aug 2023 15:27:37 +0000 (UTC) X-FDA: 81086801754.06.9937E94 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf20.hostedemail.com (Postfix) with ESMTP id 9769E1C001C for ; Fri, 4 Aug 2023 15:27:35 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="jZlRzwl/"; spf=pass (imf20.hostedemail.com: domain of 35hjNZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=35hjNZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691162855; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LfajP5IIbFpg2Y7BQRxvk1sHiLCqjf/rsCkC3jBCtoI=; b=lWagMTxTnY7m4y+oE64RibVRdrv/uVBLfGc4YuI5lnCjS9ziG01zPlT+j9P+lEQkgZYgqU O7BHJW+4V9KUBBUXs7/l4JXIQkRcUCL7VdbsNIiTst0NlMfBxIxU/9VSHoHFuhRmoCiMuL 6rF5AVggYUjCZZBG+0Jsk/PPs46QqC0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691162855; a=rsa-sha256; cv=none; b=lRuPQ/tFcHcV/FHTWJT9xONun2MMiWlfZnrwX0jLqHR1ax6Lwp9OwWuODyqCqLCVfexl7T S5qXdH+o/ym50uLWgOJ7RyG9crFtoB65yFBDhr7+rgWq7sBnk6g4AVxfgoA14405Z9qXO9 zgDorsZq8cl0VO4910ZDUjjUGGX3Ko4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="jZlRzwl/"; spf=pass (imf20.hostedemail.com: domain of 35hjNZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=35hjNZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5866e5f6e40so51348037b3.0 for ; Fri, 04 Aug 2023 08:27:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691162854; x=1691767654; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LfajP5IIbFpg2Y7BQRxvk1sHiLCqjf/rsCkC3jBCtoI=; b=jZlRzwl/sZxeKY0Q22GmSPxgFIqyr2sjFjG2dJWAnROoCqd561GTzeSUKMtg/2mqt1 ggyBLLrU1fznQHJEpV6iArmu4q/1ElFb+5c8i813a5QvYsRPPftZDP04yYsFkEpBfTFv biqZHQTtxZS7XMj4IKe/OZIiQ2Hzx3Sc4KiQkuiuVJ83Jibh5wp9rdvvqebLNTk9QL7W NA08SkaqEcu5wURtIv2bzCzal6v9uxf83gQzzLLvQVI6Kfk5jLDfOArO5bHqXdZSMQ1C ssQWic0dvDYtH0HIwM19ShxQ3BVwkNI58J3XyBg9Vj3lhfWdSX5jmfTA1qFRH0njGemz MjSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691162854; x=1691767654; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LfajP5IIbFpg2Y7BQRxvk1sHiLCqjf/rsCkC3jBCtoI=; b=C8/oNXu+GQxOUdiH0U6pLc0+4wnnxIDzugLgsJQ+MfRa+9R7Uc80ItinF1JGwpq+x7 6CJ9bBVZsH4LA7SxLhWSNhS3y46ryDJVtaDvx6I47E4cesZqoruZZzD8IPynJGLmudZN xR92WH0vTOCAevlYXBWNFtzi361/RushCL4hmgiBxOykwsaDPqyQI8Cr2S9OiFDJwTmA JACyVamVihLHXkgn56AqZByVeyre73dh9Vy6HN1Z8LLIjJ+8NrcwnwvKlqbSpJJrB6Tl L8TNIhFXu5IRH3l0rp8et+Y+i06D5v5rU56nKas/FJgGP+k52xq6yL4NNfh2vs5WL0aU EB8Q== X-Gm-Message-State: AOJu0Yz5NYNUlmU4C7hsJLEQ+HykOdNWiolo+Luwv6feY7C+K9W97gkc oxn01bay2U4u772MRUOG4tcOGqjgMqE= X-Google-Smtp-Source: AGHT+IHa9ndTaenWukiguHAw5+MvFDtv8DkvZBd31Rp3irRnrIogFfiHRjgfkWXCeFLeklAX7f0FtCBLNkg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:43a7:a50f:b0fd:a068]) (user=surenb job=sendgmr) by 2002:a05:690c:448a:b0:583:9d8d:fb0d with SMTP id gr10-20020a05690c448a00b005839d8dfb0dmr1503ywb.0.1691162854651; Fri, 04 Aug 2023 08:27:34 -0700 (PDT) Date: Fri, 4 Aug 2023 08:27:21 -0700 In-Reply-To: <20230804152724.3090321-1-surenb@google.com> Mime-Version: 1.0 References: <20230804152724.3090321-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230804152724.3090321-4-surenb@google.com> Subject: [PATCH v4 3/6] mm: replace mmap with vma write lock assertions when operating on a vma From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , "Liam R . Howlett" X-Rspamd-Queue-Id: 9769E1C001C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: jsifbrqf1y3b3xwcojqp13hgbakh5dm3 X-HE-Tag: 1691162855-585624 X-HE-Meta: U2FsdGVkX1+X/bnr7jB7h/hO3TvlE3S/72d25/eS7W5yhS8OHMnC/bYDBU0MppaqAt6tkhiZnrpcxC/lozoTpfaQ6WvDyeQAn0R1DjrNsKb3nGQ9BLxNg2hEWpi0N7YjTGiEfVPpJABlQnRdSns2NGfvftOMluFxvTURVEvwQrDoWFdDFwPFdYIZKe+lT1+qxIA4ypDkkJVI2+K1Syv6i6C8kJ9TO+W58w7sCMRhdi+yXntAWqZH0Yqm4yCHN9x3fCUSxL5LjZpWsvX7XsJnJ2380GaxYKjwoPrsOETX1B7k4OINoEh9yYpS0rA3FDVllyTSc1plouMLFSjyDMvd1OWzncHlrkq4H2XvA3wWdGmZXn9WfW54xfsej5i5Fc8hmFj0+QSf7xaKFXIGaL5J6RMo4wrq824F1ujIbuvvQZD6oeS6d1F9+P07x/Y2s15QMG1BRWP1qynGR9iAyMq8rC+vFvE1rYfUZjc4I9B7xSq2WbSELDqW4Dn13Eo/fylgmNEE+UWjag7fe0lw4AzwW1T4/Aw8wtg6Q/isUWBXBZ6v0EmfOK1mNfOYzL10IJALbJNmVQh7tg+UGEcFTKQ9IpO8psk81RkxRPRW/L7r8Hd8L5wHuN7/GqACstxHq0eClxE/i1WYY0AJsOIaADDbCvZhS9lT0F2y3Hlqmvb1sDE70YbenkYMK/L1/z+HwD2wDC5AEO0ZM4AQP/KUZS/+Wtml2onRxleYOveRdwxY0zUjfpo8Vrh8zuJdVoIL+KwwFhws6Y8kH6ZzoPZYVqDXp1K7HkzUy8J9ott1Wx0csV6k/UnxPV/4gT79Kjv2aL865BssUjxKSZ3OBWXC9yD4rozcLw19ue0CGKqpFpTbay4SwAfZADGfm9DwaNXLYhYRfCSCpPYezZb9N9CcxHPefp5PbbaUoGgcSJl7WdIS+pdw22UA9piYGGEwEyS+wRbknic1Yj4z5a2Qm97T1G0 Dp8hSMXF KpZYyUtg757AjuoGnQGfyVVtFCEZyCUqjh5QAoZ3JEVDU0caJriCZ5qzHA7hq/1cJLMtwcQcbZKNnGMwaDpWIO39YKEiXiEpVw8FFXF6WyAvTesIby785kpBPS/InQlyk0gB6mkMXjgNAidBUbBzHVxw26zkfZuoTspTPPiHOiodAxhI6KHfmqBueIA3zlEdwo14+7zIwCvAEQ1KeljX0GsnavRG5FGfLXswCaktpnve7dGtNtRnAU/Gx7HNYXcPUuZmQlGUDLbxvsordrWSwFepLh6atv6FkIAVieT71nzZoNMVo3z//nFXBIVHm6rhx3wIgvc/UZZRv76ZW0kp82X0Apu3a0zWTs59C3EKyKpvy2UhDWxlW9FckKtFTGrJ/ddhuA0OsWDqSy71aEKd9MqZjJCyKuJ7SVyNxpdrbf2e808qM9wp5Vsx89SKedzyYGaN6QPLnKo0ZBPWoVs1G/x9GCpHdY8NoU1O08nrCpAyXwS5R0hPxkrRshH8AYQTaXHqPWjt5alIPbbPXXF9J8J/sLYQKZ5mdPGzlPkw5C5dbJ6lPLwCW7aG0q5zl0rTb6WE687MYXdQWOko8YicT8Z5ajzX0zLpaFcgodBFq1+NH2vBX7Tp/XrjZWrp8Yw8RE11Q X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Vma write lock assertion always includes mmap write lock assertion and additional vma lock checks when per-VMA locks are enabled. Replace weaker mmap_assert_write_locked() assertions with stronger vma_assert_write_locked() ones when we are operating on a vma which is expected to be locked. Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 +++-- mm/memory.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..1d871a1167d8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5028,7 +5028,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, src_vma->vm_start, src_vma->vm_end); mmu_notifier_invalidate_range_start(&range); - mmap_assert_write_locked(src); + vma_assert_write_locked(src_vma); raw_write_seqcount_begin(&src->write_protect_seq); } else { /* diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 78c8d5d8b628..1e43a56fba31 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1495,7 +1495,7 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, }; VM_BUG_ON(!PageTransHuge(hpage)); - mmap_assert_write_locked(vma->vm_mm); + vma_assert_write_locked(vma); if (do_set_pmd(&vmf, hpage)) return SCAN_FAIL; @@ -1525,7 +1525,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v pmd_t pmd; struct mmu_notifier_range range; - mmap_assert_write_locked(mm); + vma_assert_write_locked(vma); if (vma->vm_file) lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); /* @@ -1570,6 +1570,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, int count = 0, result = SCAN_FAIL; int i; + /* Ensure vma can't change, it will be locked below after checks */ mmap_assert_write_locked(mm); /* Fast check before locking page if already PMD-mapped */ diff --git a/mm/memory.c b/mm/memory.c index 603b2f419948..652d99b9858a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1312,7 +1312,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * Use the raw variant of the seqcount_t write API to avoid * lockdep complaining about preemptibility. */ - mmap_assert_write_locked(src_mm); + vma_assert_write_locked(src_vma); raw_write_seqcount_begin(&src_mm->write_protect_seq); } From patchwork Fri Aug 4 15:27:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13342035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DD77C001DB for ; Fri, 4 Aug 2023 15:27:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D7478E0001; Fri, 4 Aug 2023 11:27:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5867B900004; Fri, 4 Aug 2023 11:27:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 400AB8E0016; Fri, 4 Aug 2023 11:27:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2DA158E0001 for ; Fri, 4 Aug 2023 11:27:40 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D7252C1535 for ; Fri, 4 Aug 2023 15:27:39 +0000 (UTC) X-FDA: 81086801838.08.1F77319 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 104D7A0015 for ; Fri, 4 Aug 2023 15:27:37 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TyIdh0Xc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 36RjNZAYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36RjNZAYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691162858; a=rsa-sha256; cv=none; b=a5ktt8RnBAqn4/tNdWNftU9quNMNUvuOO5IxO8sDFgd6xMlv1X8q/QEiVHpEqrqFRcNH/c 9TqQd49a/iX7Te8Oe/YNMneemL7gErl97xVvUQpUFFzECK/D5MILD7AHlcyhGfD3ZlzRSr 23rXVwPFHRFjpbibMnHwYyt00mNm9EY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TyIdh0Xc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 36RjNZAYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36RjNZAYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691162858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=520TPAR42nWNvH3iwlw29nIZA75JEpTNGHc6Ua8ufv4=; b=Otqy1vG21lT1P3FZVPZhNsBpe7nJ5PeK8xdcMtURYAFA9vKsHsYCWrgIpkjgOQoQjaUALK 8jaaDaDLWLNM43HfahkdHOGAU7ANtpksITxEUccOuufU9PrCPGV91MX5xXaE/ItPMS/diY Fc5VPqXZtqkdMvwOjLPpgOdxsmqPWSg= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d41fa6e7103so1453862276.0 for ; Fri, 04 Aug 2023 08:27:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691162857; x=1691767657; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=520TPAR42nWNvH3iwlw29nIZA75JEpTNGHc6Ua8ufv4=; b=TyIdh0Xc7bjO/FKNU6ggV/1Dv2MMNNGLkgGiA1/wDC4kswrAYhcsNTWyI3nJOXhJeP aqsdLhrq6CUrp6q3DossAbcj51QnYlaAgR/aq+EsQJ3GN3Goge3dO8goY6tsWeahbwLY Emnw4tOdwC3ADtuaCKz1puhigXXZzjx89Nbekny2jzBpX5vMePV4JC0cjOazLXCBYExZ v6vsGj8x/91TJQOhZ43Nfe9PYgY1043oenv1qnEYngwp8wioDACX0eYyz//e76Lw3d64 gfVCR97aftdiX0XPkdD7dGCrEAiTfg0V+/tVEv94gIvaHCopa5EQ7aDwdogtXW1JtUu0 wsaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691162857; x=1691767657; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=520TPAR42nWNvH3iwlw29nIZA75JEpTNGHc6Ua8ufv4=; b=ljk8BthaFyT95pRC+FuuzWvDUxTYG8nrpxz/MSSVzY4TpFluooAuJHZu7OkHJWuAkP EHbar6Ae64NRsReaQSyDk6nkEsFWJzoVNI/SVvR6JQwSnf7U5w5jbocxKL2DBoQRI94G es1YXjR9Jijp+oQpnrxLwNGPjqB/xHYyAxdWfUpakersreddr0JF/fd5+bqukS/GlWxl J5J4jXAZwXcMY8bUyZSlj+S0RemWh3LUB9TgPQEcmylvHoan4rzVulWcNEqzFtTpxEAU W+3+H7zIZdke/O1jhVb1ptNfj2vsvX818TjCyeesSnEQYvTADkuuJIiDAKBxOxWxFgAq wHwQ== X-Gm-Message-State: AOJu0YxbyhRJo+NjHLqJVCtP+HhX9lr4M3zgO9MV4dmeO/f3uaSU6T75 Dajx6RkChmbit5vGkUj7UZEpRS9hPMk= X-Google-Smtp-Source: AGHT+IGlTGXhVU4fUQkbJkgfXB3PlqbjxRUP6W6WYZII59aJQ9IDvYRKiBgp8gXvPph741ZWOBMBJXs8JB4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:43a7:a50f:b0fd:a068]) (user=surenb job=sendgmr) by 2002:a25:ce94:0:b0:cf9:3564:33cc with SMTP id x142-20020a25ce94000000b00cf9356433ccmr10206ybe.13.1691162857007; Fri, 04 Aug 2023 08:27:37 -0700 (PDT) Date: Fri, 4 Aug 2023 08:27:22 -0700 In-Reply-To: <20230804152724.3090321-1-surenb@google.com> Mime-Version: 1.0 References: <20230804152724.3090321-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230804152724.3090321-5-surenb@google.com> Subject: [PATCH v4 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Linus Torvalds X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 104D7A0015 X-Stat-Signature: xz769nmwrjwb5r3ofubkk9jd9oeoyt8n X-HE-Tag: 1691162857-108538 X-HE-Meta: U2FsdGVkX1+SteeXtCYU2AQsHU2QM/IwgHqrxp1lIlUpGZuPTYOl6weVhRVqDF7wz6h41HWoLUGstK7i9/om/4I5CuZ+P0/W15I0PSzXes8SxW2Le8dJCFg7IrniyXinoeUvpwQMbY63LRAy+DjQFbJpI8a3LBy8ZlJtYyNaHaxgesSdt7k3JV68m9dCVrbL9qbQZHe6GdqEEEg1d/RmvRsEiEiWYJM88LSZQaWYvGkysdeJahbr8w+Q4XlJI2S/7gftO+UyvXpilrSPL9SxqxI6KoYQ2ODr5a68TfK//umLljPMzEEUG3JEgTapKHGSSGC6shz/ZWFfn6FdLBtglSI3TOn/AXWmwFQhDGa3csm/+cOEsjKFh6tyl9WNuBflc+jNOKTfBMBXJ6uJWCfBkH/kqDN/itNErC4X+vnVz9LE63Y5d0MeODSjbG6ezGg5WKDY+ZGE9WGZzvW3jsW0Yun2rp8w0uZc0IxvUKjbiLR8jrd6//PgzbDifjCWTwb48sDUvibIRCYnVRDBb4tSNPUJw/YLaCsyPQToTjMdCedGGBnEJq0bvwFb0WAKR6itjGYC1ZoO0BI5k1YVwpPdct/nnOxWktzEebDWiR7FIIzk2/7/O2L6cjAZ2T73dK9nkhtbvRpvMPZGCX/7ov0naOFB09nqJxV6D1UWP7f1/6eX94v3iT0JtFA5LOPsgwM/sfxSc5Lf+YnbjKRDB6lEWsNG1GONSFdnBjpe6QB7FBvakcnRHbl+JdMi09+u5IJAminIljY39BP8use1mp+96KjAZfXSKTUoZK0x2jK20agNnW2FVvbcZUTCpt3+Gr0e2MSoei6wYjuZQvq2fSe0fNxexqxIAp17iUK0+tN8uA6kQ+v4O6GPBUx7yT/gsswjTGga98Xl/FdQkqdm6laQfnOKC0FXStUcFooFSwM0dBOkSB9vY1bQOK3OgqjMbw64Do4C6Ujf0S6k8Awubkw h+8EbreF H90lHt+nSAHFCQtQBTN+wxLCGUqzHHbUtrhLFXLN+/cLjKj0xx1jWisFYLWYIBV3KzZ0CGeVRljI5TwEZzwUHRFXCQPGSz3HZqdY0aBgNRCcL6JYMYfBdsZRK08EVXJxdsWKopk8HDPfEY3Yrb2ZE/6YO+VL2kAG4yOpFTr9PPVer5oNapiQrgsU8ZQP6ecA/F8LvKjHTEV52ruR2hKAZOk1I5HbslRxdXgn7GLp8h5eGAL3k7NV4QLngcwwJHCMWH8JM3h0u/SvGhrH1Oh90eJCiyb+THbnvxAbRZ0OUNJge/GpzgCkueJsx+L4isnkh0m08hNW38Vcw6YYLmeryeyGsZD9JbBCkKAWE0xLyr3i9RkQo7h4eZM5ojeLJd6M6znLz6vW+AkKU5GQC5AIN5Vav0ezS+SKHXh06fhmtCbKt1AZjZA4/O5uKGtuVul10P4G+OTqoTt8za+4IpfRlsuuofXh4CPgB+zfGWedRXyJtTnxNS1VDAA0RJ3aCrU0SSoDarHeJkxePjhOpUOwaSHMF9LhHkj/EAgII42HAEkuZBXhNu8wB9OdgBAk7F1xRDJOzfSAFMaCroYdAa0seS6bV7fR49HnJci9cgsaYzP3udrvZ8x83GlFbmSgPzPcLSSlPKKjim6ul6g87Fw+QUNf8zmTuZjMszho0TTxEPt76okefDB9Xwg6HcWQqUAYHkRghLkFWYSXmF8h8dahvPOBIJA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implicit vma locking inside vm_flags_reset() and vm_flags_reset_once() is not obvious and makes it hard to understand where vma locking is happening. Also in some cases (like in dup_userfaultfd()) vma should be locked earlier than vma_flags modification. To make locking more visible, change these functions to assert that the vma write lock is taken and explicitly lock the vma beforehand. Fix userfaultfd functions which should lock the vma earlier. Suggested-by: Linus Torvalds Signed-off-by: Suren Baghdasaryan --- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + fs/userfaultfd.c | 6 ++++++ include/linux/mm.h | 10 +++++++--- mm/madvise.c | 5 ++--- mm/mlock.c | 3 ++- mm/mprotect.c | 1 + 6 files changed, 19 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index 709ebd578394..e2d6f9327f77 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -410,6 +410,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret = H_STATE; break; } + vma_start_write(vma); /* Copy vm_flags to avoid partial modifications in ksm_madvise */ vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..6cde95533dcd 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -667,6 +667,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for_each_vma(vmi, vma) { if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); @@ -702,6 +703,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); return 0; @@ -783,6 +785,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); } @@ -940,6 +943,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1502,6 +1506,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1685,6 +1690,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/include/linux/mm.h b/include/linux/mm.h index 262b5f44101d..2c720c9bb1ae 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -780,18 +780,22 @@ static inline void vm_flags_init(struct vm_area_struct *vma, ACCESS_PRIVATE(vma, __vm_flags) = flags; } -/* Use when VMA is part of the VMA tree and modifications need coordination */ +/* + * Use when VMA is part of the VMA tree and modifications need coordination + * Note: vm_flags_reset and vm_flags_reset_once do not lock the vma and + * it should be locked explicitly beforehand. + */ static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - vma_start_write(vma); + vma_assert_write_locked(vma); vm_flags_init(vma, flags); } static inline void vm_flags_reset_once(struct vm_area_struct *vma, vm_flags_t flags) { - vma_start_write(vma); + vma_assert_write_locked(vma); WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags); } diff --git a/mm/madvise.c b/mm/madvise.c index bfe0e06427bd..507b1d299fec 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -173,9 +173,8 @@ static int madvise_update_vma(struct vm_area_struct *vma, } success: - /* - * vm_flags is protected by the mmap_lock held in write mode. - */ + /* vm_flags is protected by the mmap_lock held in write mode. */ + vma_start_write(vma); vm_flags_reset(vma, new_flags); if (!vma->vm_file || vma_is_anon_shmem(vma)) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index 479e09d0994c..06bdfab83b58 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -387,6 +387,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_start_write(vma); vm_flags_reset_once(vma, newflags); lru_add_drain(); @@ -461,9 +462,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, * It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ - if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_start_write(vma); vm_flags_reset(vma, newflags); } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mprotect.c b/mm/mprotect.c index 3aef1340533a..362e190a8f81 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -657,6 +657,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_start_write(vma); vm_flags_reset(vma, newflags); if (vma_wants_manual_pte_write_upgrade(vma)) mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; From patchwork Fri Aug 4 15:27:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13342036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 320F5C001DF for ; Fri, 4 Aug 2023 15:27:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49EBB900005; Fri, 4 Aug 2023 11:27:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44E5E900004; Fri, 4 Aug 2023 11:27:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2EEFC900005; Fri, 4 Aug 2023 11:27:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1EDDE900004 for ; Fri, 4 Aug 2023 11:27:42 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E2FCBA13E0 for ; Fri, 4 Aug 2023 15:27:41 +0000 (UTC) X-FDA: 81086801922.18.D5D4A0C Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 2C3204001D for ; Fri, 4 Aug 2023 15:27:39 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="HmgF4Vu/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 36xjNZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=36xjNZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691162860; a=rsa-sha256; cv=none; b=R8d0A1j3tkUIOKIxeo9S+UVK1ndyFBRC930oMm51T0WHEIxjlXA3CBr1Xj35ZaCdqcmrQQ /ba0ENdKEHmn8l7Z9DXrvdQq9uGJRrUgHjrM1fdgiKzaR9HYGafYuZjJHLnXEGEnAZ7u97 BroIocP0dXkFgc1XkCqXPEN5VZ7CW0Q= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="HmgF4Vu/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 36xjNZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=36xjNZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691162860; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GZEu7FQxJMwmMQ2ZbaGaDze4HsujzOay4eQMS+qpos8=; b=YTdYivzT18thwmm+i4AKoDy/rcFtsAZb4kw+z52QDmA1FXU4I+v5bp9S82xMG56zK/z1Mf 62AItrvWH/NRF8Z7Alk7OsB8WSa9qeXasireN7gSk0Jw/bup0nZsALLIwTcPuLufEIVv/l sNM6WVCRBHokavvCPR5hd0tixWdvVPo= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d05883d850fso2183632276.3 for ; Fri, 04 Aug 2023 08:27:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691162859; x=1691767659; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GZEu7FQxJMwmMQ2ZbaGaDze4HsujzOay4eQMS+qpos8=; b=HmgF4Vu/z2UFpEEa5aNZa6sedWXmoJ/DZmFFNhEk4pWiLAQkZQVorYTQctKH3uDMLD mj3Hcmc2cHw9IvvhBack7cSbpm6kW8wjEkPiKsi66CbqYjjpDQLrQ5x1+cVpu8yMv2a9 XNBPj0bNqztWfJDeTb2l4iWSdIBL8kDpukfn8vzY9BFDRsihRFmI5G/xZTOwo4NUFRLU +ynAJ9lUkfrprv3GNKOEl6hufYJxWORcGXt0DMN24lgUdh8KXq9YdjamSzLozV9syQ99 NFeV7GUgWtYSH9/+ggUPmVVqbv4d2kVlEfEv2775n79aqrLFQ+3AD73ymHXIu0XeY1Mj vSlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691162859; x=1691767659; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GZEu7FQxJMwmMQ2ZbaGaDze4HsujzOay4eQMS+qpos8=; b=fwyWpYfLK8bPttl0Ry2qo/whfQG0iqfr43uLdEFx5EWhEXmBRuwrtwcO+IfqgyuJw3 y2rW4IpD6LC5fnVOaX1m9fPAmjXRG6vNCQT3s90QbJatZlc2vMVZ26jbFvHZrBeyUZPX Cax56wLxtCjES5+EJzy5wA8hPznW/PCAl7JcTWG99RAaKfs98SIlPBzHYet1AbZ+uy3o OJxHcBwTUzAfVQ4PLIqJXG00S0yLwqjNTsfPaZQpJD3qkrbMUweATBbTPfiwxo1wKdCD fNX86fEwZH/BXm+93P5/tZ3g9b2ZeDK8Y09FVjIxk++i2oRnqrOLql1eSC46Sv4SgKGm 4Hfw== X-Gm-Message-State: AOJu0YyfayyLOC2q19/zl/89dXKNuaN71DczWv2E5UBpoKKHY8i3HuD+ /iTIRE19zJmKrg2H96jRxQo3vRwvRE8= X-Google-Smtp-Source: AGHT+IHAOVPLc3SFAGSwKHt32pNTYMZjrIqM5CAxOwvtZsJvngxPNlrnfOqcpU9flEZvax2zIpplAXOvcxs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:43a7:a50f:b0fd:a068]) (user=surenb job=sendgmr) by 2002:a05:6902:1614:b0:d0d:c74a:a6c0 with SMTP id bw20-20020a056902161400b00d0dc74aa6c0mr10191ybb.2.1691162859317; Fri, 04 Aug 2023 08:27:39 -0700 (PDT) Date: Fri, 4 Aug 2023 08:27:23 -0700 In-Reply-To: <20230804152724.3090321-1-surenb@google.com> Mime-Version: 1.0 References: <20230804152724.3090321-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230804152724.3090321-6-surenb@google.com> Subject: [PATCH v4 5/6] mm: always lock new vma before inserting into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , "Liam R . Howlett" X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2C3204001D X-Stat-Signature: ta8dmc6tu8hiyq5tnc5h91tqwq1adtfw X-HE-Tag: 1691162859-618425 X-HE-Meta: U2FsdGVkX1+2H/jkXOsHtgyLOUeEaGtObbV9UUwFrK3d84wDYr/ciXELFPjhO39i+WMw4swLiSpFCVleobMdmWrXHUNujKFrklD6EuUk2y/8e0BT/Cz5Pw8ZvDNR14T/sQHNv342Gp/6JIt7utCWgByZ30NfZPJSfgu70QH/JZ7gPes/M4vgIVgaX7MB9TLOGRtEphdoGrFP/DSY6KiOZJ7wAOMjtOswvjYOHCZAbxWcyeyu/PVlcDY0SPD4GrTAQAzsyCbcxz3I/rHmhaP7AHFIGUR9NipbEwfO1q04LI2Fy5GUBFB0jxiOJgM1Mbfc5Yzoy/xkiUE20On1H5TmLq2vWuYdFTkOXwowjZnoAHJKQEDef5/qHqgdvz2lfL4NKVTtbMPPxNF3lduiXQTmtvhlF3/i8TMLuVn3AGjctmH88jYdPLt5Sp43iYJ0a+DZJiulIfByJKrAhZ3tHomdDA40oU8z32zQ8ZZJK79ScfrFoKOdriilKEOQe4vgFheVk7E12UXli3LmTKnXRdvAdCieFw9DURkaqXI0GKLajNDPzxkFp/+AZrek16rOI7DaqPqYTbdswdhaYFk4mp8d9mZA6W1WxpkV9F+nO5UbZ/fjVrWxKV9Bml4CWS/nRQnwnmtsJNRYEZoh9fD0Gl6ijUst4xjV6R3TLWgQX21zYQ9aISBnxKh3wAZ3ORpWZzgXcC5lK2vRmt650kq6LnFg8Bp/l6bBZBSaDJQsYTUzJn3Ob8utZwTawSjUVXaE8PdZj6EUwTMF18jlXSrw2V4SvAsqrlsMroC+2K0RoV5odGSNfGGqTjSi6tDX4KC3DUM24VvVosnd+MWC9UZe9HkP8EZNLA6GSc2c785Xkr/twIDyRjM8k9AhtqFJNgdEMVGHQTdVe5PmH//kgOOUZlzPy/J6WEr2T/xquxRwtOJttfxdHlKUzRbNpL5nf7end+/m8cHINIPRc9M0Dq/d7TH QGlqV2Rx mSSTJMBjiXjnzrnGpv0WkujF0/FVE2eo3DnPtd8RzLf9hQ/ciNqgg3nDpasW6ezduS3FmAzc5eqUcJY8bb/bmKh+PG7S0i/GrfeYL+U5c/i+Nx/N7RkqVelxTTGP3G7wv9VdOBY28k7m2yLxmikB/Km4One/zpcaUFku0KRc7UNDVb6ySRpmEUsHu2vBJi+ccbHMRncu3wPRCNqvwc/p69qbJMD7X70/a8AopiuKi/8/ccBuPDpyxYPxBkhuJuDnZFmk2uEiu2hY1NsV4/RghmR164AJSLUt5qH+qUlY50/RozLUtqfnNWBHRbOTjEOoLdFe+EX2cUD3mcw37QK5YSPY0NClCS7PaC/Gysl3t9tXGIAp4uQThyulBxRlWUjflrqNOxQvZtOdX7X73qralm3cReZKtdFPAO6mKgWmgB2iQkwvdrk0H3DLbM4BVmlBSe+3HnG6qOOs5AwlnIMIRtE2ic2VLLVcUBo+9AVztsLYq0ottK81rQYJWe/eBTsM2mrt6MJcW3EkIOtql5sKWlf/H3rgwLt/DiUbyPHUHtmgi5VzX5buzdx6Vy9G7acD/ZY6mrCrf1lXRJ0qJelwrGKN2+FsqHVMeVkiVLHXm3AtQcrY8P1AmBNE38CzIaPr/uHbJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While it's not strictly necessary to lock a newly created vma before adding it into the vma tree (as long as no further changes are performed to it), it seems like a good policy to lock it and prevent accidental changes after it becomes visible to the page faults. Lock the vma before adding it into the vma tree. Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/mmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 3937479d0e07..850a39dee075 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -412,6 +412,8 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) if (vma_iter_prealloc(&vmi)) return -ENOMEM; + vma_start_write(vma); + if (vma->vm_file) { mapping = vma->vm_file->f_mapping; i_mmap_lock_write(mapping); @@ -477,7 +479,8 @@ static inline void vma_prepare(struct vma_prepare *vp) vma_start_write(vp->vma); if (vp->adj_next) vma_start_write(vp->adj_next); - /* vp->insert is always a newly created VMA, no need for locking */ + if (vp->insert) + vma_start_write(vp->insert); if (vp->remove) vma_start_write(vp->remove); if (vp->remove2) @@ -3098,6 +3101,7 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma->vm_pgoff = addr >> PAGE_SHIFT; vm_flags_init(vma, flags); vma->vm_page_prot = vm_get_page_prot(flags); + vma_start_write(vma); if (vma_iter_store_gfp(vmi, vma, GFP_KERNEL)) goto mas_store_fail; @@ -3345,7 +3349,6 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); - vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; From patchwork Fri Aug 4 15:27:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13342037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80BB1C001DB for ; Fri, 4 Aug 2023 15:27:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 847E890000C; Fri, 4 Aug 2023 11:27:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D2D7900004; Fri, 4 Aug 2023 11:27:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 671DB90000C; Fri, 4 Aug 2023 11:27:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 59C2E900004 for ; Fri, 4 Aug 2023 11:27:45 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 33A74814A7 for ; Fri, 4 Aug 2023 15:27:45 +0000 (UTC) X-FDA: 81086802090.16.411E27F Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 19FD5140012 for ; Fri, 4 Aug 2023 15:27:42 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Ze27H+8O; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 37RjNZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=37RjNZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691162863; a=rsa-sha256; cv=none; b=y3x1tC6bQBicVri39bRzQThwMOT5r8P/HrPMQZozCgOw2WcTWz2AVfNWYGC1Zk8sp8XjJv B809Nq0v60V1R9PGT+xYQsl/7WrbtnOct2cfOTnC51b51I/rf2joK6pCet8u1VcNzipz8a cwrgmOvpX3wkjPzBINLCNWlu5UZiiHY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Ze27H+8O; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 37RjNZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=37RjNZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691162863; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=okHSXq0SYkri7E0to28JFQj9dGiLp+hbVT5qT1e9xdU=; b=0BcLNE1l7JDNRYr0DdJcxOsy7sG0jjRMubacQEvHJpJIXFNvqBokVSQ+asxDoYyBYmg6AH mkdn26Noew8yOXRzmcwJXD5uWtvWmg3AdKYm18fPE8y13yIr/AnzTE2hQTLv6lqVNFyvdx GYF4Z1GS8Q0lrepODAefC3CMbuNoFDk= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d3ae42d146cso2209575276.0 for ; Fri, 04 Aug 2023 08:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691162862; x=1691767662; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=okHSXq0SYkri7E0to28JFQj9dGiLp+hbVT5qT1e9xdU=; b=Ze27H+8OOQhF7Oj+2ZOk+cY5b5gFDys4DfQgVT60v89OD7qi5UeZ/cauYLk5j1Z+GU XzmqYz49Fo48ZGd45wNkV/OWAGwTfMj8sHgA+WLxLRz9FINm38B5BG9wrka8I8py53e4 JxZXr7fPwYEz6xiF3SpxqoYBrBtxE6j/MgIItpyaYxeW+m/Vd5hX9SE2Cml3lUFOZW6r WM0e0CYffNX/JyZR7vi5OUlMXdaHPpAW0Kgep2I5V112BkSAgUR3PqlHz5NH22t6usrn YTi2zd8kbs4j6hx+FeLjGtgIsVutMy1LiPSegd4snkm0AH19LFF/qxuEmA9dhEOGf0FI 5fXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691162862; x=1691767662; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=okHSXq0SYkri7E0to28JFQj9dGiLp+hbVT5qT1e9xdU=; b=ge4SEwgujvTwbe/VGKZp5o1zRtN3dtTRliL3ubFjWMRRhOMSsG/Fl2XZGCdaw/2gYv VFWLx2A0RzYSDxGC8YOjzwdwlpYv0kGn1wfCC63nNO8wQzRWbh0w77ZX4V3KoLqSWrVU sfTQioTHg8lHHS8/E3ddh/zTwT45umz88TWfXsOHG9Qd5pbhy+boZPreL6tNxicvHWYk QKqPklJ9t1ZMr5WrWjrcQSLCvSX+Xwy9zFaJEo/Qtuv90JrVKwVZ01e7t4KlPTRuG20i /VKCY6/5ZBlBWqXp8ImKM96HuAAyFTQ3cPS4bmR+dGmFEhSZ+GKiO9ydNrTodJHsy9N0 E8gA== X-Gm-Message-State: AOJu0Yw6uky5WpOLY/Vtg5f77Jp/RItfK8ZuPq7zgycusXAjGsR9s3MI jlhZOD4i28iYbX3x4ZSeReScX1Rq2bQ= X-Google-Smtp-Source: AGHT+IGWFF8LV12V45AQ8PHaqPReOn9OGTO6qHNI1IHy9jYdCHxrz4hNZdLqwTqM3hhHqBDlThuUA+9rLzA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:43a7:a50f:b0fd:a068]) (user=surenb job=sendgmr) by 2002:a05:6902:1614:b0:d0d:c74a:a6c0 with SMTP id bw20-20020a056902161400b00d0dc74aa6c0mr10193ybb.2.1691162861553; Fri, 04 Aug 2023 08:27:41 -0700 (PDT) Date: Fri, 4 Aug 2023 08:27:24 -0700 In-Reply-To: <20230804152724.3090321-1-surenb@google.com> Mime-Version: 1.0 References: <20230804152724.3090321-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230804152724.3090321-7-surenb@google.com> Subject: [PATCH v4 6/6] mm: move vma locking out of vma_prepare and dup_anon_vma From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Linus Torvalds , "Liam R . Howlett" X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 19FD5140012 X-Stat-Signature: cjpkm4my3gwq4ufk6jfuz3pw46y4tc9r X-HE-Tag: 1691162862-164181 X-HE-Meta: U2FsdGVkX1/Zr3ZninQuPM+aQs+8vdm0sEdfYLHN/nWpZUB/xmO6hckVq+swa/+Ob7es2pSoUD1u3kS8fOxvcdoMbK7/BYh6pbrzZdJ/Xnv432IXxhCm4RdGX7d1YZHRuF1mQJJ311fp/Ne5rMM3MUclB7IPZEAZFvJc+aSw0GL/r7RdHx34A5HXB6a43KBb/9M+Mnc4PObfDJsdGmg9oWvSCnEB8r1iqQMJI1mysGu06f4yoVUCBrzr0+8rHKxNd4rNUqN+iTT9RXPqTe+J/Vtxu7XorItWm/K/oUKSc2LLGT/voAxvn3Dxc6bMJzppJGCQrAAPvX0ZpMuw6VR4STBR4/8L0ObbMO5dyla4vQm5IaWvHMnhTYyqm528afB5MQ7KcGPvO456SPLebe1QcO+DodSan4W6c6IP98rDd2WDhl1GUnhB1xGN2Mak6jSDqwBYSBATXycqdn0swvXSVe7lpmfSI5lcGhsQyt87XVU7+S2PCWfnY7InAFjTsBKRv5sPowC8EpWat3D/AE5vvc+tTMzUsnIK6TjHguEV4ndKKtXAZYC2UlMi2wgOcJde63bqFVBOztMstffqshXdzSQDx4ahm4Zgw4br59IrBo3ICEo8ejp9Y9l92i5cTmsaZI8bkD+ymgDlAd+/C4e+CKDlu3bGzmVP+IaHdqg30xPLG+s+s1OeaNrcl71BNGRdxLhhTHJGL1uaRJI+m4ZdZiqTFYqLxym4r4nA2pLMkeLdqgq1P5MloTYzSgWw1Y1M3GVtb+B+x4q71M5wn953XXIumt5En/QeElb44+sidIr1EBmuQOEFbXi2OwyCkwpTzWe3m7mvjHNpLLLRFbfTbemnetUY+JJTdwdKCKq2kqG1BZKykZqUxLul8Jjb46rZK/JpgXOyfgIBFqNGYUSIfkHtR5zt2bqlvXpHxYJSiEVH+pQAekmWcggXjqT7RVSpVB8gZqxppTC9/KUbvyp g4WOOqYV L87hcvl8AMpoWlFwdar78dhbagmXfX1cXyaiQ0KDZR13W9L/rY4iNycO9sRFNHkGbercgRlfRlLe9gyzFRuH9L990sIM2hrWVoxYIv/G9Pm7AafevsrAl2NNMRB8TixCIOdhcReh09ZKb8OaV/gf7nGRN8UyJgu/Pn1Yq/hn/qfjkBT+4aTySBf9/N6K9As5Xk2ec6aD2upyUyaiedF7dbCnDw+NN7LIT5ywkvdbdgHPGzOJthySR63fJz1oO6D6Az4lnjhXwtUmMighEng1OC5i7DkJ31H794CM00Eyfm0Hy2s5Ilx/+sJL+m/2zsHiZptC91bkViV0rtJi4mV3ZAWewjsaaDHJdkmyQ0B5TQqax8LAl/YY1s1LpeNwAmoIjs0jTGepD4nxsPOUTIkkOUd+w78ewAVc6V43WKRnhu4o1AAYPUWeEn6DBN8Eyl/4PKbkhEOwZSw+wRL/Rk6svBaCd7YHVrbi1YcefpHUBeoHz0QC0bgjycksVUnAzUJdzTRMT70GwKaZc0W2eIWQU4CjKg8vAAXdnBrPN9OS5KSPQtzCoJJGuecbWCgFPOzj2AakOm7TU2mMwzjARZ4nkYeG/iW3k6V+4Xn2YqMDIsIQ/fTRbtxwn79w/2ryjJs2YfDWldmJRzygCMkVEgTW/51GI/HbaTcVa2G+KNvzhZEGTly22WGMfQihPFIRkD8QWcEwKuYIt/iyuPsvLM2hIQRpIDiImALIbi0pGnq80urrYBm4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_prepare() is currently the central place where vmas are being locked before vma_complete() applies changes to them. While this is convenient, it also obscures vma locking and makes it harder to follow the locking rules. Move vma locking out of vma_prepare() and take vma locks explicitly at the locations where vmas are being modified. Move vma locking and replace it with an assertion inside dup_anon_vma() to further clarify the locking pattern inside vma_merge(). Suggested-by: Linus Torvalds Suggested-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 850a39dee075..16661427d3e8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -476,16 +476,6 @@ static inline void init_vma_prep(struct vma_prepare *vp, */ static inline void vma_prepare(struct vma_prepare *vp) { - vma_start_write(vp->vma); - if (vp->adj_next) - vma_start_write(vp->adj_next); - if (vp->insert) - vma_start_write(vp->insert); - if (vp->remove) - vma_start_write(vp->remove); - if (vp->remove2) - vma_start_write(vp->remove2); - if (vp->file) { uprobe_munmap(vp->vma, vp->vma->vm_start, vp->vma->vm_end); @@ -618,7 +608,7 @@ static inline int dup_anon_vma(struct vm_area_struct *dst, * anon pages imported. */ if (src->anon_vma && !dst->anon_vma) { - vma_start_write(dst); + vma_assert_write_locked(dst); dst->anon_vma = src->anon_vma; return anon_vma_clone(dst, src); } @@ -650,10 +640,12 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, bool remove_next = false; struct vma_prepare vp; + vma_start_write(vma); if (next && (vma != next) && (end == next->vm_end)) { int ret; remove_next = true; + vma_start_write(next); ret = dup_anon_vma(vma, next); if (ret) return ret; @@ -708,6 +700,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); + init_vma_prep(&vp, vma); vma_prepare(&vp); vma_adjust_trans_huge(vma, start, end, 0); @@ -940,16 +934,21 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (!merge_prev && !merge_next) return NULL; /* Not mergeable. */ + if (merge_prev) + vma_start_write(prev); + res = vma = prev; remove = remove2 = adjust = NULL; /* Can we merge both the predecessor and the successor? */ if (merge_prev && merge_next && is_mergeable_anon_vma(prev->anon_vma, next->anon_vma, NULL)) { + vma_start_write(next); remove = next; /* case 1 */ vma_end = next->vm_end; err = dup_anon_vma(prev, next); if (curr) { /* case 6 */ + vma_start_write(curr); remove = curr; remove2 = next; if (!next->anon_vma) @@ -957,6 +956,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, } } else if (merge_prev) { /* case 2 */ if (curr) { + vma_start_write(curr); err = dup_anon_vma(prev, curr); if (end == curr->vm_end) { /* case 7 */ remove = curr; @@ -966,8 +966,10 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, } } } else { /* merge_next */ + vma_start_write(next); res = next; if (prev && addr < prev->vm_end) { /* case 4 */ + vma_start_write(prev); vma_end = addr; adjust = next; adj_start = -(prev->vm_end - addr); @@ -983,6 +985,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, vma_pgoff = next->vm_pgoff - pglen; if (curr) { /* case 8 */ vma_pgoff = curr->vm_pgoff; + vma_start_write(curr); remove = curr; err = dup_anon_vma(next, curr); } @@ -2373,6 +2376,9 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, if (new->vm_ops && new->vm_ops->open) new->vm_ops->open(new); + vma_start_write(vma); + vma_start_write(new); + init_vma_prep(&vp, vma); vp.insert = new; vma_prepare(&vp); @@ -3078,6 +3084,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + vma_start_write(vma); + init_vma_prep(&vp, vma); vma_prepare(&vp); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0);