From patchwork Wed Aug 21 08:18:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771000 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 17984C52D7C for ; Wed, 21 Aug 2024 08:21:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=ng4d7dq4IggpkPKeImi1GR97vU Fzl1nQB4iH3JNGeXkwc+xj+nYydfav8UvQ0B4nMIcteSntdy7G1XWR9OWYap73df2JV8Q3uke1wV1 0Gc9/HsNQsvSFlYVg9R28AVej02si8PjskSNkRnzSjaP6JZ1LQDAbXuU+SkMaQn3T6Ma7+yKVDNBj mrXU0D32o42l97xSp9JDKwI1dh/NVYVMZY/7zNgFYBTfjJgWaGFUWD1PpYgVSu+NngnJZA52sYMnv NaKADPTnbPyk0HQoTM94QksUC9wLj7z1hDATcy0XZRIAyQWd3nO4D5JSbJ8FI8rudgoW6j6pxW7OB ybM8r+3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggab-000000082NE-18yp; Wed, 21 Aug 2024 08:21:09 +0000 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZ3-000000081kD-39PR for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:19:36 +0000 Received: by mail-pf1-x432.google.com with SMTP id d2e1a72fcca58-7141feed424so768681b3a.2 for ; Wed, 21 Aug 2024 01:19:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228373; x=1724833173; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=iWHv7Kvh6uIQwpnQODmhCr3jMVb1wxhyRVAyTOSaUDQQmgXXVRtvdZxMcGn1CfIjRT Gfl7zi8bjFALzPa9lCe8bNOcQc6AvX2aWsHV9TcI8RiJIA9tIDQItQy6Gh/dRMjceFfU wKPOXO8cNJ9Jd+SpU9muyjbB+Xw+M04MazRati6/SStWHm6moXBxOt+ukJu7PBXfPT3V FbMBYZ9q6NI8dzGmOawROcI76cPC1qjYLyXu6Wv4kXPckwBKg81bp62Jzh6sHqUOc/2G GgQ68ew22ecdrmWyX7eiXX7DQYacwUzimU3IQs5k/rjn1Zg/uQhHCYswnSVc3mAYyowl RKWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228373; x=1724833173; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=wWPCmIxc9FV/bTxJ09433N2s3vv+0zj1ICe7nGBWSAEPpNtWSuQ8oEh823VWdVBHhs oR2wKQ35JhAxuhVfKlhuscpHizhieLqhVzverO0F+lxOseOxc6cjI7v3RH5cINFTnbeC uz3yjclcie6UlAiELWxu+DzJUu8MNp6YwUyBu2AO2CEP593rCDFFfBaRBJKpBuKLGjIc KG3TYdXuxprQXop0c0q+75hCIPKyihd/Qqj/4ofHkoeRttbuhA0gs8PMamnz3arN71CP ha3LcWkop7o1wfokCB76FcRLhvkhZLGUuTKykZPC08tL0WL9jvVQp4xjcHsKv+nLDiMH ubKw== X-Forwarded-Encrypted: i=1; AJvYcCVrlzYwz/7ykG2IvVna+ZiBcVCsDYzhPBMaSf3Yv6A51GIG6jkhQN/PrYqCOpbMIfFQvD0rPksI2jj7/030l1rP@lists.infradead.org X-Gm-Message-State: AOJu0YxAST7esMcs8VU6QhehHx3aOg4NX4LeZKA/4x0fVj4RHynH20vY 9nc2I95DzQCzgh3TIdBF1Fhf04FhWsH3SpLaF5sp8XYtHt2Ism/tE2KMF05fiC4= X-Google-Smtp-Source: AGHT+IF11KqU9EwWDBxqOuV+x8Or0DtjlDyZMn8HSaNUPAUvyVetfGjOkbkYFHx43yYUuJqRZJ73CA== X-Received: by 2002:a05:6a21:2986:b0:1c0:e49a:6900 with SMTP id adf61e73a8af0-1cad80ef085mr1849540637.7.1724228372801; Wed, 21 Aug 2024 01:19:32 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:32 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 01/14] mm: pgtable: introduce pte_offset_map_{readonly|maywrite}_nolock() Date: Wed, 21 Aug 2024 16:18:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_011933_820366_F3A70114 X-CRM114-Status: GOOD ( 23.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_readonly_nolock(). For 2), in addition to changing the name to pte_offset_map_maywrite_nolock(), it also outputs the pmdval when successful. This can help the caller recheck *pmd once the PTL is taken. In some cases we can pass NULL to pmdvalp: either the mmap_lock for write, or pte_same() check on contents, is also enough to ensure that the pmd entry is stable. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 43 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..f54f717ae8bdf 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_readonly_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_maywrite_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index 00501f85f45f0..1fe0ceabcaf39 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2954,6 +2954,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..29d1fd6fd2963 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + if (pmdvalp) + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,22 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_readonly_nolock(mm, pmd, addr, ptlp), above, is like + * pte_offset_map(); but when successful, it also outputs a pointer to the + * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without + * locking it. This helps the caller to avoid a later pte_lockptr(mm, *pmd), + * which might by that time act on a changed *pmd: pte_offset_map_readonly_nolock() + * provides the correct spinlock pointer for the page table that it returns. + * For readonly case, the caller does not need to recheck *pmd after the lock is + * taken, because the RCU lock will ensure that the PTE page will not be freed. + * + * pte_offset_map_maywrite_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_readonly_nolock(); but when successful, it also outputs the + * pdmval. For cases where pte or pmd entries may be modified, that is, maywrite + * case, this can help the caller recheck *pmd once the lock is taken. In some + * cases we can pass NULL to pmdvalp: either the mmap_lock for write, or + * pte_same() check on contents, is also enough. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Wed Aug 21 08:18:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 81B65C52D6F for ; Wed, 21 Aug 2024 08:22:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=emDTb3AIhwVRzU6+fiEdhKdwwc DSTv7znHjQCdRQBYMjhicEHXNhhdjyZVKFIPMJ1V75H28a0zMx/YdW1cqCgtP+eraLTFPj3BNLC3q VSCLBxRn9gmLII9pE0zAO0wu8BYOwKQHeqrdHG++4B6CUTml1d7v9hwPX/sf92MW1rNJZQ4FgkvkO +VMfIqIAvGY5TZAElCFNFeVI9u2rEUyegu7SecSV0PMMA++rj23jTXawfkfwtIEa2d2dB0X/SQ7dk 1L97s3lJBTUwt6pmToYXGG7Ff/SXFF7R8woYkgO0iKDltFAHmIlsqsUphI9UKg9t3u+JGLKj630rR RBNlDHfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggbG-000000082WH-47me; Wed, 21 Aug 2024 08:21:50 +0000 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZA-000000081mL-1eAO for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:19:41 +0000 Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-7bcf8077742so4723022a12.0 for ; Wed, 21 Aug 2024 01:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228379; x=1724833179; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=jvnN9L/bpXerNTA2J22EiTqRU3eL3RNrzi7sAnlbfrYr5XeDU5sMKp7fRsrzuM5djD auQLvHH6g0rzx62+hhb4ZuP/A//lMGSHlGrC8KCuxoOo3/tgE70PVlajAdizTXXJgNSB hKaIb07LC41oNcsaMCatwZKA1QmF+sni1Ix6MVqx8RHbJ3j+uw7OhLofb0Odzr7x+gTz y6KG2775JHziebTjzDMEvq+o3aguyOsFVoENGchneoTy2w//W02NkDrkG1B8tSF2UVMm A8H1MTFDQ6BxJgQhaoOZ4h/yyneZE9ToL1UwBTX2/7LGQ6669tInEuwmBEQ1z9onm4BK CNHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228379; x=1724833179; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=HE9XKxpdt06sBFB35nQbgrAj82t3GOCzjmPs9JIKEyGlMe8KQBeJKbfWhT0ROKkIPh ciqgnzaMz8YX1jsV/GJkKRu1NzhUKMYBAbl/UqlYGfJ6aiuTuzy1IP3DivzCHtdBCWia LuTDInAJz6+NU47YaMVGMSG6GmRqvfPNRQN/pdAF2Oc+nAr72VRIzMekLo9zE4YE3kXP 2h+vJ+qVvR85j10rxP72F7oRH4PFxmbLjLesE++3BwF6EfcQDSxzHO8r2ENGJ8eFL4HR WRR5pHk/NyZvUblAnIYgL7El/7lTo8TJ38u9osSaWwnmCgqmdInTFS53AwSL98Cn2/K5 tTug== X-Forwarded-Encrypted: i=1; AJvYcCUy8UZQzCEqXVWxE4f4wq+wlcrPHK3+XJSszaCV9Iaz2GON1kwq3zoeTNDl/v6CjWqLTtfc/6RPzJ/eM0P0BzIh@lists.infradead.org X-Gm-Message-State: AOJu0Yx0S9NRipP0+rpn8uSN2J7qENfrxZB+jt6t1yX23PFzpscMXKDq 8QMY7Doy66R5mQ3JNwaTtYWyrBGVWlSvTPRKPlN9VaiwEJXHi4MpVbaFZ2coJus= X-Google-Smtp-Source: AGHT+IEvHWRxH/KV5glvGK0Xiv2dJ13pnLs9scNXFJwmOSFepkjBPO6rW4EjBZpYY7GPQ4yq94i7jA== X-Received: by 2002:a05:6a21:168e:b0:1c4:d05c:a967 with SMTP id adf61e73a8af0-1cada179499mr1734402637.51.1724228379371; Wed, 21 Aug 2024 01:19:39 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:38 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 02/14] arm: adjust_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:45 +0800 Message-Id: <3d151aaf876b93a97fd3f070780345a3bf572123.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_011940_455140_EB328604 X-CRM114-Status: GOOD ( 12.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In do_adjust_pte(), we may modify the pte entry. At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_maywrite_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..5371920ec0550 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_maywrite_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Wed Aug 21 08:18:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771002 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C0262C52D6F for ; Wed, 21 Aug 2024 08:22:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=D/30MKDAO7w6vEuXhPE+Ihl6p3 sENDWoAluSHd/U+i6v5lAJBRXEY/ngxIl1SPT3pwEodDVPbm2AmzAtWzliHoa741kxBE6Y/y2N7iX 4qCqvndvHCt/soDk0SfGzvISV7CtDIFUUGl8CP/UC1qYtk3Z+lsuEb/W9FdScEqmWAfrTWGIPHHQM l2nUuODp6dog/K7JKa+Qq4FOAh8XjLFPM+Pla6AE6C37HyZQhZ1mOkHUKYUq93oevC+a1kMneCJfn ltNpzcs6GSFa1cS85MmzhhaKFSlz8C1iDpbu7fGdI8LKhoPZA44x/5o0JBjEaRHV04RojY0LZyASl osj9fdCA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggbw-000000082fn-2eSs; Wed, 21 Aug 2024 08:22:32 +0000 Received: from mail-pj1-x1030.google.com ([2607:f8b0:4864:20::1030]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZG-000000081nz-3A0y for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:19:48 +0000 Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-2d394313aceso4979204a91.3 for ; Wed, 21 Aug 2024 01:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228386; x=1724833186; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=gwTDNKV59JbeDpl47MSC2MM9GjT8D6yxka+tt0ArErlelr58U8aFnGS/rGI17yRHDz XrR0XAw0SKyacX3vZW+nRPkN7bfOhfAsXl44A76a4r7SJ/G58VINDSAt8ipbJpGYQose sarxhAJuYIythvh3bPga8i4CO5QNE3MU2S3eJmm6Ov/IXdFJn2BR89TGngGlYLBoXM5q 09PePz28MSbSPW+BQYPz9e6b/s3PlB4qmKIpdKxBPOHbMCvAZFi+2IAYZkw+Jc0g2Md4 eUpGzV4he2YnDBkCAhjxeWBqJe5vCqZHK8HYTEpkACQK2u7oLqC8tMhTT1F8gnta/V4y CSrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228386; x=1724833186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=kfPuTvDsRGAk9VQ4Cp4tyfN8lxv+D+CZIhzIwVCmw9RFXTRXc4fx9D93FazVeK46aP 2UmYQv/gyGQ4JnzFgItgwzV499LNM8amZVFISpw7fjTUMo6ZvFYe7mD11Sp+3ySZzick W4Yq8e3ZFileZj2B9GWKfOdL3y52Uro+6xd9PWQnQes4Z/0eHZI61hVQR9ygPKO/wSIc E8Wru5vWDcRdmq/F24ijHdfq/busp3OfxQnYA7ewf/IkilXifkqGHISQb2/cZUTTFuLx /ITqnYtPtS3pjGyA2fI8lrAC2mXNKQZFqyIBO/oSg6s5qFiAFrauq2Da6XDcAlWqQdbV Ktsw== X-Forwarded-Encrypted: i=1; AJvYcCXcL2IM9sl1T01APjrNAwgEIR/3dCG2Vw/HNOUdm4s/A+WUI5iuQZdZBwGmnnRG4vplNn+9qq8sDUZ6dnfw9RcH@lists.infradead.org X-Gm-Message-State: AOJu0Yyom0Iyxneiesr7Wpy8uCeU0pp4K0xeTljx+ziNX1JO1KZJOe7s ERkTmAVpCWQZJfVMivj7SYEk75wEsBDp1Db4Q+k1reuBe0oUiw90o4a0A+kbXUQ= X-Google-Smtp-Source: AGHT+IFyY3NlNyztu6E70clqH4irJgHSXlPUHA4AtOW6RrWWSbwAKaoUYDBEK6KZMX7nlR++kviHPA== X-Received: by 2002:a17:90a:e7c6:b0:2cf:cc0d:96cc with SMTP id 98e67ed59e1d1-2d5e99c8b75mr1585510a91.9.1724228385781; Wed, 21 Aug 2024 01:19:45 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:45 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 03/14] powerpc: assert_pte_locked() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:46 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_011946_819908_6BCACAE3 X-CRM114-Status: GOOD ( 11.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..ae4d236b5cef7 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_readonly_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Wed Aug 21 08:18:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771003 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31717C52D6F for ; Wed, 21 Aug 2024 08:23:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=tXsWWEkofpUVqR52JPObchJQxN AI8AAUeKfg8QDc9ErgtGXTEruTNkpvRgRFvGzoUfWGqYOi74yVfwvWCwUlZHOCpd+pp85Vr5SVct7 zYGEbOETs7EvPopFfN9XsmhunSC7N8rW0mFouzilXsy0kWRIXOs4r0RGmITCjvWYlxt4mQmxCgttp YMuyfMQqBYrOy8+Yjnf06NvCMrJTomUID5GI1dga9D1rvmFq5NZYbWT9WkTci+ETQL3xrZP9PjdvE zCV4nbLQRPZXugoI55JwhNgrdJBfNzP2tqcjr2UuBgmN7/0EUxiltxRiyaLLIcX2W19/A+nNh8vB1 DRTiZUig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggcb-000000082o3-2TBT; Wed, 21 Aug 2024 08:23:13 +0000 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZN-000000081q6-2hzg for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:19:54 +0000 Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-7a18ba4143bso4177807a12.2 for ; Wed, 21 Aug 2024 01:19:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228392; x=1724833192; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=kFXXLpgg/2BSSdlJZSXyhd/D8dDO0ONegEPNw0/i7FxfV15Bk7EXW0oZK7pSocEIL5 YBPvbBvdroGm0T8IwdAPp8yoaORVfNadZ3xyz8w8j+jgCG3YzRZ4EfTOUpQLS+dRH5Ju 9xSJv7YbQyHT62xd73XAKM45AnURtr0W7PxzTc6dy0Aah7TxlG5waqOocg5LUI+u1MBq MGoXp4Ws9tki9w6p2C2geFhGj9BcCui1gq6tVdskBRZ8/0WBK6T8okTEcSGiBPmHPiHp iOMoYKqYIFy4lneJt8hXJ0E3qKhUC8iB+Nls1SJmXoYZSmgdLqpm7k1Ta6HKNeedykVM 6N0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228392; x=1724833192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=ucSmblfkIeE42VQgAB3kjOk7C9O9FeZiabVZfTrjeNP7dd+0zRh20SHZfBMRn5qR5I ghmJV9DQXcdPor9kVQOEuCnX34PZfmXKY4ZXdpxVke+cMNAO3zDz8wWQf4dnXtN9FWab jWPkvXgo2z1JyaUygpwoBfxMU2EbJKvtwEim5MfRNwHjj6R4Myz5VqF9Eg/fyAbodcur uy7DBnWtdzXFzKaX5YByI/WVC8+HX0eEVR+HiACtyZYXpBV+6tPbMK3Jd1R/hB0mKU7b ih+e54SqjlRsGxYgQuw83peVaYlKTbAb7M4Zo6mE7Fo11WkSLhmWWMArgd3Vy/K/9YqR c6Cg== X-Forwarded-Encrypted: i=1; AJvYcCV8Z0uDG3xOmNQ5HuJ1BXFrgA/YqqqqIrhekrFwufjrSbdD+VkESZIsFcavZDvyi0oMNc2FPxNQaLBCscS+CClb@lists.infradead.org X-Gm-Message-State: AOJu0YwSHy0iJyAQ5HJDiJR72qWZYU4btls8QjMq4GR+KgkFjeEQX3em 6iSrtPttj82+Av9mJhT6JsHoxaGhWVwIeAic8Ww4HWi5cgMufXarz8NuziJYDHc= X-Google-Smtp-Source: AGHT+IEDKdhDrkAt8Xw9HjhxvAn16yLBj0ykk9gzeBgjUCoqqbqdejlnt+3gntiVrRuxkT/2XR+s6w== X-Received: by 2002:a17:90a:d98e:b0:2c2:c149:ca4 with SMTP id 98e67ed59e1d1-2d5eaa9d141mr1321075a91.43.1724228392525; Wed, 21 Aug 2024 01:19:52 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:52 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_011953_689735_7B82A4F6 X-CRM114-Status: GOOD ( 11.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index d87c858465962..491eb92d6db1f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3228,8 +3228,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_readonly_nolock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Wed Aug 21 08:18:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771004 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69679C52D6F for ; Wed, 21 Aug 2024 08:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=LlXee6enATwm4IkBSgKa19mWTx 4lfkXH0OB42Z2RTJpv9x4GD497nQxALo8wXHT/7DUyphhZLBITZNljkDAjVhu6F7Zw0Yc+RCH/Tc1 sqHlLXIAxgPmJ6ZPUATtAQsKRg14uNV8FzLJslUEN/I6J/LajHeTV9gU3tkjZD32Q8RkjJ2QZSkFJ Wm6mBeXylQoFbkhwZrtGWQYengq/xGSt8Sv/xPmRBt9A7khFaQL5GOEp3cU3ZR4I5V4+8BPr27kkZ ILUT2U9Nawnez6ZmIAsDWpM+R8mDQu22b7ONP6M4FR2NMWZbtrQ+2tuNHK00qU3MM5VjYY5ZBcTPi 1Um17yDQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggdH-000000082wV-0dN6; Wed, 21 Aug 2024 08:23:55 +0000 Received: from mail-pj1-x102a.google.com ([2607:f8b0:4864:20::102a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZV-000000081u0-0q56 for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:03 +0000 Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-2d3ec4bacc5so377357a91.1 for ; Wed, 21 Aug 2024 01:20:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228400; x=1724833200; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=WaCDmtKugcQMV42UCoGRIU5U631Zfg1JLiBnpNY2ygHNcUByEC5f6EevamZVtyh5Mv AalUinlzfQhIptIN2P6UlaEf+AMHoFCq1dvVN8bju8NrmK0KO0ScPITQbTf6W/Tcf6L2 eKiJeLiV0I5OxxibuB5m0DhZvu+JMr4vpGwdPwJirqa3NuT/wajL0qPf6Tt9mp6Cl/LC 7maqboLoWQHKhV7OD0LiYRKhdf+YBAEFq9GHHqDNrtrVOMcgtqdlqKf2dabTj/kUqW/p KqAMJZDutcMfMtKh4zWSE543TahF5sjD1BESKMtXr5TbqGafCaCix9lu58HE3FIDXCJb D7cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228400; x=1724833200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=SG1RL1pmYcpPoFQ0QsYO3Q02w86hEqrvs61egIczO4LpatrM/yotoqPF1cOWHQLNfp 81DlopGxy6p9QrhRGOSzHSlercLsK/MXSsz3tQ1o9vxV5KqmfgaUB0kr2PF1lhy84tW5 EOTI48xJAzs4dxcw7l7QYYrAqUQUg/7RYqkTRPINFoXrbj82QgJd3DDknjDc4GkPk613 hZdYcF4VJLPeyG7zvFTs/LacphmVA9fkh8+Wdvop9Bq7wItihUu45P0D33tyg3Y1ynoU cqRsXate9DSIiHr3n/7pEe6y5O8OFvdfkw+FCH3O3fy3DbMzrWQEHNqm04rHt9XjnJzO N7Uw== X-Forwarded-Encrypted: i=1; AJvYcCUu6us902GjbiZlIAWJS7YKBfxwqoJyJ5TNMUpN9EvoWOZ59vFpXcbOQyj6bwscyfDexhFU82Y2S/k4IZxf7qak@lists.infradead.org X-Gm-Message-State: AOJu0Yyg1OBMDkEYLZSoTwZSZ0uCdIa6xjH2TvTPI5zioqcGs5t5Stck VQJeWn+NszsmkugbQSYNyGk4H4FZsv0Jn6e4ONdUJ5+6S5fa3hHAbThIJebQ9Rw= X-Google-Smtp-Source: AGHT+IETLNjbMtdResL0gCQrMShCwCpeGL/L1zL6e1zJRM+ER4woSJMvk4wtR2Cz+Z4m4+aIJL98tA== X-Received: by 2002:a17:90b:4c8d:b0:2cb:50b8:e59d with SMTP id 98e67ed59e1d1-2d5e99a5e05mr2805975a91.12.1724228400217; Wed, 21 Aug 2024 01:20:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012001_252650_7A87F54A X-CRM114-Status: GOOD ( 12.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cdd1d8655a76b..26c083c59f03f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1009,7 +1009,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_readonly_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Wed Aug 21 08:18:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771005 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A48A2C52D6F for ; Wed, 21 Aug 2024 08:25:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=IZz5WLov9aExuyDgjyTqQRy8dY frdrkNA0Sq1fmSAjrlGdL1JJMwfb0wPGShlgf+cnwyEzlZTxLUgVDZhqvU2R2nAhbdCnJlHok+1qV Fm0Ma8MZlDCHM44jY9Wzxdx3YolqAUi+nMmsAT4ValnX1Dv2frwYoYdsMy9asuxrK1xdAnWmR4zMX fyjL8TAPIhPZeoaNBZLOeKXfHZkpHfEHDyEkhnyTe8NDmG5/WDtIh2FFHp4wNL4+vu9Uy557XGkco FIXxn84q3ML6CSLDmlyOb6T52JdP9U7OXzF8dXb2zmHhsmV31bI+emuvGYaL6SBEY/fVG0/z1OFFt gnFCtgFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggeb-000000083Bj-2Zm3; Wed, 21 Aug 2024 08:25:17 +0000 Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZe-000000081y7-058E for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:11 +0000 Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2d439572aeaso379667a91.1 for ; Wed, 21 Aug 2024 01:20:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228409; x=1724833209; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=Hmu1wH35xIwxGHHxlejwQyvZCJllXS1fJtJRnAzPxJjMhkQrUh3AP6LDGXFyFD6Z9m Ioy6miHFW+ej6W+s6HmqsecW7TK+VP6fg28Mzpo++yhLy/7ixKoYTMvd8W3KJly2p5cT KhqQoB8tJVQ6AmlBFjoSQUwB+L/wVMvoB4otGZh7yhEKOxIqgO6q/THKjJcFiFdyEogA wTCf0Rvx+TRlu1VyQnW8QlIDw44bc3O3lq/fwIEEeRJ38xHHssjdi/tTE/OuZNKefAgx mHKIWM6zlzXUBgmmc08VNbI2s6+5bo3mm88qPHXafpQBm8vOjKTp4u7K2HU7+hVEWL8Y jQMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228409; x=1724833209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=a3I2ZUlzROlxtcOJDDVP0pGR5KS6RVOUDNJ490Hj6DwotQsTELjFB2HR8GRXyJlIGP jVKbKplw8YmqXjNyQ/oW/RslqpuC90wAlxohZp30fbCjhUbFGJQo6Rp8zQBIS2QUZpjV 9FEtRDcXyspyE0YM6zZ65dkaP0ex0JNnBW3tSrt9m1IeOLiTjAwcFgi1ZIbfKAN7kfF7 nDqWrB1HdRZ3+CIvDTVLrE1zuEwRynpfJsi2EKldCjEJ0YJV8UzPWsqn+t3ukyWhMGMF 1HddowfzrRKKv5YjZf21nyrQ5WVsQUYZUISKB9Jjcy8z1zwbKb9QNWk10R5MoA8J32pw 5i6Q== X-Forwarded-Encrypted: i=1; AJvYcCVrzpZh5w2RHRTjG6WHJ6ModYBy7geMiYDH4pzawVeRsMl5sE/11Kr8rvCRGZ52Ek7Hbt5GrrNiDjono3g91R/c@lists.infradead.org X-Gm-Message-State: AOJu0YzrLWDVha9+c5Sk0snlyDEBP88ZRBJx4GE1M7JIkJG+h4XFHhv9 Mf19zoXcXAn5+7gUbt+tcCc5l74JapTBZW0QF52wjvsOmJ3RpDTqkuteA1ZcBoA= X-Google-Smtp-Source: AGHT+IEz6hEUMU7QDWWn6q3mNXap3/YJ5XtpoR3UgLNTk8qimJ6ZhRH05YhkApxF5Pk84JIDmuV6pg== X-Received: by 2002:a17:90b:4a10:b0:2cb:5dbb:d394 with SMTP id 98e67ed59e1d1-2d5e995c932mr2717849a91.4.1724228407428; Wed, 21 Aug 2024 01:20:07 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:07 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 06/14] mm: handle_pte_fault() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:49 +0800 Message-Id: <239432a0bc56464e58a6baf3622fdc72526c8d57.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012010_083613_731DD709 X-CRM114-Status: GOOD ( 12.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d02..d3378e98faf13 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5504,9 +5504,14 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte will be + * modified, but since we will use pte_same() to detect the + * change of the pte entry, there is no need to get pmdval. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_maywrite_nolock(vmf->vma->vm_mm, + vmf->pmd, vmf->address, + NULL, &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Wed Aug 21 08:18:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60EDCC52D7C for ; Wed, 21 Aug 2024 08:26:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=VuGD/bdcQ37WEkip7Q0j/V3FkD 8vTj3HOF1piL4mlO0jF7a+Cvi7/xE+j2jEiwdvKOqtin8Vy+go57TTPZDLgN4/vS+M6oLzfB8mm4y Z2JwlhtA7gao+ZQK1wCxsPgKIeO1TV/hGHF1C36/sTTKnL1s+va2UghGffmsInWhiZT9j6/RPxagz 61ffHmli9xKtSJvpN7zmZtfZ8/8Ecb86nPfjosCHM6+b9j5GGtrK7BYsu62rAGv/2+2XY6UunPQnq nmMm/yXaBwFYr0FZJmoNxsITeVxTK9WiAAr2r2yLl6pm623Kpd5J6oC9BGmJ0j/UQ97ap3DmkzTN1 2ryPe16Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggfH-000000083Mz-1ZPV; Wed, 21 Aug 2024 08:25:59 +0000 Received: from mail-pj1-x102f.google.com ([2607:f8b0:4864:20::102f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZj-0000000820b-1xYy for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:17 +0000 Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-2d41b082ab8so2511549a91.3 for ; Wed, 21 Aug 2024 01:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228414; x=1724833214; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=b2wxgmkHZU1yojRIT/gEr54m6DCL/IdmNIkcooylsXvh0vgWwmOmNe0Ki9hEYQX6sJ eDcd5xE0lxmgWmc+F4iT506XahgNidiHvXGtKcjuYLW5tbPKsVWgPygURkn3YTsYa66Y k3flxpWvdrCnkZT4OAisk2YXuI8VAZDGMDKc3PhDw9DuUvFct5LMFeu6XDVHqQl7h9wt qZD1WAm8w+JBWcWgrXuH/KvI8h1yVJbvlWADdne5R10p4hIQkJgzvJdRkGMJwwa/KK6N Fqlf8TYISN3NpelpONJuqLA4+AQHmVHktuuyKJ8pENu4Vcl5dThvl/TWxvc40EEE9Dte aA4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228414; x=1724833214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=DvkqNTzKxBQt7JdSSB+DZG3zDLePuVLjRCJczAiKeMA0FQ8b4ZrlVhC5HXF2byW+4K XozjTyliiWkZf1n6ILUC8lR3ngcbwEdK0y+BFpncV7W9sOLF61VcTmsivZRwSEtOl4oJ QA8MgJ6djDrhWGv/BfjNAuPnzZE8sWMePc4LpO1zjc8I1iHmAs7pNV+GwPRn3pGduTzj SuX0I6wEJmq1KupNbajb34cCy+pEWdCdsDiaJSdZriN+0TVouhdXm0Qnmm31bNH0V8G/ Ce3eQ6R+jdhWV4yaM5QmPKTDOoJdYLeKjU2Tn+QOFJz5PZ+OMrAXKLiAh9yj0K6cHnFG 60hw== X-Forwarded-Encrypted: i=1; AJvYcCXo7PYK3yYCLP2bIbA+pWMkYr19f4TFgxbDXhOEcRCZLiAjfjKTEgvj8G4EUDE6gR5j2Fi/Kcie520+/m6FUreU@lists.infradead.org X-Gm-Message-State: AOJu0YzuljtYW8l/kJe3Bj1JfzhtwsVHb2f653XlWMqMfIGNhfZOlWgZ SJvE4yKYp9fC9VqR3rFAFXtU+t0PPkEEajs+lr07wJmjxeuQ22b5pHS7AjVpdrw= X-Google-Smtp-Source: AGHT+IEASO/tYfbmKEqODBO5ptoLkIYqajvyb9bjTzXqrTBGFe58odiOJRUvoUKiSb3oI+1Cc8Ov6A== X-Received: by 2002:a17:90b:1c01:b0:2c9:9658:d704 with SMTP id 98e67ed59e1d1-2d5ea4c9ab9mr1366119a91.40.1724228414402; Wed, 21 Aug 2024 01:20:14 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:14 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012015_525909_87A90936 X-CRM114-Status: GOOD ( 15.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 26c083c59f03f..8fcad0b368a08 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1602,7 +1602,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_maywrite_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1610,6 +1610,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1655,6 +1658,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1686,6 +1699,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: From patchwork Wed Aug 21 08:18:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7ECB4C52D7C for ; Wed, 21 Aug 2024 08:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=zAlRey9B5LdPZ9/z3IBGLSdr6k KdEyf9r8G4NT043bLFXZlyTpMl3OeWCl9zgERBzcNCgX964Bg4JvYupNcpskwlE2QAe0ZfhhU/lCc aDdzpf7cuubpWx98py+V/06aRbYNOcd/+6Xxvk242tFSSafBSM4+oj9C6b1z0vbS4zgLNI0UbI6Lt X46u7imwK0GkRV2AfjcFlTuDRB3K7sEBGpnP3y1smJSwJ23SHDAr/PWMDmO8CofC4AnkyN+n64LuR 324DZqqaNIrnmUrVEs3T36hYZ5OXrvMpDOTfkMRIqRvmo7C+SajRzdwXcC3mvTqOTw4mRwEjHg1Mn /nAuzCHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggfx-000000083T1-0GkT; Wed, 21 Aug 2024 08:26:41 +0000 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZp-0000000822z-2ZV5 for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:23 +0000 Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-7cd8afc9ff3so221235a12.0 for ; Wed, 21 Aug 2024 01:20:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228421; x=1724833221; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=fJKw0ZM7hi1YxIHUEFLe1oZ0iTA20XknAXxqQ9BxPk8/iQG5o0/s/NHLG2rWmj0T2U SrfVd3Qid/41PtwMQ5TWWlqaJbFTJtw+d+shMLEzafR/F1JnM3getDcnSAGrXGKdzAVH EAifPRgy19hQgMVk9BolrkKvYD1bipS/wqm7g/Fmn5ug3RYhmc2Rgo+fHHfv1CVKoc1F AEQbXpSIA+tQYsnRSSYHpxuURdqq84+QbsfO1MSHD4Ig6pSUbo7/UPYeheuPpI+1H8QI WyjlVzWgNbveGO0beRrpcYnfxrQCwJppMnogewMgkvxNAP+aY4EPe2Sfc0IY3m5tqFsg TxJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228421; x=1724833221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=kw/nroaJogc2X86cDZaQTWwt553CqN5ngLREdQP0dGKSU8GDjibUezT1eSE+5Wn0mm Nz3u4cRAzLW6dFtBQZuRS1sKvH2P3lbN9nuDK2Uwu/MbglptY+bdJEGmeYCjZl+T0UVk kYezv4F8JZs7tYurERK2+CFsI6dg2Tuh00nqbZy8yRO42aQxqPKSXtzS3G4yFpwFM4d4 cCz4iMU90yF2hEAfjX0Q4c8eHLe40EqRcp7MDgkYor4EsykVUoJzZM/iXvWn86pxNxBr 6VCTguWP2I3qn9yVid2sZJE1lVSR4m0oevUDW9ReUsHqjF4KhaLyxyu23vvMovdfE1Cf wbFA== X-Forwarded-Encrypted: i=1; AJvYcCWXioDb+MLgNMQWr7iuhtsBR1FLhneK+6hR08Jhsb212w/6/4hXOyJy68C/IaJSVelnDGD9j+t2GZ88H+DuUCXI@lists.infradead.org X-Gm-Message-State: AOJu0Yz5bdGQME+Vz10J50iIP8viHOuVPiHjtB7YmgC0TWbVOtM18XRG BNmfazgdpsvsoHLqQsr0t2Y/Dj1VuccvJSq2Gt0A/BkZyPdnVrcn2k2bzbgweWQJpVD6Jf0CT+F I X-Google-Smtp-Source: AGHT+IE93wrdhvj1jPWy5zvxTPwSbhlcdosDYnf+NOlDda6ytSjhdKrpXWKVADSrOUal4AF7Aqrd8g== X-Received: by 2002:a17:90b:350a:b0:2d3:c084:9d93 with SMTP id 98e67ed59e1d1-2d5e9a2f0damr2610948a91.14.1724228421082; Wed, 21 Aug 2024 01:20:21 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:20 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 08/14] mm: copy_pte_range() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:51 +0800 Message-Id: <05c311498fc8e7e9b2143c7b5fef6dc624cfc49f.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012021_679741_12DC740B X-CRM114-Status: GOOD ( 11.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already hold the write lock of mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index d3378e98faf13..3016b3bf0c3b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1083,6 +1083,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1108,7 +1109,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + src_pte = pte_offset_map_maywrite_nolock(src_mm, src_pmd, addr, NULL, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ From patchwork Wed Aug 21 08:18:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 109E0C52D7C for ; Wed, 21 Aug 2024 08:27:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=VxVhHlyJGdrJXnVIcsoFx/IIcX mYwSK+uq7EOlcEwECIN90hQqVBBAiKQBxarowvIc4iguZW8cXmsJGMaVt+P17dtZzLimV7O6ITnpu 5A3UP5BoInsZYenPHZQWhYW70s4bQFz1XBFURxA2EBWhQkssHRBUt/7qB2DdzKAszjbTTl2UXT77M veUQ8i498STIa1Z85C4ZZCKbx1LJhn4kKKGvHd0N+WrdjYxaOM8Zwx1Rae3du6sjS7oDYcPxf/is3 MJV5i3+L8kt/cweDsRFFploZsYvR437YI00B/U09dByEWcVvtMh5VOMYgFBsIjSLmETBZ8uDLXx2k AFKtzrxA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sgggb-000000083aO-3Dm8; Wed, 21 Aug 2024 08:27:21 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggZw-0000000825o-4ApU for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:30 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-201f7fb09f6so46451875ad.2 for ; Wed, 21 Aug 2024 01:20:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228428; x=1724833228; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=Dhws4CPNo1AM6BZi9bloW0HDTRL9pU1EK2LyR7JmnnKsieKb610UBTLh6ff9kqBWd2 voy13N03kmoZnXKRdXSPfpm4KRq1PTzTemPoCn+Xm+QsaW6aOthWgO7UYMcqpRpSn2AZ UjUqu6X2LiuvOgA0Q9Ug1C/4jpAtlGgH7WuXSf7IbQc9aN/AAGrWK2rEPjI/jaMF8/QO T8iuJEyDxC8ypGWQXtNxB3G9G3e5g6d2yHih8BllDKi2WYj6udlL3+gnFwuR2FN6tSna ZTvBU5/rN9jERa/EswcdHtxj9gq4uUUyDURLyWOMdsI4bNsAHU4fdpuSJ8BuuW9msZ7T X8dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228428; x=1724833228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=Swu5xGvEOU7UrmEP7/4I6c48rxzTAT9DLxHy9NUV0a2+v6mAQXlDahX8iKTrt0oZrY fcz8PDCqbjw+DlUebyJNyRnGsydePeEVE0TGd+uob+HXpveILuwopZSe6LI8nUyfyu9r 2fB3PzbNb2/ANTvpSJAMQweUGTdemP1/kg53kTnocVO+bYtS3/siLJo8Plo/61po9Fc7 aBHLhdecwm0apGFkvqMxdSw5ZxkJc6fT1pZgZ3C2cyqavX2TWJ5ZzZfw82HrE3MJGuf3 hxiTb4a7q5XFCDcOiYWnOAKiks4ceeQ6yJzAVEc6wmSLAkihrcRYn9sj6jNRt07PO0zN KsPQ== X-Forwarded-Encrypted: i=1; AJvYcCVyGzF8bbXxLV5XrUUn+b399deQY3IoNDr2M5P2VBpXCU2eLodElvKmW55nwwW3lVX6HYp9G+GZQA/4XnK43ROA@lists.infradead.org X-Gm-Message-State: AOJu0YyhzdF56cT2jPhCpXMAc+PDpY2kk4/MIgSb5bb88kHZpJCJ6npF F2QuP1uFsQDq7f5lzm42Xk8gnJIeWmuWlpShTDqHvEi/3bdnxXUuEvEZ3Cw7bKIC6hEgQhZTFG5 6 X-Google-Smtp-Source: AGHT+IGct1lfg7s+Av92JTuFI7wGr/k4q1ybIYiVEVcgmdImld0nRYlRbV5Vh9eNiEYSyV0nSbS/hA== X-Received: by 2002:a17:90a:7893:b0:2c9:635b:7271 with SMTP id 98e67ed59e1d1-2d5e9a24ea2mr1616047a91.21.1724228428100; Wed, 21 Aug 2024 01:20:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:27 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 09/14] mm: mremap: move_ptes() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:52 +0800 Message-Id: <89bd1d48db1986ec83dcae228defa748f38238ff.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012029_063459_386713AD X-CRM114-Status: GOOD ( 11.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already hold the exclusive mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/mremap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index e7ae140fc6409..33a0ccf79c32d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -175,7 +175,12 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Use the maywrite version to indicate that new_pte will be modified, + * but since we hold the exclusive mmap_lock, there is no need to + * recheck pmd_same() after acquiring the new_ptl. + */ + new_pte = pte_offset_map_maywrite_nolock(mm, new_pmd, new_addr, NULL, &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; From patchwork Wed Aug 21 08:18:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 33CB3C52D7C for ; Wed, 21 Aug 2024 08:28:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=4woG5CY6lhWcQYI533LpVldaAj tGJd9yE3eEEABY8sqOxgQUcZsX/CdmCHEJUfUeBSxRAuSii1557mxinvPWZkikv3Eqbd1BpN3wLnk DH9O9t72c+T9Z1yXFk8Jpmy2iCkGJxlme4UPDdr0Ri0tE+UP8vvg1fdi8qs45D46y3ldZw1PLCjRq DmA+5R+uBiZcN1IgrJZVj+j9//zVI3db1kuPEIdzLS8Nj+9R3xPr2dlUCeCFezfuXiYXOQvD1quPI 5MSVhmV7wcTp+hgXAlGkC1KD7w6ZEpE3Z+VRcE3Rd7caS+OmPRFSZ71GwGu0trbPfV7NkBRbGNvmO Gn7WgWPg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sgghH-000000083qw-1qgW; Wed, 21 Aug 2024 08:28:03 +0000 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sgga3-0000000828i-3Yiw for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:37 +0000 Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-7c3d8f260easo3925130a12.1 for ; Wed, 21 Aug 2024 01:20:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228435; x=1724833235; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=Tsm8WWpPWG10osffoodun6sZIFEfXfhCop5SN12CXUyHh73miRWxykrOkzSmaOaRxW Vt0oE+dYicNF6+LCeGFmZSLIGmh3mtECM/OqBBfA99etSzkz6TXPNCbiWw2T/jaKAW1t 43P7BFxieJEKeFDQ7x6loiBJk9U2MtWstpafeGYM/Hsrk+887XU7l0+K/K/DsYXs+6c+ oSBOhMRyfYbncFTSANM+zNyJ7xWxgJ3+P4/fUh9PKfQ2QGaCOTOKRq+FEGm0RXK7rLe0 Vdi8EeTTTIaa4Lob1dgZlks2X5jGx0JRj3L5nR4CwfPShvn+qA8qCTPAxPsAat3vE9Zv +q3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228435; x=1724833235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=jhrEgjWiLLo1bY8c+vEVrztqqPtvG96tu75BFW/I8Djc3GIvy1oNxQxUjYD3f/RLXn fb3HF5hiU0cRlhDZRgQTt/S+DI6Fw2uPSXoXQAZBVdbgu4rJAYsBSUs5u2kORgitaB48 z1mMyhYFpC792ZeJEkiP5QN9l7r0p+Pp2hwzKrQ1Q7J753p45vOnFtGL/tpTUIOnMnpZ 88HxQc3yOylsjVd0CCDoWjVhIDhnXmsKzducbXovr62NVIJatxFMykzxj6FRw/YvwZXd hTYMUB1djW859sthWGDKp1HTCiRUyHBQTKP8XKFCkuL2MHfvM/zVog06Xb4eR8Tb2dgO 431g== X-Forwarded-Encrypted: i=1; AJvYcCUf1TwLg5VTYgRUSVFzcGZroPNw2KL113fpgUlDI/3luctPuZOO80VNJfF52znZcu3PLQPZ2O9bhrIdnD2u5JHq@lists.infradead.org X-Gm-Message-State: AOJu0YxjC+iYFhNoKMzftg+3WFmycP/RcVwjOWWhPKpUuT+Dp74Oo4cI TcCFUjTr/yz6wf4oliiPaBNRKAz6I2VKCRTbQYUYfSaDcocytyZi8NhHCGMz69c= X-Google-Smtp-Source: AGHT+IF3P/4155uMvmGYpoowuXrhy2v6aHwudYM9PJFMCLkX9j2rCBHO6+Z7X1wkaLdhxTlExnQxpQ== X-Received: by 2002:a17:90b:3b41:b0:2d3:b5ca:dedf with SMTP id 98e67ed59e1d1-2d5e9a0db5cmr1626393a91.17.1724228434854; Wed, 21 Aug 2024 01:20:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:34 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:53 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012035_921539_73F7C8C2 X-CRM114-Status: GOOD ( 14.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..da806f3a5047d 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_maywrite_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Wed Aug 21 08:18:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A488CC52D7C for ; Wed, 21 Aug 2024 08:28:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=KM6vjAMobddi2mhnhTIC86CzFq wilGPfLdC6BlMbDxkDuzA2EI9fhX3zkP6Q06rYHi7IUdJiuDRz/vJOGNSlpFU9mb5180MgG5z+R3Q XN5djqqx2dql2bvjR5RzZrM4x2Mwd2fbaLIHB1HuO7nYXTNPOIATJwnDMTuCQQL9NsaoIwl8/r2Yy NJTU2brCxhfg6UE4vAGjx7/rcuE40Fd+hCSu4mE7HJE/sobqYoxUasKES1P1jPLlCTUOiSqYpMS3+ lNvWvYJ8aTzAjMD6N1PYF/lNoVkwEGQUb1qw8XoX8EB8ySV6Ce6fvF1Af9rVQJCb1XdN4PAtiUqu/ Xkebk3kg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sgghx-0000000842I-0Ujh; Wed, 21 Aug 2024 08:28:45 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggaA-000000082C4-1R2E for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:43 +0000 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-70cec4aa1e4so4316350b3a.1 for ; Wed, 21 Aug 2024 01:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228442; x=1724833242; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=Av0sGT8CHOhksuiSoFM+NWmvLSK3gUfi8Dq4bnHqoieXJdb5Y2IPcGTOWcpp8DyC3e CLQP3GenDs0WTRy0APqOYZdpeZ/Kc995CIB716KbeG4SaDvTl3xaJXjB+fG7inupE7Y5 J8GAd1iaLGX16iCKIWCklETYFM7r1wkYxIY2ZyLLYT+yIkCtjr1x02QRgHVj8TWOnjyh 0BwSN3WrUPRs9e5lhb4ReVS0GbfEYR7v9E3vB/hRchwfWPYtA5kMWDhDr1xFA4ZJZKhR FxTBT7FBlICHuxEAd/WJDAYnISmJ3v8PXPp4N5AYwmU4V/HRpjXNau7D08UruZ/N0L57 vweQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228442; x=1724833242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=OsldEmlljlxuISOFtFbGenGCaIjFoWeWloziIZTeL6Gn7y8m0yHMu8WVkgRj4WUv/C CtqlFhM0Z2M6o2HsgY/MEMUhI5grKHu8jR23i2CQlbnei/Z6Hi8k6XBSARYtV6gERFQu 1jw2J5RwgNVhPg0NLN1CBInTVbtEhPYT5C1LZtLaVE8Md+jsW2w8Y5liZIlkrDe+iqlT PvK3jUIGjB6fZZ/Q6MVQaR2yLGsggPEs0uJXDs4k4Juk295gYsNA5GIRb4C0wxczI/83 wlm1dpSZ5ra5eA2vNm3y6MWnffsvQ1Y9M9FS+ntd4vyL4P/IVIj3X1e2SXs1MZBabmiP JePg== X-Forwarded-Encrypted: i=1; AJvYcCUkrvFxsix/LRZAk354QGuwoXbPHhzYy1ZZqmOajy0IKP3YnVunqzF/lenusp93+ayEGH/C0FvxFx0fCRdEHzJg@lists.infradead.org X-Gm-Message-State: AOJu0Yx8n+tEA/W1DVBFLz/VPivvDL9N6t55Aa8ZM5r4/AH4z33K+wig 2BZ5cxIWA+0Q/1Lr6SuIwIX4OK9b44USbccjRTafs4QaxI4ARc4rZt3pgVrTOTU= X-Google-Smtp-Source: AGHT+IFM+x1tDjPFZswDxs94u/u9nyN6BfWsdj/MLtJd+fBRDvg5mEN2O6omshjTkNyUCHM1AQzwKQ== X-Received: by 2002:a05:6a20:e605:b0:1c3:ce0f:bfb7 with SMTP id adf61e73a8af0-1cada055937mr1798278637.21.1724228441599; Wed, 21 Aug 2024 01:20:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:41 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:54 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012042_403168_ECFDD383 X-CRM114-Status: GOOD ( 12.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/userfaultfd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 6ef42d9cd482e..310289fad2b32 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1146,7 +1146,13 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval. + */ + dst_pte = pte_offset_map_maywrite_nolock(mm, dst_pmd, dst_addr, NULL, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1160,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_maywrite_nolock(mm, src_pmd, src_addr, NULL, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Wed Aug 21 08:18:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53A02C531DC for ; Wed, 21 Aug 2024 08:29:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=suLQ0j1ZKZA6n1pZnkL70v6RDk mlbFO67lqvXxNx2OhSEjR6582slA9UOMhd+YnxzRRxvSHGteNi3ytU/l+wMUgknJHjKZJju8XpVnO 56we1k7D7ElrdzrWdLw7iGA/afTH3BF34KfmWoowlI2fuDXYoM8dW9Xq0sPE71E2BxDouTg6jlwEw Fc+61vKBSMpTlbgpNZ9FOIwRlKcVH9VTsa0LHxJ0C4RN+Qmo04KkSw5lTGmldadsftTOObwwb/OoN p5X5/5T27LYZX1zPmZJ4B5PWj9lFy5ZqcbFXxeDgyJToIV6LpYXo5YaU+r0pmFebJFo/Eg3JhhImB 4gWwSmbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggib-000000084Ei-3Uux; Wed, 21 Aug 2024 08:29:25 +0000 Received: from mail-pj1-x102a.google.com ([2607:f8b0:4864:20::102a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggaG-000000082F3-2hem for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:49 +0000 Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-2d439572aeaso379998a91.1 for ; Wed, 21 Aug 2024 01:20:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228448; x=1724833248; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=EQQd6qiHQeZU6f0WD5Yo4o9UezP7oj0Zf9SkHvD91Jo7g093lWx+cWwBH0GqFS2kb2 EzujHLzaO7mq3rTYTqccD+rXO2V+2ymuOxmaK1r0ykKw5dbzKbe7oYm3FjMHBuJzmGZ6 UPT6YzGerL8ZtrRMH+vKge4Orth063QD11JuyRHLG9NRMRWtflSYuZLJ/jh4j078GM+L wXf05akX6C0KKAB077KhDXQR4u8NvE188/OVTAYLgU407pxEJ54XVW8jPQR8C57RpaPn OS0+BuO97a2DXEevns5S7rn8Y+8yUVkp8t2AOMBnHPKfBJFxSipUW8ymkm5P1mu/1CXX UTvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228448; x=1724833248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=jj5ScLWUANGQGsezqLayMHn/d9+o1gbgZH5HkBom/htxHG3zOHod7XpzORA9rS+5Tp 09lrMsfOZ3kWrGYx1uc1AE64Sgq/dmWMPiTRYWdSFMStDnhKNDra+GD5wSlOjKTVX3ZK RixZepA4opwzpxHqRNQh/lRN9kc3aIWpyvzZ2A9fVBCD9EM6c7z5d3cQuTpZcwYrFRN4 bX7PcMFQmQapKiTfPVsfePrbE7Bz5cBdZacwwp4KjZRHtF3e54qgph8F6KZfhGzBpo9C W2wgIXQy5ElPhmwjf+ZdDJc5CCCkLTbD8Aa1vYtjZnCGN/0HZfTgX1DCtO5Z9gxQEBJ/ N27w== X-Forwarded-Encrypted: i=1; AJvYcCX1Y7oo43uC2S1Uj1nFuxTMbMC76IEkdQE35eaf4mvxQFXR7BC8tiWg+otRhEsdS2QM4N8hOxOc/fAJO1Q3y+rI@lists.infradead.org X-Gm-Message-State: AOJu0Yya3qhlSk0hb9vpmZbWn182pKcKHOyQQd/5QsZCsNq+i8w+KlRZ hft/8yFjyNN2zNC1uxMgdu90fejiRzrqwKCSP/G8HUVmA+Lmo4V3S7jeAZvdxzU= X-Google-Smtp-Source: AGHT+IFBw9G+gRGiVw886HN5nxLIybNKSDx7re7KNY7nJCh1bBtNFP6GTlUB4MiVeFJVy0IWyCdYMw== X-Received: by 2002:a17:90a:d987:b0:2cf:fd50:a2d8 with SMTP id 98e67ed59e1d1-2d5e997c3demr2593063a91.7.1724228447873; Wed, 21 Aug 2024 01:20:47 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:47 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:55 +0800 Message-Id: <1bdb1ec4a30acf6b3a376f746c96236328904768.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012048_704963_C06B492C X-CRM114-Status: GOOD ( 12.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5dc96a843466a..a6620211f138a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3396,8 +3396,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_maywrite_nolock(args->mm, pmd, start & PMD_MASK, + &pmdval, &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3405,6 +3407,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Wed Aug 21 08:18:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13D08C52D7C for ; Wed, 21 Aug 2024 08:30:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=ES9peVyxRCF9bvSNovk1AQSPzs xlDyQDdtYWWIwvoQKqXOrt2CXWRZhIkHw8jzUolBzM0ZQPl8ktKMx/Gikvqgne04iKh0l4qaR2q3h 0BPmZ1md0+/IEwxk8QrIpLmx14xLnMsnbtthoyFvrfMJe9aOHxv12r6Lxz8qOsQeGKJpLDHkh7HJH pRqHHmW1F58a/nVwCIpGuaJS3R8xJYKZ5U4pysYiuNu94Pj9ZcVYnlT9maO0CDW7SLMHnWyiu05Iq MCe2COdv0B9hjprXjDAB+apJFJGwGqeAqYq3HfMwxU72koMS/ubQg3hObya/YB/QjHPPSz4IYeHWl Zm9L2EoQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggjI-000000084QJ-3IdQ; Wed, 21 Aug 2024 08:30:08 +0000 Received: from mail-pg1-x52e.google.com ([2607:f8b0:4864:20::52e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggaN-000000082Hn-3t5H for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:20:57 +0000 Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-7a18ba4143bso4178392a12.2 for ; Wed, 21 Aug 2024 01:20:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228455; x=1724833255; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=QlQckOwMgFp6bfzfOVUhXWs0W9omEsNduYUQp2VJ7WGKKt2YNxlHn3RMRCXVWacdFA 3yeN0Z/1Rb4OD7Bvj2Y369mqCLbxNUdc2M1wh/+dphROVl6jW8STwlKRwYzzVl+ZUD8d 4yKKUzehsyslpjInq8Vco+4jnLN2aNNP5a24LUVfGacJ/kDV0qo8sYYQAxcB8LfsswkV kaiQIXoJtckElZj5rcmBOvrWx9muAy9gwhQ6aXoXeqQYzxH3gonwGcd9tEpdkrr4+wK8 RdOq1pnyo3zcFyeLW0739LKm/F8f65OZk0wmqM0Uq7kGL7tHCZ0BgxTAHeiL8XhQOhBi qyng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228455; x=1724833255; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=AVvYIiLnpbkOfACQxOBSnLfzsWAMPCn2ALLIiLxBwj71FcdVoT/AkoZ296djf237Mb 3kU/Ypwst1mWMFyr/EFZ89BF7OFWOx0ahfs1/430kgcAzuJa6p6d85GTqVueh8Re2t9d 1vyxXqQOGh1DSDdJXwK3C3NPz1cgkd4rFPItW7U0qoMk55OHw01giouGqM2m++NnCHLS ZcY/2y5EFFoyP7ZiNbrqZzHFheN+Uin2607qErWYvjdwZsrELQt7H5hBaNKc/mqWPV5D 07fAdWygkYCG2LBb75WIudMk5IeFkSktnJ5vnvL8tpUcB06IH1QFCqTRkyOoXVJMFkBW nxmQ== X-Forwarded-Encrypted: i=1; AJvYcCV2Eq7iFmyKhFvmCNnBqK54CnscVkIoPw5BKa1kcT2+wnd84XSSqy31dbPvHVldke3F6tB/Mrw0U3tORAFTitB2@lists.infradead.org X-Gm-Message-State: AOJu0YwxMrXgYDbtVA85ZERVkf+9XDxXURfy9/195E2AACY6CND22JQz KjVAz6U15goTb4X3UlRwD22tgF9SERaAcKnkqHCA62R1rx/gzrhe0WiR3MHoxi0= X-Google-Smtp-Source: AGHT+IGrVkekNFESU70hIpUuBMPasbPeBaSvK0LFz7NLCJlpTpTqHUrWHKc0kKTsT//aSMihe2hbaA== X-Received: by 2002:a05:6a20:ce47:b0:1c4:a49b:403 with SMTP id adf61e73a8af0-1cad81a4248mr2186429637.46.1724228454750; Wed, 21 Aug 2024 01:20:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Wed, 21 Aug 2024 16:18:56 +0800 Message-Id: <129f11e3434c5af898096c1cf3fe378f50bc94b3.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012055_977798_9DF1ECCB X-CRM114-Status: GOOD ( 12.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index f54f717ae8bdf..596b425fb28e6 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_readonly_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 1fe0ceabcaf39..f7c207c3ab701 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2952,8 +2952,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 29d1fd6fd2963..9ba2e423bdb41 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_readonly_nolock(mm, pmd, addr, ptlp), above, is like * pte_offset_map(); but when successful, it also outputs a pointer to the * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without From patchwork Wed Aug 21 08:18:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13771058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7D881C52D6F for ; Wed, 21 Aug 2024 08:31:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=vRX+G2sDqNZJfT5q+AQSulD7Yb b9pib2D33sieocm0S3cnFFm8YRizQrNkd5QFRUk3ylmw+1KaLmd1oq46UuQg7j4ziFlFpVyXvsjpf GANNolv000nHYorRALh6AdH4Mdr5G0gKZPy4I4yr2rVFvuNjvD0ppiFebffEO3cJGbAIvvefuAKEx cEbNQtV5wrZ0AUCcT79fEtWiwMtzyxeP+TPLXz+bTOnD3ShaLgkLBg+vn9N+3bPuBPuaa56TQrYsu RKpAlFZ5391oH4g3F84dyQfbxAtNpZXdUObhwgrej8acqfEqkfSn7dWAmKrEHltyb3LPDYpcf6qPi EY8Lthtw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggjx-000000084YR-3D8h; Wed, 21 Aug 2024 08:30:49 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sggaU-000000082Kj-0SKO for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 08:21:04 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-20203988f37so46225005ad.1 for ; Wed, 21 Aug 2024 01:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228461; x=1724833261; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=kdmP2nKZPyEuHf0p+Iq8XHMSgDaS5s46zAkYviWH5ObzYq5zNq2nXVCMJNUN9/DQbU rkC/l0TDbOSDacFiCGAbcM4ik/oCon/cwoyK9VNk6DsYPRjZXrJ5g1yUf5WzxcqV3Fe1 umGD8SokaT1DzlnDCm9oSKH2I3XsbSnkPvmT5OGM4vHlBf67P16yxHQdVJ5xY45hTnEx kBzcd6bTc+3tEOBH04DSakfTFheJDXR0CQI7eyVLIOXhGJXC7uoaG2HBUbKe7kNjSCEa qlL9lyjg1ToBfaB6qRjS0V2pTk/q+QD85n8vGK3wXg680o1N5v6vJZgzXzTJDr3uLFHN QdGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228461; x=1724833261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=V1+pAjN6ReBNJKnbbYJ2Rg+TKUdXrp0R9L+dpKTK59G5Em333Ji+pfH/6bytL7fcMO mK7v9Ar6FaYytgeq+hX2k9NZ8VYAsCspCQ+kXrYEGmixo89PjTzji8Moy4RPhHBYE6uG 7SfCZituqjI6PfyApDudNqljmyQDg0RbQAFTKu9/76cvyxj0BNphtkruTzEpp2Tr4yvI 5TAy5zopcUqs9JxEBVH3r/iWjEvcC2P4AcHSnWoJjDdnlNROOXsfzAg9FR+UmSjo8aWs IZFJ+q7XDiYwWPijPWBcIQzeiMPy36k1mnWHRrl3aqN/7CW7RIHxbYxlxy+ElPovRLXf QmiQ== X-Forwarded-Encrypted: i=1; AJvYcCUM4WZ0YKwo62UXg7ConGLE3qkyBrP+UmiJXUh+nqyGvU8JurFjdcYPxICs7CtiM9T6mHlnqHqMr7yp0uVi0NIL@lists.infradead.org X-Gm-Message-State: AOJu0YzaNWO9qIJapyH2XkUwRj2gCsTgZDa2Yq2aHdLnvvHBcWPz1XXA 96rFMtp36fSOAgNVkqwsMi7WKa+pjsamb8fasgI9yjujdp+9pLN8JEedNVlcShs= X-Google-Smtp-Source: AGHT+IGVCW7LuSOfGhGGSqonUfZfQEf1Y51LpMxCXHDPBrQX3ltYUdmnoycYhvy3ErzvacrcFVYd1w== X-Received: by 2002:a17:90b:3851:b0:2d3:d239:24c2 with SMTP id 98e67ed59e1d1-2d5e9b99f57mr1469884a91.19.1724228461120; Wed, 21 Aug 2024 01:21:01 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:21:00 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:57 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_012102_164553_F45C6F3D X-CRM114-Status: GOOD ( 12.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_maywrite_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8fcad0b368a08..821c840b5b593 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1721,6 +1721,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd = false; + pte_t *pte; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1756,11 +1757,25 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); + pte = pte_offset_map_maywrite_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml = pmd_lock(mm, pmd); - ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl != pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma