From patchwork Thu Aug 22 07:13:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05B6DC3DA4A for ; Thu, 22 Aug 2024 07:15:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=LXPbr0vKMyl70xuwyR4hAzJSap PrRllb9P9x9xiWrnbxx243Jup7ru+A/hSjFnKMmYhzvGJkoyYZvFZLoBsPGZBRmIorRord/5XsFQw xZwN9ydGhj28+9O5dZowYN7Ib2aihyf80Xy0m6FtyN52cvg8ZB6KkaJHAjCeWtaHLLg6NZYvQ/fw6 U/oduy/1VGcz+3Q9QTZfk8uzT/0v2kkDhhSgz+rgTCUS3m5Uu42Qp7ehvK1mq5GkzjMdHcq8OvnVz sn10g8cj3npL9hIw4PECb5orTI2T/fqKPrI5ITpgAbHsJn7tsoP6SSw66j2Ao7pEKIeBHmld1iXbK bfyWB7tg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh22l-0000000BnqG-307a; Thu, 22 Aug 2024 07:15:39 +0000 Received: from mail-oi1-x22a.google.com ([2607:f8b0:4864:20::22a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh215-0000000BmlZ-4Adj for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:13:58 +0000 Received: by mail-oi1-x22a.google.com with SMTP id 5614622812f47-3db16129143so245231b6e.0 for ; Thu, 22 Aug 2024 00:13:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310834; x=1724915634; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=JzssvCK5VVkwVbKlIUyc9upVJ+aRmSfPPjYrwvasiA3gbkIDXWaecaQKHn7n3DxiZG I4cN+XgbwUfgicgQjVndXtcO3vCsbx8zuq1tl09ZomkhLZUiSPcuL3doDMaWOJzWiiQK ehVIvC2SWhRsGx7ZlvbYbEqJlFuNdieVd8VAZsLru//OvJMyPlWKtFZuVclZ1QvCvckM q5Tzi8aYmgMK9v+gkyThZuAy6Lrud8wHudkStJLs/5l6XVy64gkt7eDKEMsR1E7dXwdS RjmO9fEWSGSauLXK+Tn1YRiwQfsYHUukDeu/Ji6xd4ES/dJ2FiPmFHP0pRGPf3hQLlrH CWVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310834; x=1724915634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=d3m1XX+L8bD/CTAB+hSPGjwJUt4sh8AtCMoPca+cPxACOBEKa8CDEYOHH6d7WWeL81 8xny5RCUOWxS26vrxWZ3ISzMNyv1QR8iX1OoWMKktsxR9fCUxH19XUHFc4AHGX+XLR3k W/yOqa6anhtpTyWyOITCk59oKNSZq9bfq+KxQjfv53irO1n6mAEPJdevJ8MDKHirwBtq DGdrBa14l1nCR2o1ZWzZtSwk3aPQ+OUzCmpenv/gdn8dJZb+d6l+hwnoRBbNw4TxyAgC OseuCuKj+eJiKATH+NvK1LiI5TgA9/GdSrUuwYEVEM0Y0fDkezC1G2FwQhAUxZKJGgmP ccOA== X-Forwarded-Encrypted: i=1; AJvYcCXrxC51jeiU6sZOLfswXSM6mtYC8ArExkySLDryJLcaOYc6AtRHpmINiCTfKUkgq36crB6dNxNzUGGPnlaGe45x@lists.infradead.org X-Gm-Message-State: AOJu0Yzpd4HPOVOKhTgryZdsu7pf0VpKZ6fq3WIHa36lu+/mgTx/7rcs eH5PH2+94SGmZ9jF25jEUDe3EE5PtG6mfcqGmpr/GICQ7ZJCOxiIgWT+uVMM7Y0= X-Google-Smtp-Source: AGHT+IHZvtclVw9AATG67WMtfqsJZD4bpWoHNpEdN87kwQejpgQ5CLrsmOGdaNcwygIvYs6P1O1W6A== X-Received: by 2002:a05:6808:148a:b0:3db:15ed:2a24 with SMTP id 5614622812f47-3de1950bb78mr6135912b6e.24.1724310834483; Thu, 22 Aug 2024 00:13:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:13:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 01/14] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Thu, 22 Aug 2024 15:13:16 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001356_078292_2FA54B1D X-CRM114-Status: GOOD ( 24.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. This can help the caller recheck *pmd once the PTL is taken. In some cases, that is, either the mmap_lock for write, or pte_same() check on contents, is also enough to ensure that the pmd entry is stable. But in order to prevent the interface from being abused, we choose to pass in a dummy local variable instead of NULL. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 43 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index da29b066495d6..a00cb35ce065f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2954,6 +2954,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..9a1666574c959 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + BUG_ON(!pmdvalp); + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,22 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like + * pte_offset_map(); but when successful, it also outputs a pointer to the + * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without + * locking it. This helps the caller to avoid a later pte_lockptr(mm, *pmd), + * which might by that time act on a changed *pmd: pte_offset_map_ro_nolock() + * provides the correct spinlock pointer for the page table that it returns. + * For readonly case, the caller does not need to recheck *pmd after the lock is + * taken, because the RCU lock will ensure that the PTE page will not be freed. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the + * pdmval. For cases where pte or pmd entries may be modified, that is, maywrite + * case, this can help the caller recheck *pmd once the lock is taken. In some + * cases, that is, either the mmap_lock for write, or pte_same() check on + * contents, is also enough to ensure that the pmd entry is stable. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Thu Aug 22 07:13:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00732C531DF for ; Thu, 22 Aug 2024 07:16:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=CobBl77/CaXlISUZBRF7zVm++N EyI166pU2Okul97hDwfq9MkRn7SEk6/bk5e4k8X0pV+KgeONANHCHVYUVXA1Jray9CAHZeCtTGfOU vVxbQjueAb+a24BOu5/JBrNilnZVXm/jGZvaFaesgjAAlnrySB41h+Elj9S6dyZFLRA47mFklsQlL kA4otoUSK4aDG6ZVhTH1XvKrgFjcEaYvWO3vQBA5M2ZZEWnxt+9nsct6c/+aQykdyNnIK48A7aWhC NHPXYQGHZwNiNlCWCohVo6oHRF6yH7u+YnW3xQdzmYOPYWPKozf42mlp/SLKVgw47rvr/OgnNXAEo ZfXEGVwA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh23X-0000000BoBI-15Xs; Thu, 22 Aug 2024 07:16:27 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21B-0000000BmpZ-0oHw for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:03 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-714187df604so1006290b3a.1 for ; Thu, 22 Aug 2024 00:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310840; x=1724915640; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=caSGutUze0wUd1bT/j1bP9AFpVoDzEBspl8zIgOtjMYxH5VR1Xsqy3HJ7W0owFmiwq hFR4GclqKlQxdYkJA1XOZoKRLLjrKiUU7ovpTUFdxy6v0nT/aCVri6HaoCI7bRDg+2oK dlccR/N8xUx5uUskAjp7+kfObJNlIQSn3X6XuGU69+Z15kGJl84OwLmwOGw9yRx2GaUn wlG+CXIfKQv5bOaT8Da4VSJqzfoefQB1o4l9RsZTs+0C5inYakip60GVD/gChsYx9W3j 13Rswiv6ewPemwGb8ZQGonNyDFuu/lNiy6zZmFXRPxxtGsbgHUW5L9U9/hOUT+XQlORk 5J7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310840; x=1724915640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=PDAKlQwR/GObz7DmoKm5CX2K4JT1soBMqjJy985VOHk3dIQbqTAv7MF4WcyOoU5V4D MfFJwp1SHbiqvDPtm//UrokK8laMXSBYZ0TVUmgwo3vjAOEpGjixJztBcq5NGiwVzAGL QwaeK4KYBVgIwOXDiTbRVTOoW2jVIwiV57tlY+AqjGEOXVWG+kaCs5Dr2EwETXzBHI7s 2q6bTqoP3tOEEm/wmjaIdClD75/BrF7yhlZQ88akGf4NCw+B646Nne+VRAqDf1KhaM7t JzqbinpErheuIwh2AfyQmHhJLd032pMquP12FOgd+ZFaJUY3ccPF65lGuMLtK8uZh/jA H0RA== X-Forwarded-Encrypted: i=1; AJvYcCUV4Cyv6IEs7Se0qb7fyxr4JXAswG/fH/z+b8Y4q/qJ0a9yep15QDUAsbHvi40tccG5Ifp1bNbHkXf96PdJdZBd@lists.infradead.org X-Gm-Message-State: AOJu0YwyN66ccE0JQkVqwTWXBq4YYtRgR7S2FKM2OsXjT/YUotqnBrH3 NYwRkCcF4d5ysRwWVzsw6V95rbycPWJfqkEoFTWFs1sPqLZ5BXs3wbEGRdOkZ44= X-Google-Smtp-Source: AGHT+IEhsa9RMy8FC6eeC7Dc/qzW9nWoiqPGY6DtNfgP/dDTOo2T0oKgHsMPC17UGp2n78kSJG5uNg== X-Received: by 2002:a05:6a20:6a2a:b0:1c4:9c08:5df3 with SMTP id adf61e73a8af0-1cae51cf01cmr3205085637.7.1724310840022; Thu, 22 Aug 2024 00:14:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:13:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 02/14] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:17 +0800 Message-Id: <7915acf5887e7bf0c5cc71ff30ad2fe8447d005d.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001401_275444_2F84CAB4 X-CRM114-Status: GOOD ( 12.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In do_adjust_pte(), we may modify the pte entry. At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..de6c7d8a2ddfc 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Thu Aug 22 07:13:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D4592C3DA4A for ; Thu, 22 Aug 2024 07:18:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=w2D3MFoJNHXF2FSPywc1bxYolS W5gmDQywPp37DQZo+fMir+KA4YcXQjwVEHZVEL6/xERcJrtatj7BrdGRNbeQ8jGvCfDxW439CN9fm 627WlpWOQd0Koj0s0YZNwT3krp84JeMV/QGi8MJ7wcM68AgOQ1zY54NU+RhlpO09PLGY/RxVxCUwr nAP+EAPqCbAcydhnSN7YyR0FM+j48WkTTAzOuT8YN3SOnfR9KclEJL9nFcmD+rKGcsbY+pCTkRJDO CUTB6hisXc1b6/rpXNiHQobo3PFS8sd6CWI5svtKCiuYgMx7Vysfk0RWzOUWCR2qGgXq6trAhTQ7X BpbtJXbg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh25k-0000000Bp75-2A6d; Thu, 22 Aug 2024 07:18:44 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21U-0000000Bn2E-3k4U for linux-arm-kernel@bombadil.infradead.org; Thu, 22 Aug 2024 07:14:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=CqA1wbD+NvsZeiJ0oJ4UzEdXxN ava60kGundJlGRqSrkE+gQlq2Nr26wL1hT4I++9LhmPL+Z08U5rrO/Ow0wVH331s/mjSYVUzDaC/1 Bm0BrXZrgDwdPLc4R1l0i/gIZvjZiVOfGxCKh8CcTzRLoHXPwSrY0UWfG1p57xlxkl36OTfPgKGeP 6WusRij7+wC9qNSgujFIsbnn73kbtGDTbPSFcNpG3HfWrUHpiIykeDqd3RYzhYxB8WNc7r3A8ZCje WbtvYK1MKn+b7+QUT5izmR51/QILWWXXQGezdWG5xQQ2yvxLI7Y6Kgnnz+t7RnhPkKcrFNWFtlV5Y l+Rwl7Qg==; Received: from mail-oi1-x22b.google.com ([2607:f8b0:4864:20::22b]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21Q-00000009sLX-1OCg for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:19 +0000 Received: by mail-oi1-x22b.google.com with SMTP id 5614622812f47-3db16129143so245284b6e.0 for ; Thu, 22 Aug 2024 00:14:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310846; x=1724915646; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=iPGiGvJRzP0COcxCeQnOAcJpioE2aIR+VLTaGZ1P9uLhW/4mBKwOvPOjJ3G9MuJM// 8XJoMNw3z0ZKAV9Ml7Rft4R4XJl/Y7t0Xk5x7s+s3MNEg5jamklpaDTuglDwJWZw7d79 PydBGdT5U3M9eD5p7Dhmy9OAfGUVxcN9e/MP8Q0XyApZAIdZCfq7WXiMpEd2NGP3RHLi UEKsLuLSI/HYw5Tb6F8QaF7iStmsOKKXxXWxyHx/BsMdQTujzxIYeTVPjnjfCiQaw8Xe E3+pI6XY2d4UfdjN5ggLLhJx/7l10tb4OyVTBCDAqqO1E3Xc+bXv34QYirAYoNS8cdts gY3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310846; x=1724915646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=Wz7nc3MuLfY3tG4rEru5aKu62vxXgiGDpa9ncwfNSJRmoIwo9oVc3E5cgQ0JDZ8jQi lFshjd9M4YQbfS555nVWgdNQoy5GOzl1JW9iSr6Ip1+bZWWEDr4etb6V4w/rItNyZdWK wEYTNuarbWFiX9BYa+ZJ9vMnEhCKKQAuquxWklbjMw/4OnkXRJ2LMZlip9lvdUgT9DVV /inaGyrQQ9VR9B6aF7J6gF4kM+dSk5rdx7N53lNBFkWmWOVsjZo0R+klO2nIViY4Ijea U8TtlZ8aPjtySoxREcR6P1huqBKlEYccjSIfrYkSLuMkJ1o9OwlyFicRpq0BBYXhKLJI yEUw== X-Forwarded-Encrypted: i=1; AJvYcCWajeOZGk7AS9RdWeowFMyYQyFbtjHOqTOumnsy89NqO/A2xls0Zl58gsCqd6jhAwGRdueYA8q8RFSiHFiDMtq1@lists.infradead.org X-Gm-Message-State: AOJu0YwObHjjraIxPioy03lYtMBVSb3JcQquAI/GkiOBuwyPmz8vDZP8 xfeiBWyN9EJ8UzKNY+VKifdZ+MQKv7QeYKt8LBU8leSJgYMCmHI26lsFEtIUpMg= X-Google-Smtp-Source: AGHT+IFDX2PqWoIoI6wCA8pFxmw/BmBf/PDUvtd9hijDBvTgLhn0KCKCIBnod1yVuIa/+/M4c6qbig== X-Received: by 2002:a05:6808:1a08:b0:3da:a16e:1764 with SMTP id 5614622812f47-3de194e0e97mr6243309b6e.4.1724310845744; Thu, 22 Aug 2024 00:14:05 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:05 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 03/14] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:18 +0800 Message-Id: <89ae94f722996b4ee7ef20d44cd7f5e2f0495c47.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_081417_655292_B053D983 X-CRM114-Status: GOOD ( 11.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Thu Aug 22 07:13:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A3A0C531DF for ; Thu, 22 Aug 2024 07:17:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=QbS0gR+5p3kcu9k1X2H6rHlSAw Z9/j5fleRENDyJRSo46MPGHrAPVlTn5qXQLJ8Caq+ebU/fkf02oIiuBV5/Eg4k0hS/jMNuQdeuFKQ rQTA9aMLKKsKwazskTjDqVdkIgaL5B+RqNCvp3L7IqsdBAoYN+TyAIlfXnmONmQNAwtxZV7acVGka aQvUnSxI/C+OyPAtNbYV9qi5LvaSsoZANoBImdhjZcj2JIni5Wr/jjs8t7q/Iime2C7nbTu5XtW6y qQD5wRVsTkGFkJ+f3TT5Ecleo33e4WWjmTSbLhzgjh02TLphFJKfiSulLxBQ7QBoODSvXc1zPRk5d 9JF0vQHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh24G-0000000BoWh-2HwN; Thu, 22 Aug 2024 07:17:12 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21M-0000000BmwQ-1wNY for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:13 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-71430e7eaf8so391869b3a.1 for ; Thu, 22 Aug 2024 00:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310851; x=1724915651; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=CSliaIDXDQ0LxAysN3lnowwcw8sWpTj8ai9KsxyM635XDDZl12WdGgG+obuZdVhQvU DEe06+tO86bosrQxA95z6uPnjGsCkLvfC0e1HiaBo1Y3uNY4ZD/UT2eR2dSpr36R3fgA tRllcR7nLpeoJ00AHbE0WgxMN99h+myp0EUjZ01OPAaoUDi2akAdaVrqK1vxoo+Q+O+b sBZiQSyfYCn0oXWK+0BTb8rNoRUFPvvNCCSwWikRQ7I73itjprQw9Qf7ojrJg6gDiVFh 5SE0dihQ9OXcNffN/qFKL4LxtXmR+qfgkG6ImkW/sovK+3i976nNjJ6DrMCTv3ZQHVKR vawQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310851; x=1724915651; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=Ij5N34AdApFgH/nJJ0NruhRlMn2LWsHjdiGoTnHogMFpzxQQQBdg4A9FGY6Gs+ffGP PcZsd7w8GiYQKPZ6dZDvJrnns03JMg2DJGyaUoPTP/2gsZ7QV+az/xGA7RxvZgMznxpP 1F4kOCelt8SDaGFL0XPGY42DJgca86MX1eIhpLjKt9n7KsaH7xSoN28O75ktsEKT/sYO 3TLm2ms4IdNCPofPEyLZDIczJuCQvmRXCHQKzNbRKxyb1vwzfiY8BQSjxUC0b9JtV5KS q0hYMDeiJdp5ceckjEkV8xxASPZX09pj80kSOm/1K/GP5VqdG21uCI5Lpe9/Datibznp TLwQ== X-Forwarded-Encrypted: i=1; AJvYcCVw46vWRUR6/e29jpaagqGaIEIWlzjI8c58tdR79ThQ5I2V6++h4qK5nA9TDAUSbRuN4BkCVhnIJadSO56F6fxD@lists.infradead.org X-Gm-Message-State: AOJu0YwgryRlEH759kS9Fa4pSbuf2QXzjUpzmWz/s75hdcedqfGCDsdR whOfOUS+WrmR066rc4+0h31qAISnOUj3l0kFqgsILheKC+tKxz1phm3KdTWGFa8= X-Google-Smtp-Source: AGHT+IGXmNRuv+mQEijx6lEwkmYjHk6XirTogWRG+N5Gk1h3xzTz9eMzqUkZv9fzF0cNQyeHFRUhTA== X-Received: by 2002:a05:6a00:c93:b0:70d:3777:da8b with SMTP id d2e1a72fcca58-714235a5404mr6267050b3a.25.1724310851578; Thu, 22 Aug 2024 00:14:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:10 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:19 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001412_532580_49375A68 X-CRM114-Status: GOOD ( 11.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0f13126b43b08..c98da9af6b9bd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3234,8 +3234,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Thu Aug 22 07:13:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A748EC531DF for ; Thu, 22 Aug 2024 07:18:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=OptSDUCAl5fa3bbobuEoGGPApE NSP+GqB0vUt3KZBMUuKJzP+Sl4X80vpgbHZEUU1xgnBKlaOxoHH16JHxpFkylvN/Pm6A1NpSUxUZb wAO7DqX/7aZ+TF8TEJA9YOckUJug2HipMdVQxsBQXuMoz9LI4nAd4VWVbYWKSFemoxDyzDWVF8+UY jY6TayHgSXMhvhGBuYSl2nIljEDrhbn/Qbu9Qt+9ESopExMdhFckG5eI7qnbXSkKzYGou2e9b2zEb TNB/Q/a5zSMOw9rWfBDqQBCmpxJIqcAxTiW9HX9SeslFFW2sJAM6u68EouvP9fVpHdBKu/YVAaOdH Na2+eD0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh251-0000000Borq-1Ks5; Thu, 22 Aug 2024 07:17:59 +0000 Received: from mail-oi1-x231.google.com ([2607:f8b0:4864:20::231]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21S-0000000Bn0S-1jrY for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:20 +0000 Received: by mail-oi1-x231.google.com with SMTP id 5614622812f47-3db13410adfso298689b6e.2 for ; Thu, 22 Aug 2024 00:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310857; x=1724915657; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=kbgVWPW41c40VN4U0L4EVHMUGYRViD7yikXg298Vxkqy0PnT6d3rn1knB6a2vL3CIR 3cqgHSnuDS3nkBxOWq+fwh0qcCvEmvPsGbr17HKrP05lAP3JuNfPy53Lz8jiZvVscOb3 Qv771IjE13+bbx9sFssRwf72etnUxRObspY+Hg/U0BH24YxKQNp9FO55rB2pxYdY/XAN XQcEXmDEgNu4roH1EEJslfOxtS18A9zcDptD3JQbgsXcoNgFvBZsD8i2vs9XyaQFQ4DX cgxscM6sc3H28re6uxES90s62kiUfTUQ1qhBh4pbBiYP6UcaJIiF/w0mTyTPXzkJd/Gm VK4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310857; x=1724915657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=O4WbDuDVigFWJd/EzSqI9MCcVWxXJYOPdXIBtLnJ0nTIE9PB5f5x+7xNdZ/rbtEEwI FGyeg+OaWAwrU6vTWZQpu3SLwqiUmlwa+6JyxKgpj7QAO1yzEOtDzK2h+dWEiuZRHupO mdV9TEweFqDsEk9VL7Z1NJO9rs2OWcaoXFjAoQv59t2wAUUSRnM5LgVunUGYxC8UeT4M Mda/FiJiiD2PfoAQZiuIgaChDUmIBDIoH6Nij/yuTqxW9pWtlw6SrKGdbSZDqyErpSWs NnL08ROOHFwu2pJW+kl36O3dA9A0NfmfMK9v9I2IWi43MDnO8l4eBBSM185HkooRuzKD RuMA== X-Forwarded-Encrypted: i=1; AJvYcCWlBuALtYK+3UprqrPsU+GFB6T3kU8zi4dMKvTAWzRwHIkiRsRsM8lWO3fu3d8e+LQYR/JiNUPSNB/LkxRJUVdm@lists.infradead.org X-Gm-Message-State: AOJu0YwYIZ++sOa9cILmrOQkJuHlkeYnx+NSQ/1+X7sVIVushO8gRMfH NtAiN7zLbrwZBQRNR3P+/RQG/Zpt/hCxhBrZ63TWJM4830TzUl1IcipHqm77Yfg= X-Google-Smtp-Source: AGHT+IGgjighdWtbrKc2Se12Iutexgq8AwJLXy8Rdygo/b6qt6ZS1aEL6K5S14xFU0XlAJpbImifsA== X-Received: by 2002:a05:6808:179b:b0:3d9:38e2:5392 with SMTP id 5614622812f47-3de195c380emr6158328b6e.36.1724310857179; Thu, 22 Aug 2024 00:14:17 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:16 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:20 +0800 Message-Id: <2b7dceaa12da273c6decf92e3bf2ebc4f4e85ec5.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001418_472526_90448D8E X-CRM114-Status: GOOD ( 12.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4a83c40d90538..53bfa7f4b7f82 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Thu Aug 22 07:13:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 505B8C531DF for ; Thu, 22 Aug 2024 07:19:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=lnfcOodVVAADL1fIDr6J7rZUKD yINvwnfCYtVKWzfx94Y/u8OJ9FnjC9dXOtOjh2XXXKvptqIlbqmzgfimuwG5iAJB4YW6MUihR8AA4 BPXwajcNVceIo5hUGptPg4yAgoNmMzGlnZ3bUZh8G9edU1NXU80pTsg+CkdrBXvgGIa151zYkDr6v fLx98yTTGd9oJF0gMlwItP5nD6CM313iKK4OUx256cuUTM8SwUHZ/wH4bTCd7cw26pdcFvGTiIeDj BdrRNJoWR0RF9U5rV/tyGEn0scoE6AUqdPEr0hOT0TwAvneRjsVJA5ijOLdBgBDucn3XzdfJ27Qnf /GpL7ujA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh26R-0000000BpM7-1pqq; Thu, 22 Aug 2024 07:19:27 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21Y-0000000Bn3k-0KcK for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:25 +0000 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-7142a93ea9cso394348b3a.3 for ; Thu, 22 Aug 2024 00:14:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310863; x=1724915663; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=KV5a4q0pSLPYlBSNjm+kGIpEDvZ3pmtNnHufz/3cqo4kmnPsV2zo0BHycWwXqe+RIg zn7CllPEn6tRBkM2YUEXSI8KuutXUavI5Ykt7ya6MCIfTuEK+pv480bN9gyq6hucBb9m jkOvStK/sLEw6H3NuQ9g7yMIURrFSd+bQ9bHpm7lLOdvPeuDhSilpq7dd8nnoYhk+8Qs Hh/b6PTYSbVmT3wLZXmmILJ2xzP/JDRQvmQjr+BBbFPVcYYCeQOCEWu0iH18C3HUqdeF n35WmQkHVDK17Y4rLEzts4hLcMDDURDpXR0UYJp7t+h6pW7WADargQvTSjzq6BV5bQbD loRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310863; x=1724915663; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=u3vCsTu+8U0o6NvecXzy1gIq6b0weizqde539NUt+a9HJaPoSCFmmfSCaXZXtIWk8z zCkRy5E3DjkpMOimHR+cE4Ncr30uCldvwTKzyKDmOLXg8CBlgfaN0YTTzysMSU4lIe5M yoP+I2fi9GERvXKBBIRF3FWRxAWLWmggqjOpACQmqGBN5XOPnOCw1j1j9O35Kk82agMm bWwdoXi8se3McSKnRjOo5TlwL3BAyVA4wPu9sCjhy7tGtMc60DaVxJoF20hLcctS4jSF KtpzHG9+ZpfXPfPdDxbqoQziw7b6gRRx/j8lbEN6nPj7oVkEPJC8eGs9MDZcrLpQi8+l oRsg== X-Forwarded-Encrypted: i=1; AJvYcCUvXXqx/nghuYdX387qadUilGv3h2Llph8N7bfxLhYm9wABWVY+ELLGSOPATahPAHp7BCqb87n+zWhuTAmxm1gq@lists.infradead.org X-Gm-Message-State: AOJu0Yx9lTVF/dbOq/DhAYVb7huA6gerYKKVa66bFBEjll8MMrR0QVMb dKN4jCrXLVf9tD9A+0L6nqKVzfisGbh+vt4dFoilfkRrQqIkZgVYIfv+jHaF4uk= X-Google-Smtp-Source: AGHT+IEJaUpX6CQIEutPHebqIOZXfyMaKxdLW1UElMR3gb10gY3PgzFG7UEaK+UwdtskMN8KUpfIGw== X-Received: by 2002:a05:6a00:cd4:b0:714:1bce:913a with SMTP id d2e1a72fcca58-7142356e960mr5852557b3a.21.1724310862916; Thu, 22 Aug 2024 00:14:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 06/14] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:21 +0800 Message-Id: <5acabedfae7ded01b075960b4a91f2e15b4d76b5.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001424_166307_857A307A X-CRM114-Status: GOOD ( 14.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d02..7b6071a0e21e2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5499,14 +5499,22 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte will be + * modified, but since we will use pte_same() to detect the + * change of the pte entry, there is no need to get pmdval, so + * just pass a dummy variable to it. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Thu Aug 22 07:13:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46AB4C3DA4A for ; Thu, 22 Aug 2024 07:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=4OxQx2adcUK7HZbVVGKyj/BfXv Mc7GSzLZ0tN+j9R5DhMzuAzhoGsMjhBd0848pZSYcuqoja/+Tz7O5PmE7Btw82+9Gg3lrPqNe9dLW 294pKZLZxPe1FHryXivxR4LQuL/UWT2Ai/1Q71yI/vAVfl05gsHcoO5hwt9FUTavvqwyh63V2mXse nYieC5k1Cd0nYvC/ArF0i5gpsVE+myu3bxeSDzlkn5dUobT/VO6jWQ63fAhEz50OS3SJ/HM4y9geh OsNiXMWe/KrpcJWixW5tYFSCtnoXe/px0KYRcSIRF5siuUc1LkN0AlRZjK61UGFYiSq23nFF7ATg5 vRNyOrmA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh27A-0000000Bpbu-1TO0; Thu, 22 Aug 2024 07:20:12 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21d-0000000Bn7e-0gLv for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:30 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-71431524f3aso398601b3a.0 for ; Thu, 22 Aug 2024 00:14:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310868; x=1724915668; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=P5u+4yig81cfTv723SGU0jczkpKMIwJUVRg0K9V9/jDd5+LvaImVh+1XUiZGYVCSmb ODlDyIFVxOsyhPztZ+D9AzeSZJ7GYtx5Xcc19Y7+GARXBmx4vYN3mmZq0FPerbMK+thM qWSWpDrAALHnT1hPTFkV+P/6IfbcZM25WUqy02UncXtyzo1/5jzqnski0WtIyTH9BDFR wvaTi4SvVkt5mCoDrnC1IBj5sbbb28UmoNx2myITFfatZSVhxmZx9Kde4wtcO9PQtCnJ 0XarNXA+kFXDcWPpNtCFKuGm5rqhkeuETPPZwwhJF3k32016JP1fxmM7V5ghA7+7vh3U yyQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310868; x=1724915668; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=lHUGlqcYXOGB1bru/EW0XDdbfXz7qr4Ak8rJejPrfqdSddwcGJoUqRy+ssUS+ithSp Q+zuoTuVcwjCEsTMWBjGMpSUxwRCr9ozcHiDwLpABYo/fSpgpBcbXOukchRXCtIOLk5D m8kzSf1eqpHUfZelNMfFfIiD+iLAZ3ODcH5RAG8XaW1N7oeT3Q/g7yUe2pZ2Ooo1qsnq McH1poxJCg5evRqF2FIksk9/1WLKZzfdYuvL7yndYaVJIpbsB8BVHM1s+gcLkLsDo+x2 6rhZHXqj9gm9iF82cK6EnPEkyeuNmDyQXJ2BW4k3UgnpWVaeSz3mWSq+7O3TzumXF8Ca qlOA== X-Forwarded-Encrypted: i=1; AJvYcCU/Tpy5Mxm7mWSP1dXdO72+0XwdWvPfa6gCzGRjsNEgfY//t/NHDqMtWUQdpX9KSxhTzGPaJ+KbFp/JZDqqStMz@lists.infradead.org X-Gm-Message-State: AOJu0Ywj5TIlRlWWLGqZEM3cKaEdQ83FDDAuVmRH3jzgslbgvJ0VVZfK IqOV09m9b4mqiamM1agtACsaR69oCLsZwyNRSGwWRfW0LjSNZRMp3iij9+ewAlU= X-Google-Smtp-Source: AGHT+IEV/3obR+4TCbDGR6rVI1h9FT7v1/u/+fxVcK1o1ztnph09CHD4OJqmjxMXnPNwi8/vaSiIPQ== X-Received: by 2002:a05:6a21:6b0a:b0:1c0:e728:a99e with SMTP id adf61e73a8af0-1cada078d94mr5247385637.26.1724310868515; Thu, 22 Aug 2024 00:14:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:28 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:22 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001429_222319_9B6423B3 X-CRM114-Status: GOOD ( 15.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 53bfa7f4b7f82..15d3f7f3c65f2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1604,7 +1604,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1612,6 +1612,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1657,6 +1660,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1688,6 +1701,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: From patchwork Thu Aug 22 07:13:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12C74C3DA4A for ; Thu, 22 Aug 2024 07:21:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=F6kOPUmEePGMU/YOehRRkOOAMS Yd/x182lJnHAbxMhXkJ6XCTPlfn6c0ntpcHpuTo8IT49R9fyxayCv/nuNRotNI8un68xdYZDsvhI3 K4qvyM9gMV+WUAR1qCMn4rhIxgGANDTL6IiCKPKLuS+rlW5vIEGHQs7QfTnO3miBU8/qJGJcKRgNA mMYyLvy+dQVYQaSdS4t3n4nOYMowCpMZxo+bu66gqgJ+AOwLAR4NyfErqQr3TvlB5tJ9EtT8hLqp8 MAdgIEpM7xVkY6qBecs/CDQhmmUxaEGUhdTeUluul2pF6vXfOIWsl+fhHGftiLhzczHPPVG4M5Sk5 IVxNb45w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh27t-0000000Bpva-3mpR; Thu, 22 Aug 2024 07:20:57 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21j-0000000BnBi-0QOR for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:36 +0000 Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-7143165f23fso391262b3a.1 for ; Thu, 22 Aug 2024 00:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310874; x=1724915674; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=J9ZGoyH3FDkynMdnMZSg2Ae8mLxPV5ZxcsvO6TnScqQaWnKfe9DaDsKKEZTu/6u4sI k0ta+02/lcRY/y5M4TnwfRQL9hnVrtEnk3x9hXImAiA6ftl1kv2CQBZlC8DFsxEunNbs OabP+Jq5kIjm2M2tARwuKpVdzxKQ2Gfg6vYuzTOIL/jASLlupdh/6EJyd6Y0Tnh3noer qncwDlusnTE0nKZeU/6knt9d0sUkwOJM5iic8MH7BsAT3KrxgR579rxReI8V5gyl+N/I egagFt+hn0TASStXWQ7Z5QfFoZuO1OwdEOnQafOT29R8YeumvUp18caUJx/7/LPjhoOj Qv8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310874; x=1724915674; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=vMLnWHIwqom8a1makHHmi9GRdZnmPiJfrLKer/rkebR+tixT8I/oJGZyMtdrPPqGR/ vK3jXyYxyXb7/Rtf9ardERTtOV/2c7ZZwsvMQaXYbpoeoG1IZoXfYQrcZTyw8aAAT1lk e/4ci7lIvmU40Q/8fyJuyjGh9kZe8OQp54v0CHq3F5IvcAVXsoTZ8hkzDTwFUf15Htxe kZ7EADgGIrwj04BVJ41sRYKIQPICo2AKyfPyj0QuuzLtSV0VGaRoviJ8GgtHa4mny0zJ z4LQSkC24/dL96SC8vkFUYbUGm0drKOdGz0YGQPybP1qG3MhfB8fIBou/gccfyZFtYLZ 8L4w== X-Forwarded-Encrypted: i=1; AJvYcCUU/+xaGVFszQoTZ8AI/IE4zXus16VSr9yrHCoen4qO1/SX4ecdOuUKZFVXkw+pmU+RMsigeFyCWe8jg+TWvh4B@lists.infradead.org X-Gm-Message-State: AOJu0YxfgHvYbwUquo+6j6EXEpIOV4n0ja09/0seU/ILplqBMYen2CDd XUqMpomeI31KRPnCN5NVS3+xb4CN7E+cC9OXUZSSwspVeG9KyIpJw7I0B9k+mDk= X-Google-Smtp-Source: AGHT+IE1Azw7tBspdYFUO9+EydXCiNKc3KqrNzafyoF+69WNl4yC4JRDYA6cJOAAfX+V+yHVonAuzQ== X-Received: by 2002:a05:6a20:9d91:b0:1c4:f209:f1ea with SMTP id adf61e73a8af0-1cad80f657emr5853305637.31.1724310873985; Thu, 22 Aug 2024 00:14:33 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 08/14] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:23 +0800 Message-Id: <71100c3867c4cf6f5f429ce9f2db8432066d0e99.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001435_159734_4F9E7502 X-CRM114-Status: GOOD ( 11.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already hold the write lock of mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 7b6071a0e21e2..30d98025b2a40 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1083,6 +1083,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1108,7 +1109,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we already hold the write lock of mmap_lock, there is no + * need to get pmdval to do pmd_same() check, just pass a dummy variable + * to it. + */ + src_pte = pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ From patchwork Thu Aug 22 07:13:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD4FEC3DA4A for ; Thu, 22 Aug 2024 07:21:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=Ra70lZP5eENWTmEg8eTmVEnyYQ ei1wr5EXZW/yAf6B0qX41Wp5p3HKMav/y3mNO94K0J4bPXOFk3TgTLQAMoBQCZ5afOCsd3NPNhkl+ bzMr3p7hpRdJm4zditAoXbvBPRc8w7Wlv2QPtYuXegnVePWQZpfbJ3MRiqMzJeIwa926GERGN8Lnn o96qIipeJ9bB04pg/9gaonpIl7O6oiTlv+9Jag3b/GT5W0279Q9v/h4R9oJtnWxCyB/lcDF/CF5J5 T4NM8PN4DPSENEjB19dYVzh9uH64Lr64h5SpUAqKeZL+yfqHsoTjMQ7t7qMF8euPnp2VFEqIJF+tt GFpUPhVA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh28g-0000000BqEi-2SsN; Thu, 22 Aug 2024 07:21:46 +0000 Received: from mail-oa1-x32.google.com ([2001:4860:4864:20::32]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21p-0000000BnFW-1Fls for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:42 +0000 Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-27012aa4a74so250300fac.0 for ; Thu, 22 Aug 2024 00:14:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310879; x=1724915679; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=Z+zWqv1fJfCR0Y6It0hmpE70eNYz/31M1EPLn+GJi5v3bS8LRZy5Ypkdi+1ot684Mo OcQl3DT0jZCMYq1PBw1lTdbAfTLwh3ZjFlX1gz2/seb75IyiTAt/PnHIX+YO9uWEkV91 1DOg+BLUMrJCTKKKP5pkrNrS+F2Rt4QBTLb/qPzkQgxRlNFw82GtYrHPneMNPgSMePih KflTH/GZax7wlUvGzR8kHTJgN9H1iuiMhToejdx8NMkMglhLoz8cG8owYZIDi618fFjW qQohzPG4cdt515cJQx7qAQFfcKAmW15ZFqW90PVvO5iZvG3UYRrUNkK4Dz6pzbmkJj4s xm7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310879; x=1724915679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=pf5CBqWcemS4BXPMgyYN33Gk4tqO86gVR7wcpA7H1i09VFHG4Cvn4GZbz4TPlx+UJz mwncKJihwDPlKp5jYBC6cVC7/v9FJYjjr59KjncMoUbj3L92jGWjz+2R/rXem7tnnJjC ShOXhq+F9io9/s5E3/CWLQW7KJA2Xwxa9FrQ01yv7yIp8cAq7hp8y5pZoxOQdfWPSM4h nmldftJmlu7iSqaiH5dyVIWkEUbjotdE0xWIyccoQChOWb75+yEzIVBYH++Yka5G1+tI AxN55wSnTknY1AEp4VekHa//G3bkYqqcX4IDnhjcFxJVpXamtstvlaCm8vG4EOce1S/S m7WQ== X-Forwarded-Encrypted: i=1; AJvYcCUBtkmwODnRfE/HsJOyi54/0QSwNwABOrPmvfOjngXIzb/xfEt8nLYIJXGWBYfUXCONOdEUY21y8ZVpZ6pS+aLu@lists.infradead.org X-Gm-Message-State: AOJu0Yw+Gyhpnk/w7fA+QTaZ21jp7FzJjD1eW6J49EL/ZGsUYi2al3cA Q0EABB1HFPhoFI368d2EZOOUXN1eEXG3aHXCwWAJiEAGaS/Uey7hgqetnLYWvYM= X-Google-Smtp-Source: AGHT+IFsTn+Q5A1WOhbyPTAjHclYiOvt3nhLZV/Hi5KOu9NOtbxQa/3JykhMvHE3PrFxCVBhO2k7lw== X-Received: by 2002:a05:6871:149:b0:268:afc3:648e with SMTP id 586e51a60fabf-273cffcb557mr1073020fac.42.1724310879532; Thu, 22 Aug 2024 00:14:39 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:39 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 09/14] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:24 +0800 Message-Id: <12097a187fee5f21860d4f87e7aa103f822e298e.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001441_360310_6ADAFF86 X-CRM114-Status: GOOD ( 11.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already hold the exclusive mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng --- mm/mremap.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..f96b025c09079 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + pmd_t dummy_pmdval; int err = 0; /* @@ -175,7 +176,13 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Use the maywrite version to indicate that new_pte will be modified, + * but since we hold the exclusive mmap_lock, there is no need to + * recheck pmd_same() after acquiring the new_ptl. + */ + new_pte = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; From patchwork Thu Aug 22 07:13:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0990FC531DF for ; Thu, 22 Aug 2024 07:22:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=LFY3O/RryG8umnfi2kTBtr9NLo h/WZ9DkUlR7bQkI9PC7MtjIlGby+E5+Mw7cajlt85Ct9xIVfsApvPyuKUze4CYApBP2kAFjbWxUGK RixV1LmMjM3WawuAO6EF1bV8Del+lWMQcBmhy8ujIS4UMMHGWjTM8l8hVDvS5dPAuhWADM53qLHeZ vrD5YLW+GmnL+waZkpQ9HxhERzS/XVz18dwYW70hmLKKza+ivmccyXP9HhaTvBLcxi8wD0nuLtswP Ytm8QPJrtHr8twEgPGm9QbtduEMidWcqPUP0+ZmlgXtKLpfBen8Nf1agGedMSG2Ki3jcnYoo1q7Ya Jt/nhk0g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh29O-0000000BqUH-2vM2; Thu, 22 Aug 2024 07:22:30 +0000 Received: from mail-oi1-x22f.google.com ([2607:f8b0:4864:20::22f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21u-0000000BnIr-1gLw for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:48 +0000 Received: by mail-oi1-x22f.google.com with SMTP id 5614622812f47-3dd16257b87so212307b6e.2 for ; Thu, 22 Aug 2024 00:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310885; x=1724915685; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=XOdCr0+E36lbKzMUG/HljCBAjccDgM0Bv6s/LbTsj//4YT9J2cwJFk7jjx/1esdfXy iSyFVVVbV6a6QB9agKQjh73dv8Q9mnsZsUYdOGdxuLZA4Wjjoy2GwlRNncGLwWIHo0lO WFH3pZtWvXuNCFXXwc88DHR8HG2SncmvL91gaEbbPbFXqNq3AmZL0i2Y3rNKxtzPbr6u uDTlAZn9XetbaVYAXUXaaRZQ591oB2zDIUdBZXoxsFDYvSUBJ+raTKa0czmv9cSiGWLs Sy/MR7zNjYbcuSPP1YjpvnDL9vrMwKh1xX7xiXUhr+DuqLon1cuzZkIII79g86PemADN 6KVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310885; x=1724915685; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=sIsHTUUq3zO8JJQIYl4aMzWaTmK3pZJkuJ4crD/3PYtNZ95PiUd4VhyQr8nLdGL4Zj DUKjHxOcdvXeVWK61vuRMbkuitBIqtrbI6RFugKPPO0Hk3NH9sKiOuGALg3rkT6LvPED bWy2+s57kfjVdrVhrQ1PXMk86k2Z12ykFom6eirAcNAyWWZ/CMiwpNSNjEnF7z5aGW57 hLmxqMYQa8pSmtFkep/yhIJajJNqsoSHQOjBs4VssCgOIKKCKSpzVhLqgBOPdCGcLuGp aYKX3fRpQfj/G7KD3Rw12/6MMiaWIcWrsOk93Qa6tXCuxP7FsNFUPhQpaHy4yFCjupZA N35w== X-Forwarded-Encrypted: i=1; AJvYcCWN7aTGlFX+aGyMtkJeLr24HQnWkQmd4Xtj5Rc9h20MwKXzwGqeIsMAhI4Nw4KDHAQJE8/XhSukbCXMLrH/C20H@lists.infradead.org X-Gm-Message-State: AOJu0Yw7a7pivxVfQ7BwEDUbl/jOkBqATb2IB4QhYMFLSGrzVajDNzCa i/7Spt9iqjql6YHUCgenkrgstUc6FqSlLw545hkf1V1iZqXihW6jNdlI0VICA/Y= X-Google-Smtp-Source: AGHT+IF73IAzwqsx0nomJCVJryAV7cndKc0EgZ0+2TYoDy25bfQlq+9JSwXFUBQmZZcxXSkt111lxw== X-Received: by 2002:a05:6808:d46:b0:3d9:29c1:be5c with SMTP id 5614622812f47-3de194fbf2bmr5329807b6e.12.1724310885053; Thu, 22 Aug 2024 00:14:45 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:44 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:25 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001446_489748_96DCB4D8 X-CRM114-Status: GOOD ( 14.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..f1d73fd448708 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Thu Aug 22 07:13:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 27ACAC531DF for ; Thu, 22 Aug 2024 07:23:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=uUWhYy/XoK8f+ovTT0FDa/Vu65 MRz2D34QB7sPndOgt9EGQpG217pJx/ve+bOqPX4Ui8z6eymzLBBkAJ7dmbwI9BDkw3/MZiPTIV/4c r+yWWGjcxtNRsYDIIFF1uLPuyLReEfDk5ESDsNzvYNXSC14C7T4OAKLtdWZKH1KudqGz/Owmqanlb M/3FntPjswo+Fnopf82sHufKuXrhvcMUdU+dbMyQsUyxCw103IHdV+vfrtx3lnJXblciVCWB68Z5L hS24AojQP2bBlgRVmfedW0GthiDQneszkloB8hDa4u0GyLXIx5nUtiY7CmKGM3PO+5/9DHR6YlE6a 4wlrGrFA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh2A6-0000000Bqhq-2g1y; Thu, 22 Aug 2024 07:23:14 +0000 Received: from mail-oi1-x22a.google.com ([2607:f8b0:4864:20::22a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh21z-0000000BnNx-2QEI for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:53 +0000 Received: by mail-oi1-x22a.google.com with SMTP id 5614622812f47-3db14930a1aso293534b6e.3 for ; Thu, 22 Aug 2024 00:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310891; x=1724915691; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=WRsCp2PxdE6BMAF5zsiFf3msfXt6ZJFf0QRpBZT4fBJnocbsK/JzLmbHBOFzn53URY 3LYuhUq1PjPhBNkHz0Wqu0sYD7i3z/oWOuWfwWH1eBcdIcdi4CfKvV5DrJ0oxCp+X7v0 TmCmwEJG6++tOHGMqpEIfctJwhnC5/2NUsJBHHxr8HQvZub44cVNRZ+gYU9kksIip+ZA 3iJh3rgCS8PR00R6pGiD7WT5+VY5pF0TJ3Zhf2vH5fcqSBqqy6jeiUSMSO5FlTzeEUy3 Ig+l30eu+Vx827bLPl5F4dLfsnbexyt1qfqiPWqnLdxtkozeWi5R+o9shzakfF/AV5xd lnPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310891; x=1724915691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=kVNHwcs5Ot/ttsyBKCAJkat/ezq3C+W+/r2eP7/1hdmv0iKjFDVU8eaobSf4IBh5HU 5d9ag2GrYUH0Gp6rCwVDdwuaau9I6Nyh0+IArISITuP/aOPzS1RJWmR6UIYOMiUVX/ZB JSvKuNVhC+6uqoRLuaOombt9OBcwL56fCxqRsISnjWfh9hM+477xRd/VQoHn8dcYNrnc 8uTaEJ+ln2rL1Xl3IAfdRWra8WYcBgc7n/nwONcJZuZJmQUefPYzpJc9lg5mmTL0TjBb Dox2WFtjOoYjQHKYRevo0gVLu7ZnAfsMnvoAziUMQbg67c04H2Z9ZfDuZ1tGF6gk3oGw Hxkw== X-Forwarded-Encrypted: i=1; AJvYcCWvI69CK/e4f7Di5W8VFeN9rdfoBuDIFQ5a3oxcHtjwsaBNMALX2SRvAS8gAPsL2oHpR1d2gV+SZtsQz+GkuLT/@lists.infradead.org X-Gm-Message-State: AOJu0YzYY2Xu6VurYB7B4XwUBMAVwi8F02yfA2MdHV7b+Q7WcTBmGAgy /0eOKlMfaZy9sB0sI3+5T8ULblFpDQTucmE34ltxdvd1SioZjpaLlW93OjmElHQ= X-Google-Smtp-Source: AGHT+IH3dTXEnimLAtKAfnTveP36n8fQueYjLUMvugh9RpWMjyLI+ldrjyQ+VF2Ub3qToXnGQhbMbg== X-Received: by 2002:a05:6808:3847:b0:3d6:2b12:7dc0 with SMTP id 5614622812f47-3de22ff6ae0mr1325714b6e.20.1724310890896; Thu, 22 Aug 2024 00:14:50 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:50 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001451_674746_D6AE47CD X-CRM114-Status: GOOD ( 12.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 6ef42d9cd482e..414ee744257b7 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte = NULL; pte_t *dst_pte = NULL; - + pmd_t dummy_pmdval; struct folio *src_folio = NULL; struct anon_vma *src_anon_vma = NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Thu Aug 22 07:13:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44C95C3DA4A for ; Thu, 22 Aug 2024 07:24:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=swE3VXBzCGVYtgAURsL2Ci3jGp JuKmtwb1Q6J7Wt6hip/UaRvX0/130VKRu29q41zCXdjYWenInscoC8vrS4APSMi6teNsqtVRuPpAB DV+kUettf746hYsBGC0T+NIpe60Aq4TUHmpuDNc/cUUaZC22zA+wr4kwFKMg06ZdXn/Yd42LSBODk gTfYwKXJ/u9lj8+lcm2tJns4pUi871irHU5ycFBZ80tyqNYshXNmPX43VNyntsLc1bLkxbajLPbX1 j2STt2e4SC9IR1BeeKfnm9eAmAJ/sepWJHLKHdiAs8F9px32akVRrwsou5686jJZ5IWnvJSi+px6x ZiyqyfAg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh2As-0000000BqwL-0p0E; Thu, 22 Aug 2024 07:24:02 +0000 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh225-0000000BnSy-3iY9 for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:14:59 +0000 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-20202df1c2fso10661875ad.1 for ; Thu, 22 Aug 2024 00:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310897; x=1724915697; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=gyFV6xLgFegC1I1XsynOjzCcm/LZomG7EfDdbqMdtA1tA+hcq+Gx1AzsJfbfRmlW6a dfQ1L+EHdoNpXxhnB6cff6lHsYRUQWcF+1zfi/VIhKnwlZfcIhSOJvxrmUkB5JxDZdE9 AOf0M43dSFbeWrqp0hC4a4m0Eb2SgWP91QUy0k+EZhHzhu7mfRcEq7WKfQz8LSmnxMKO iE91dwUXsSfMqZEkeEcssufjut/RHHhUxzjhwDK7el9xSKUuxTZyyR2mpTnBjZWZ7pPS q+kclYjWbiQ/Yyxob5pjGYWgNZua6tanRe1bvbRGr1QviNW0Iyja1ibNy+d8B7dEhUuy 6G9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310897; x=1724915697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=ZrsS/l0IdDeLMy8fNBG7vyetl3bc4yeB9eBfDZ3YIC1a90XUbhJSyydNuEjvkVZJ/s NcFGyyMtQI0ChvOj0Uuv1h8BRA7u4ste8Tzg91n1kIDJPsWnY3ngVMQdSCKUj/0lLrRr BqPmr9Ofg87c9bSXwgUc/Aw/hpT2Pdw47dBwCWH15GZ7oEQlN8N2xmmAt370Q072HDtI lfoahnsa8mT9RitB6gezlYRKjtZFYbLD33ISL0lUYRfzi75+YU+BWDomljtXOS/Vmn9e NToXVcHwa3orUpMRskQixmn2Ekm7nF7zzB6V3a93zQ2gDRt8DZSY7PvUIiz+JcvhKdrn y6Ww== X-Forwarded-Encrypted: i=1; AJvYcCWJlyiFL3QcP9RK+qsLzrKckLB5oem9kDP5XbXsnqOCxwNCBJG8rMUXhP2YtOnO/ewhT31CF/v5ohbO2x0dEE8k@lists.infradead.org X-Gm-Message-State: AOJu0YxDjlE1Gs2wJ3xglcCeYKMtHYWEsPbi551TpuTAI7GPYOoEX1Y0 4+MphPz3TkzjPz5IkxiSAnlWjlwAMRB8BfCKilqxO9IU80JF7FN+SNKcl4y7g4o= X-Google-Smtp-Source: AGHT+IF6H0tDfgQqKcPaVFXqnNnvtvkdaP6bepcTCK17Vph5rD33YztktUCyS9gQ0weOULsun7jJQw== X-Received: by 2002:a17:902:e88d:b0:1ff:43a8:22f2 with SMTP id d9443c01a7336-2037fe188b8mr31159405ad.24.1724310896888; Thu, 22 Aug 2024 00:14:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:27 +0800 Message-Id: <0e2980e9a31d3503a5dfb350c04f4ede4d05220e.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001458_040243_A345977B X-CRM114-Status: GOOD ( 12.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5dc96a843466a..89ef1ac8eb1a6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3396,8 +3396,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3405,6 +3407,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Thu Aug 22 07:13:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D0D8C3DA4A for ; Thu, 22 Aug 2024 07:25:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=t2Qsu9n2qtjWRlJIlolpGR5SZE /m2pGbuKKsAytR59wZ+LxI2fXL9MP/K/pvui5O3YZRdEsJP5k63Q7upPIAEEg1a3VasATcw0nLBZG r+JLJ5FOyIjwPzxA1EGP8UOFqTNZg3Gn4qyF1SRi42cRyXAYd5fW0l3g+PpCeYEPicW/3P+Dxlryt CNGihOIMcz3DdzHMASEvEnCCv/md3CPJitFUETiArgOozSbY3/xPYUTupZfXstA7e75w7wdOfRYsW i2PxRocPJnA2sw36rE1DvbEOy633kU5tBVgYEImvVnoFEGKHgfS+KHScNoCnk2KfOLcWhXXcscjNE WgQvh9Pg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh2Bf-0000000BrCd-0ZJu; Thu, 22 Aug 2024 07:24:51 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh22B-0000000BnXs-1jR0 for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:15:05 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-71430e7eaf8so392204b3a.1 for ; Thu, 22 Aug 2024 00:15:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310903; x=1724915703; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=PIuSwzJ5yrffXejcFffHeoT+FkZ5msDhEUgmXXdw/+5I0RG8kcRLmvPwccBe41Zd8i XLYtYngw2oW9ltawOSugIzLCD1J3VvglnlhyzSvPvx3dWGGeuveiWKSeKtKvohH0zu7K J2iS+UlK6wfw3E7ug6oFB5veAAmyuZGjJx2AckCfWxW0XSHOXUcAexM34e6Bk6JrS8S3 sdafmJuPVM/fZRxbzw6ke0hc81DlSDWErdgf9/Y9F+zM2Ro41Q0IWzlBiRQR+a0Hs5S+ 1LGO8G/gLbrGfWaFNvys8/2Wzy8iIYgW0tNmOmxPVvl1gO+rVoVb5h3iBkvTY/JcYrP3 ztUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310903; x=1724915703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=BRh9pZ8vQtV95AtIFFTJDWlQg33V1jMmNwLI/agibDyeP7wmf0Lj5M4LidBy6b2Xl1 T94U5yNq3jiQX89fXS5tBnT1lPBYjDtQdVGusNDgxW89P5v1I/ba7kXaYQ3NcNDMdqJl ESYTBy9QwjJaFN3ak15eS3ZufoJ7Ah5TSCgQlt/hG6Gl9D49snNI0h6BVeI+V2owvP+E LPCI+MOJTylNqd9OVvB6d0y/v4qnrWijduP6LHvM+gZYmyyqweHcwXVRMHQxMxW/k4tT y3LqFPl5xdpvlNap/QiDVRVD3JnlKhEGUtD/kXl6LxNny5wrAfmlzwHTsDIU/9gbPF8n ZRRQ== X-Forwarded-Encrypted: i=1; AJvYcCXCHgUp1SDwM2pZtWQmIDDDRQ6jF+9Y/wX6CwU2r9FrchnNy13W8WwSHPz+Gm81y1tUbHz87xZKF52EWiMYeTYN@lists.infradead.org X-Gm-Message-State: AOJu0YzDam7/crQOKBo/1aWK7NkD0f6ZKKqcSC2mUjQVfM4FacEAQed0 8xZK3Tjb0amwPwQil1EmQTpan50N+MgD8Kob3eF1PlDfntKWuFCygrzq1NAHFdU= X-Google-Smtp-Source: AGHT+IElSFOoF4KMPSsF0Gy5tbhJtfEKgqkCJ+pjJm8bK93AAf5vkcbpbvWiZRIPotD+TEYG0fABhg== X-Received: by 2002:a05:6a00:b53:b0:70d:2fb5:f996 with SMTP id d2e1a72fcca58-714234922a8mr5431869b3a.11.1724310902611; Thu, 22 Aug 2024 00:15:02 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:15:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Thu, 22 Aug 2024 15:13:28 +0800 Message-Id: <061a61c5871948680488ad02fd1e03547948a500.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001503_484318_9B9141C6 X-CRM114-Status: GOOD ( 12.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index a00cb35ce065f..1d3e8882f6ba1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2952,8 +2952,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 9a1666574c959..3b6dd8bb5b134 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like * pte_offset_map(); but when successful, it also outputs a pointer to the * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without From patchwork Thu Aug 22 07:13:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13772879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD8A1C531DF for ; Thu, 22 Aug 2024 07:25:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=piU7DMt+x5fr51kpiLeBoXwmwY 3d2hC+aokKYEtyMN883h+56lE0V27tD6Pr45YOVgJofMaDEA4RsoqzsNUnRwfInD3xwgRBrh/DKLN KZ2fAFL0/LZ7yBdnHfoK97HX38d2UpebTKDui22zkAfqld5hMuZZ7lLsld59s40oYsWLfmkLuvOwM IHY7AT2AXYRbimlKyKnBjwn1VzFWI7Q7GeIMRqn3Q/4KZ8ynJt7a21zBxozxtIBZBmYy7pqNLsPCf mxyQk2brrWClR9vF374l7hNTVZ5jLMZ1R0STqCx83UfG9FnJak0bai+f88n9ZdiBuReaeLw5V40XI /eXUKdhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh2CO-0000000BrRM-1Iqe; Thu, 22 Aug 2024 07:25:36 +0000 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sh22H-0000000Bnbk-0ScM for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 07:15:10 +0000 Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-714261089c1so332092b3a.0 for ; Thu, 22 Aug 2024 00:15:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310908; x=1724915708; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=UoiNMxWMHnRRmzLYphH/qXukT70LwkogG4AsUhsvMsGC1VcZNomh0zP/bVWIXEElr1 q6XHiAQJ3vdFzr+zGU9wKlCxGcPVoIJoYSxvZHFf+u9/YRwsnEqXWCSJFg0DvFvuQCsT N3BzPxYxDoG3tAjfHusRu66JWuUpL7s6TgUGBF7pCg52+fk8bT3tMnHupr7F7RC2T+7Q ZxAsKVOhbu+bmtxK3b1VUCFWgGjLF18B9Zsg5/yv7ANjCVvYBx9id0w4ropaJ256eCO0 dHp4PcxA8S6je035FY9cXvg8yXoq+zXZ4SXram9qTG0fs/KDhnKs5GVzrF83knmOcD55 6n1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310908; x=1724915708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=TSYQ3Pxm26wMx9u2h+lo/hDJaxxZ2HPet52LQfHEwoiRLVO1KHvwp77yZuxD+T7v8Q VbsMesYz2ec8J810K5olynNXME7iPOFrOmczgvYFOi0IRtz56F+6VPEpI5amIJPnxEvn PQGjSVnjWIWtn+ixwdnOBSAXQhERWGcYMVQbpjDyJt6AHejC/NnZeLNTWa4TRv0Q2IKI xoDr6C+I+n2OLO3ElScZPG9zcjOznCrnJTDiTgQ2ibUsJu9kyJHGZs2ZlyEI5VHgAgy+ k13NmOV+u1aVvi3SL/QmkYItAni01vM0mi/kHXA6gzY1A6h7ExCnFBHO5JOzcnKMqPSs cojQ== X-Forwarded-Encrypted: i=1; AJvYcCWV0jnwCHsnqNiLF1MlN3YedPQOFB14GjBOmvmmeg4FkLzTAvRRBYokzmJ5Ei5mAyqp2hxXtF+eujNpZQUV/3uG@lists.infradead.org X-Gm-Message-State: AOJu0YzgUHgI+GVYo127kNV4TdBanOQ1jYkh4kb4b1LgyUiX2cleqvUl EEup6FZiCMi+FHPBZsJRKCQ1jQH6meziL+4YghwqjoQw+tV30A4YOKghI6a9dkA= X-Google-Smtp-Source: AGHT+IGjO58kdb6JjGpOfA0UnXFn2ByeOkp/uIZzxQt7YtlV+9PVIdBNBOEj4fOrvZDCpXHiWUn49Q== X-Received: by 2002:aa7:8244:0:b0:710:d294:a2b with SMTP id d2e1a72fcca58-7143176773amr3032765b3a.8.1724310908168; Thu, 22 Aug 2024 00:15:08 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.15.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:15:07 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:29 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_001509_311404_BB5646EA X-CRM114-Status: GOOD ( 12.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_rw_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 15d3f7f3c65f2..799412041d262 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1723,6 +1723,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd = false; + pte_t *pte; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1758,11 +1759,25 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); + pte = pte_offset_map_rw_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml = pmd_lock(mm, pmd); - ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl != pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma