From patchwork Wed Sep 4 08:40:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A1C92CA0ED3 for ; Wed, 4 Sep 2024 09:06:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=cbfhF1aUsOURBqKQRsuJiZJvaG bj7qUspTuHGy44Fk97CtW1vXW54dfRMhuyzWhvP/eGYBrseaK2hKBkeDW/tddFA51NGf9NsEqrXAc Q9sxtzXyzraCKeSSuQ8Kof6ItZE6xlpVXJ6MuGoRvV03YYQiY+1enawbTSfxsLs6vsrJvEf+NZYRb ycNAfvy+coFxKM2rtN7lwbs/62tn1OyPhwmUpIOBSxz5UUCzahp0IfE0Zk1b7auxJECFlq4LzJSBU qSycMq8Bf5m/YW7Go0mTo6tsoEmS6Cd//prlRF53hgpWZT0vz2jGInlP5lZsHgv6Qn2HlB6yQvd7G +yO59TMA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllxm-00000003aTt-2LCC; Wed, 04 Sep 2024 09:06:06 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZL-00000003Su5-0NJK for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:40:52 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-2053f6b8201so31851675ad.2 for ; Wed, 04 Sep 2024 01:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439250; x=1726044050; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=X9UGxNucroqwoE+ooEJ37n8dsiJV4pyRlDGU0przED1PBsNT1ALF+YZ8/tFJHT2xGN QxajCmNrGLX7nFBieNL6N3KNloDgWpF27piYkQgmydo+DeyUMikZv0SZeiYQbRGUnNko 7VPp0LwcrMPM3z/6MeMJKsMdT98s/GyANYxHZ575uc2vwOk7UFdObYcf1SQUqchwYOoO A8mZM40J9r01LVBXsCYZuJ/OzeLSHF4s4fI596VBQaggPhOoQncQ+cvdVWbCX91mbceQ B6qaO1iCv96WtG6YsQI0OzWiwhkeWToH2QJmsV7dzVy2w3LUBjBj57d16q5ckioF6ZLv Wctw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439250; x=1726044050; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=ZtygsLlrrEHyyDgFWIUqZFQW2NaOxNMGflVvw2aQPAUiQJkx42mmc6yj1ESRvelG1J C+ThYtRXak0WibEtzWaK+KjO6VSW+0/xoWaKlH2/RPxlfMTQZeq+v9duz6ezS4KZg9in u+DlY8XyWFI7TcRnFcJC3EdsP/uGx+x99xdlNBFTjMPvGaGLy1Kagyyy4HK85CgvsGtL 4zhGt9T+QIytm2xvVbhYW0TIuC8WvAfymhuV4DVeVv/Hj+YFMRWMRO7/MxuW+RwF2Xb6 IQ4O/yuWJ3p33Q5ImZdKtvQGVpHStg4Ax8sAwbHqOALb2E0hJY6OSfAmfKQKoXQ1WUY2 Whzg== X-Forwarded-Encrypted: i=1; AJvYcCVQ7ev8UCoJSo51uvOvQZ/2iRt6/yoshGSpynkDP+ipemPBgk/yJxwVbILdkKm1fJbBF9hZSVKax61E/tBsIkku@lists.infradead.org X-Gm-Message-State: AOJu0YxhBCY7wImjHf4Gd/clqyGU5upTW+zJftgNV1K3U2h8TGXlTifr LsV2YkVBYpeFFKAUPHSfhHxKciXzn/x6wdrBark5H1Sh+G0qekLPYQHi3bHGcg4= X-Google-Smtp-Source: AGHT+IHLu4af/0RdmwIJrtep3I3PIUPy/4FB7EG8zsCDexMsa/uyvNrQYTXwy/3KiLtDhw4LWBklXg== X-Received: by 2002:a17:902:ce0b:b0:202:3dcf:8c38 with SMTP id d9443c01a7336-20546b35d3fmr190359795ad.44.1725439249775; Wed, 04 Sep 2024 01:40:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:40:49 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 01/14] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Wed, 4 Sep 2024 16:40:09 +0800 Message-Id: <20240904084022.32728-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014051_245772_606410EC X-CRM114-Status: GOOD ( 25.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 +++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 50 ++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index a7c74a840249a..1fde9242231c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3006,6 +3006,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..262b7065a5a2e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp - as + * pte_offset_map_lock() does, but in this case without locking it. This helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct spinlock + * pointer for the page table that it returns. Even after grabbing the spinlock, + * we might be looking either at a page table that is still mapped or one that + * was unmapped and is about to get freed. But for R/O access this is sufficient. + * So it is only applicable for read-only cases where any modification operations + * to the page table are not allowed even if the corresponding spinlock is held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pdmval. + * It is applicable for may-write cases where any modification operations to the + * page table may happen after the corresponding spinlock is held afterwards. + * But the users should make sure the page table is stable like checking pte_same() + * or checking pmd_same() by using the output pmdval before performing the write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Wed Sep 4 08:40:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BAF50CD37B6 for ; Wed, 4 Sep 2024 09:08:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=UsfSUldWwnJu9UkzyvcZ4pZ4ln dOnoAyft17fhIa5mcKS3ceQ8IA7vDRh5jHsMkq9chQM10gp68PB41zbGwnoBEaQ1ct7c9jnv9/BEH Q5umM7M8FAGI3nq+59ed9QZZExKr8B8JOnm0A62YrWVCm98dCEczHleqQZBIT0vV/Yu3fTktzzTKq J/62agxeB8oi4psN1FUwX1FcHYrrKCAOP8DQNVBUL6OGpEDcNVdO5ZE6/YDytSfo1Fy2v/Z9X/UIG Lccxel0bDsMx1kh34QdRR6A0vLRzSKZC0hKeqDKhw7/7ZKICpXmF2l9HwGzdfjSe+81dQo6FnqGl6 8HDpLqtQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllzf-00000003bAb-0AQb; Wed, 04 Sep 2024 09:08:03 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZR-00000003SyJ-2FJN for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:40:59 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2053f6b8201so31852285ad.2 for ; Wed, 04 Sep 2024 01:40:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439256; x=1726044056; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=Ire2maEjNqfS3vpM1R5Ly0ZjL6J19buFsGqBrCFmEYUfvc3gtWxcDaM+AnUKk+Vfah 30IQvFIYBC6ziEmbNOwuSjv3IVbcNYmxJKeuL7ZqWfZfILP3/Xztw1+vu4V46q8BNT8R 1KfTVKz9e4QfloJntV3gqMuF87Z9RDBAG1jm0PcHWI7NxxCeteLLLT5d0oNAwBuE7Xjs qYNFOzlFkVAR9Tpve8XLgbzLoSgCuB5VSZg2c/N6KtLGfruXywmoLM8mElcq/o3w61KB 0uucS0DC+Yo4+fP799ju80KwdGqohDNgTrOTziBauj2jm5Bnq3z95Ai6u46Ve28u3UE7 D5aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439256; x=1726044056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=FCnvSafTBCuPsQdDEopRM4LGpx4FGvg6q97OZLoZFaGgSKzxkB2taxY072gnfE4rDt Mgd92IMXdoXndJb26Nkm4Yg09/lbqwteHJ4NQnjPO7I6OHZE4bS9dcSPzioUm9ml9q9A jhMQIAksPW/AnutUjATfRBP3a18oB4Bmhij2n+B8z9+HQv1BQcYQLAFYlOeTiNlepA0h L/ZZO1w/aiRQUpAWWgMuJB0m9XoWYUlUV9U91XAO94whyzDN+SqCp7zU/kF2Lell9EWW zmLvUdWrx3IkFZ4KvIViAhI9CFbEK9Fvl0CoHDYc6rWnuMtc1nT8fZwN7oqDzkH1uux7 aYUQ== X-Forwarded-Encrypted: i=1; AJvYcCWrWfrcWM7JDGoGjgQsxAWY70ctUoAUSpfhEToBcG8UJFMdrH1JW7JRQmAy9GOwDlrNeW9gVWVEaWLS6ysjYveK@lists.infradead.org X-Gm-Message-State: AOJu0Ywjey2F4mVqUnHNsnSVW4OeNCgQ+FQyX8G5u0QysX7sh9sRfSYn Pi4nXX+ESCP+YyTVUw8HVW1+Zfca1/MeeUznlLxLXVL628s6UiedCvVu8AbHdAs= X-Google-Smtp-Source: AGHT+IEEdhBdCGQBBk6sF5V8bi+PhpxKTNotIZgwSj7kMgQxLZV/04loActQGvyTL5xDqZZl1Kbj6A== X-Received: by 2002:a17:902:f547:b0:206:b618:1d7f with SMTP id d9443c01a7336-206b61824c9mr14802495ad.37.1725439256381; Wed, 04 Sep 2024 01:40:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:40:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 02/14] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:10 +0800 Message-Id: <20240904084022.32728-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014057_692743_FC4E9990 X-CRM114-Status: GOOD ( 12.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..de6c7d8a2ddfc 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Wed Sep 4 08:40:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C971CD3431 for ; Wed, 4 Sep 2024 09:09:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=UdLrdy1dQBrC4Li1JPD0fRR0y7 WeiPjPMbhBzggHcpTvPWkH+QJcO56A5Zud9nMRTAE+4oGI8ks6WsRX7bBWMMVQUQWaUCVZuGmPZoT 9u3ulu39Oa+eUPf1ZYlumMhAfMMpwdNnqcz1axTi2Yxwvu6c/iv38hgPxY2tBB1N6lmD7QuwWfVgF YyyvTgmh/r6TwpmVZbCW+EcZwA5C/g035sA+m2Fo7Iqp4GdbRLTuSbIAhbEV6w5p4FR5LN8a3klbT L0+cEjgm6WbQArNLiul612ztyLXxy4GIcQIv0GvaTo6qkiGlGyjKsnWHO2nQvIwgirt0XhY7Ddcn4 llpJPdyA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm0a-00000003bQc-1jpN; Wed, 04 Sep 2024 09:09:00 +0000 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZY-00000003T3S-1Oox for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:05 +0000 Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-7cf5e179b68so344310a12.1 for ; Wed, 04 Sep 2024 01:41:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439263; x=1726044063; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=CM4E813Si1BJxiYTk/vShEWDvl/QCMletLVAxFkNWRUZLipJAwwXvkWBB3OgablKak E2R9r5G+2PUGUHQmn9LxgUf5CcV8qGZYnxT9kw3YiIbvmoZZeNCJO36twE64fsrz5iV4 1EE9DELcGSKbttW3MrpRcXZwllRXBs3UDZEGSEuBplGndnypG0wNRZZ45gVmxFGFZGis eYlG3nBwrA8M3TiG4ajGXxZlwn6FYMauZc8ZtljalbBGP8eUsSghZFpTemb5AqXuXQAi 7vVWhbL+rkBXO0oILdYJ3gdt0E99m1gYsmOMCmd33U/6iHLm9dHOJw3wzRK1sFGTW+i0 Wz3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439263; x=1726044063; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=v56v+A92RzS8z6VExQqg5yLXfiKqIKjQHBf8kxeTxcpA0sKeYmm1HgiFz44MD7ntay j4VuKRd1IIdForTg2wvm0h6Bxecy2c5nlMGompDX0M2u3C/M1Pya2vLu6I3axYNkGyaa j2pKjRym3jRN3CiIqQF+0I3SGIodZh1ZXUrkbBn22UsNn0Utx9/G9PWNigDzgJTzZByn JmTE3gRP9OYsZLSxsoRB+Zvg6tvOT6LoblnjLFzltcYIm1qj5b4LiT9CYc41so/6sL/f aJHOZaMwM4fsyRYGw/LywAwOhJ5UyRYHv0Ss6jyi78ZjeJp0pUqaimWENbcNv8azPzUR 5Tgg== X-Forwarded-Encrypted: i=1; AJvYcCW/caECyoADTnQv1ABl8LpOVFjqwjZQ1265lPJabdyCtalWI6j+XiEG/3xkXBeCjusYXBp26ZKSixNJSTU+1fou@lists.infradead.org X-Gm-Message-State: AOJu0YxBuQKpbu59u/fFsI7TONXWsFHVn4OP5PptAjyCUTurnW9CM8p5 cGpt2TkCgjw5prEL4eZ5h/Kow9taB6pDN7jdcigajyd0nDoG9pm2EbciHJ7sszs= X-Google-Smtp-Source: AGHT+IFosxGKKraD23O7QyExDWz973MicB0gfFUEF3j/RrDnY2sLd35727GFsppJnEl62/xkyI7W0A== X-Received: by 2002:a17:902:c412:b0:204:e471:8f06 with SMTP id d9443c01a7336-206b83566dcmr20294065ad.17.1725439263160; Wed, 04 Sep 2024 01:41:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 03/14] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:11 +0800 Message-Id: <20240904084022.32728-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014104_404824_864524FE X-CRM114-Status: GOOD ( 11.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Wed Sep 4 08:40:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1F13CCD3431 for ; Wed, 4 Sep 2024 09:10:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=FkcCTWJbPel+UuQeAu31ZFym2y nexVsZG1dwpPEfrD1jki4RV2RMZ7yLtwrVWP8aAxTxq+UnZEraBEBspe0PZwhB0RntCgYaRX5E7CT NcQhq3CbvefwMMoWNeilMucvpiJ9AgNv9VIPkjdKXu05NLWA9FhNeTBn6zQ66ysVat/X3SwNPYyk+ Io4l373HKAVUy9rgwrYhSncQfCEdBp1JCxCW73SdB79OV5OKVrJiu7hwBRVpfG0VfEGVfd5mmhvZR m0So8o925fzJgkbsJWzNcn1bJGk+vTHzSi363nSSJA02RrhPxM4s8DEkwx/cll9r3CHD/Cb6ZLyO4 BPDlFSEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm1V-00000003bgB-2KtQ; Wed, 04 Sep 2024 09:09:57 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZe-00000003T7S-0slP for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:11 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-206b9455460so3336475ad.0 for ; Wed, 04 Sep 2024 01:41:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439269; x=1726044069; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=iTe7RcUahah340lKzCQVm9utZVMkhDS5IWZYJ3PfgZp1W5Kt8a2BiAMV+npgIZAnNQ kXuVmVhnhze9Rq/BsN6M72b002dJHQ6J7SktajZZ98zP/5vWG3dT+T/r/LZKJxyyDKFO 93Jn6bgwqVXcgDsXeYsz8iC93yJXw1zUOujZQYYPuRnkwgoc6OiCR9+u1axTbF1iC4hj 8yt8YMZ/ba4gFT9fYhV5baoGKAFXJzX2CknCKNmcdkBQeMKRon2yUvZ3aEEf/L/2Fc+6 ADYjKCj4d1XXCeGMfA1GGr86YabwzdgO0Rsb7yJO+l0NeCdEQeocIdvVy3ZATDopXt78 MJqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439269; x=1726044069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=meAXHDMwx+1JgjTRursDH4UzSzyv0jR3CQ+XBlnSRWIVp409g/GjkSB8Ps9b5JSR9K YhkHNxjLVgppSfO0rWvwSTub4lHJRg0wY3MRsk3h4rTuqH+GlUFNnF67Ghn7ndrc3Vvi wP84xD1UqCGciv/fnly7JW9FLE0buUZsAVIe72S+WBDblSYAR5ZVktS9deheArlokydl VJbNdpFIVIWHY/MePLjj/5SIW7zXsnLLeY5PmNyOXbQiKzb/6cSRcdurrVcmblvsMWVA cIFfu0lkCfI83d/FSTbR5S9thIA1RLmbeKMT6un+BgzBdiUJOyW+MWrtyz/2PX1oXsWv kPAg== X-Forwarded-Encrypted: i=1; AJvYcCXeGP3f//LG6zVXH3rE4jtbOzHG4tymhp/TDyKYAQCEV362qsFPTjPpbk9ERgkwkMHrzqxXmlmNvvpkZEm3HlQV@lists.infradead.org X-Gm-Message-State: AOJu0YyitzjPoqDaW/du8Qp6p5FIQvDj8X0AKg9UQ/aIWLAiIz1P7/40 XfarJMJnOQ1v1knqljx0t8HsUvithDVJNlH9WErQrqlKke8LKZwF+zx0vb41m/M= X-Google-Smtp-Source: AGHT+IHVi0h+YpQWstt2F+C2N++FuREpmzHqlVlqHX+adh/zaJT2MbahvdDpXAnk4gZu2G3FJSfhgA== X-Received: by 2002:a17:902:f60b:b0:206:9dfb:3e9e with SMTP id d9443c01a7336-206b832146bmr25150715ad.10.1725439269514; Wed, 04 Sep 2024 01:41:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:09 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:12 +0800 Message-Id: <20240904084022.32728-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014110_348283_AA49FF2E X-CRM114-Status: GOOD ( 11.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 20346df53df3b..216405ba497ea 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3251,8 +3251,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Wed Sep 4 08:40:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 384EDCD3431 for ; Wed, 4 Sep 2024 09:11:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=tk95RNkkDehbkhamWLayl+Zcs7 nY+u2P0pjaAdS4TDbhn7fzycq3LuB2iwKmvR8VUkVYmrTN4Sa8TD/L32j0XWmZvHedGy9GFex+dXV M2386IMX32VNFuAB5wqvSV8GBNG9jVGk49+PAKL/nO6KtdjtEsHh1PSXo7IeOqChFNOHjE5hpbs8x MJUJ//N+EB/+37ZLAHP6bXq2lC9XK3WE02gyECuEthJF91C56I+STvhj3c7VYMnMWocFi2GlMH6rT hK3QAGP1Kv+CsHwK+vZ/x/R4dt3aJwZqFKtl8/eqZ1+chbTjWTbSVbTGyBdzLzHt+TfH0G+VkmoNT CJB1Sa6g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm2Q-00000003bwj-0Z1H; Wed, 04 Sep 2024 09:10:54 +0000 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZk-00000003TBd-3vOy for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:18 +0000 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-20573eb852aso3453425ad.1 for ; Wed, 04 Sep 2024 01:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439276; x=1726044076; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=JG4ABvCFwlyto3OQfNEfuh7r1skbu1Lx5vFKpxfNV0MZPj5KQK4MRNY8bjUp958Jwo K+/3S/qgD9H6gMpL3FMvNrkGBDIUCG6UIYeu799vfSh3syVevlq4s0Kf+YhWkC4knQ+u qEFl9qMPk1pBr+NLRgAY1PbbWtDPDoclK4t4fvmQrbKuUneZhgbQAxQQEO3yNzFqI+7u /ZEZnbYEvaH4Xz7/xskux82pG2r8/UZKdtiPkAg9e7Dx82044O4wCX52Tb2G7Y0VaHQo ILVWAw+KhXM0xzhO1SYrKrVLOwhq+g0kfPbfUJ4PekCHDgH9RWCEF3HpBAhoIoiGjpim mROA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439276; x=1726044076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=DG3MOfXuJTHwsDZPvCyL/VurQWJHOi537Mde86dDd8dA86xeEo3FCr+L4GV1zVCt2d fLrctLuf+2L28inPOxbDcAaAp5V3jrxVNS6gwBDshbtzcNvSqTQA/CM+HslmeYwvAZSa Y+TdJD4nrfdFeTja1E93g2qIBEV+bzhpJO3LFCZdjlKqJMxj6d/XvnjSNKojeX5lUKPI 3l4qRXj9aMjMt6kQ9FDEJ2+0YMTBzxCf6S2lElNiSE8EIMoztutO/7/EAc/MQRXZYJwA hOf918HM8JkoDKJXwvhlAuTJUx9cuKKkAAWW7oejnoppQbgGDqQQwbAUfNAok67PHL2O 1vbw== X-Forwarded-Encrypted: i=1; AJvYcCVIVQzrknYPZ9l4OWb4cWQNNwqT2snVmCrjRLWqMyTPDxB/v+Fmjd60g2gpbaZ2wXbsD/PaHFxe0Y3YzMf7SUui@lists.infradead.org X-Gm-Message-State: AOJu0YwD/pcvaMJOeoG88s3tJoGbBkSRHN6cs2iMo7ijNXVApU+7mftN LqofL4v/3eZ3SnsOQrshbUeQRWk7jPx8ODOK/uilhBzdc7U2Vet5E5Xe1WOS6PE= X-Google-Smtp-Source: AGHT+IEy+qEnEqpj3sReiKH3tPFUkwjqR7spuVkWXeQJodyHWsfCM0cenE/z+pCdCppofQTZA+ekKQ== X-Received: by 2002:a17:903:283:b0:205:76c1:3742 with SMTP id d9443c01a7336-206b7d00d3amr22901025ad.3.1725439276249; Wed, 04 Sep 2024 01:41:16 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:15 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:13 +0800 Message-Id: <20240904084022.32728-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014116_999198_FF286B19 X-CRM114-Status: GOOD ( 12.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Wed Sep 4 08:40:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FA98CA0ED3 for ; Wed, 4 Sep 2024 09:12:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=UtISB/zQOwPZrSOpdEVX5ukThZ 2zGamhQUyEmWdE6Pjcg7MtOSrkLJz2G24ZJO2siXPktZ3o6+bMGNStkEnx2BbDCTdGZCh0qRytFCL SrxE2o7gDqD9yxDHyU+SolR6CQxOUSaxrXvl9mlhHDrGy+nuI9Oon+P55vqHO0PJHkujTZEAYlS8m p0L+QCD64kuc42P5vWWA7VRHCkTnHG1sIZOJyWA7oPINEw5g59Pv8fpl/xi7nesyV74HP25/DNtAW eUnNjszotVkYskwiDe7PXIZuuK176WacCGgKGtcR4szDBLk0mpujxWKdjHseEKQ9+NhjyMWkWMkll fSU3wgKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm3K-00000003cCP-2rAN; Wed, 04 Sep 2024 09:11:50 +0000 Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZr-00000003TFO-1Ccd for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:25 +0000 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-7d4c923a3d7so1410727a12.3 for ; Wed, 04 Sep 2024 01:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439282; x=1726044082; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=GDOjZ8XQ24safzyCsCDbpJEM6N6MtSg/m/8iM8f1KWf+ScV4Y28hbdVP68OdxsSAtq 4uSuTEVzrOECAXp/DxTkUwiL32CgLcEv2NpBvA7UpqPAUyKUhcWYx39Soc1CLy0+nn4u uEP6lRK/t9cyvKs0W5fnRrTS/atAqnlRbpRplBZ9us6JAj9+ExsxgZh59Cy9N+f1sfLz r2jCy6KoGIe1Dnw9jVKtZq7F7htcFgWVDqaNWx1aui/QSqnR+xaHXvq69hinR9xSZWox qEmmmOpgRDAA+jIpzZPw0jnsc+saJ2VyMDqBIGw7a6P3Ql5hW2WBGDkClKJH7BQZJUko whBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439282; x=1726044082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=itTFVm19o1bLLIayziXZSLhRDTu/Mr1rkP3se8CNFAQoBHfok/+TeUYUkZ/NkjFVTD udygtA5yku+hJpxiBO3mbtb+OC3GbkFAUzCcb9FE2mgrovP0SxucEdkOQ+0sc1r8hitm vAAVUJQrNAkpathsJGS8X6cQDmoePcss21aF65plMEsIKLV2ZQSx2Vqel7fYYzYWDf+W fdLVoYjgYiAaRRSEoDN3VKrPuKdho80+14PV4kx7qSp/9OlcozNr0TCRaI6oCTn6ppMe aiW/ZAvIFVvYbQ3HngOCS19cdDCSxGIvN7dhZc6BmqkTYlYUUgAIl14QSLD0bGLjfPNO NbsA== X-Forwarded-Encrypted: i=1; AJvYcCWwx/ZiqliWsUZWJxBqXj5X3ABIhhS2TOnwTU+eOtO30Y9awAdAl+qxXLKoexq7vhMPDq7pEDEj5Z4OOLC8Prpe@lists.infradead.org X-Gm-Message-State: AOJu0YwQKjLVVS5DiwC7ZL8vRivOMpGKtpCNqVq4R7/xdTqAHYp3bpRS rEsgVrzTgww4XE3qss+JrD9qWXkiUI7ZQBFQlvckA8PmGb/XHFXruyritVtiing= X-Google-Smtp-Source: AGHT+IGQLG6wPNgDOgfzFUGIly89eF+GLm9izjxQGbRYP1g8ZSKlFJK5pYdbKQOnPMpn9+eqCP/7EQ== X-Received: by 2002:a05:6a21:2d86:b0:1c8:b65b:3db8 with SMTP id adf61e73a8af0-1ced607378emr11775108637.9.1725439282360; Wed, 04 Sep 2024 01:41:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 06/14] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:14 +0800 Message-Id: <20240904084022.32728-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014123_364293_1FCA6145 X-CRM114-Status: GOOD ( 14.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 42674c0748cba..06674f94b7a4e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5498,14 +5498,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Wed Sep 4 08:40:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 39881CA0ED3 for ; Wed, 4 Sep 2024 09:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=soDUIL7d+sczPYDo5mUDt2jThT 7b+A3dsW1Uf0/04vyCitC7l+Un77BAD6W7JZeTNn4JbTsbew+bBSPX2ZKf3/eKTe03BZi6Px/6nH3 ZL3T9of3548u5Ec6KX6KZlaG628LSqTQXwlcSQI4jfwRLEomyus5erMSVXtB+22D6cGvqaxI7Bnuj 9xbRmX/X+fljQPJdqFPM/vxusjqWDm8K/AsQegW2ci27yvpVtqfpjVxwUM7zCpPlpCE4U8SfgDkir 5NvvrXFFqQzYjJIrrnzQuwXwXlizFy0zHNo/N66foNHX9hs2zuwa0UhA/PyR1pRZY/nAz1tIaiC1i ZrifDCcg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm4G-00000003cSm-1Nu9; Wed, 04 Sep 2024 09:12:48 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllZx-00000003TJ8-3VXz for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:31 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-714114be925so5098900b3a.2 for ; Wed, 04 Sep 2024 01:41:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439289; x=1726044089; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=A5EDDJxNGJMdft6ZPp1Phuo+iQOJP14QQrRa9LMmYDR0s1mJOXpJdCd0C39E8oMJOX KMl3rIijwMmcL2GnNFQm5hcSpD2GEdXPCCkGKyGOb7DUIm07MTNLKW3Kf2B72eQHBee7 0AmLI2+IHTybhraGc4/fycv8Da6qGFKTHtlA4D5N/yW/HnUrEqsNAE0DQZQ/+k2M7M92 u/LuB4zf8OZL9nXrCrLKn0bTDutAI4VCef96SwzCihVhSlWmWezfrouANaDrXHSx/4MH oRRj7u4sjNlZsth7X1rKuKIebxJbiTvt9uc3hK4E4R4r689mS/WU7P9EUw6dEGQ/51IU vCHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439289; x=1726044089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=uuT+QijDfzQAgRu/SuvJ6Uxb3Rbjm/j1FPFrhixEM1ukyPnS0LTWf/4yP4YlIoVrsg JUs/HZD7WQ9mbox+ZJgmjUSXkB99DwJQ7fOrwdRAw44Ee8d/M+h7HEjy5X46TT7ED7U4 xT4fSMDYPjbJNYvznyuGSanFNICXcCk1hAXWnnAuX79qtizpjhuGNw5E7OSKzbjxiXcj beUzSzTTTqQ0nD1V2fCu4HHNaYIusLcDi+FNYu1Jf4wGN3BMgxzXeuZ6ET1UkG3eDjhU Z+5PJypFlc4FzbT6Op3wFgMQlRLTnhwRfP0p76RQsSm0GKX87qnHhWOiQV9/M1AZ3RU1 w5VQ== X-Forwarded-Encrypted: i=1; AJvYcCVaOAxBc/GcIzrJL/bq+2+Yqup2aVVF/a8AfM1zNwOFGo8r0nwVl78sOwt3HuGD8m3zOsCxqNZY0yI3E0DPNJy8@lists.infradead.org X-Gm-Message-State: AOJu0YwN8bxTzo7kEJ7O9jcaqyFSgpAUHBPL+YiAQEhQdOisV09rFQZB hCcaUsedbZ5/VoHTCrzRDWI3kKSSFaW7x+GMYJ1tDB+HoPBGy7uZD7+U+2gRl5g= X-Google-Smtp-Source: AGHT+IHVhSodGEWLJbqfntGARXqFTBke8IrjtWuvc6P4tKsUnW/m6YB1esVTiG0+MVp8pwa1UVDERA== X-Received: by 2002:a05:6a21:330b:b0:1c6:edfb:431f with SMTP id adf61e73a8af0-1cecf788518mr14875554637.44.1725439288977; Wed, 04 Sep 2024 01:41:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:28 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:15 +0800 Message-Id: <20240904084022.32728-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014129_886969_22B1AB2F X-CRM114-Status: GOOD ( 15.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..a117d35f33aee 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1658,6 +1661,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1689,6 +1702,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: From patchwork Wed Sep 4 08:40:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5940BCD37B6 for ; Wed, 4 Sep 2024 09:13:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=XujGvwIIIYTNs7iw4Qtf9Hyov/ 08F/BxjmVLG5DAZl+sFXxLcZJYykwh8RE3j9sYZPfkdjKfyh5XjKUGmGtG+Rfi1A8QXoXqnT27h/H P+RzNHeDmHzqMGHO/2wP81dGsvM4tOCtTy6oARZ4jA58RsITXpsMlw5FUcp9cKqoEnJxev+utEAjE FrujwZajCSxeMSIdDu6xM29GybWNxevOjEcqUXGHr2N2MSBgH5qUzZTyHSqkLXhNx7PCrVgR8MtX1 HU/U0QakYK6FU9JG7PNftXdRe1Fxrsx62C12Pxbhf1DiLlwqetIKsv7QIC4eaO297TtOxGXC1Yp3U yzNCG29Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm5A-00000003chs-3Pfk; Wed, 04 Sep 2024 09:13:44 +0000 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1slla4-00000003TNP-30rO for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:38 +0000 Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-7d4ed6158bcso1440728a12.1 for ; Wed, 04 Sep 2024 01:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439296; x=1726044096; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=HV8x1aZ6/h3zsoAhWqlbBs+RH+lq61zFbEaVi8AfNk0JtLaG9QsZ5TcesjOzgq8SEn 0xwAfNiiRhvXAJUiFKCv1IdT0bGtXOQSshjrxgI1Xd8QeB3/yss/+EEUHwZyUMxouUxU 8FUUCN9UvY+xiBZllsWqbIEJRoEY2BfS33Bs7Ea/Cqqs/cHA0o5hmLlDV1oj1ww4BHqr 67dRaTqz/jnk9/06GPjt6ZS0opiYhkIyy0v0iqT1J29UDUcG4Lzq+H8m4MBi4P+gU3SV MmRfvg37gzKwxUVivt1sv5PNNEV3zDM1vx/n78PJg/8ZJpY6poHVOT6zCKp1C6/qEpHD 0cTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439296; x=1726044096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=G2IAVM1FXHgjMDxZOze5J80kQRazJDyMlLUcM4C8y5kxUZM4Xi85FagIpJqJfYrmiS AvMKrnpKSGPXEQFcxGp0tHgWF/cnbbMZQco+H7KpM4f50ycjcNvbEFTYdRvZMKiImTdI zN0JK8oAScyNQctXAKoLIW/vC8ZaDgYX8lIuNaSgmc/t9nC2czfso/4AVu6bYhqOHyH6 u9BzB4JmMawhO0x9NvgKRDFUFIWcZ/rKWVWz8UpDJx0d28csqNBF7QqKNimncZFpM1ts zpMF4rmtpwau3UMl3YR2knJX1pGDpwUC0U2es/PRDec+eF8vukYURK0eUhnLtZ6fBPiI 9s3A== X-Forwarded-Encrypted: i=1; AJvYcCWP5krHprIN/uB31wrxGM2JoxiLVkM1xUxZ3FYv7h4cMQCRUQ38FGyDLEAK8jXCUFf9w9CBR9wq9ryA1o9boziK@lists.infradead.org X-Gm-Message-State: AOJu0Yw+W/7z5u/vm4R1pf9xgOy/Qg/s5QHyXtAhMXhcjHkHl3SceUHS jHGfTq4xOy8+FAbEKNVwm7L1UoUvsjsaIwY+zxj5gu8v54ol+Jh4LgKGEmCJZSY= X-Google-Smtp-Source: AGHT+IHnX4UPdOrG4UbMFan2fo99NWoUyUAqXexLL4zg80mEgnx1zTpF+DDeMcB5gD3OIXAOIo10hg== X-Received: by 2002:a05:6a20:b40a:b0:1ce:f77a:67d4 with SMTP id adf61e73a8af0-1cef77a69e6mr6594806637.6.1725439295802; Wed, 04 Sep 2024 01:41:35 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:35 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 08/14] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:16 +0800 Message-Id: <20240904084022.32728-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014136_777777_5E45DC19 X-CRM114-Status: GOOD ( 13.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we may free the PTE page in retract_page_tables() without holding the read lock of mmap_lock, so we still need to get pmdval and do pmd_same() check after the ptl is held. Signed-off-by: Qi Zheng --- Hi Muchun, since the code has changed, I dropped your Reviewed-by tag here. mm/memory.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 06674f94b7a4e..47974cc4bd7f2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1082,6 +1082,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1107,13 +1108,28 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * Since we may free the PTE page in retract_page_tables() without + * holding the read lock of mmap_lock, so we still need to do a + * pmd_same() check after holding the PTL. + */ + src_pte = pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ goto out; } spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(src_pmd)))) { + pte_unmap_unlock(src_pte, src_ptl); + pte_unmap_unlock(dst_pte, dst_ptl); + /* ret == 0 */ + goto out; + } + orig_src_pte = src_pte; orig_dst_pte = dst_pte; arch_enter_lazy_mmu_mode(); From patchwork Wed Sep 4 08:40:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ADC5FCD3431 for ; Wed, 4 Sep 2024 09:14:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=PueaqdfeMRBivRflD69kSFKQsw U4O9gch6tBDG3zt9UnMCGlp4caKQ+4elgHSs5xlUlQWKjQ7Vh4xPdn8j7KrzTT6DnbwUuBS5gYH+e DlX8I7ZuFYPdJn1Z8zUc9QzdfU11B7f7yb4FHL42XenjRGRcxlfLO7dt4Ulr3NNV8mPIyxthM7k5o 1bq+d9CSBcQxCsOhtZMP81Mdy3lds43bUEjbXWcLpDm/kY0+dUh98rNYz5dZZbvY7I2UHTEv0XgvS pA8bniWZjPJqJ2Zq7xmok1rFzMUqKqMwUFBU9zsbs6aNzrd/4wVURQpNBCPlSEluz5fJ+Jzr6ldfH G1eD25WQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm65-00000003cxF-1UNi; Wed, 04 Sep 2024 09:14:41 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllaB-00000003TSR-23cB for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:45 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-20536dcc6e9so30417805ad.2 for ; Wed, 04 Sep 2024 01:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439302; x=1726044102; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=ZuEtgJwfuNqKb65wO2tWX2bqPzR8IZyNtqx3LObsIE4uwnm2hntUwVztj8XE5vwwQk iw/V27leT6/rXNC0mjTMkshARr2WiqwsZoFrpsyHuRqWoZxveFmvRlVOrXiHUqoF+jYZ Sza7kARpykIjyXlnny6KwEGpICIM2chCSfyHDWnwXr590LsgFQfDmqyiAJOc7Ds5Nd8h lrb8JqFCJjCaevb3fn7VXMvVrJVZF2aHwmesTjx145a4L4hTtEhBPLd7S+/c/x4CRdZ5 XqdvbSm4705GFdS3PHPwOr9tziZsXtjouxBfZXmNCFUTjrbt3tRjeg4bqSdY3ogImlJl 5z2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439302; x=1726044102; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=R5E4qlefr0j7WU4JqqyaavfhfZyFZ7Vt9dKHSKBOqcmq2Gu4KiKR9BUu+wqwBoHvTl cY1HnciGXNCZTgy9PKTJtg0WGbnwc4EODzH+MjyHbZ3l/6Y/GJy9/wn3TgAdQHbH1Dn4 vVXJvuOBTK4UAfXROdQBQ66DdTmgPPh00ctYC9cIhKizI50r+yolxXif3aM+I4yNnrCA 9Tqah7Cl4Befk6BO9PPTv9SZH8e4Pqdb0iqM2BuWX1w2E7pJVfTkwVyVn4h0oMWhFRmX 0UCaVBFglwueMYfQzIy+qhzHKpeesU+ubRt+vN8I/WWJV8snteFAJFpuZnYkvii81q9Y xuXA== X-Forwarded-Encrypted: i=1; AJvYcCUFMJoIJvoRR6Zj5Qe7icF5RK1YodxhZJa6W+DVS/LpxEpYEhuOsqljv09PMjQjxM44b/eEzaDnAo4B1ZCIbqX4@lists.infradead.org X-Gm-Message-State: AOJu0YyC5ZUXsS3rSijIltU6acQdph4TjzKl+muTgMqih9hs5PNqefcj ssMcuAhG9SdgWDksFr8RYe8E27EJtKDTryfQHPPlPq9zJv/ES4zAhYcgD4lXkIs= X-Google-Smtp-Source: AGHT+IGlDE+bZkJMe3LtnR3dMefm4RFnOsn/qpiRzdld3O/wCPCkO6r/LjjSl6iGWtDIKmNPthI/eg== X-Received: by 2002:a17:902:b68b:b0:202:4a24:ee with SMTP id d9443c01a7336-2050c45e710mr147539585ad.55.1725439302519; Wed, 04 Sep 2024 01:41:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:42 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 09/14] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:17 +0800 Message-Id: <20240904084022.32728-10-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014143_534173_AFEAC0F2 X-CRM114-Status: GOOD ( 14.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we may free the PTE page in retract_page_tables() without holding the read lock of mmap_lock, so we still need to do a pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng --- mm/mremap.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..16e54151395ad 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + pmd_t pmdval; int err = 0; /* @@ -175,14 +176,29 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Since we may free the PTE page in retract_page_tables() without + * holding the read lock of mmap_lock, so we still need to do a + * pmd_same() check after holding the PTL. + */ + new_pte = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; goto out; } - if (new_ptl != old_ptl) + if (new_ptl != old_ptl) { spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(new_pmd)))) { + pte_unmap_unlock(new_pte, new_ptl); + pte_unmap_unlock(old_pte, old_ptl); + err = -EAGAIN; + goto out; + } + } + flush_tlb_batched_pending(vma->vm_mm); arch_enter_lazy_mmu_mode(); From patchwork Wed Sep 4 08:40:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A5E3CA0ED3 for ; Wed, 4 Sep 2024 09:15:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=0sjhqII0PJUqv1umtQHfoGGydQ 6cfQErxIO0fEtmRwmA5r46KPRLjK/Dl8uNLtcoMLgOZWNwXLbv4PfSxD068Zv5fYA9RtikoCqFpBD KEAOGI+WqhCgNraMX7Z3oSQ9o9DLfZ6YxO6g5IhWR0ickDGQIXtdWX53WYjg2i/fvWV853F1PNCUk zrURJml6+vidOCVsXBrppuUChGoQglaZzwNBg+whGY2mDT1JGQuv108qLv053ZZ0j2KSs3oI5CtYu R+t85ppH6G5ElRmQ9NCvIK+Jxm0A65za3qGMnMiHYp1pUBlLxaxHICqORKzLrP9H6GUlW2M2rSpDQ gMvpOIOQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm6y-00000003d4y-1FZT; Wed, 04 Sep 2024 09:15:36 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllaH-00000003TW1-3o9g for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:51 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2055a3f80a4so23925995ad.2 for ; Wed, 04 Sep 2024 01:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439309; x=1726044109; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=dKuFGnePNJ63n3Dqef77nssW2ShbkJs38yE9GBNGyXROw+AFxQkaPBrXE68uK6HGne SDdWMHgAIKuYtrTNJIQmMHBq8ccSEuMZLgVXeS71P4ERK/WhRC80jFOoTLxtaHJ2XYH+ 0Rz8+4KqmFlihKt6tS4ZLAAw/KVjBdHEpHuzzeu3I7cwWcheLEG2OWc59L1tGiSZ7O3R fepDHj34xPzIYh+Hm4i1WL0aCtK/IRp6APgz2kk4iTxr/QrS0Wh9sn5qYyhFaMOIbmLs SG77uZ6Xw2O4vz2eMWqvq1cOwojCezDGLOY96X9M1Y5bHy1TAHzoBcyAU7gFTwEsLagj 38cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439309; x=1726044109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=VUeMNtmdjCQfutadMohSy2qz4GScj6xDemEvO+ctqEnJ2ZzJBYstkQcAkGjc5Mx8bX f52ETy/kvX93XuYQD5CmWkrLP553TUDEMME68R+T3GBgY+6YZtr0KThN8S6o+GrMFgfo 9lSr51z4vY+IlPHku87LmQ/0l65dIjvhC2ODJ5er+AIz3Gd4MUFSio2syoxXC8DBJ/oR SoCPr1zvJ7vh3v3QAcV3ZjXVc6feE62NbxGP7Y4NLm/2XIE1thqi+6TpnL7uQGBwqxe1 3nH0zq6D0TJwa+LYOWZbboUb8iTPsYZ2aKNygmEwjuOKuvgSXr6thz/WpsXR5vBl2Jql +Iaw== X-Forwarded-Encrypted: i=1; AJvYcCUxUiKeje37otKlXpCW4+9h8ztCCB3H5M3JCO0f3GWpNCFZvW0Vgo/h0Jb6sspGn5asgCh5mR1aq1ffZsHfvKnc@lists.infradead.org X-Gm-Message-State: AOJu0YwLYw08QFG+aUXhJteU1aK/0N2QwqjskH8FlzaX97XO+siY9n9E DLgKCju7Gx6Kr4SFsJTtgPH5m6EOrFhpx7igC2a+18/r3Y9JLnrM/63eQ8jJXiSpuWktWx9whjC j X-Google-Smtp-Source: AGHT+IErn+gfx0jNqN8oYReDzi5ALFGdG/LuObb3FXvUpU+trUkKEzs/YdM2MbdzLHq+FDgFhXw6+A== X-Received: by 2002:a17:902:d50a:b0:205:6c25:fe12 with SMTP id d9443c01a7336-2056c25ffb5mr96453155ad.34.1725439309345; Wed, 04 Sep 2024 01:41:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:49 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:18 +0800 Message-Id: <20240904084022.32728-11-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014149_984490_57B006CB X-CRM114-Status: GOOD ( 14.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..f1d73fd448708 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Wed Sep 4 08:40:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1900CCA0ED3 for ; Wed, 4 Sep 2024 09:16:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=kPSL0GblNksUxTj3iCc99aLowx W2B/Im0ZkFVyf0KHWUppBtuePURaYEr4vRg6H7bpmSMglw/eMhK1HQvVh4PgUuBR4PnosIMD+3vCs omn5MMnyr0sN/TD76PTXTc5TTvR37kSVEDyCoORqNRx4HZpjgygFSEJsvqs9vAhaaRfBmZ6CGupHl cgeiwTmNnmfNJmpT5dUdAIyYqH2pJClyLvAwcqIoKuO1AmUepqcp0aPkUaSxLV/eGFbyaQH7UCRKg As6JqvOylvPN1wcqCIyh2ELXcZ1RlHyU1v/6Dk32Z4eFUYQi91LulHTm3pO81zMkAWOXrpI5GlKwA GX1oKmqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm7r-00000003dAB-2PJs; Wed, 04 Sep 2024 09:16:31 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllaO-00000003Tav-3Kau for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:41:58 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-206aee4073cso6775615ad.1 for ; Wed, 04 Sep 2024 01:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439316; x=1726044116; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=bIHL2YlbPA3LTXS69CWY1EVqyv/2XxTO7VXqbv4BpAI6Xvtmfonh820fXTv/N4ARX2 6PDH5bL3xcQSM+wtx4SpLncX7k6kPFqPWXMPN3hRJFX+1yXPBJLv8fUMSO/RVt7aXkuL cSdDTzkHVpoPW1m25TjPHE69zzzytzlkXbSexUi7bXX+RrMgzxy/sqY1ZsfUHBo8WrWk ZptpumULWuho/eMgZeaGKB2Cs3UBmXGZgyhmKh/fjJZ6s5Yy06ZA7JISYyVqwvK7wPma mE4mU7n06D9Z3YO2CPV6c3ssPrwH+mfiJRsOBhFM4/ZwWVD0Mz+R6FS5fy2NA5ZsTifm jM2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439316; x=1726044116; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=aj9WKprkmWOZOgUyy1u+G+QD/xvg0eCwf45/fRUn+zk9NQ+XJv54y5JurGj/FTYBTq 9oiX/TlAyr4iNGAu51JU5vQlm17FykhnY4UPbMQhEFSSuVIsta8NnjZc+JYF4LnnnQv+ 2iHBeu4qG2EgRsCD80GEY5PuwmS6Eh8LfWj1wzciktbyU8xgNEgRtrthGzTET4l9w3Ig X+i6co3FkJyAIJE0sF3HfJJX7RJ95YlJesnxScq8+8Ut5ObehjSGu3aoEELCcNNUiti0 6xPguHb0xjk5TkyxJhP3MlsGWcIjMeSeNNo5rmaGwTQaHGw64rq9wFTYMkRVwPNZpfbk iMQQ== X-Forwarded-Encrypted: i=1; AJvYcCUKO24y9ivr0i9yoUH9AndLcmaK3InDoJX8eUK5YBeRDewL/x6U4nQeZj20EJZJCQ/CMdNYyD0tmDd/VO+TfBHS@lists.infradead.org X-Gm-Message-State: AOJu0YzNbdgs5BlKTo8o5WiyNKsQGq2s7VYjoMG7qXrL3ajFtezjdDt5 yv/l+OBa9uK0WatieMoXn+gIHIxnaNbqokRJAlaQ000v0CWDI/pLLmfGiUC/+Ao= X-Google-Smtp-Source: AGHT+IHUb0AH81Bf+t9sl3x8o3nWMR3fxDcIYnthyKOt2Ee/pQCOOrlcmsn4X1RsHo/HQ+Mvmo6mhg== X-Received: by 2002:a17:902:f94c:b0:205:4273:7d65 with SMTP id d9443c01a7336-20546131866mr119665875ad.21.1725439316150; Wed, 04 Sep 2024 01:41:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:55 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:19 +0800 Message-Id: <20240904084022.32728-12-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014156_897624_9DF50E38 X-CRM114-Status: GOOD ( 12.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte = NULL; pte_t *dst_pte = NULL; - + pmd_t dummy_pmdval; struct folio *src_folio = NULL; struct anon_vma *src_anon_vma = NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Wed Sep 4 08:40:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6003CA0ED3 for ; Wed, 4 Sep 2024 09:17:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=NgsRsOqm0OJj+hzinsWZL/7KBv TKpnsPeufrrlKNt6KVRdn1ZntXXoFr+tspUtNtLlTdg6KAI5rgnsOKFuL3IZ6xNJhq9DlcXM64aIH UDgqwcMRRbk1bjHRPtdxI/Eg7DOVpqrpMb3KZtHfyHHzXW7RV+DPqe/NeVrmZ4jILXA2BB3roq4N3 GsNzn8HT7g83JQ2FkF43/VIpYVX6d4beMxDlq/OxN+CnBVaLhowUF6A7QazDgHNmPJ6AsVUSY6vBl AE7VsqoBA7uNhCA/pGS4oQ4k19GbQtJ29hg2U3hetr1jJe+1M5eZYJ4dgk4pnfObXqLX0RbyHp0q9 tytaYoyg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm8m-00000003dMc-1iSd; Wed, 04 Sep 2024 09:17:28 +0000 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllaW-00000003Teq-0FtS for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:42:05 +0000 Received: by mail-pg1-x52f.google.com with SMTP id 41be03b00d2f7-7cf5e179b68so344694a12.1 for ; Wed, 04 Sep 2024 01:42:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439323; x=1726044123; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=alc82p1Dm0Gzx50K96eMxuHQcrZncugzu/o4jGHKMEZkHsULfVj8pEdR9jsCW3GVh7 SrYUB5JsEDRD9xsuj6th+2woHyOGIjTCb5VcYzneX4N9UpTdhE/JvKFSuoSCdPND7K6Q lldcdlCWEwWRxVMe/rGfO/WDJg9fHnM3Lu6zqh4AUjcya/Xgd3YPS9+1Vu7dsNRxUwAr Ytx1gDMurXySvir6M7IRcuAR3fpfJKZRzEngGUevYMmLqtkm27qLZZk/0kGr4aVYTv8p bvR4+6IEqp+pMi3H/VGfEbHYd6EFaXWaAMns1Z572+ybG51BeG8tIYHv7v6cnA6/Scyk eHgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439323; x=1726044123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=Z1mZUXowW1subpXURVMwqtX8Vr80FdxsERzCmTF33n/PDIj+omIu2p9cWZ9D6gVlNq Rc71XKRC8VOsbjU0GKsCbv7b5gmQjB3/zkpXbdwrYtR1zbdnNsWPb78Spn3JQmFxTjF4 ZDEpDGQORf4BdwVZpOIXqHcAeBcp8pP+OszEOfXfUfKAgLZaceP4uPhTTCDjA3ZZM5DA jIVEPF/w9areEnd+bSihJcbVPcYKy5v8PhIigHzf7ifc+Y9b0cBWrk+/agLatmRpadBb MtiCX/jJ5yAMm0rwueFhqENHAXXgnuEz5ACCs1G/sVs4hrT6MdgO2Jn6xkjdxO+hjyIF 40ug== X-Forwarded-Encrypted: i=1; AJvYcCU9auNFSfh4L44MpDhHQSOPoX4AS2XBiRZY+qGmF+0K3dGlM6S5Ak+1xTV+54uAkh2JBoeGPC6jdiIdxPShTXHu@lists.infradead.org X-Gm-Message-State: AOJu0YxTv2biyWA5Kzawz4zrae8vT0kTO6BB6ALP2pcTzN7Pkt8RDrYt FSvonNwVQq8PVCThyiYXpvlrDF/W0RA4/xd9UB/doySFy68B9CD02VGVABI7ejc= X-Google-Smtp-Source: AGHT+IGXU/qy5PPe2L4ehUmGkYFpO+LVdbh/09mdfTo5RpVJKjaH3T6TEVRYBI0feUcEJkjAodGgeA== X-Received: by 2002:a17:902:e545:b0:1fd:6ca4:f987 with SMTP id d9443c01a7336-206b8341fdfmr24676655ad.15.1725439323059; Wed, 04 Sep 2024 01:42:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:20 +0800 Message-Id: <20240904084022.32728-13-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014204_190909_B9347F65 X-CRM114-Status: GOOD ( 12.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9b6a8196f958..36b84e46cd7b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Wed Sep 4 08:40:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5DBF0CA0ED3 for ; Wed, 4 Sep 2024 09:18:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=Q9ttmcJ+6lS7eFmcRYVGbG8DhS S90EU7MfiWAkH8MTyrret/ltVwJ3R+NRo3nDPDqlmO84NnAkEHpz2DdlKYcjh2Isj4XId2IxQ1Xlw XbpT4w/Z1Hh4EyX4Jlx4hOp5U/HsDwy0DNMQBcHiCcJBcWPDxbVvpda9LV4s46kZbcXAKRNG+e+He SlEzZBHye3sIGboFo1wZUvbjTkcWO8ixUS02gypVs6KWBKTievAqbsBF1XgneLfDXqsK7lG+lix0k WnOd4gQsSt7WO0BPlEaambI5rgSPsZ/Sl/fAsamh/4Xb+lu2HQ1iJ79x6Q0JQaEPT/WE0mTWBOVF7 zvc/ahEA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slm9f-00000003dSw-2CNl; Wed, 04 Sep 2024 09:18:23 +0000 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllac-00000003Tjj-2BCN for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:42:12 +0000 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-2021c08b95cso4030265ad.0 for ; Wed, 04 Sep 2024 01:42:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439330; x=1726044130; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=k5GPpuBtSvFTaR8U7eSkpHqu1s+pa32VqOKLIiXdf7kqL8DhqmCErcUs2fBHnqSj8/ d50mthPa9eiwINZtzKXlz3nqHv7W3yQ98eTwOwdwMlLWGfxPm4gP5H2jGpOTpir0z/rf q03dD3d0JgzqgvlMkh0RJqpfCuWEbsivV6SCZNf3QQMQAKLoXVI9kViXWd8V6I67Qn4O yJgcFEp++s/Uw46cHyu3TZrDhwRl7mqR5YBvsqqz6z3DQROas1+bWSR//u4FaZIMAZEV zk+c/wex3wgD+Qm6Okx4cZBzv02I3tBfu4y/czMAA340NbO9TpN2/jMf6gxrAREJdcGw Fjbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439330; x=1726044130; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=KsHslQjSmABfx11h+2GxqZgAGQM68h0S1djRPaEcPnKzF1TkwAybQI4j5dPte1qAca 90SgumixPXfKED1Q+GwZqxNBCEtv8lwE9UPf9k5ws6aMCZSExXZcsHu1qxl10Mz8N7wa A5v5SQVdS1fjIaWTLQUw6D0pmdVznHvVr4sCv+LBj/JeY7BaL4QoLZlIv3/5lPO67SRx N4EougsyJvapo4ny1IAIYHUh02NCa4efQnMNHfsfyYl3CNYkeEaI5pT7O0ty9FStwXzH x2VAZnRSlZMj9X0/cyoKV0BHtBi38bHfQs5xmXHpLNKZ1iVN98PndJ4vHZVjk3QY2Gb2 npNQ== X-Forwarded-Encrypted: i=1; AJvYcCUzDS/DlF8xyI825D21v35nPA5NsuN92aRRwL5n4D/Qeo8hDUQqnXWKpYVJGl9czu+DuPQji3Le1PAOkaKIE6gy@lists.infradead.org X-Gm-Message-State: AOJu0Yx9vlNNLVcYMcf2VqMQfCIqU43ZU1cbrcGWEqgYm9g03ZhEN9aC inyYVbKyin+MJftyZBGM5JfyoOdIjiIRxaXVLOEB+OoFswTTWT1pGdUc77dLfV0= X-Google-Smtp-Source: AGHT+IHV13v46mfi8eJv03VSK3v4h4kaXbg2iDfLcHBwpuweJiwY6TIZiDRC9qw17xCxMuqJ20qctg== X-Received: by 2002:a17:902:ecc9:b0:1fb:7654:4a40 with SMTP id d9443c01a7336-206b833e006mr24427095ad.14.1725439329649; Wed, 04 Sep 2024 01:42:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.42.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:09 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Wed, 4 Sep 2024 16:40:21 +0800 Message-Id: <20240904084022.32728-14-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014210_586935_15243736 X-CRM114-Status: GOOD ( 12.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 1fde9242231c9..5b5a774902bd6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3004,8 +3004,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 262b7065a5a2e..c68aa655b7872 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp - as * pte_offset_map_lock() does, but in this case without locking it. This helps From patchwork Wed Sep 4 08:40:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13790206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F706CA0ED3 for ; Wed, 4 Sep 2024 09:19:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=a8foyKxDCZr1RHhS1QzlP1/7fT 7rJs28Pqx02lbUJK/yGIWiI/i51vy/q/93u819oKTObxSBcwYWG96Jq9b90D7MOeHuQqT+5F6pcZP 9Nl1OEDcRw10wqLH4OWzdijA5dtLdfVT9Tnh7xahJL4aJELA8i2l8Ctn0/C84/HtlBlZGNe0uKJs9 Wu6j4BFG3fjAHzKmT7SM7Hmc4XfsMPu9DZUK+wCqBKvsvNnGdrdbVG6yhX8S4Ru+DpMFsqrtlueqc uq+/g4adCP6wSwi5Aqvo2IGO61RU8NaU927ikUV/2KWODMxa34gJraXAh9iwi35V9O8GLr75Ha/YU K4Otcm9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1slmAZ-00000003dfo-3M0h; Wed, 04 Sep 2024 09:19:19 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sllai-00000003ToD-3Y35 for linux-arm-kernel@lists.infradead.org; Wed, 04 Sep 2024 08:42:18 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-20543fdb7acso28484505ad.1 for ; Wed, 04 Sep 2024 01:42:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439336; x=1726044136; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=WT5N1pe2G513tFMmhl+Auv6oQoowuHX8dQFLOrLksMhnqahdF5aoIsNf0xqFFoKs2d L4+/H8fdDqRHpfh+E6bO7YaS6UWMejc1eEHVZkkSBi0N+VFVt8OQuzxYPbTA5OEeyQFB cE51iPhD1gTd6O5LnsBY82iF5kaa0APbH72nDJ8SCh1km4GDp0kZmZnIsmb0BbV0j+E/ G51qdsdr8rGIc4kxkjedoWJN5ZZ3+5vkN23a93j/eN9frg+iGj5JltHSm3uAjhcW0TaI 750okV18S0y8sLhelceVRhezVvsckXpnj+SryZjuEOuH0ihJBmJjv85h9+aFbP1c/0gr 9ShA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439336; x=1726044136; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=ZE2PXNCBfIavRitvSHRemKPMsI30KZ4oBH/6lDVdNq8YBRgkM1jcYJVz3vjfucQKbH sqidZuzlOsTWwf7qGoo0cMnrLVurJgaPoRo5nI5bDYfl/yMn0I30oeDUz0Wu/J31m/Wk 9Jf/jVxr9MXx4/yf8H7SiUloUPb/MgXi75KNpSh7V32k9jgHczPYLlyyAcWNFzS8+68p w1HXbPGwn8r+rkIGwVwsLykMg5CRHFuMpMdh3fNDXoZBQzSST+S6jWt6Hknlurx/2VJX IATkyoYOLjeQWjXFfAO7JaZ/Cc/SiQq1x86sRn23t0tA1bKmjf55npU9QIUinI8Yf6jW yk6Q== X-Forwarded-Encrypted: i=1; AJvYcCWlLK0xjRI7SdoAOcrUdRYYqopNrMOhRSVv/0gudlHRLS8616Pu3r5N7NTsg5sX3kz9NjK7xy7Y4sFV4y/bFi++@lists.infradead.org X-Gm-Message-State: AOJu0Yw9ddgqgC5dd2AuD0JB0piidHhK190Qd2hCx7E3cKrKwO53gQGh EH47br/siumFMboggOzn0Ow9pvrOK2HCmlQfuLgzy5M/yyV4v3l3fZLdSGF75lE= X-Google-Smtp-Source: AGHT+IGRRbNTDVHBE5FEDByUohEYiQ4MupDiF5X1fYOKyMaX8J1f74Xy7lHxJ4SbBUityp3jftUwsg== X-Received: by 2002:a17:902:e848:b0:206:c43f:7896 with SMTP id d9443c01a7336-206c43f8dd5mr2410565ad.21.1725439336115; Wed, 04 Sep 2024 01:42:16 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:15 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:22 +0800 Message-Id: <20240904084022.32728-15-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_014216_903299_AF00A4CB X-CRM114-Status: GOOD ( 13.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_rw_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a117d35f33aee..318cc3eefb040 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1724,6 +1724,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd = false; + pte_t *pte; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1759,11 +1760,25 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); + pte = pte_offset_map_rw_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml = pmd_lock(mm, pmd); - ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl != pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma