From patchwork Tue Sep 24 06:09:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 571EBCF9C71 for ; Tue, 24 Sep 2024 06:13:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=wFV2Gg1HPn9r0uTPIEIMJioMCNlgr9zWJ7S9CBl14ro=; b=MGTp6bTJBIBK2vROXIhVzHQ/MG IT6o7/EzQX79MtPTQMGxeH1iwDXqe+AE7f2pPTHMrk7PpCwl0XxkW8hV0f3oL1/6ucTB12EN6JqA1 slW1xg7ZXXzB/FfRvKDxUTVIX0rqhJl3Qw1pffNmKSf8NHFW/engyD9ltH7PAljQ21U7MzjPBFj+2 HGRB4bEQ5k7iA2rMM/vX8hDYDvWMS2ERr9Q5HzgJIq6mKtEAF7aNWFnl8paQyPP64m1I9/PnnH2QL yj3y3i50w2AA3J71Om8g0GbpwijJXx2UZtEafvuYkhME7w7HnYY5VtSiL/KmvojxwriEKISIvOFoP zqD+Ha4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssynF-00000001Cjp-43uK; Tue, 24 Sep 2024 06:13:01 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssykt-00000001BdQ-2GCg for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:10:37 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2057c6c57b5so30983225ad.1 for ; Mon, 23 Sep 2024 23:10:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158234; x=1727763034; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wFV2Gg1HPn9r0uTPIEIMJioMCNlgr9zWJ7S9CBl14ro=; b=WLDVZ2pK5Xck0Yag08xAWa7ku8G7lQjoYYS8Q5RvTNQLQL6MqZydSL4+cQ1Vktia9/ iyQ1XlKFpXlcF7xF5J8vc+IMv1Htb5boKOgEgjz5pfDrlxge3UWMu+1Et//ACn55gWnJ yakRuCLsqNHR9RbkIo7BSq1YdNVaExak+N0eABOHaNmZ9gIHwRb0E/bEcaOzRzE71KhI WylCz8DvnF9cCfMX/7/3uCUj7B5kb9kIpRAtiXCTzudCDAN7JXv7iFH7F5SVLgzFAUq2 aJaOhx7gRr1W0vZ753hgH2KmbyiKqrXwRf73uyoylrNHV70LnXAFaNYTGryz/ZyDHjL4 BDdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158234; x=1727763034; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wFV2Gg1HPn9r0uTPIEIMJioMCNlgr9zWJ7S9CBl14ro=; b=fP1JLsw4sglqH9nz5l7a6pOQUGKFiGYQFG2DVW0RDm/PQExyMoNe4Nmki+KO32otdG kdIc1vFu9rltWCyJYW773heTXdo9+mE9/qZSbXMlmHihLc7c2kV0zqISCZ8GpENZvalN 5Xr45bN/cc9RVV6kO0399+gl2y+IOLKFHy0F6uW/JMsldHISmf5vkjkdEj0y85iZ9xjv c8eTt3u9mwRq3OpC8GIN40sWh13S+HdFEdwxZLrq8wZP2t+tYL7uGegENgpSrVP+Xk68 aeb58dqv4aOOYsAC0g6LZX9H8616zvdUeeAp2BFpo95RuXJqkHFFSYgvr9USLHDnu4U8 KlAg== X-Forwarded-Encrypted: i=1; AJvYcCXLlEHjWon3NbQdR9qMoTghVZCLtXZw0Pg8cZ4+AuX3r2apfGk47WHRGnxJe2CX/Jnlc8NS5AaGPF9+tJK5/ikO@lists.infradead.org X-Gm-Message-State: AOJu0YxDLRKHxeOQZlwDbPi3HRDFNUb4akadnPPZ7JjJWNMVcVkLNHBG cw8NqbaC8NgJObdTd0Cd/ujQiR6vb5mu4dBfG7Rupbz9rgts5IuXYgwR8DY5LEQ= X-Google-Smtp-Source: AGHT+IFurxEbSfxWnkgUvSmQpfQ7iOb3of4fLdIGxM8iu8OEamvgIzhTlXEYzcUx/o7HLvoMu2wctg== X-Received: by 2002:a17:902:ce8c:b0:207:6d2:1aa5 with SMTP id d9443c01a7336-20aed0b299cmr27872445ad.13.1727158234439; Mon, 23 Sep 2024 23:10:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 01/13] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Tue, 24 Sep 2024 14:09:53 +0800 Message-Id: <1a4fea06f8cada72553a8d8992a92e9c09f2c9d4.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231035_612794_21E2D424 X-CRM114-Status: GOOD ( 24.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 7 +++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 50 ++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index 546a9406859ad..9a4550cd830c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3017,6 +3017,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..262b7065a5a2e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp - as + * pte_offset_map_lock() does, but in this case without locking it. This helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct spinlock + * pointer for the page table that it returns. Even after grabbing the spinlock, + * we might be looking either at a page table that is still mapped or one that + * was unmapped and is about to get freed. But for R/O access this is sufficient. + * So it is only applicable for read-only cases where any modification operations + * to the page table are not allowed even if the corresponding spinlock is held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pdmval. + * It is applicable for may-write cases where any modification operations to the + * page table may happen after the corresponding spinlock is held afterwards. + * But the users should make sure the page table is stable like checking pte_same() + * or checking pmd_same() by using the output pmdval before performing the write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Tue Sep 24 06:09:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF1CCCF9C71 for ; Tue, 24 Sep 2024 06:14:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=eTCRoeg0isJN3FsWjY36jjsXQE i2MsvsyZ31KmLghLIzuJziVdebQRlfH+nVtFdrbVxV/ew0Bp9gHF+Wkj/GQ2Jk+dKURpvpl1cyryP E+VJKFKIdwoRROtexLP2OktI/IhLqM+x8AVODOHJnsELAhd/+r5dk/mvXv4mDPPedncUDmrYnZzZV 7wd810Q059/SAnDjdQQ7Ey3C7zN8djKj9/Ww/8NDbCBUMboL6ZSWfpYTNKqkamNpXdGuLT0jxI+ER AywI8PdbbKLqYAEy6/UM8MIX46urPmewG2UdX9+XMgyh9eBigRq06pd+VEORDoNNDKv0TxYg6f9IJ Myit6dtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyoM-00000001CwK-1TkN; Tue, 24 Sep 2024 06:14:10 +0000 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssykz-00000001Bgo-2HO1 for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:10:43 +0000 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2068acc8a4fso47159365ad.1 for ; Mon, 23 Sep 2024 23:10:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158240; x=1727763040; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=FIV/BvAb3RB4rw118q1HmPIDQ38nF5WGiZa8DqMDE6EfaHWOpPuboM5FEPlrFqC9iX jZv4MqsPLlcBi4gwHPhWeElZk4GElXxgACn4BolM1xq4UvF2E/dwVChyxD5wQEP62LNA 6hVyrw7VvbGJg54G3tVfeCTuuNfX+4Ihq4MsBHNwGebiVZ8RXnsNx0lI7VrcLVwShlPt jXbcz5obUahsfcR9EmBaVwwlnBeTJmqfHi/LDd6Lkb0vyAi6g7KuDcsm2W6TSagdczq4 JABiK6Bwu6Mj2UiIpO+MiOiMP0UE+9PaA9KMZoEBRXi2H8PPPCPUskhgEcHxNjQVTs/p HDXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158240; x=1727763040; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=d5OY/nQmGXT1hNbKTJX9Ql3irROTgja/CZQJOikT4TC8ELwvWYPkNCtwxHpoeeOMmK C2nl1SwKq2OJ/4Yz1z9kU6I6xalYbxAO6QxxyXocy2C1gxpc3mY5ozGTVP2THdm7j+8n Rrf5QFKcB4bA8fvxJHA1t1eZB9/DvrcVmrnf7LkEyDn9Jx9qHJNVLNu89ZtwapSKMbq3 s5t3MEIO04y2PzuxN1AVrxJaU9GmTpjQIbebmcpV1Tzq001KStTvBLD+OYc3KT+1Nc9c PeCdXD/MV0bsGGnP5mipRRgfLPPTkRPOacjcn9MjHw/uozH4o9+0wAzr6/rUFMNmrgvx mKfA== X-Forwarded-Encrypted: i=1; AJvYcCVZ2xioafy/RAVxaeXsCADVam5RBVxkq2k7moXyK8GgjLqf8zM+lca07iw3y7IE9LPp277rb/psxVBovd1IoqF9@lists.infradead.org X-Gm-Message-State: AOJu0YzMyVvVjgxXkmXl4za18xFqRMukVxPK4g4oMVaj8HnNMEnKYA3S dEbOSfZOdQDo5wUaTCUdLXMcqckyEFqE6z+JaxLrjd+AX8rS3dC+9kY+i+MdNoA= X-Google-Smtp-Source: AGHT+IElcy6/dcdtPcpdDMoiIIDu5/E2AQfQDwpdj/dfRLSSVL2tlA+KvKWV7BmaFFYnOD/6ZCb7Sg== X-Received: by 2002:a17:903:2311:b0:206:a935:2f8 with SMTP id d9443c01a7336-208d832a910mr203885105ad.2.1727158240521; Mon, 23 Sep 2024 23:10:40 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:40 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 02/13] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Tue, 24 Sep 2024 14:09:54 +0800 Message-Id: <597a93841525dbd58fbebcfdc547f2080694835b.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231041_606898_0E6638BA X-CRM114-Status: GOOD ( 11.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Tue Sep 24 06:09:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CBA0ECF9C6B for ; Tue, 24 Sep 2024 06:15:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=PR+XRPp+4P2ynGvEB942DtHW13 WwXSsKgYrXl1789XUr79zlS2+rgX18OhNvEbVqMn/kaeH4KKkodvVh7Te+hNHRXcFYkPVYY4x90mg YJiAbe1TsEej5Moj6luaYPjNfoBdqjN82NRF4KGdEKq79BZFJaQVvcMYJipHh4Cob4hEh3l5oOnol vRconI4trMUnN0XRxk2b160oqbMOBDpZq/jTQCRWBNadIzS3vnelZdqrefreFReCW44O/JQOEuaFG piMHuhpQxk2dckkEWMemqdYqsyikTNLDv7vQS/Zjyt6C+wRAssd6UPZT1yYDCT4xF5rXGSVRAqEmG iT3Wy1/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssypT-00000001DBb-2lhq; Tue, 24 Sep 2024 06:15:19 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssyl6-00000001BkY-0Dsv for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:10:49 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-206e614953aso49436345ad.1 for ; Mon, 23 Sep 2024 23:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158247; x=1727763047; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=BFOP747hCent+aEfpuAUNsFO2/yD9fGO15NqWF5Cfi0WjhPR+G1n9rz8z6NKibgAa0 QRh6x167NrvkcSgKVo/2dVjx30TM9vKxj9RgGj29HG1U/9FbLonIvLB1z3aLFby9huXf IRoLJOB6PX1EfyaKfVg5NheXX2oszO9FnXfybyWdAxjPGkPaxPJ7qSoreJWtrcm7HUiW jkaPgenvMuEq56o7nirInEIva7N+NdH6+xutZ1tXBgqCyx/h3OkKd1vqL8yRwIy7dAh2 ZWKwElyZ/XdqjpastPdsAF+lT+gQB+FBh6GlxwB90RB/HVxs7pfK7FbzSRKHmuas7aRz Omfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158247; x=1727763047; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=AjKgeaLLGgFdm+dRBDC1iSjb2LmjiH6fyDFSw1N8bv+Snv/ghb0c2/KsKMYpTQc5ld BzkoMNDvuHbv+Re3zhSef5ZpFxw5mxpqqUrciWrY9MzbJrDs9zGdGqZDJ6XqJzv0/vAJ EGE2Et6YIG9tnlfqNBHNZbRby4Eq4y5+IoSG0PQw1vIavAg5bag5lvbfDldBAHF/wkSI znGV1RS662dL7TaWhYX04R+lxN4yBg35k9UHjK/ToXhco2HdvQNLuP1kBGLJyvLQ0AJE 3ZeTIqtLcP4NPigcWWTEsJewFgpXEJt7GHNK0DSuqpmbzRBkB6+gaSZ7G1aRh2Y2I+po 3n0A== X-Forwarded-Encrypted: i=1; AJvYcCVXZbFWljPayfd+OVlE6totBXmeYx5VcJWLhgFL7aI3ivctXMBW0XlfI5db4IReomti1tD39V0cHfUKB3JE0nhH@lists.infradead.org X-Gm-Message-State: AOJu0YzyauUQG4Qv8WM3T1oYcxdwzx3OiX9wGQJCZdDnNQkhU9r4GEFh hR08/HIOgRlVycNJ1ax88tInWFqM2gq6pxtvLPpV0vrvdtZcrsoJb8q2/FYrCeA= X-Google-Smtp-Source: AGHT+IGm87aqttqV2Aef0OLuq60VY3Mp4w41J7dgMXq2Y+1L52786e2U6HUOab2ZHlW+VZaRsZgruQ== X-Received: by 2002:a17:903:22c5:b0:205:6f2d:adf7 with SMTP id d9443c01a7336-208d83a8387mr177369015ad.21.1727158246651; Mon, 23 Sep 2024 23:10:46 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:46 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 03/13] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Tue, 24 Sep 2024 14:09:55 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231048_119125_7280D642 X-CRM114-Status: GOOD ( 11.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3e46ca45e13dc..6c6ff8722550f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3271,8 +3271,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Tue Sep 24 06:09:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE66FCF9C6B for ; Tue, 24 Sep 2024 06:16:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=vxXPi8cziOGZy3e6m0Pf+T/E8m WqwpxTPJr1zZPTvNnu201gl1Y+v3e98RDTKLtQZX3udClEOCUcOyKT9EpQSgSXQmflK2tt4pulA5r p4C3nl6PFHdZamwpCAPVv8Eu1v9KGn+gIIaEaNhBWMzY+RPzgOyp/y+PZbaTHYSvmp84pU9irpMV8 Is9ipr9GYodNrWPJf8pjxo7dHSoWuD37wRAmBbYANvhAZCTUnNphrFwjokzs84pJYh7AJYrATL8SZ +z6+19hogzLcxGN4QgqbUj9ycZaSIGkqiOfPxxUMgZALhzr79YYbwlMz5fa2apd4LeYgUyb4EfnfL dI4jlRCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyqb-00000001DXO-2y4m; Tue, 24 Sep 2024 06:16:29 +0000 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylC-00000001BoM-2kyh for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:10:56 +0000 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2068acc8a4fso47161285ad.1 for ; Mon, 23 Sep 2024 23:10:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158253; x=1727763053; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=TeNaieQtHUQBOF0qFk0bvfgu/Zv1lHP2hmhoLDfH4FKlay8Du5T/s947bC/LIeJTnZ 4lqmWalV+cxt3+0BhfoR+/tuxR9wRze8zV5v/VYD76j7sMKJ1rN8di71sTghw/tkTwNZ Ym2M70jo3F8ruv81tKMIydt9ojNPf/34z6oB4yw7PAPnFrsS+ZtBV7W78b0qhM8uvdbp +zCeSDbPaEzfqWDX44nxLr6PP6Cce/l2HEHOHQGa60Ey9bdRynoYRD1VEpZ09/zPko7A kFvwCIp1tOUdQgSjck6bdlPUx2CKhhcF+Omu04BP3ljfbnGW64g/fgdh1zxcX64QsH0S +yuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158253; x=1727763053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=OH0NGj1HJ4+/p8dYpovZnAzwxiXVKwYxBoPJHrpO17GFiSOe2vUFZCoB2IdK0SrIIe me5G/Vd8NlbEpjoJGn42HUFFpCv8/NXm5mAwmBToa4xjK+rhz9jzJl6sw76oTYfm4w4U E8a/vslimLkGwqBz6+/kQsC1ykDIHeUBmeAb2IlktH9DWvHMuPpZKeLS/UwLMkmUxtxX F0mf6lft1O1TQgbdo6EhE1ukzuwmk8jHHDsKplHdt/jIBcmkL7jc75oHACrLn8XpaVAJ +cD+gejjrqpKC+T5+7ku7Fc7NfAVRUTmjmWSOa9G9db3VmBEdgjIInMADrCv8tyTaLpD 1zKw== X-Forwarded-Encrypted: i=1; AJvYcCWUGN4vvU9dorO/6+Ad6MSero/4k3X2yU/noVNmrravTiMn0K7XhNPj0ox88o4rd33KRD71vnJdHe5f1zxh9wdZ@lists.infradead.org X-Gm-Message-State: AOJu0YxqpyTNrNQ7WbUc2bk7G9f5Er07ydU824wDHmdwnSpdgNaTrC+H UOLJy1dxArlX8xnQQNbPL5Ct+4GVtlkK+mptEWfLrkVcRHM9mD33q8pXesv7I34= X-Google-Smtp-Source: AGHT+IEXIad82sT4qTFz/mgMdJh1108B7x8i2K/1dZJvW9IHButimBDIW5JZoOsyX1eaQQUsC7JSYg== X-Received: by 2002:a17:902:c94c:b0:205:4a37:b2ac with SMTP id d9443c01a7336-208d83c8698mr211905735ad.34.1727158253643; Mon, 23 Sep 2024 23:10:53 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:53 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 04/13] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Tue, 24 Sep 2024 14:09:56 +0800 Message-Id: <69e1c4483c2d679d3e2c8446264e5878e5b52a97.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231054_751191_9EDBC79E X-CRM114-Status: GOOD ( 12.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Tue Sep 24 06:09:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F3F9CF9C6B for ; Tue, 24 Sep 2024 06:17:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WqTOjUnPHL3XRjrt8I2PlSfGhzwzG7n3dPRatvbgECo=; b=zNIqC0dITdB50MQmM6Wr6xHbuP UkQhY1URfmI/zSW9nsM+rN+c2EUAjXAQLG/8ca6gMcJigYCbou8Hviw4dT4OznQBBcrvyl3ggAmIA Gecnm6/1a20aKxwMSD6NgvH3LA/gMbgtafNyI/2pPtcHk4Ephnm1kYd071K8M6/3mya+a/NV/V7uY EEnl4kYV3w9S4D/LjMFr3U2cO2pyvMtGcJeZMS1CWEZaHFIR8nXEi94DPZRfBU7dbocvj+AQQPDrV xFNLzOWlFURser3RoHBgOhNElqG5ty/NoY6frlacdkFlm0tj0vAKaWAOduKBHFYHnVsZA2LXjRekd vg5xjRKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyrh-00000001Dg8-0tB3; Tue, 24 Sep 2024 06:17:37 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylN-00000001BtD-2iYf for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:06 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-20536dcc6e9so32524105ad.2 for ; Mon, 23 Sep 2024 23:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158264; x=1727763064; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WqTOjUnPHL3XRjrt8I2PlSfGhzwzG7n3dPRatvbgECo=; b=j9gLlkp4N4Z+f9AcyOCvvXQNGAAuextRxJEQmFEbyT47/viVgca6+m8pywAu68TOss v7dJNuLm582KsY/u5274WB8I82Bl2lPKMRL+VEIlXqMM49CrJE88i67MQt1jP461DDSa Zp61epLkRrLMIgbWZp59dHjiLfA257dvTHgSV8hEXYA/JUkl45uOFrjfqy/8wU8BnvcD 9IfUj/n3sNrn9Ku5aC5PQ0yGxwz5VCFPa015oBjKStNFrm7Ml7CeblGllWLLCLGMZWeS ezNiw66eTAed7PvhDym/86UcG39LrSPKXod2V8lBgJUX0xRJVMSS9Y9PgfJgh6/SEGJ7 K57A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158264; x=1727763064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WqTOjUnPHL3XRjrt8I2PlSfGhzwzG7n3dPRatvbgECo=; b=TwQQ5MYp+lZ3zTKPtixSnCPpTmNIK6kPxVVUPvzmPjOX6NtVX/nk6zuLyM7pteYhK5 GdTfEh5QE310bzcwWWq2iTNf1C8O6J5ZG5fEHIUM33fk8ihdSrQizVV4IMgjJoH/4QQD diF29TU0Ad47yLEPbGy9C4VAFirfid2ofD0SC1BGk5lwRkcKhFT+95WuJeu211lf3MWM HJOGS23PXRV0xgCH0tR7vIxDgMDkGTKs0VvqG28XXjAfwXtlAtqYVGwR59oOra38yNvp U+FTADm3/2r+Gnml3tEp+Rpsa4h0hyGXk1DBJAN1/MKuAX5hcY5joFLYLLhDrhSARk4W /Kug== X-Forwarded-Encrypted: i=1; AJvYcCV/6kKXfa10cIWj7NZKDR5WW71i8R83MJgK8nw0FiTJ1QH8XBolOuzq12grkmq9VKQvo9Hak9Qdwq/EYpN1Kg/H@lists.infradead.org X-Gm-Message-State: AOJu0YxAei4aXdNmTdLreJfPeHjjcGK8s1DL/iLheZrlZcX77+J9wYDo EgTer027hNBQ4LIDy9ezUUnHZrUKC0Rpt9rwbOj46WBIo4t79kv4Da5CyqZo0KM= X-Google-Smtp-Source: AGHT+IGIKI1reF9Jfo+dWimmN1cyjgiGD8qn8mcbfIJc2tOglwlNFzTXQQXYEG1VSSXByHdLorgo7Q== X-Received: by 2002:a17:903:944:b0:205:4e15:54c8 with SMTP id d9443c01a7336-208d8440f7amr165611365ad.61.1727158259613; Mon, 23 Sep 2024 23:10:59 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 05/13] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:09:57 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231105_696115_AADBEBEC X-CRM114-Status: GOOD ( 20.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. All callers of update_mmu_cache_range() hold the vmf->ptl, so we can determined whether split PTE locks is being used by doing the following, just as we do elsewhere in the kernel. ptl != vmf->ptl And then we can delete the do_pte_lock() and do_pte_unlock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- Hi David and Muchun, I did not remove your Acked-by and Reviewed-by tags since there is no functional change. arch/arm/mm/fault-armv.c | 53 +++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 31 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..2bec87c3327d2 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -61,32 +61,8 @@ static int do_adjust_pte(struct vm_area_struct *vma, unsigned long address, return ret; } -#if defined(CONFIG_SPLIT_PTE_PTLOCKS) -/* - * If we are using split PTE locks, then we need to take the page - * lock here. Otherwise we are using shared mm->page_table_lock - * which is already locked, thus cannot take it. - */ -static inline void do_pte_lock(spinlock_t *ptl) -{ - /* - * Use nested version here to indicate that we are already - * holding one similar spinlock. - */ - spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); -} - -static inline void do_pte_unlock(spinlock_t *ptl) -{ - spin_unlock(ptl); -} -#else /* !defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void do_pte_lock(spinlock_t *ptl) {} -static inline void do_pte_unlock(spinlock_t *ptl) {} -#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ - static int adjust_pte(struct vm_area_struct *vma, unsigned long address, - unsigned long pfn) + unsigned long pfn, struct vm_fault *vmf) { spinlock_t *ptl; pgd_t *pgd; @@ -94,6 +70,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,20 +89,33 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; - do_pte_lock(ptl); + /* + * If we are using split PTE locks, then we need to take the page + * lock here. Otherwise we are using shared mm->page_table_lock + * which is already locked, thus cannot take it. + */ + if (ptl != vmf->ptl) { + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + goto again; + } + } ret = do_adjust_pte(vma, address, pfn, pte); - do_pte_unlock(ptl); + if (ptl != vmf->ptl) + spin_unlock(ptl); pte_unmap(pte); return ret; @@ -133,7 +123,8 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, static void make_coherent(struct address_space *mapping, struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned long pfn) + unsigned long addr, pte_t *ptep, unsigned long pfn, + struct vm_fault *vmf) { struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *mpnt; @@ -160,7 +151,7 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, if (!(mpnt->vm_flags & VM_MAYSHARE)) continue; offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT; - aliases += adjust_pte(mpnt, mpnt->vm_start + offset, pfn); + aliases += adjust_pte(mpnt, mpnt->vm_start + offset, pfn, vmf); } flush_dcache_mmap_unlock(mapping); if (aliases) @@ -203,7 +194,7 @@ void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, __flush_dcache_folio(mapping, folio); if (mapping) { if (cache_is_vivt()) - make_coherent(mapping, vma, addr, ptep, pfn); + make_coherent(mapping, vma, addr, ptep, pfn, vmf); else if (vma->vm_flags & VM_EXEC) __flush_icache_all(); } From patchwork Tue Sep 24 06:09:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE389CF9C71 for ; Tue, 24 Sep 2024 06:18:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=mY+j5WaRObmxeTc6QNxImUCg4+ y6SqOPZVKUUptI7DRcwYNDEdSmzK1rjk9iYv1cIAWBQpjwUfLVpsgmfs+dM1hKzzEh43TlyuB7FHb Hx2cOR0hSHlRwoYmKNwR5jzdXcVTw6hkNEHzVnwzI/1GrtsNp0/oCQ8aY2McfQi3r37rFfJFvBd1W WVnVcWGAm5j/qSkdX8h6XbpWR7rCw3RqqwfZ/GAJwHmlOts52JmjBBdIkoSpqsTSiX2orbY5fshfS LUDSqSbcCFgy8GNUHQX59f4xW0/YobxdniWT3NyFudfzu4RMWUb4tWYMFo0NDHSQqbiLMLwQ7KNaJ +abeBXLA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssysn-00000001Dro-3NBk; Tue, 24 Sep 2024 06:18:45 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylO-00000001Btn-3pBx for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:08 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2055a3f80a4so32066595ad.2 for ; Mon, 23 Sep 2024 23:11:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158266; x=1727763066; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=fcuG6MXcbXlKzUCF1Fn1/3zWDsry0M6wkznXq5/mwudxtIM7vsn6WdTMUH7+B3dfDe SWtxFMcpU23c/BQyeGYRN6xgJhFUggR+9XbEkelYGkkB8DkZSgRv0V+YZ/fxH+fFyKiH F+R/VSqkbAxxueUrCJ8oKSivEmxUo6qCKslhUwwYUM108HGFLWOMChTBQc7urG6Ej7w3 WfQClRGWhRX9zTEepBxFdftDgEEOBTvSMjdHu/l4Yre7Yky1X4x7QD/pVozWJiPdwlic XGe3v2fmVGMdogxadLiX5qb5WYCdwomRvqAdCIWUWm4nxE9IUA4PE68magUxQDpu1exL X/JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158266; x=1727763066; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=T+Pl9ex9GjNfhde9Ndoc3h8g0abzJ7b6i4WIkXaYsDCvf3zhSOUKBBrHISn2y68PWi Nqt4RXevZCyK8RLtLpSBplttX/dFCXRWvXwX+OD5ol5pbLCYSuIaYC0Chf7rD6kBKpns y6/k6HyrYDRHroQEgEb0wNQBXX2gqnfokgjx9iG9Vjs5DasmZXSsXptD+0rSIgjWFCDK e588EDPKJTMp1nBd3fHEkcQGlOxR5T36SlLeU4DymunAjG8nv31a7X2sQx+NiKZky9Rc C/+HGwywaVojJlBnpKHC6t4h7fD9P+zUAirKh40BdSTaWpF7tdppswTCIByw17xPhatb Wd9A== X-Forwarded-Encrypted: i=1; AJvYcCW98IKAtCNWbopXARv9Rnj6wZRy5Sqo1sZ4o3E26K7VeDk9+01NR532GjzHhZ9VtPnOOTGtNfWDaEd4byslgBMY@lists.infradead.org X-Gm-Message-State: AOJu0Yym2vtvxC1kKg/8j9sFS57UNKeHbGb6A+OfWB4UKL1PrIYaiykP xGl1PaLShZPq4SP8yFR6jtLQf+uVATgltJmKBJ/VUjfNS841Eztq4jJ7oFUP7sk= X-Google-Smtp-Source: AGHT+IE0gKFIx3/KW4QPIJwQt51ZMLGla2X2YU2KRRy7vcX0w9t/15+gAAip1FE70+J7SP+04M4CVg== X-Received: by 2002:a17:902:d511:b0:205:7574:3b87 with SMTP id d9443c01a7336-208d980bb34mr182109485ad.15.1727158265764; Mon, 23 Sep 2024 23:11:05 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:05 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 06/13] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:09:58 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231107_004535_AF2330CA X-CRM114-Status: GOOD ( 14.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a245d28787034..6432b636d1ba7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5750,14 +5750,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Tue Sep 24 06:09:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7542ACF9C6B for ; Tue, 24 Sep 2024 06:20:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=dtqkBA88kwvMRDkdpXIuTPjK06Yw44S9bnA3B3TRVKU=; b=IY1YB7cVRGYroA4uJrjFSaFaKD 22gMtXRUTzlTyAX7tLdvwgn9po6ND36373A8OB3dmzsfe6gPQJyIJ9nHqfze0+PHuDK8y0diFCawF zh2ON8+Qau9zcxPUXpoWDhC6HWVMNoR+3MG/k2gYN8QC1UAxsp6G4xtLYUtP8xwtG6IYfwYZ+pUi9 DSUBHk6Umh7SQOnbusg53iG7zG85rePYrsN5WIhbhaZoSGxlB3gub7YRVic/vNTvsi7m8bTIhZqDL QE5OxE75pvkHJ1yjcl8dfHbb4mUb7vs6wdLIDQWFhH9QERRUTaphGGNizZmTOadPtJi4ARCH8KbeD MG3FI45A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssytv-00000001EAJ-1MeA; Tue, 24 Sep 2024 06:19:55 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylU-00000001BxU-3OHz for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:14 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2068bee21d8so51320565ad.2 for ; Mon, 23 Sep 2024 23:11:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158272; x=1727763072; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dtqkBA88kwvMRDkdpXIuTPjK06Yw44S9bnA3B3TRVKU=; b=cxC5iF/RFl9w0krj8Onij069kEnnvghfqFL0fphtmnOLmEzVZRYE12FOLNEVIqEYMg wTqeboL5RIhiTonBad0j0PoTpy3USgBfUW+lgTNVPFNIYweE5nyLJLXN6Xvdy6RpAsnn i9glMS6Eyh0ZdC7N+RlZdmvWWyLIVIXYseqgVt97CJA9pl6ds2ADeSnzHZmqxa3qWxNp EkgXBk6tl/oulTaH3dNd9X7Bt5BbhYbRYUTG3SVVxTLAJE/QtypdEvAcBHYGDawsaDtM aL4B5rZ+9Zbup8jcs2kIUeBH/wH6tZe2tf/u/LbCceZHfbVMlErQqzRkXCT7YRpEcZJZ 9IHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158272; x=1727763072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dtqkBA88kwvMRDkdpXIuTPjK06Yw44S9bnA3B3TRVKU=; b=j0EGEV+vFIqqKfCl9hXX2CooNqfRPhN6UQiNgYTlVZ5/0ajp4ajP3aVmCAZXGYoasK qluiWmTdPzmB3ZMCkBtPcT2/tXcIUrclehtK+LXNUlmKYmNW5lvHpseEDbOeGw1/GMAw srMXaQbQlnc3ZEgJrwWbF9GeJ8Ery3gb12KJhOPRN72XGqa16Usb9hv0nvJwoE18EqVK k4fR6wftRQjW7QJU470qwaadE2rIWkuGb9GcWUJz2JaOGLQV3UZQ52IrvFCC+//W0ZJT AO4xG36+1y8ZK2QIUeebGeTxsdxQretyXZfhGUlM/3TDoveh+N0WwDjObVLdewjVFyLG UCUw== X-Forwarded-Encrypted: i=1; AJvYcCUnZfWjEHakmsFZeH+kqIcY5Z8hDmch8ElUmn2L6raeyxs7qus5IMCtNkSIlI7mDYjIk3+yiWCSZULDTzQLl5Zj@lists.infradead.org X-Gm-Message-State: AOJu0Yyc+nRfj9w1BhfNZ2H9rep+SdEZSf3G5JGc4kNqM1mwf3fpKuKM cNEhArIBrswfgEC1SYWyrrFxVBJ9SldCz266m4csK9E7TaMe9T6wuwMKAKkW4Uw= X-Google-Smtp-Source: AGHT+IGXhl7eW49ih0T9YK4dpESD/v76/2vHb7m+lE9WqCGFaYPlARnoD/c7DbHxV22Za7qr6kwfpA== X-Received: by 2002:a17:902:f54f:b0:205:755c:dde6 with SMTP id d9443c01a7336-208d83cc8b6mr212103045ad.30.1727158272107; Mon, 23 Sep 2024 23:11:12 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:11 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 07/13] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:09:59 +0800 Message-Id: <07d975c50fe09c246e087303b39998430b1a66bd.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231112_875739_2EFE754C X-CRM114-Status: GOOD ( 13.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..8ab79c13d077f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1645,7 +1648,6 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, nr_ptes++; } - pte_unmap(start_pte); if (!pml) spin_unlock(ptl); @@ -1658,13 +1660,19 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); - if (ptl != pml) + if (ptl != pml) { spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto abort; + } + } } pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); if (ptl != pml) spin_unlock(ptl); + pte_unmap(start_pte); spin_unlock(pml); mmu_notifier_invalidate_range_end(&range); From patchwork Tue Sep 24 06:10:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 81675CF9C6B for ; Tue, 24 Sep 2024 06:21:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=a5exoCzehntR4bd+ahks+h4WFwEAjFaMLu5nKDHss+4=; b=O7DW/jfkYiQkrY5rmwwlOjRoqg oDEk/yU/IZbq1vE2Qja1KX2fbOCEqu1HJxcfJuTBqxTMFiaTGpHk0CA5LXdJK4KJdLPEzW2n18ma1 nwaOl8ZfT760hdQy63kkndj9SwINbgq+DB/ANWRtKufYk1xA4SYnAlIlxNgnkdmO/8b08mhR6nTLz jCPW68waM8DM9NRROeRnThKoH86lquuFGvHePZNbuujFvYJOTwDmIeRQ7FOM61xkPHZ9hOI8dyRZr F4PGZN/pZVeJQ+Wk1II7FNzbvxStAYiVviB2om5WGEgKKBBrtYORJYp/pDklapqxl1WsC6lgJ/icn OEF1OVEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyv2-00000001EUQ-3ZGA; Tue, 24 Sep 2024 06:21:04 +0000 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylb-00000001C0b-2EdU for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:20 +0000 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-20570b42f24so56153075ad.1 for ; Mon, 23 Sep 2024 23:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158278; x=1727763078; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a5exoCzehntR4bd+ahks+h4WFwEAjFaMLu5nKDHss+4=; b=SUglfZbY3mlGfOrHA5Wk6EEiP/TzVtpnX7iEPkBy5nr/5/mPD4N0Cw+yK23nPSY+A8 rixI1KZ3/0Lzbi0LR8ALnng1ooU3GnGavFNOazJlhy00PlRB9QIn6RPFEX+jUzHz3f0g HYCWP6qyVhfldl9kxkyI3cg/DIz06j/4lOuOSB6pE0KeUGUeIye9pZiozbtP12AWz6Zv E5Fs8jZ4uHJqbYjXN5sxhFVzJ9BMDdWvZfzXy/AwTYIO7ymiGGEaD22SRiD3Ir+rCaEY x8dBWUBaZP5kfWwn+cY0A9298u7yAacViwnoDoCI6X8ym+tIxYPCn+M5idx+nog3I65j q/ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158278; x=1727763078; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a5exoCzehntR4bd+ahks+h4WFwEAjFaMLu5nKDHss+4=; b=TnpgwNbsuEqAezQR9ezh0A2aFnKGKHPQC4joBezTY4gYWuRXeztLGB60cZZge3w9un rTyYeCzuOntRw6GiIMEv54KVg02t5wKQ2XeAoUQqYmqy8lZyVuPVGqJ2CF5rZxjz8pFk 3tfE0euW31w7a141qQEXtrAnZsiiwnbFhm/xSGw8QlLApCWd7PjgLMqGVNT8vVOUZ5LQ Qgkw09isbw24F+nfqSqGELFahMQ0uYhe/0/Gc9+ODtIULqRmtOwNraHitfxg2TnV/8VH NhqSrj5ZYI9FgTEGwrxSTf+pE02w9GvNsof4uB5LpITeAdRjQcOPevHImf/SlQuN3dVq kEKw== X-Forwarded-Encrypted: i=1; AJvYcCX78lGZ+kErjGl1BVPU2Thr+EZh2DfwGi/b8EO8OuqRtZh/mmRT/6JHeC1ziYor9wsyN/m3zwjTlza7HYMgmMOg@lists.infradead.org X-Gm-Message-State: AOJu0Ywzx/ieV+qYD+VXErbFXFKAa2y88GZyZZIgXJKonT32zETycdFR p1pRxKsVIy/RlRoM4LKXVpfyol/CtrqgvJJajarkzB/juiWJGjVd6ueZn7VDUCDpxh0LVoH5Q0t xdVQ= X-Google-Smtp-Source: AGHT+IHw+OqtuQ5LCBY4YmqC9EJn7stUTlaTSc5oItaKMOqNO6mqJktvnwlzFLYt5xYGl9LymCl5gw== X-Received: by 2002:a17:902:d50f:b0:207:1709:380 with SMTP id d9443c01a7336-208d838ede3mr189786005ad.27.1727158278506; Mon, 23 Sep 2024 23:11:18 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:17 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 08/13] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:00 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231119_589746_2CFAB349 X-CRM114-Status: GOOD ( 12.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we already hold the exclusive mmap_lock, and the copy_pte_range() and retract_page_tables() are using vma->anon_vma to be exclusive, so the PTE page is stable, there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 6432b636d1ba7..c19cf14e1c565 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1086,6 +1086,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1111,7 +1112,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * We already hold the exclusive mmap_lock, the copy_pte_range() and + * retract_page_tables() are using vma->anon_vma to be exclusive, so + * the PTE page is stable, and there is no need to get pmdval and do + * pmd_same() check. + */ + src_pte = pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ From patchwork Tue Sep 24 06:10:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FB9FCF9C6B for ; Tue, 24 Sep 2024 06:24:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=JYDx6olGBZytK5syFvtWNqesQWJmZ/VcuSiPjSyD9NM=; b=QjtSS3u6liLRGbE7wl/jMMUQ8p jPyQYZDWbfLaI3+74aXRMFOc1XycMwjWYnShew6CTY7DycJrPBZLyXsU4qQKzugGI1vJbTiKHbJmj Hkjkr/whvnugxi7onAaRqGH31cKkM1VHKc+dT3S9r5MZzNIft3Qznn45/YlrHvjnUUWzNJns+xsY2 aWJ0jHcxwaKkfIV0/174HBNb7qZd/cZGCei1VqDdOyAS/F4t7QA7rEXKop2bunx6tJbuPAjD3ICrJ HSepvVrVxmN93POaAm8HSwxc8hxHuCs4QFFuRqZ1DLdHjNdbUJ/taUlelIdKNz+i+5bRYNBD8prdf ufxerhjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyyM-00000001F3B-0ZS0; Tue, 24 Sep 2024 06:24:30 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssyly-00000001CDw-0fkd for linux-arm-kernel@bombadil.infradead.org; Tue, 24 Sep 2024 06:11:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=JYDx6olGBZytK5syFvtWNqesQWJmZ/VcuSiPjSyD9NM=; b=TvIwKpjUQb3oId1COm3OHRt2an iyDKzDNS0ZGepSZf0ItB8uh+WD8XGYGOWRopfc/4B+N8/brZfXHnnJ45EgB9Qx8IjW0irCwzULdfG V6iO59zk9DTdDjKSey9BUDeszVULz333JwDe1n9VvWwcDPigOJh59PGrPoIQSQcc9h8V42TkmqHJd 16RD8xchhNHKjIX/OdNS6gl8raGUU/wp8dU0f7mpAYJsD0PyEQBhFLdbGDJszawVMnA7icB4BKfuq 6AwRSDf94NvrQnH+gcD0rjsEirqCjk4mJFXkhUlI6Hb7iLozYXOUIEU9c31tG/VLhrBYri5NDvRsy zkqpcSiQ==; Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssyls-00000001jzC-3dxj for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:40 +0000 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-2055136b612so61986395ad.0 for ; Mon, 23 Sep 2024 23:11:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158284; x=1727763084; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JYDx6olGBZytK5syFvtWNqesQWJmZ/VcuSiPjSyD9NM=; b=VVZucuoDBhGUG03btOCA67Hce7XVdUS2X8gQLqKZ5E4mBW6PU/FrnRJgxfVfN5lzh1 4yme97FtDMIC6Efl4//x1XcJCBY/ENjUemdgK9wO0FHUqOUEn+l6KQWPlIt3PVAPglCq m0fW9dn+PMFMPXwIIPQppIw9zl7pMkdMuF5X+gB7S/BK/qD8GKK+9iWSWm1YeyqUA1OJ RY7HmXmctXqEJC8mYdkgbzTdTQZLccqsZNAMhTlJSl3anOi9ZTH31rEdXmCU4rj1MZbo tfOZ5qVBs67f5DiOSSaj7IHYizrGJXCZa7pAlsQh/8bOX+5yZs57kAAvdXnPlBXNYUqD 5Ltg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158284; x=1727763084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JYDx6olGBZytK5syFvtWNqesQWJmZ/VcuSiPjSyD9NM=; b=HkVAfmRu6BSxkFUyNOR7PnhfCC+/eqRljBLSw8qJws2IZoCgjOe+cW3dSLllJAGPsP XnC/rm1AifkjT5B1LoZZ6qoRd2rBlHcYekpKkVNhaFOBLaUJqbxjR/6lSWFpVq+VC7Eq A1eMmTw6+5if0DOv2uHEAWhfDPBK3UvyxJ4AX4EBaCuYDZ8My04EBuuJcGjlg0tmM+vu GgMyIYL/8Cgwv9JxBT+5ONuz5Gug09/PZLyWMUYZ28dYjGrBi2dV+nNVOhuWaNdTm7NG ghDojLbcWTp+MJKRDSdwWwMogGun8izYLTww8tXwNFv8lxP+8E6viz9nbL5k1ApYzGHT KMyQ== X-Forwarded-Encrypted: i=1; AJvYcCU922SoyP8UgqFLViPNIaSFjrFxoOswhee1z3kxrkLn5ddTUQZ94Ujvl+vZL0GRqJwBV62F5H5XTL7zDKrHFrkd@lists.infradead.org X-Gm-Message-State: AOJu0YxQOcidNe6q144nUYOGCcR54/0+gRvidEfcO0injRWZm5s5qprK oL6gO2VSh8o673r0McmujEu4+Q2XuKBdXmKa3b6U3IkcYI7BM40P8C/iF22AwVU= X-Google-Smtp-Source: AGHT+IEhn+iGj1vkzgTAalYoZbG5psMaFd6djM60OU+3J7JbD8pdYTlvGGhg/wgUQ8cNcQFEVSSWnw== X-Received: by 2002:a17:902:d483:b0:206:aac4:b844 with SMTP id d9443c01a7336-208d97e3f78mr215574535ad.6.1727158284398; Mon, 23 Sep 2024 23:11:24 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:24 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 09/13] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:01 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240924_071137_894489_AC855B09 X-CRM114-Status: GOOD ( 12.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Now new_pte is none, so hpage_collapse_scan_file() path can not find this by traversing file->f_mapping, so there is no concurrency with retract_page_tables(). In addition, we already hold the exclusive mmap_lock, so this new_pte page is stable, so there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/mremap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..9dffd4a5b4d18 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + pmd_t dummy_pmdval; int err = 0; /* @@ -175,7 +176,15 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Now new_pte is none, so hpage_collapse_scan_file() path can not find + * this by traversing file->f_mapping, so there is no concurrency with + * retract_page_tables(). In addition, we already hold the exclusive + * mmap_lock, so this new_pte page is stable, so there is no need to get + * pmdval and do pmd_same() check. + */ + new_pte = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; From patchwork Tue Sep 24 06:10:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31702CF9C71 for ; Tue, 24 Sep 2024 06:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FxMRr8GFLSu76sONoxW9925XRbgCGcQv9BKiL/bcjLY=; b=I0CwQIZ1DB3q9hycfQeza/LF8I xRXDGomeRACb4HKLjZk+8YlQWs4E3/47r9zl7ndOHkxFC6CAU+UOFZrolFRuqu+NMrQiPG5PYsod3 lESwr/6XD1Ixjeh+ulja8GPyy2kcdk+HuUlQNZ+eVdqMYcyrR3SschSI1vE6Kssnmfm/vh5bcoDyJ 8wLjtOizECf6Ilj81jHSiapfKZqY30mtXQ+sMzWo+Bnc/aaykehr+AwTW+4NU6KK+o6Yf85MeCvSe 01m+jDNZnQ34Fa/5zdFiHr3DKajAD7P+yJC7PWKakp7bHQzOrdG2eStAHj8STlvwiMZ5Llysg62Y7 vdTMPcRA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyw8-00000001EhH-45dE; Tue, 24 Sep 2024 06:22:13 +0000 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssyln-00000001C7y-0kM9 for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:32 +0000 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2055136b612so61987335ad.0 for ; Mon, 23 Sep 2024 23:11:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158290; x=1727763090; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FxMRr8GFLSu76sONoxW9925XRbgCGcQv9BKiL/bcjLY=; b=NWjsdsSkynf+kBFnvA15h5j4t+i9DHo8+afv1bLRr3oNlq2+hE4c81dBkT1gmF7yN2 8BCP8uPqbmFTkE01gbR7V4/Lb/n4TiQ2GZQx4Y5w0j1rLDXBCFmj2GaniefGkMlk2sQc VtgOi+Nr304s6HwYhiiTCiK8N4YD4zu0K+uUCF8TyZS+ic6enpfo7k04SaPgWBSD2K+s TGQ6ZMd82UgHonk4JAJ2ZFFwRUzpLvSnBf2h1jotu5H3011GLnHQVqCycLQviZocCyu+ 7EEjIgbt3Z2gwrPzUWTx9kMgXErxnjt9UF7j+R02oFRd3oYbrSaFY6AB8dKrVRunkf/y dEMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158290; x=1727763090; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FxMRr8GFLSu76sONoxW9925XRbgCGcQv9BKiL/bcjLY=; b=nxqGp/jIUUAdtnuKlcf0SFk0dO30RyK3zfFS8ikDxIMpYDtPwaXjjEHMrGAGzYb3Rv xNlt/GTTSB8PqXXKkaZ063iWcphcfRRPSrkJGyipiE4H/HARviYOS2hCCVhRNCdAXv47 1pDrC9bS8BTonNd/yZPCt5Yfeb3I73MppraUAxwhHZMdW/gs5RIxYVqSo56o60QC9yb+ csRL1sU4P5RetIzeURR1naXvGxdgZeWZ6Skj5mqtB6dzPM5ENX9Juucq6SKEKEajiwhY QQV9HQyKJnaPvEFtV5SAcGtKjjt6KmtZE/5cFeXlLPPnUgDctoXeDZro+/fjCJTVe9k+ XbyQ== X-Forwarded-Encrypted: i=1; AJvYcCXpGJqyqcNVsVJ1KmrCvsJlkOqu3aAmGC2VPcdP12oK4318F1PjTLeB7jCYTXf3mmgbd6cAwwiox4yJIvKGxnih@lists.infradead.org X-Gm-Message-State: AOJu0YzZUwhWn7duzsdu9byPHVcRJBLUyVB8RSJnIU4Zs4oxLEByB1Ve elCWH9zT440tGtjz0MhxmuhYnEB8JNEAjywZC21d2USkQVNDYJqj0WVAo7ATZAQ= X-Google-Smtp-Source: AGHT+IGuU8VFNPNElLsd/e1qu4NWf2AelC2dd7GFKvCMDCyRlgITelf6+XB9J1CmvT70UykrnIRqmQ== X-Received: by 2002:a17:902:f542:b0:201:f70a:7492 with SMTP id d9443c01a7336-208d98603b2mr202721885ad.53.1727158290456; Mon, 23 Sep 2024 23:11:30 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:30 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 10/13] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:02 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231131_239721_CD419105 X-CRM114-Status: GOOD ( 14.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..6410f29b37c1b 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -67,8 +71,13 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } else if (!pte_present(ptent)) { return false; } + spin_lock(*ptlp); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, *ptlp); + goto again; + } pvmw->ptl = *ptlp; - spin_lock(pvmw->ptl); + return true; } @@ -278,7 +287,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +316,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Tue Sep 24 06:10:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 150E1CF9C6B for ; Tue, 24 Sep 2024 06:23:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=OfC1RlAMiynS5u2L74+wFKTmuv 63I6GJVprKkVsKqXqRTBlWid1L5EGEk8TeNqWznKbFOxAu15Rj7Oh4ct7q0u25mFx5GzuELoNDix5 1RDKeZeEC63Xqm2cvbM0o1f/BeDygRptXa+Wpq9IO8eUB/eM1MIdBtsFhoEo3yFAMMX27B2UWiX8M z4jJ17JaYM7t2k04si1TUL3VWn+lPfWqDXyCAl9UHmzKa3Z9e+0meP6NImQeh2EjKdfxZjLUZb3Ma ynUl/E3+qea9yIt2706RRBy1R1HszIp5bR0jA78dpHiw/Mc4ewtlsqirgqUA9mP8QOeqy0CjYNHDs inWpajfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyxG-00000001Esb-2O8x; Tue, 24 Sep 2024 06:23:22 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylt-00000001CBR-1Bff for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:38 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-20551eeba95so44279735ad.2 for ; Mon, 23 Sep 2024 23:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158297; x=1727763097; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=CzPGHGmRFqEGbK8h0wANbgJs5PRtl4UvTcG/TlXfG4R+Auh0gmtZl2TG761OpF3RO5 7uI8TwvEPxUmnJaA66GtncrmLM345QV1lZWuXIDUyJaDgsew1IXixt+E2Mn1KskooLiX jttVQJBUc7klIB7+GBWL0ERd0UzcBAycH5Tj9bBVA4zvjgZhoRlYvCU8AmGZE3nhscCs dAB1nMGwPDC7WC4VPjNWbenX7TBRH74wqQg632Ff4yXfwNGm05i1TxuQYkHqI8xvFEOa pWWUGl9kO3xpmOELpllxFG0ROCWeYeHpsoSiDW2BduxM+bWTWZ4EYUMG2507aQo96e1+ aZJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158297; x=1727763097; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=TOlKDUDbqCC/YlTf2haksyLlLoFuJbBvKMd7Vz+Dv09B5jHTubQ1mlfuu0nTfikZaT i8Va/yVmEb41HmPswDacZfRThpc3CgrRfIfrSLhPNb05MCaU16DiTXkQP+tUM2XedU3k JeJ0K1A5+4/nuaQHX6KoxkI6QNB062Akn2CSRlc1BcTKOwvitbCOAti50YewkAdvX4Bt VlbnFMJ/hhN5mmkocD90Yxy94aMJJzqZTRNOD6gQFr3O/+TrpsLrSjdpLEdrMvS6baqq FOVswtPd6Yu0Jgo1RqUY/BowckNEw8DkjlGfQYoiVufgN64ekcuZdIVhL0Idaj2tU9OX fbmA== X-Forwarded-Encrypted: i=1; AJvYcCVV4R2IGKTtE0b3vrmQP5u1L0OQ/ma/w8ExW36kUl+HeLftrHBp4inBxvOu6Ijjcm/FN6GNY6RHW+SOEgARgffJ@lists.infradead.org X-Gm-Message-State: AOJu0Yy6CXTR3k1Pbd2YTn2W+Y3/DGi1VlZSqpsCCh3sGbMYwyNlZuCS eQEXZsaV2v9um9uLyZI+SeV0yRVCQlcrV9vuDsSnFQ2dpfp5pAMCqaahFPJoJ3E= X-Google-Smtp-Source: AGHT+IFBI5Ned/vbTuuRRzdnv79MFASKoi0/m2fLpf4am0H6y8GNq4qW/RgfzVT9no9zcYiBkg/Xzw== X-Received: by 2002:a17:902:ea07:b0:20a:f013:ddb5 with SMTP id d9443c01a7336-20af013e29dmr19152265ad.59.1727158296610; Mon, 23 Sep 2024 23:11:36 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:36 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 11/13] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:03 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231137_345480_57BD93A0 X-CRM114-Status: GOOD ( 12.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will use pte_same() to detect the change of the pte entry, there is no need to get pmdval, so just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte = NULL; pte_t *dst_pte = NULL; - + pmd_t dummy_pmdval; struct folio *src_folio = NULL; struct anon_vma *src_anon_vma = NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Tue Sep 24 06:10:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF306CF9C6B for ; Tue, 24 Sep 2024 06:25:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fwq8DT6JeNdavhckGZI91GgKWXC+kbG6stmMQ1HgBAg=; b=mfgzeaMpfH36a26m2Pai8gfJc3 bJZQT2tTwR+30IyOms6xC9O+3Va9I2nlmDqzJvQtxZAb3m9/2eqWWQLPShIcypr9i9zd/XdRko27S evjDMCh4c01pRgGDTDmZWaI74Y9ZnsSBGWz9rRQMarG95frny9khzZuYqowKs9SwNfh5+6+QDTdeW mJjUd2gMfxb2/oc33ZnqjuYHqQJGYFqjKgbCtMbQ3VyAqQ6FbxJOkP6Qizd96rI1u5DgwhTBpoTxv thScvue4/V9dlT8mCVnw/uJzyHiy0NUHgAk+2KPdRoTekS380wiifGmMVJirJDAH4DaSrKEjs2Vwv 5sq/LpMg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssyzR-00000001FDq-3Jj5; Tue, 24 Sep 2024 06:25:37 +0000 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssylz-00000001CEm-1gd6 for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:44 +0000 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-20551eeba95so44280205ad.2 for ; Mon, 23 Sep 2024 23:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158303; x=1727763103; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fwq8DT6JeNdavhckGZI91GgKWXC+kbG6stmMQ1HgBAg=; b=NzdtXg3tm6bkQPoBU6x/QDbp1fRv8xChyo0i5VElmAOU73lnyY/+4nBhzmfzMLzE7G pAN9MpaSVOVKXyrSEEjRNP+XjpNWuILggZjYRdroOhT0BPy6q2v8eZyZaB+EUaLXNMkb h8767MSNVkCDCtkrUBmm2/47tGS0Gr1WPi+QIgIUpnl0mRsu/Qb65GvyNEIO8vlL9FFv eSGodDkFAB8ZYyJhRc2+ZLHos3WiFI+rjcSA7cdLQplTKnZT1vzOCjS0GCLm5PEgn4QV LhGMeTL99ZXfwmQB7E3D20np/4OEApDukPg49i+RqPcdWPhmzXN8lPsA3rAtuwzEbzKA GH3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158303; x=1727763103; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fwq8DT6JeNdavhckGZI91GgKWXC+kbG6stmMQ1HgBAg=; b=eVpPvnbD1FlSbG5C6cD56RbDKaPoiB8pxpm93M9jukS85tIX2vYUB2vHFo2v4QJGZX oiLjn9/3fHoP69xq5oSE0j4FnqOxK6FUIrtn6+tvPcqpiA8smBMmXJ+jW8EJoD0YQ0Ty XuogNq33yN2NmbQC6dK5yEatTh5625ahKjgHqUozsYHkejRGmJ+YvyqDLWAADVW5lrBU /HgayZBVRO7sQulWREzqqaPWlvvy+BErY5D8tFC3c2x/UJu4H5XUnejL7XDHRtlQ/4u9 O0ohMCT0qduH7eOjFIpxeB0VXCtG1VzWWHUKT6BZkloJvCFEAfI9hT8YpHm0HBGGDnvG H4bw== X-Forwarded-Encrypted: i=1; AJvYcCWi33MLd3R2AyMYe/lPgeDfYYyCRtWqWVtBWGmGa6fVK7BUK7qcm1CX9NqHpt494aSheXXLtp4XbTfvdYW8HqYj@lists.infradead.org X-Gm-Message-State: AOJu0YzyMMYwQkKWPEeK5KzTixz2JY4eK3H1lc0MjhSCjPj4Ie/78l/5 l1k4tZa/WNDscdAb9KZ2eN+VBIXdGjB3O6J0nDc81WvPKir1m//qBsirRf5JcAM= X-Google-Smtp-Source: AGHT+IFl5AsK8a8BNnydzmhf2ct6wAj0Fi3O6gM7MY+2JS5/M4Off4jZQYnq6Yk2iDK5ETk+W1TbBw== X-Received: by 2002:a17:902:ec87:b0:206:b915:58e with SMTP id d9443c01a7336-208d83af7b0mr191498925ad.22.1727158302727; Mon, 23 Sep 2024 23:11:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:42 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 12/13] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:04 +0800 Message-Id: <6efef5316d7468d13ed5f344452a0b872481972b.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231143_449850_11B912E3 X-CRM114-Status: GOOD ( 12.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 749cdc110c745..bdca94e663bc5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Tue Sep 24 06:10:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13810181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C625CF9C71 for ; Tue, 24 Sep 2024 06:26:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=e0T9lvI1pcCPfr6v8oRbivQAnqkZ5MJutHJ+/bNj+/g=; b=u7dLhOQds/RhTCFU2mPmNETZyb bgSd1XUy8lQ3XEbmNDtU1uuQUSFp47SZ1KIP3xtotDHBFcYsJ2mtF6oMYU0ozpA7RQmPHLazizFFw V77o3fKrxfE+Livij4bZ+DSnNApWVj78IhKP+FuBgi3SpCu0z7LdBAeFI34nMwUq6cl784FEGr5FG 1LgFCgXExFb2qtc98aIua6GdEPZwJeMvq8UCiSOBggbHoqjl5EUIHGBGenaOagp3vJkW+bVJRn3Bg Mz/4i9MMPaasdkajz7qNw27q7icgvSlKYIyaakiacyz/iRV3b/7NAepa+Xr8PXfjR412lYo7AHtM3 SrT6gV6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1ssz0Z-00000001FP3-0ggw; Tue, 24 Sep 2024 06:26:47 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1ssym5-00000001CI0-26Du for linux-arm-kernel@lists.infradead.org; Tue, 24 Sep 2024 06:11:51 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-2053525bd90so47138375ad.0 for ; Mon, 23 Sep 2024 23:11:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158309; x=1727763109; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e0T9lvI1pcCPfr6v8oRbivQAnqkZ5MJutHJ+/bNj+/g=; b=T0+2FsvZ2LFT8azDcXZEbzM6nR8qFOF+Trh4qXio0rdlYNhr0nnDes/OXS8IZ/QfaJ /l3OGUm4dvYKWzJ2vSWJ3WrlJlkKMlgqcCFpSTKTfCHtd10gxSbWBzIpX3uu0V/oeXfw SsR1/a76n7uOaz2cutIWZ/BDuQP+HboGRrn06enMzOkf6LCN9Lb8Jq3Hyfn0uprPho6Z picGsXy0eTHEJMAO0v0Yhhf2U1DBhXeDsf7OIe/BvKweSIW8DRqpWLOXH7MaUQlbFNtg 3e0mf44MwCaEZzBOFgey8Ym27Ijl5qTphnheq1yf+PkrjViSKcbqRmzyMN96EG5KL0e4 GhNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158309; x=1727763109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e0T9lvI1pcCPfr6v8oRbivQAnqkZ5MJutHJ+/bNj+/g=; b=dqE3X92syS+bW5eMoPPUnD2ZlCFKVm67+wdLkME0tHjG+qVOcr7TBxM+2frOs5mFUd 8EeSC6Xa8mb2R9gljtd9miFY/Tg74+CpmPEWT4Ai0pmsDICi2HpmeqVDBck0wvCYQp9p hQ1jWlmHSwUtVYI5GAa6Io7OG6vbr9XqS5HLI2jDDZ/iD/Lwyum+vCZzs+cr03wc3jFd pdJLc6w6Vt3rB/Lu2tBNvWGXEwIQJAgihWGcVauCKog/MxJr/QkOngSBF+HS544pTw+P ENOW60Hv6ifv5BR0RfmLgRMX2ecdaLtpS5ROQgb1Lkq66D2sh+kmxAbwTBoG4jpkLx1R hDhw== X-Forwarded-Encrypted: i=1; AJvYcCWoSF5xydJcYJtlkZAKRggGQo37T2yvkITHQj/M09sdYa2KXHPBrssLAJG2Gsf9PjSFo1ar8Esn9cJclf4lNN6Z@lists.infradead.org X-Gm-Message-State: AOJu0YyuzNTfHupup2pdEc9FERr3QGcEFQ1mSK1/deANY5UjKsFydrDf zn3HIVAgM1VN3vx68jMAgKOKfJKM4UByc3lS678N3XBf7Hw8Y3Mg51EFsEX1RsM= X-Google-Smtp-Source: AGHT+IGkJQOlzg3g752NhI9RquMqodCfvTZ7obYXRPzPmN4sDO5P6Za1vWoZJJ9nTCdoHXRNF+rvtA== X-Received: by 2002:a17:902:ec8a:b0:202:311c:1a59 with SMTP id d9443c01a7336-208d83b6ce8mr195135205ad.27.1727158308785; Mon, 23 Sep 2024 23:11:48 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:48 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 13/13] mm: pgtable: remove pte_offset_map_nolock() Date: Tue, 24 Sep 2024 14:10:05 +0800 Message-Id: <8eb7fcecf9ed8268980d0bd040c0a4f349cbca8f.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240923_231149_702289_5E7CF715 X-CRM114-Status: GOOD ( 12.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 9a4550cd830c9..e2a4502ab019b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3015,8 +3015,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 262b7065a5a2e..c68aa655b7872 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp - as * pte_offset_map_lock() does, but in this case without locking it. This helps