From patchwork Thu Sep 26 06:46:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC639CCFA07 for ; Thu, 26 Sep 2024 06:47:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D66A6B00AA; Thu, 26 Sep 2024 02:47:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 486EC6B00AB; Thu, 26 Sep 2024 02:47:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34E866B00AC; Thu, 26 Sep 2024 02:47:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 144456B00AA for ; Thu, 26 Sep 2024 02:47:45 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8AFA3C08C6 for ; Thu, 26 Sep 2024 06:47:44 +0000 (UTC) X-FDA: 82605958848.02.06B9075 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) by imf01.hostedemail.com (Postfix) with ESMTP id BFF024000A for ; Thu, 26 Sep 2024 06:47:42 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=BL6Gv5VW; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.160.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727333226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=EQNY8yTZOcTDHgwTKoRBlcBws4lmNKQuKYflql/gRaRXJzTmD2GmlM5tnmnWZQdq0plfU7 99zwsA+dws+c5sIHcOaT0Bzo1DQezIr+/yXom1pfZT3YJ6nEpqFE/najE+cbOagx6AQEIZ csiu7yraIf02ZNTLOwJQQbYP7RgaqSo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=BL6Gv5VW; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.160.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727333226; a=rsa-sha256; cv=none; b=qZJVs6YHQBYkj2Jdz7/G/kzWPrFNYx3+a/f8jzHKZ14nZUWxIrkfAwBcq9CQ+pdB7zS/OR H3HzGoz2H7mL42dacHN8kn4DZnUgWdm8nqMheLcpR8a3aorFZ0uCVo1gK6LU5Qak7cnu+W KoIdFqwvMb/HYGGs6J8jKieZiRKrt+Q= Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-458320b30e7so3909191cf.0 for ; Wed, 25 Sep 2024 23:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333262; x=1727938062; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=BL6Gv5VWUNAD5G5nxV2rlGzSJGnEl2drmvi0VDgz0CrH20QRKbPj8gMp6Y3YgufJoG 3dGmCNS5LMZSJvPUnUX4CgUNGlbp8oX0DXLG7AfiFTxhpX6iSuuMB4Jc2JX6kG72S1d8 TAV2DPz4JOUvMHc4KTaXNWU4cGaMNcr9ObVUn/OgANn81Nea6rT366ydpXKNTLvWqbRK 0mO9ilxogKM9g3hSmBqN7YVTmBs8NFoNmT4w1dfImly1L8JZLevioWmcPKyyV6OTT9qe f7Hu9xl6A9Vdnj2usgfqMlnrvmCR8JRipe2/QC5rj9VE8B6Rwe4xuPqmeX7o003YZ4ye edVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333262; x=1727938062; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=qIYvhvWB0Mnmfj3d2tYFfsKEuh/qR7mpOzIA4Yr608QhALHCorJcknUKRv9utw+luc eHeajtg2KqnoB24FCLNG3SuyseWN91pNihF6/QVcQqnp3327lbytC0sl/HTT+cNRhhm9 esclQ34ICi91SL1b00j949WkQ0FMqdj5Qwwzaxkz1S0x/q/RRGTt4VtGJY4OAAmChJ6g 0qOomkPqpPDe3vWEASaUR+HjVJLtpaK6d9DwNyMijdqT12/hGvg6I4GcR7uoyK/0tVn/ ILIuSVOAJfNFSleBFknpPw60kQS7+uWwUb/lVbI/N6nsBN6vQum6Fj0P36JLCaogBI9m l2og== X-Forwarded-Encrypted: i=1; AJvYcCXd5OsOoRvbaZcK76HW6Uu4McGULaqZywVfsCXO/9yv/0TTWvhdcMKpGiIrqCBnL71nkbtHT+hf4Q==@kvack.org X-Gm-Message-State: AOJu0YzlxGOYfVB2KP5B1pPLiUeLFq7x0RPyNdauk+l1/m6kf8EZ+tjE jzId91pL7vHAvq35DtHzkcUblvall0SSQNsqDbgjv+MB+7Fwbtv46ClzF+88BRmWFSPUQ/MpjME 8 X-Google-Smtp-Source: AGHT+IFPPE2WMfZ0ye6LGa3/R8zXQ/6KjVtS14uw6772ox/YEy7cH2DCZDAwEci91jHXPd4jVDRFUg== X-Received: by 2002:a05:622a:1493:b0:457:c747:cf3c with SMTP id d75a77b69052e-45b5e0534ebmr99370671cf.57.1727333261763; Wed, 25 Sep 2024 23:47:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:41 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 07/13] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:20 +0800 Message-Id: <055e42db68da00ac8ecab94bd2633c7cd965eb1c.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: mi8k8yofp766knua7npkrpmh31mar4fi X-Rspamd-Queue-Id: BFF024000A X-Rspamd-Server: rspam11 X-HE-Tag: 1727333262-23894 X-HE-Meta: U2FsdGVkX18fPxGeZgOuZacLH9JKk+YWv6PlqVbhcu1sMyfnBkPn4cyC9Cs6gMklZn7+jNryftnxgQPfn9TZTXhDwrWBPXWM7Ugf4QeFc0G+hxHKAiKjPY0s/0SkVOe7UCrIv3bCZE0Q27VFs40jrCBBXVejgGlb5TwY8k4qXLSOAd0Kr74C7MkfQttzLDVIB+/LO+5GsDzPMciCKfN9z/HIHHPUryA3Yh0jEiEV81KAJK5JnHh8mOUJqIMvn5SKxXOm2qsnmEg/gv0VTOZn9tjU70JG+httyJNhq+VhTHB5nhNVVSbZpdCKp7tTsUovc23Vh8cxqbNsJu4DYu9RqYAigBwNo90na+rI8oc75O61/JxcN7HmBR+4lvU3+K2XGqDgNwJDJQEeX50RPSJu2+X/f1p3JoVNXxNW2amail9pv/Y/yF2LUCC8FVdTzkJ8tGfDfAY/UIr1Ht1IdIQatJVG/Goupm9mtcS3H310Ej74t9i7arvRuc0rHMKRr89ImPKlvuONL5iDFyv/f97xgOBekmy9HWPODpOBFG3y+Mrz96X60o3jPgcES3khvGcTu5ayQrRaLcmBYCLimvYRAvTZ5SH42LdXZ61AlRWhuELFwO9sd3LmN5gqB2rF8DwRk3ztnTTCf1kVdh8gKR7Q+wlL8hOfA1Cvc9lxF1GrSR2iKpN9RCyCResUfZjn1i6YuWdC5h39UQADaLVD356ZtNucud83BELW+w6HFjn1kr5h0ihwzlH/qpuvrI2nzz87uHY7y8swgQeY5SvDojerprkb1dcGsa2PzCU0j6GnxPW2AzAjmYMjtr0Y/0BtswwvpMVw/aWvDe6C4lenJTdRNdSDlzU22+R/2Hx2Kjq7IZ9Zo9HUV6I+7dkwGrULPaNaVjUpZYV1RKFOwNMrfUpwh8eYGM+hHPJR0JAlchWZKTnUEgU/ZJtq95dZx6R3X0MiyXGRdx7Cz7mBjWG04/n ve4Q9oJu ePagtuxvvuz330W4dmaxndnBANly+SS1+4TdCAHcjAF//ec0cJp4wG8kjR2lMjarWFAhlJOrjgeTtW9jyvjUG6QxfytwBbHK19PGvoBU8rzXvIsJaoKDWq/fP3D5OyQ+96G4fIgZ5kuwwm98Lku+yUHoGuseiC385CAerMu1JSJgtYZ7FM1PZZIm3neFisvx3YkznPLHC03SX4M8c5zjJW5JAJGr75nfqA+2FGvY8ulHvqehyAQx/fNz/XiOmSLnESpG15RvbB8q4h2cdQCegJh7MDC5Y07A6/0rj8QtTn7VfjYq6zfdvKKwltdjqL2Sihn/26BocGzfbb8/Z6LU8i7q526xTlZ2IsgYLjVlsVOpi1/5TVo2f3g7vh02sMYS+G+XBtEATJkRD+BQYh0+Jx4IWkuUyDPXfDLS35RlXzVV6mOw8yu51Cmu1dg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/khugepaged.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..854577f39957d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1645,7 +1648,6 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, nr_ptes++; } - pte_unmap(start_pte); if (!pml) spin_unlock(ptl); @@ -1658,14 +1660,19 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); - if (ptl != pml) + if (ptl != pml) { spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + flush_tlb_mm(mm); + goto unlock; + } + } } pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); + pte_unmap_unlock(start_pte, ptl); if (ptl != pml) - spin_unlock(ptl); - spin_unlock(pml); + spin_unlock(pml); mmu_notifier_invalidate_range_end(&range); @@ -1685,6 +1692,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, folio_ref_sub(folio, nr_ptes); add_mm_counter(mm, mm_counter_file(folio), -nr_ptes); } +unlock: if (start_pte) pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl)