From patchwork Wed Jan 22 23:27:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13947708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 344F0C02181 for ; Wed, 22 Jan 2025 23:27:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3976A6B0082; Wed, 22 Jan 2025 18:27:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 346DA6B0083; Wed, 22 Jan 2025 18:27:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20E036B0085; Wed, 22 Jan 2025 18:27:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F20106B0082 for ; Wed, 22 Jan 2025 18:27:32 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A1882160967 for ; Wed, 22 Jan 2025 23:27:32 +0000 (UTC) X-FDA: 83036676744.29.AEEBBB3 Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf25.hostedemail.com (Postfix) with ESMTP id A7965A0011 for ; Wed, 22 Jan 2025 23:27:30 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=qduLD1F6; spf=pass (imf25.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737588451; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=/gVgCDajiZ5FVHubL9gu4gQZ78OrbIDAjdC1qsGbA80=; b=S1zSNQrAcNCMYmexEcCihNAgr0iaX2DM3WU/u8P8wf5E/m9swSRdC+qOIXodcun62dFFfa AEK2+ASHygpL1T+Fi05+nMwhBTsGWqvQgzmf3S0InlFi2z+mc79AC5vm0v5I6pgTAXk0AI ry2miBg2oaDZehDYECDu/yUs7pHfmxw= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=qduLD1F6; spf=pass (imf25.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737588451; a=rsa-sha256; cv=none; b=8Ghn4JBDy2iqyfzqBQKQEpX+U8a+2p0Z7oukcbcWj5UElo9aJ8RGrvnJGNw47STGSzIzjo Hyim0L+L0yA6QYzefPQTE6oMED/z5z75I+/d6NqSm351XDUfyDg29qsro9BgDJJqYWoGd6 OCOf+jNZ+xuM69Hz5NqyOPmdhNBf9kI= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737588443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=/gVgCDajiZ5FVHubL9gu4gQZ78OrbIDAjdC1qsGbA80=; b=qduLD1F6WnVvqtatJhvmCJmkj/mC+ztYelFTH2KHW8h0JVkwzncD3WUcR5YeCmw9bEXgjt YEDtM9zHnABi1Ao/95Q2orkpTyhiDvC5J1j8MWIGBq2Ib4t7c7sIAtCGC7+1w/xN95UooT q5wx7vOfb1RU14RWBAPoloMznH/YUdw= From: Roman Gushchin To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , Roman Gushchin , Jann Horn , Peter Zijlstra , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Hugh Dickins , linux-arch@vger.kernel.org Subject: [PATCH v2] mmu_gather: move tlb flush for VM_PFNMAP/VM_MIXEDMAP vmas into free_pgtables() Date: Wed, 22 Jan 2025 23:27:16 +0000 Message-ID: <20250122232716.1321171-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: A7965A0011 X-Stat-Signature: j7tijh1ddkgmnp6h8tiegtiycbasfwpg X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1737588450-951638 X-HE-Meta: U2FsdGVkX1/HIw2MjHpqfMj2Qi+5u7SMYwi+EiMsUkGA+MMLJl/1QyF0stkMzBDI3t/u0mf1jpN2Jt4vY0vhRgy1U6iJFnpO9zI4y5xKJjLfg7puDN7Exzh6PdL4KP9c7kK5o7Eetlxy1eVGpisHDkJIConyhn5kEmhuuI9uPFmxOCRw/TsM0iLEUfjQkUvZQEkF2W7biKKvEWZ38ozycqmSTWXVTxoZD/dx3TCwu4w6Ttad08wShFwhYEwD5NZAumKDEgluyCVB18/SSHQlJRW7NeOwXvPPWYPHh5zyI84gIqmpgCbftmHV3wZmMuqqx3ooVRHO1maeOZrLl9nwIUT9PYBUbGtqzTRUqWcoL4JZkiEp/J6BPso4h8S42Gg/Cp3cNSFwX8f5g12pLwYZL2LCRanyhWb0ZVcVPaY7CB04NnUq6kMw67s6VzY2p5RYgGD+5Y37zM1IYA3PfkW6GARfNxHDYQIZjASSwhXG9zQyfnKk6PDlOVhp5jgdenkXwF2QnbPEFPVm+YMiv6Sc1+6BRQqSLkaVkic1Hvixv/sdCCVcpve2Prhe2/kDEwzaVdD7DfVY5lMSqHRD9fGlTGE4ddX0DePD+9bRRvjY37gl4k+5wKrmcvmIFeXfDnltZaT3tXgAFg54HEO5HOMLF61KsrY8KoOkYdTwE8WA4LJozfLPYPb65OEcEov3glEuXK2XUYZiE4rRJtN/miEklkXdkJoTX88V6prrSnR49WCHuY4NF3/ldR4lRTa50JpznwQ4Z4i5DYwT4texlDMYOFFdSC6bEusoNXpWoiaz9HvStDnH4LrPnctu7+ILhzcupZJhA3ygQ4O/1TLFjwciNDE7yW24wLlaCnymjn5QktFu5vTDSFVac8bqvo2knh3sAkvY40HXfxbXMdXO0kV9TnXqrit9Iky5ZhvZ+Im+4OACVsRDAw/HGDx+8ZiPVefNdmhhOdKfZsoTKTNdd6z 9nm6dk9Z ZOC7tESca6l4Ryh0n2IVSSA66C8QPmmijv0Y01JBi8EDhQ6cKcf21i5T02d+RionWNHcDH7DhgLtjKGpUBM+vf1sEF0VpMtfQtOfBI3ljobK2cHCZKFKc4sddCmKc2XgVR+cKuUzpW1iqsZg1A1WccZQlA/Xw41yb2OiTwkhPxkhW1Ky2TPHyGcDoqco2AjYC7pnDybljseWFZ6MKchRjlfHnR/0QobVLOnCnQsyWc6EXTiby00mfIDll86mshbTQ7Se/uBwpyn/m4H/TVkUqLh4If0pQtS8ppWLeefpQpYPIhPO5MFBS7pisdkX0WzUolqFBcm9cL7Adw5jey85cR2tq82e+YCDq/Z+7q3hCEsBmHVPdZRirrV9UQofP32Zjrf07XIx9gADfr9sepxYhnAwC/shT31biWPNiK8XReQIvAUseUHm0Y0UTMC4nxjjtD3vX0abnnWfUtSxfSrjMjFC6iQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit b67fbebd4cf9 ("mmu_gather: Force tlb-flush VM_PFNMAP vmas") added a forced tlbflush to tlb_vma_end(), which is required to avoid a race between munmap() and unmap_mapping_range(). However it added some overhead to other paths where tlb_vma_end() is used, but vmas are not removed, e.g. madvise(MADV_DONTNEED). Fix this by moving the tlb flush out of tlb_end_vma() into free_pgtables(), somewhat similar to the stable version of the original commit: e.g. stable commit 895428ee124a ("mm: Force TLB flush for PFNMAP mappings before unlink_file_vma()"). Note, that if tlb->fullmm is set, no flush is required, as the whole mm is about to be destroyed. v2: - moved vma_pfn flag handling into tlb.h (by Peter Z.) - added comments (by Peter Z.) - fixed the vma_pfn flag setting (by Hugh D.) Suggested-by: Jann Horn Signed-off-by: Roman Gushchin Cc: Peter Zijlstra Cc: Will Deacon Cc: "Aneesh Kumar K.V" Cc: Andrew Morton Cc: Nick Piggin Cc: Hugh Dickins Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org --- include/asm-generic/tlb.h | 41 ++++++++++++++++++++++++++------------- mm/memory.c | 7 +++++++ 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b75..fbe31f49a5af 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -449,7 +449,14 @@ tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma) */ tlb->vma_huge = is_vm_hugetlb_page(vma); tlb->vma_exec = !!(vma->vm_flags & VM_EXEC); - tlb->vma_pfn = !!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)); + + /* + * vma_pfn is checked and cleared by tlb_flush_mmu_pfnmap() + * for a set of vma's, so it should be set if at least one vma + * has VM_PFNMAP or VM_MIXEDMAP flags set. + */ + if (vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) + tlb->vma_pfn = 1; } static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) @@ -466,6 +473,22 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) __tlb_reset_range(tlb); } +static inline void tlb_flush_mmu_pfnmap(struct mmu_gather *tlb) +{ + /* + * VM_PFNMAP and VM_MIXEDMAP maps are fragile because the core mm + * doesn't track the page mapcount -- there might not be page-frames + * for these PFNs after all. Force flush TLBs for such ranges to avoid + * munmap() vs unmap_mapping_range() races. + * Ensure we have no stale TLB entries by the time this mapping is + * removed from the rmap. + */ + if (unlikely(!tlb->fullmm && tlb->vma_pfn)) { + tlb_flush_mmu_tlbonly(tlb); + tlb->vma_pfn = 0; + } +} + static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { @@ -549,22 +572,14 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct * static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vma) { - if (tlb->fullmm) + if (tlb->fullmm || IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) return; /* - * VM_PFNMAP is more fragile because the core mm will not track the - * page mapcount -- there might not be page-frames for these PFNs after - * all. Force flush TLBs for such ranges to avoid munmap() vs - * unmap_mapping_range() races. + * Do a TLB flush and reset the range at VMA boundaries; this avoids + * the ranges growing with the unused space between consecutive VMAs. */ - if (tlb->vma_pfn || !IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) { - /* - * Do a TLB flush and reset the range at VMA boundaries; this avoids - * the ranges growing with the unused space between consecutive VMAs. - */ - tlb_flush_mmu_tlbonly(tlb); - } + tlb_flush_mmu_tlbonly(tlb); } /* diff --git a/mm/memory.c b/mm/memory.c index 398c031be9ba..c2a9effb2e32 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -365,6 +365,13 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas, { struct unlink_vma_file_batch vb; + /* + * VM_PFNMAP and VM_MIXEDMAP maps require a special handling here: + * force flush TLBs for such ranges to avoid munmap() vs + * unmap_mapping_range() races. + */ + tlb_flush_mmu_pfnmap(tlb); + do { unsigned long addr = vma->vm_start; struct vm_area_struct *next;