From patchwork Tue Jan 21 20:09:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13946511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D555EC02182 for ; Tue, 21 Jan 2025 20:09:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60DC46B0082; Tue, 21 Jan 2025 15:09:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BEAA6B0083; Tue, 21 Jan 2025 15:09:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 485DF6B0085; Tue, 21 Jan 2025 15:09:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 289276B0082 for ; Tue, 21 Jan 2025 15:09:45 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9936F1A0F7B for ; Tue, 21 Jan 2025 20:09:44 +0000 (UTC) X-FDA: 83032549488.25.8E1B03D Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) by imf30.hostedemail.com (Postfix) with ESMTP id B35048001D for ; Tue, 21 Jan 2025 20:09:42 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=sCq8l6Vl; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf30.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.187 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737490183; a=rsa-sha256; cv=none; b=wI6cdI/TOI8oJNGXfuOOnCMqhBNNDPDPgGT/kEL+26ZxOOxuCYigL4fauJRdwhjroeai2w TW9GRXWGR48W/2RJ98g/YzYUNdGvBW+5P2gIUCihmo+/r5512terW7wLKDx++Ox3+vcAiz o3j+JufCkV5y2S7BRyKKWo1RMKZqsVI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=sCq8l6Vl; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf30.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.187 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737490183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=7MsGtIYaWKOulbwuqadA1hPhFKScuzlkf6Zojvdyofw=; b=fBZNeycMxvqzjUR1/VQAjkigLfVViCVzDbTYfUhY9WEseO48pLeMGYt5UlTmS0qO2wZlwr LxfNekNgxKz8Sv6nZMKPUjI332CRyHhkzuBNe1YtW7v2LDVcIkBKxbqJTKYXIaeEatoX6r 775RhLDmvsyK+tD0kNG9H+WhHaoznmE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737490180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=7MsGtIYaWKOulbwuqadA1hPhFKScuzlkf6Zojvdyofw=; b=sCq8l6VlXiQ+Y9MvDQ05R6I1sl7MwDLsW1PD9fe5Ce5IQITIdW/tjnum91Sc0CC90YVzgx T5wrFIEFiywHonrhA82nF6Uro3G9+eEkylPuLA8yEWZp/oKvhevw1/aQ1xwor4viyzSV/W ofzOHQXA63oNzUfhj1vEBR/PzqOKl0Y= From: Roman Gushchin To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , Roman Gushchin , Jann Horn , Peter Zijlstra , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Hugh Dickins , linux-arch@vger.kernel.org Subject: [PATCH] mmu_gather: move tlb flush for VM_PFNMAP/VM_MIXEDMAP vmas into free_pgtables() Date: Tue, 21 Jan 2025 20:09:29 +0000 Message-ID: <20250121200929.188542-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B35048001D X-Stat-Signature: i8u9ytrymygd4qg3pnc9ek6oyj6mifxx X-HE-Tag: 1737490182-560205 X-HE-Meta: U2FsdGVkX19nW0D/F5d0BEDTi+psVHCZBaFDfNXZoeoWzinyvgPUiHCqr7hwAlcC+mlgh4zvv1kZC7356EYOoA+cMrIhb4RmxcdL5cWUC93nvm4ed1zGS39ydFrmEYZ5y/2S627BJPVf3+nQn3LmJlZCgxyNkMceUYn9UC00y4HViGZhwc9Or02D45+Mf4btsVvQF1LuG1U4LigSS6RuZQopVdxIWn3k61KkRQkMET87WoJNWGCfz9R5ZgEQrIZPyhiKQ82UdyVrfCny08rD35uO9wbkPQctNVeDCuC2K6508sdRRVZAXiwHQzK5PsgVNthzcHLANkd4O0SBGCecumFNW9P+0te3UMga6GZOQ6LTGw2w+h/p3Qx/3jEPYc/mcLBofdPhj1S+N+AxbMGER/fxigZbIypgV1oaGyK7ink7tTEHP3MYEqiBWn9U76TOYW9s2WVtY176aETb10x0sGjIbH6GGgPoRFz8yeUj2VWVyWh9UGBljF0iRv4J6mniCq9WdNikqq4Ehhtp+MIutkZayHZHWAz9JKC3NKd5XNoCXi77ojdUm9HAnfxm7WlcaqYI1Jav07ubfZDY6JJvnZUnkvKMAQwZcs5CBuNIga742TilV5k2QnT8v20OONkWb5N41EJ9mBLWcKzuwB7ZkES5D2F3rB74WMDWibX4xBw7e9d7wZRq39Hm9k4AXrpxIipgkLGaDsITwHpCuvj/VXgu7Aqo0eMZd7rwSjTSGEy1RmzlQA9clKFfXSaJb5+24VLfNNEOkXrtApdIjqyQcH6il38u20klEDkmlM5L/mnAKkgyTiAkBmrolzw1TEs7FJw32NYddS6YId0fEMTUMKaMYP4NLxMvDtsmnmHjH7Q+tEZDHYJbTDClTcayXqCKabC6OfNUZNKyqGKIh/VGzzdt6djx04AKsQkKCjxrrtTSifRNEACQKiP1fBRbzUou2wTIceJwzxxa5K7JXKl fxFU9ifu 0MJ6sk3Y2v/faIRCmqw1oTlRy8P+Hd/8ZUW12Rd2YOf5rfu0rWU9H2cdWN+KdNqryAiK/EnP++s7Owsg7mexG1FnQIwYeeZVjHbJ6NFXz++Hz/F1uBrxPOEY/R6OkIYT6uBxmW81MouAj4ULD3CRRUll40yZ9s6Pt4TYWd9z77VFsUqbVS1nunWWEpE4imux9pcqzVMtRXW83insIhF60p36Qkax+6XGVB/eD/jslaeMPd01tPUW3z/tQNz+snirxhyqF0G+iLjEbfqRKSorM7SX3iVs3/2lU2sWT86rld4HNg2qMlOv1V2IJwowoKdwYXar1huKgrNmRyDQ8lyzs30sL6IDzzcy6avLMKWRLOtDnZPgvieYLrYWjEjlQxQnwSlZznmR83EkG0vs69xv89eRb+LDXbq5fVGdpeVOy/gBqRRpBpBJz9CQQUxT74Mn1dD1XYAQhlts7My6eQj3f0qJZAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit b67fbebd4cf9 ("mmu_gather: Force tlb-flush VM_PFNMAP vmas") added a forced tlbflush to tlb_vma_end(), which is required to avoid a race between munmap() and unmap_mapping_range(). However it added some overhead to other paths where tlb_vma_end() is used, but vmas are not removed, e.g. madvise(MADV_DONTNEED). Fix this by moving the tlb flush out of tlb_end_vma() into free_pgtables(), somewhat similar to the stable version of the original commit: e.g. stable commit 895428ee124a ("mm: Force TLB flush for PFNMAP mappings before unlink_file_vma()"). Note, that if tlb->fullmm is set, no flush is required, as the whole mm is about to be destroyed. Suggested-by: Jann Horn Signed-off-by: Roman Gushchin Cc: Peter Zijlstra Cc: Will Deacon Cc: "Aneesh Kumar K.V" Cc: Andrew Morton Cc: Nick Piggin Cc: Hugh Dickins Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org --- include/asm-generic/tlb.h | 16 ++++------------ mm/memory.c | 7 +++++++ 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b75..411daa96f57a 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -549,22 +549,14 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct * static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vma) { - if (tlb->fullmm) + if (tlb->fullmm || IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) return; /* - * VM_PFNMAP is more fragile because the core mm will not track the - * page mapcount -- there might not be page-frames for these PFNs after - * all. Force flush TLBs for such ranges to avoid munmap() vs - * unmap_mapping_range() races. + * Do a TLB flush and reset the range at VMA boundaries; this avoids + * the ranges growing with the unused space between consecutive VMAs. */ - if (tlb->vma_pfn || !IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) { - /* - * Do a TLB flush and reset the range at VMA boundaries; this avoids - * the ranges growing with the unused space between consecutive VMAs. - */ - tlb_flush_mmu_tlbonly(tlb); - } + tlb_flush_mmu_tlbonly(tlb); } /* diff --git a/mm/memory.c b/mm/memory.c index 398c031be9ba..2071415f68dd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -365,6 +365,13 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas, { struct unlink_vma_file_batch vb; + /* + * Ensure we have no stale TLB entries by the time this mapping is + * removed from the rmap. + */ + if (tlb->vma_pfn && !tlb->fullmm) + tlb_flush_mmu(tlb); + do { unsigned long addr = vma->vm_start; struct vm_area_struct *next;