From patchwork Fri Jan 24 23:54:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Yznaga X-Patchwork-Id: 13950032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0169AC02181 for ; Fri, 24 Jan 2025 23:56:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D7BD2800B8; Fri, 24 Jan 2025 18:55:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 085DA2800B0; Fri, 24 Jan 2025 18:55:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7E562800B8; Fri, 24 Jan 2025 18:55:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B92582800B0 for ; Fri, 24 Jan 2025 18:55:58 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7572DA1A02 for ; Fri, 24 Jan 2025 23:55:58 +0000 (UTC) X-FDA: 83044005996.18.CF1DAEE Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 3F7BF180008 for ; Fri, 24 Jan 2025 23:55:56 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=QeFtJJdY; spf=pass (imf06.hostedemail.com: domain of anthony.yznaga@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=anthony.yznaga@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737762956; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OSIj2mJijhCNko8dcXyJfuTUdjY7NtJAOi00OHXcvao=; b=vaK5q1doCkbEvEP+/jPQxj+KTEpsuEji6Po3rcDA1FLvDpsbV9qeeKI8fJ87Xm/W+yS6zu TNJrSFlWGSBk2FeL5Q3totA3N8owLw/Z3Fo1a7IB+UcLSOItQxDQDKLgjwIMQgt2UvSKhd PtnbTUzt9v2vTqGgWNKHsDG86szaXLM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=QeFtJJdY; spf=pass (imf06.hostedemail.com: domain of anthony.yznaga@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=anthony.yznaga@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737762956; a=rsa-sha256; cv=none; b=AzWE4Uf+FE17JtpPu/6FZK61cPJgB0PjS6AedzZ3RJrzfsnCFFiyOURq4zcFsKZQ/pHIRm MDlhK2MrbiFzJUY3HS50VqFZbDJlA2IFAewlRDYJgIUw+3NylNsGlmuXCtPTB1WN3uK5bQ +v/me8n0ZIg6RVwyy6JboeopBAZOzCE= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 50OIExIp022002; Fri, 24 Jan 2025 23:55:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=corp-2023-11-20; bh=OSIj2 mJijhCNko8dcXyJfuTUdjY7NtJAOi00OHXcvao=; b=QeFtJJdYB0wtvXXNjl/QM MPDJrbyYUVLf9oTAT9Q1GPOvNcwMRB/wQjVpDMKhajG22HNnFiLBqaUY4k9lk+Ch p+HSZLNHXxr6eUHaE1S7kk8q9oFnK7c0bR3w9v/pxvPddRrPLiJ8EFvdkZ+MksWF kkNtUZMettwjyChukyChxUmO4z5Z6wpnm0a7epnj4d56SjuEL1ws3QX/AiXktMNC kWNWqqyzD5/dcni/8xsWCkA+gvfBNoHQTEyIJWLR4DRHSIgRunNbVBi2Zx0tqZ+x ony2WUw8mZyEiA/55WfKntKeHX9zSZgPpASTVJgawqgXcW31dZtBBWMWuOFY2Ra+ A== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44awpx5vpy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jan 2025 23:55:43 +0000 (GMT) Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 50OM0FXi036437; Fri, 24 Jan 2025 23:55:43 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 44917u4af9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jan 2025 23:55:43 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 50ONsxQ4018051; Fri, 24 Jan 2025 23:55:42 GMT Received: from localhost.us.oracle.com (dhcp-10-65-130-174.vpn.oracle.com [10.65.130.174]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 44917u49ww-12; Fri, 24 Jan 2025 23:55:42 +0000 From: Anthony Yznaga To: akpm@linux-foundation.org, willy@infradead.org, markhemm@googlemail.com, viro@zeniv.linux.org.uk, david@redhat.com, khalid@kernel.org Cc: anthony.yznaga@oracle.com, jthoughton@google.com, corbet@lwn.net, dave.hansen@intel.com, kirill@shutemov.name, luto@kernel.org, brauner@kernel.org, arnd@arndb.de, ebiederm@xmission.com, catalin.marinas@arm.com, mingo@redhat.com, peterz@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, tglx@linutronix.de, cgroups@vger.kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhiramat@kernel.org, rostedt@goodmis.org, vasily.averin@linux.dev, xhao@linux.alibaba.com, pcc@google.com, neilb@suse.de, maz@kernel.org Subject: [PATCH 11/20] mm: add and use unmap_page_range vm_ops hook Date: Fri, 24 Jan 2025 15:54:45 -0800 Message-ID: <20250124235454.84587-12-anthony.yznaga@oracle.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250124235454.84587-1-anthony.yznaga@oracle.com> References: <20250124235454.84587-1-anthony.yznaga@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-01-24_10,2025-01-23_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 mlxscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2411120000 definitions=main-2501240162 X-Proofpoint-ORIG-GUID: MwcrHgPHqQLpE3U3E5SYpIbs-4Wi_R5L X-Proofpoint-GUID: MwcrHgPHqQLpE3U3E5SYpIbs-4Wi_R5L X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3F7BF180008 X-Stat-Signature: 6xsm6s7cuupus9kxuzeo6begdr4uaeiu X-Rspam-User: X-HE-Tag: 1737762956-449340 X-HE-Meta: U2FsdGVkX19QWapvkHFZlS43C5M7p3M1KsSQVXLxdfIbwNyVs4dQ5Px705Z2kIMg7BCLxdE8zxWwTXhFm7mH4tC1n47QwwyWHFTS7ZFex+q1GGdYzz5QvM/jENv+sX5iCXgPA60DlWLPbmkSCQVwXfXez0uojpfElFgfkjsd4MEOq4kmteuzVH4bx5q/LTDOEgKqGzLgUZZ5XjrhmRXZVFn/m3PV44j14zXQpnsZ9XYx5XO/6aQzktmOzH1lMnBFnPnk3kRy8jsSHr/jX5+U4ZUNAhuNiFJ/PlnwyV9CFIonYOQdJAkZEyv79QjF2Ri85kMVpMrk/T3SvwR4afKCm+o+x7wmKg9DYkW1zFhZJJ1A0heuwsy/3H88aVQwoZWqP7PcNVmvzgymwzRD96jfYLkT0XLVVuxXGg9adjr9JXHD9Y4cAM/jO1Ar909cUttlGG2wgyf86jxsb7pVmuWzc2kWYq+EHg7B7DKgP+EutP95bMWtPS+afSXYFIezov6Sv3FaXZcvQhzom88ICDGZvoD6vhXlSlzeW0KkpE7OEdHePCEcDsMJFd/EMLPNEmL9OmSYs6lYei3uyudSqhtqIBbqWowD0qcVerBbFLduMrj9Kuj/Wx0h5hTxv6emVk2hv6M8ZGs5YnndkO5pchvGdEmbN7DFLDwaGINcw6ZJp3BNjZeYVhbeHjx9jBNxGC5Eo/JGZcAh1y3DugiabQIlgtujKizSZQn47WQulUqwfrlsbaHN0d3Fz7U3tGzjk644FoH70/6O9GZp4Tq+wywp4VQHy+2xhSXDz1vBbAADVPnViqLjw14059npKAUOs2Q75PA2xvEWp7YbiuBVsE/47bVpj4+o6LG5AOqjBkevw2+EU41drIhiAwYBKLf7juZGjmTUec8Y88lQtqCGpe9+gPQGZejkDMFCL5giD648TSbS5NFjUc60ARmFPS9Y2+NGSnq8iypWDA4cZh2f1B/ qVCnYc67 r8g7BXjx/6nz3+pOTNQxE7CPIaW3/r+rSbrvzhpFIYc2HUMq3FpBJuTHdc+FCrhYrWrkCBe1ouobaSG6PlfAFV9ZMYLdUC0VsFY56gvZbCFIknEipwYNUs43lTG8OMpSrRK7BIptG7nYpk66vEopVz21qnDXoeJeTOXpyqRZqYcsXncl/FNUpbQjM+m5Rx6c1DKqH8zMNMwVFBRYCCL9bWppn85LC02hbmVq1+DIScBb9RWdQDQ1SCgFA52G0/ejfBNhfMV4ytSGOlpTvwv+mOonQAepHhXlVdKuD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Special handling is needed when unmapping a hugetlb vma and will be needed when unmapping an msharefs vma once support is added for handling faults in an mshare region. Signed-off-by: Anthony Yznaga --- include/linux/mm.h | 10 ++++++++++ ipc/shm.c | 17 +++++++++++++++++ mm/hugetlb.c | 25 +++++++++++++++++++++++++ mm/memory.c | 36 +++++++++++++----------------------- 4 files changed, 65 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bca7aee40f4d..1314af11596d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -39,6 +39,7 @@ struct anon_vma_chain; struct user_struct; struct pt_regs; struct folio_batch; +struct zap_details; extern int sysctl_page_lock_unfairness; @@ -687,8 +688,17 @@ struct vm_operations_struct { */ struct page *(*find_special_page)(struct vm_area_struct *vma, unsigned long addr); + void (*unmap_page_range)(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end, + struct zap_details *details); }; +void __unmap_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end, + struct zap_details *details); + #ifdef CONFIG_NUMA_BALANCING static inline void vma_numab_state_init(struct vm_area_struct *vma) { diff --git a/ipc/shm.c b/ipc/shm.c index 99564c870084..cadd551e60b9 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -585,6 +585,22 @@ static struct mempolicy *shm_get_policy(struct vm_area_struct *vma, } #endif +static void shm_unmap_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end, + struct zap_details *details) +{ + struct file *file = vma->vm_file; + struct shm_file_data *sfd = shm_file_data(file); + + if (sfd->vm_ops->unmap_page_range) { + sfd->vm_ops->unmap_page_range(tlb, vma, addr, end, details); + return; + } + + __unmap_page_range(tlb, vma, addr, end, details); +} + static int shm_mmap(struct file *file, struct vm_area_struct *vma) { struct shm_file_data *sfd = shm_file_data(file); @@ -685,6 +701,7 @@ static const struct vm_operations_struct shm_vm_ops = { .set_policy = shm_set_policy, .get_policy = shm_get_policy, #endif + .unmap_page_range = shm_unmap_page_range, }; /** diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 87761b042ed0..ac3ef62a3dc4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5147,6 +5147,30 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf) return 0; } +static void hugetlb_vm_op_unmap_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end, + struct zap_details *details) +{ + zap_flags_t zap_flags = details ? details->zap_flags : 0; + + /* + * It is undesirable to test vma->vm_file as it + * should be non-null for valid hugetlb area. + * However, vm_file will be NULL in the error + * cleanup path of mmap_region. When + * hugetlbfs ->mmap method fails, + * mmap_region() nullifies vma->vm_file + * before calling this function to clean up. + * Since no pte has actually been setup, it is + * safe to do nothing in this case. + */ + if (!vma->vm_file) + return; + + __unmap_hugepage_range(tlb, vma, addr, end, NULL, zap_flags); +} + /* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops. @@ -5160,6 +5184,7 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, + .unmap_page_range = hugetlb_vm_op_unmap_page_range, }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page, diff --git a/mm/memory.c b/mm/memory.c index 2a20e3810534..20bafbb10ea7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1875,7 +1875,7 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb, return addr; } -void unmap_page_range(struct mmu_gather *tlb, +void __unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, struct zap_details *details) @@ -1895,6 +1895,16 @@ void unmap_page_range(struct mmu_gather *tlb, tlb_end_vma(tlb, vma); } +void unmap_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end, + struct zap_details *details) +{ + if (vma->vm_ops && vma->vm_ops->unmap_page_range) + vma->vm_ops->unmap_page_range(tlb, vma, addr, end, details); + else + __unmap_page_range(tlb, vma, addr, end, details); +} static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, @@ -1916,28 +1926,8 @@ static void unmap_single_vma(struct mmu_gather *tlb, if (unlikely(vma->vm_flags & VM_PFNMAP)) untrack_pfn(vma, 0, 0, mm_wr_locked); - if (start != end) { - if (unlikely(is_vm_hugetlb_page(vma))) { - /* - * It is undesirable to test vma->vm_file as it - * should be non-null for valid hugetlb area. - * However, vm_file will be NULL in the error - * cleanup path of mmap_region. When - * hugetlbfs ->mmap method fails, - * mmap_region() nullifies vma->vm_file - * before calling this function to clean up. - * Since no pte has actually been setup, it is - * safe to do nothing in this case. - */ - if (vma->vm_file) { - zap_flags_t zap_flags = details ? - details->zap_flags : 0; - __unmap_hugepage_range(tlb, vma, start, end, - NULL, zap_flags); - } - } else - unmap_page_range(tlb, vma, start, end, details); - } + if (start != end) + unmap_page_range(tlb, vma, start, end, details); } /**