From patchwork Mon Dec 4 14:21:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478492 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A844C10DC1 for ; Mon, 4 Dec 2023 14:21:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D71936B0099; Mon, 4 Dec 2023 09:21:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD41D6B009A; Mon, 4 Dec 2023 09:21:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B26BD6B009B; Mon, 4 Dec 2023 09:21:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9D03C6B0099 for ; Mon, 4 Dec 2023 09:21:57 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6109840285 for ; Mon, 4 Dec 2023 14:21:57 +0000 (UTC) X-FDA: 81529349874.08.5DFD395 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 5DA7AC0027 for ; Mon, 4 Dec 2023 14:21:55 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UmTkjA6P; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DtTuMDxg2eg7y+J7YOtz8EtHkmHKA1dRnnRsMSbHZZc=; b=q/WZqKaPN+wY0OoEUAfu+uZeDIKRHDiTMXdGpdqDbDUepRpX1e3rLwE2dp7iwXdCtTbhtO BjENRobZQzDHTo+MR9H3oQTwmRZrKm2LGGQ5EBKT+SbHQ6wvx//kY+K39WvUbnSiODhrl3 uSRX4S8xXcvgGPI98x7i7Mpv1G+Ie2Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699715; a=rsa-sha256; cv=none; b=GGfzZ7FtpQ4p73fLkHQ3U20y8QcOWYdeCMjXqFqgfj+TJt8YhnC8Q1on/LCPogCrLjWHuK PfZFi7KXXVjlqN2/fyuqXnmc+q8Yabrm8wtX5H5Co3v5XCikBOqnAFRikQsFUARglXOJpC Phl3VqbWg9y+Z5njVCyiGy9prR85O6o= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UmTkjA6P; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DtTuMDxg2eg7y+J7YOtz8EtHkmHKA1dRnnRsMSbHZZc=; b=UmTkjA6PdjbOoB9qRhe+ovTH46/TmF62xgpvWl3/ErN9IaxVJRxZ8tzqI2JUiSwAZiGPd2 vfZOeLzwhciUvxt/2Moqm0p0j4B+MBQapxn0oTZA9e+bOtieIAnvuRJUY7NIOfT4yYsiYE JVoYXK10bv4OULoIfuEfxLEoeHHL5tE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-158-Hayy6FKhOPqtL2ERdoY8Jw-1; Mon, 04 Dec 2023 09:21:51 -0500 X-MC-Unique: Hayy6FKhOPqtL2ERdoY8Jw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 899C08F9201; Mon, 4 Dec 2023 14:21:50 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C71B12026D4C; Mon, 4 Dec 2023 14:21:48 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu , Muchun Song Subject: [PATCH RFC 01/39] mm/rmap: rename hugepage_add* to hugetlb_add* Date: Mon, 4 Dec 2023 15:21:08 +0100 Message-ID: <20231204142146.91437-2-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Stat-Signature: yua6qrns1uzc6fehuxtmauneiqjo3mgk X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5DA7AC0027 X-Rspam-User: X-HE-Tag: 1701699715-214369 X-HE-Meta: U2FsdGVkX19W8xs15KP/tpTrIlx+A0Vy9xO9nJ/0E4OKP3JK1+fRLLOM/a/Pl36h0JtuVR4UFnkT3C+hq5j0oHSNEyOP4AfmKYKObkmAN4TbjVEymZh6Rj9feKbVKBn4tPpG42UhK4d++QCwad4WgT3/IL92d0mwUrDOKRRzcyqee9wEA6auepclBu1gw3XkH79pw5layX1ONBRwk6UlkOXLF9wXxhIs3q1c1EnJHBIDAL4A9tVUR1QtvNtnLQhaSytdIQR+L3aLrkz+qF0eG/T53qP+lJiJBIhttNEfhgn9CRWHGD7XxbhUjdQCPrAOExvwPOpFtiBztJ8AwFHvNr9ZoOsGlB9qY/F/yQgc9JBRLMdjwXmyr2B/zBXCKvCzCERX/wsoJl5gMeXyYRz6vL7JoyooMEzZfettHVnH/a2KL7w74K4IlIWcPv/D3xC4Pkejj45quCKc1Z7AtDJGQXTJ8/vi3+5GtCmVa4TfQgXGjNMs7EMkr433x7HYOyu1GdhRK/Sxj1X6Gzmi30jpfCUG9YkfBH7HqYzwgavafQPDNHedmJc95PgldlAqsV+S1OYkksPbOJ0sC/LUC8YU7KZZS3HJf7WRKvy1bDQRWq3qeoQTL1Oo3Ys/RcUZV4lED4DWzzJFcBydxFeAfmbhXWfn77w6MWf40twraBQTYog5++glzTm24FCVO1WfWWBoh8mHtppxpLU4z+R8HXahMOgj4nkqCBtScX1NScpLJmmmvtEBk+rIPdeqjmMGhnpunI2qws1hknOgZ51EjtnBlFVbI9036JhU7l43OevRZA3w7GRDmZkglaDhXOiTecRnS+7KBWW7qHTQBHmuSCraKGGmNU55nfaXi9ZBNPcWA5Ve4jBW3bbCpwqFIdpYXzjq/fu9TjzTfbwsmQAMtMjgrRxBNI0OdhicbIHg/6FceLpUASTOjmkDLYFMl7YWwq6JKlZnJXVsGKY7h2RThud pnyvh88p T/Pu7j307iXA5kWPwxa3g0tmY8Zj/YaIk3EF3uGmEoclnY3y1+mkFPKg3jSXMFOZv4BxQ/ef9BW8OSH1rMJceytt5anyPxsiSSvVRfYJjZM1VDuclCKANzU4mXGApsUu6bGV3og2GAazrdrD3Zv+c6vKPGcqpZwlaImk1plY7OaSI8wGyTvba/a7gJHPSoee88YOJvU8TC7thQ9U+I3JZFJJhlMeRpGxDWRiTNP2njc5TfNXxQ3DV3tkh3r8lSUx3Bx0iNe1UjfUIKQ+jRqjJAjfJHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just call it "hugetlb_". Yes, it's all already inconsistent and confusing because we have a lot of "hugepage_" functions for legacy reasons. But "hugetlb" cannot possibly be confused with transparent huge pages, and it matches "hugetlb.c" and "folio_test_hugetlb()". So let's minimize confusion in rmap code. Reviewed-by: Muchun Song Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 ++-- mm/hugetlb.c | 8 ++++---- mm/migrate.c | 4 ++-- mm/rmap.c | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b26fe858fd444..4c5bfeb054636 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -203,9 +203,9 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); -void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); static inline void __page_dup_rmap(struct page *page, bool compound) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1169ef2f2176f..4cfa0679661e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5278,7 +5278,7 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); __folio_mark_uptodate(new_folio); - hugepage_add_new_anon_rmap(new_folio, vma, addr); + hugetlb_add_new_anon_rmap(new_folio, vma, addr); if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(vma->vm_mm, addr, ptep, newpte, sz); @@ -5981,7 +5981,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); page_remove_rmap(&old_folio->page, vma, true); - hugepage_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); @@ -6270,7 +6270,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) - hugepage_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, haddr); else page_dup_file_rmap(&folio->page, true); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6725,7 +6725,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (folio_in_pagecache) page_dup_file_rmap(&folio->page, true); else - hugepage_add_new_anon_rmap(folio, dst_vma, dst_addr); + hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); /* * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY diff --git a/mm/migrate.c b/mm/migrate.c index 35a88334bb3c2..4cb849fa0dd2c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -249,8 +249,8 @@ static bool remove_migration_pte(struct folio *folio, pte = arch_make_huge_pte(pte, shift, vma->vm_flags); if (folio_test_anon(folio)) - hugepage_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); + hugetlb_add_anon_rmap(folio, vma, pvmw.address, + rmap_flags); else page_dup_file_rmap(new, true); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b418021..112467c30b2c9 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2583,8 +2583,8 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * * RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) +void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); @@ -2595,8 +2595,8 @@ void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, PageAnonExclusive(&folio->page), folio); } -void hugepage_add_new_anon_rmap(struct folio *folio, - struct vm_area_struct *vma, unsigned long address) +void hugetlb_add_new_anon_rmap(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) { BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ From patchwork Mon Dec 4 14:21:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 724E2C10F05 for ; Mon, 4 Dec 2023 14:22:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3F6E6B00A7; Mon, 4 Dec 2023 09:22:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEAC86B00AB; Mon, 4 Dec 2023 09:22:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C60E16B00A7; Mon, 4 Dec 2023 09:22:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B2AA56B00A4 for ; Mon, 4 Dec 2023 09:22:00 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9264A4022E for ; Mon, 4 Dec 2023 14:22:00 +0000 (UTC) X-FDA: 81529350000.05.8A4123A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id D63721C0018 for ; Mon, 4 Dec 2023 14:21:58 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C3W7JgbO; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699718; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ShB35s+ZfhtJKsIKjmsv8EG/47pRc0gWRbXgiSuPDog=; b=OghzYGxtVVy2VfwPyEXr9RfFF9Y/pVa2iT+OEvqaL1zxbEypgPSjOVjKxf3PVAI98S4pfy EG35Ky9kazVw3SmzweAnHdJwON4HMLig83up7d8RYyxU/rVsuVnBK7lCeta/ds4eARXrNp JRok7LLI7YFT/3vU9orwur6ctJedeUM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C3W7JgbO; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699718; a=rsa-sha256; cv=none; b=CmmO5jmW7H0OYNGLXVA8eqBLR50w/f5kXL+PasepM0X82Hn9n8ydhqkZUX3y5p8zybmcAX KvlH8Jzk2kEXJ3WiAwdhinxm0qCCcCsKaMsF30JwuIwGbFEUxgg/UO/eggHnpcpzNBuAMF 2x2g0eVFEM9bEztePBFIbcwETQ1uGGQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ShB35s+ZfhtJKsIKjmsv8EG/47pRc0gWRbXgiSuPDog=; b=C3W7JgbOW9Hx3dVHGdM2loPZ5nwXDS5luSyS6VlDarxz24LYD+TDZz+cwp7v/EkanNUe7V kEJlmdK/y/uUyHtvHTmEDequHnimXsxrn6XDKseocBXAUfueng9yovjet4WR4Mko2A2GEF 1++381fvYn0FkX4HqYRQKsJyxy9MWiY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-526-PsZ5JvICNICyMCT8BrmXEA-1; Mon, 04 Dec 2023 09:21:53 -0500 X-MC-Unique: PsZ5JvICNICyMCT8BrmXEA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 669068314F8; Mon, 4 Dec 2023 14:21:52 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C83B62026D4C; Mon, 4 Dec 2023 14:21:50 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 02/39] mm/rmap: introduce and use hugetlb_remove_rmap() Date: Mon, 4 Dec 2023 15:21:09 +0100 Message-ID: <20231204142146.91437-3-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: D63721C0018 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ffkygimdxyfafmp7etzgsh47wzsh6es9 X-HE-Tag: 1701699718-188643 X-HE-Meta: U2FsdGVkX1+SPvMKLwZigUPa7CcF97RgNq9kQVqv+bJwWJBcs+MG1OZk1oyyc1cwpNLmP0Z2u6tWu+1ESFw0sJ95kzt2e9ekRWpHRXqBgcrLxGM9ulxnzehEFa5isPHFNp7lg1JKpcq9Q1oeF8VxT9gDhT0lmtTC0Qn/eR5S37PHdY4yuYQ0KXrnYKG/yDL48kzp0pfyU9XYYVKpuaDxyjH3zx9Pvn+llqHmRA0UF48eI9SZYrIzuPkTiNLkmkW77favJuVRfi+Jt53sNmIu901MxFjadqfTKmD8MAwa/OpcEwVlrmOGzBE1s+NfO7+yH4DobmswVGWIcFnfZy/s5yxKG5Ku0IhlA9WgdDt6q6XbfrewWITvw9u+X2R5EmytmPIbppNv5lhNcemg7vqdQmn6B2sjGo+BfVoRfYz+2Ps0MGKXMUkm1DMa8I4lzpDkILZNwxg1TN2uRZjoHVl6fiHpkFj+4lIBUFT1AGN8Tyr1UUUK2TACEFlIqyZJ4J4NRWafuUjXNewDjF49KUEnGDwQX6Gh/BEUccngg/YX78GK3TFZbal9MX3pjNMbhuL0oKQZ2dnABvfBYYC4E4c6+RjUUibvdl1BcHLa+t6CRDbk8SO7VloJeUPkUS7ppjeAe/lvflzrxef/eRz8E8AyAjXH5uXDLJrmuwm9PAddhFPa+su9XtcD28dXSk5cMoXNT7LY8a2PjZBAUOaRAkw2c8RLFsCmFa81ElQVa3g4kFX8lJgqPHWff/5Y5S9C2BuJBR9+6qVUMtzbX237jyHzfU4Flxg0HXNpNgOutR09trJP7PEibLmYvOmm8+Ckie0DpjqObOoMH43w/y2bYFpsc0Chr4miZ/wngz+SxMLKSimLQU4WIC5dUDoAyDymZWR7MrjGK1S3AIkZ3vXwfeSY/oDB2YfQaCZqXxSBXIiap6vf/tFn3CcFyVlV1A8ufluMhTPrYENmVGGLVheUsoF lGR8khH6 p7goDwLqP1AU30cWJNNr7g2ECPWT708mqfECMfiHxy+1euzwtVjJq0d0La8LfFW1ScGajeoA/Yv4EHvQh5MArq9MVNh5Ey8WPM2xYkX+/0emYJh31bd1FqcQeWaTsrR6vOXRGIjVBsgUg+pjMVftMyRSYnGGs19PUSwiwBmZTFeDtiGV6GUKjPDFpTe1KkvJt5+KA5iKb8jZCJ3++dMbgIBSbFbYBmxYRi68S X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Let's introduce and use hugetlb_remove_rmap() and remove the hugetlb code from page_remove_rmap(). This effectively removes one check on the small-folio path as well. Note: all possible candidates that need care are page_remove_rmap() that pass compound=true. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 5 +++++ mm/hugetlb.c | 4 ++-- mm/rmap.c | 17 ++++++++--------- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 4c5bfeb054636..e8d1dc1d5361f 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,6 +208,11 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_remove_rmap(struct folio *folio) +{ + atomic_dec(&folio->_entire_mapcount); +} + static inline void __page_dup_rmap(struct page *page, bool compound) { if (compound) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4cfa0679661e2..d17bb53b19ff2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5669,7 +5669,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP), sz); hugetlb_count_sub(pages_per_huge_page(h), mm); - page_remove_rmap(page, vma, true); + hugetlb_remove_rmap(page_folio(page)); spin_unlock(ptl); tlb_remove_page_size(tlb, page, huge_page_size(h)); @@ -5980,7 +5980,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); - page_remove_rmap(&old_folio->page, vma, true); + hugetlb_remove_rmap(old_folio); hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); diff --git a/mm/rmap.c b/mm/rmap.c index 112467c30b2c9..5037581b79ec6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1440,13 +1440,6 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, VM_BUG_ON_PAGE(compound && !PageHead(page), page); - /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(folio_test_hugetlb(folio))) { - /* hugetlb pages are always mapped with pmds */ - atomic_dec(&folio->_entire_mapcount); - return; - } - /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); @@ -1804,7 +1797,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2157,7 +2153,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); From patchwork Mon Dec 4 14:21:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478494 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D585C4167B for ; Mon, 4 Dec 2023 14:22:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4755D6B00AA; Mon, 4 Dec 2023 09:22:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4235F6B00AB; Mon, 4 Dec 2023 09:22:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 276E06B00B6; Mon, 4 Dec 2023 09:22:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E4E346B00AA for ; Mon, 4 Dec 2023 09:22:00 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B78EA802A0 for ; Mon, 4 Dec 2023 14:22:00 +0000 (UTC) X-FDA: 81529350000.05.8709C9A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 0B82980014 for ; Mon, 4 Dec 2023 14:21:58 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DgMtRCae; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ED3P7Gmg7VM9PqzBG8DklxM7dI8K8CnTi9/+BO29yAM=; b=5ak7vUSRTcgFQAs2OXq4+BDDP+TZOyjGdPAOoH2NMffXIKSi3JYMaa5geht59+HHM/Y4pX /Ml5HsR8PoHh7qjeZqTC6Vkktpw4UyJtYmSWqR9vmNCqf03zPqPckwrqx5EHsx6K9Dg2Sv pbmjn7Gccv5RiUEDdIxunBc71HpOvFk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DgMtRCae; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699719; a=rsa-sha256; cv=none; b=CqEGIBnogpNV7gkzrZb2pBnGIh7jNldsePQrxU+UmD+FyiIcHtcwRWoRA4Dktrf4WVfTlp sCa4QRpiCwBnrITgAk+nFDv4oVZ8xfL6NLFlx+EwiHxHPCpCuZILJxWpS+3VcaH/q4MzJh ryAzulYkx1oYRshOu35/8yIu/h22YeM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ED3P7Gmg7VM9PqzBG8DklxM7dI8K8CnTi9/+BO29yAM=; b=DgMtRCaecFSrXO0XC6touXlnpRUhPmG7WAfHI+OYVYp/L1atg/MaBM2G5vP9cgD7ioY0MZ qQpKD5tvxGbTBakeSalr+QyeAZAkgx+inxYJUFbBhG/aZl7CjGwn5L6WJbSBcITAx5v4tx 3oBsAw6NGN5+R3txrN0bXuCnd2QBrpE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-460-Qm87iSsWM7CFHWpPHaPMdA-1; Mon, 04 Dec 2023 09:21:55 -0500 X-MC-Unique: Qm87iSsWM7CFHWpPHaPMdA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 41BF2811E7E; Mon, 4 Dec 2023 14:21:54 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9EFFB2026D4C; Mon, 4 Dec 2023 14:21:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 03/39] mm/rmap: introduce and use hugetlb_add_file_rmap() Date: Mon, 4 Dec 2023 15:21:10 +0100 Message-ID: <20231204142146.91437-4-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 0B82980014 X-Rspam-User: X-Stat-Signature: 9utxfy1hai9hhrhim6krsoikj5muw9qf X-Rspamd-Server: rspam01 X-HE-Tag: 1701699718-235133 X-HE-Meta: U2FsdGVkX1/40j80Z5YnNXs1BbowwZurattUq2vMxKPhrN3AgME47Hb3BmnoVE9H5cCy8EjiobWKPXwO6MoE0aklfglCqbOtWg1se+2AuKtR8xmpe5+i4lx958RSQZO6RHwnnPSPnHmBtMQnwE2GKdN+cAlIRPBxCXu4wz7rORPcc+UMawVxYsWvndQ7Ku89cRK78EDpdpu7QTC2k1lkbSTlDNsw6nhiOCZkE0wcFK+dB1EX7Odfrss6FN0tF7830+liGrnm5hMKbeITPn0qjoIvIFtnWA33R0/KkSTDLLd6/SwyZZq/iuaDq+MeaCEVe4B04uPk2nYkzYadc9CCeB3WvCQDAp7QRTG491oEYEcUwptrtFtHy71/icNqFouViWhnp7MafqAWJ/ldvmbm7YfTeclqkqh/IIbHmgzU0cK8dC3Uz89ARpkj1H5ZpCYDhEs89gjZ0BVmcHsUNXMvXQo/c36VLVwRGzdPMvJHoVEC5SbZ+jZafVHUcmYH3GHxi7LeR8Fa9fWV/QT0yJ9bGLal5sz5zxggJisWiLIIqZSvtWOBA6BFZ5D9H3IDJvOWA3Y6wa0K9nU35oGsLD0g/eiuz5pcuPVoQuwQkIM36Rsbh+lU2Ftzqk+tfNey3x1XXWPi7IwY6XICydPCS2FcCa41EeL6QS21nzlYMZvFKczeX8O4K8thwCZ10C3EwibeSvy/UTYJP//e5SaqebkU/8WXEgxOvDl6OCrh0HRN6uFGef6TFXLDBPa4ovjgXDoWuk8hvTe83cxpBrx7Rotr92bD90lGEhJdAjIIxWdiTNMDqt0c4oSsXOCPKOxEn3iiDiSzNPb+UZri2C9A1D4WcO85wRwlvj1YEpDoIUsAx8BxqE2sXWC4xkfLzicTaUyiywIG+v7gYVBeGFuTtK20usA/SffztMwiEThZ4qeIct1tJniJa9bKX2CsS6D5ciK3O1137ye7AhcSSgUOf+W ExP2onni UYhOeL3cqo4DWdej+Thv0papVdiSwAUrtn2TCTW1uu6JXm9MX05mq6i9sKpjVZNHqQSvEzcqySO7Ph4SDinyJxKRFExioJMAk54G4nh6oukE1C6FQ8VgsZfAVIXXmhWwZdJ2mb30lerk6gtx7gF14pi0NUXyyOhuRgANVhqObmeCcEwtjHjOtrxuBCz+g2JgNK6MWgU/q5aw45Xl7caMViIIOpCFDzxAjwpMB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Right now we're using page_dup_file_rmap() in some cases where "ordinary" rmap code would have used page_add_file_rmap(). So let's introduce and use hugetlb_add_file_rmap() instead. We won't be adding a "hugetlb_dup_file_rmap()" functon for the fork() case, as it would be doing the same: "dup" is just an optimization for "add". What remains is a single page_dup_file_rmap() call in fork() code. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 7 +++++++ mm/hugetlb.c | 6 +++--- mm/migrate.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e8d1dc1d5361f..0a81e8420a961 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,6 +208,13 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_add_file_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + + atomic_inc(&folio->_entire_mapcount); +} + static inline void hugetlb_remove_rmap(struct folio *folio) { atomic_dec(&folio->_entire_mapcount); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d17bb53b19ff2..541a8f38cfdc7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5401,7 +5401,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * sleep during the process. */ if (!folio_test_anon(pte_folio)) { - page_dup_file_rmap(&pte_folio->page, true); + hugetlb_add_file_rmap(pte_folio); } else if (page_try_dup_anon_rmap(&pte_folio->page, true, src_vma)) { pte_t src_pte_old = entry; @@ -6272,7 +6272,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, if (anon_rmap) hugetlb_add_new_anon_rmap(folio, vma, haddr); else - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); /* @@ -6723,7 +6723,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, goto out_release_unlock; if (folio_in_pagecache) - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); else hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); diff --git a/mm/migrate.c b/mm/migrate.c index 4cb849fa0dd2c..de9d94b99ab78 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -252,7 +252,7 @@ static bool remove_migration_pte(struct folio *folio, hugetlb_add_anon_rmap(folio, vma, pvmw.address, rmap_flags); else - page_dup_file_rmap(new, true); + hugetlb_add_file_rmap(folio); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); } else From patchwork Mon Dec 4 14:21:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E9A7C4167B for ; Mon, 4 Dec 2023 14:22:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 114256B00BE; Mon, 4 Dec 2023 09:22:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C4EC6B00C8; Mon, 4 Dec 2023 09:22:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7FC46B00C9; Mon, 4 Dec 2023 09:22:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D36BC6B00BE for ; Mon, 4 Dec 2023 09:22:03 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7688C1C0518 for ; Mon, 4 Dec 2023 14:22:03 +0000 (UTC) X-FDA: 81529350126.12.E526475 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 9ACE4A0020 for ; Mon, 4 Dec 2023 14:22:01 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=idR1Ngei; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CldTAIMRJWc6rWIvSdlg20t+rIQF+KSKFiNVs+JtPQE=; b=IrnpZtv4U6PJQWR9Mo2uTRvYgtXbxE7jIAlglgrM4jJ6TqeN5biCBpSmEckQsQjFLQ7tKM KD9omEtdBGGIs64CyF+tE6juOswSWTaDbzNQpScCFroNNnVeE0peKXjvRg60j/5Ka0yjEr G+wspR6/gXlm6bvrLSaSZo19G7Ac3ro= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=idR1Ngei; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699721; a=rsa-sha256; cv=none; b=6p0ypY3C+xpj7Hcl+faCtdK3mP96aBlWO5Jx6v/0BqZ/kcrpA3AYar53Q27FmecPUv6V5O hnxlWjTDAHr27cycJVHcNXIOVroHjaLRIN7R9FtsxPH/JmJMUPQRSKx62QZTvVD59PSfXX FEvygMTsqtUQW8btRSWMyoimGAYaa4M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699720; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CldTAIMRJWc6rWIvSdlg20t+rIQF+KSKFiNVs+JtPQE=; b=idR1NgeiIABmoiRn0w+lwyd3Q4yTs+EqTNKeYfMyfHk3udk7v9ToE4Tlzg3yICeb3EFYNn mfGztWarEMwFyigTUs8HZiBBZAcnsrgLcOI61vmIEyg8AXfpxL4ir2euh5fm92Y65/3akW Xj/MtCogGU4E8QXS1Mg8e86XZmv56Pg= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-119-IrgCM3BPPOeYAKNDeamZgw-1; Mon, 04 Dec 2023 09:21:56 -0500 X-MC-Unique: IrgCM3BPPOeYAKNDeamZgw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1E5A73C0FCAD; Mon, 4 Dec 2023 14:21:56 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 79B642026D4C; Mon, 4 Dec 2023 14:21:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 04/39] mm/rmap: introduce and use hugetlb_try_dup_anon_rmap() Date: Mon, 4 Dec 2023 15:21:11 +0100 Message-ID: <20231204142146.91437-5-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 9ACE4A0020 X-Rspam-User: X-Stat-Signature: js3axfauc7w3tur6bq1ek4wthffm73h4 X-Rspamd-Server: rspam01 X-HE-Tag: 1701699721-175859 X-HE-Meta: U2FsdGVkX1+8l5Da1hYBpDelVGowRsmhDAXm5JIdhU48oAkRnHnZh5ALLmr1I+DsNok4c3Ud/7aJROfGgHUcHDbQZwG4JSo7fmqU3XFsyPnrwvDDmtF9UJQTJX+TyhsZYDGyXkx2JwMh4AjeEMpOVXsKNevYwCgkr3dAx5UebmdrU0vcpf5WkXsIouj5rjtn99+Cx1upB1gVHtg9+R1tZK6M17gyEakWsthz3dOYqjGcv8aUlXGjQys9KnZAyqf6UXOJayVMlA8u7mrqyyrbdrhCFTt0YeUOaJSggJw2VJGEztLbSj2w2GAeeFVt/6+hWw5ejBW3G8o1tjr6LnprMzZXaq7jh8LVupn6Rfg+mR4F0rpMPdFlUMYU9aw3YfKUzwyPSrxzrl4O9OP9ht6G7Cn9TevD7gHHC7V7teCAvg+m7rZ2ejRGa0IHq55cPKNYZ23VtzKFzFlNmSDl6as+SpTl4BcakL4uoU+tY0h8DXKHqr/gCyb2D4c2f7UdC+KsQv7Tpkzr5M9oSr7ShN138zMESbaX20A150FDfL7LzOQKVCZZHYmbaErULVTxhaz6Mdl/EeLaI10c2bJinIagtqgp/A932Mv0Fp0y0LV9rxWheQO3y769hQ5RIwrn55bVqbXrZlAUB5vY2KdY3dipZphCNWeXtc15CooTgpSoBTtNWia6Lqzs3SIH8oaeIMNacIE2FOudE1GAFKpsIfjQbBmwIkdUdCNDAxwKJgsqj+zwslketGrHDJzTG+ztXBbvNH4asKN3+bCCnZzMbndeXpRin2Uz7DZOvxpvqpPGJWnMJIC4e/C6Lge6xPgJK+FLhpMiNbK6IM76Ia+hqY3qPZ6il/AE3sSawDiaUPS3XkDuw6AZbblj3UG+fi2hiV7oICPs7/h+ThzmOzN6iK8eG7TbQ7Qj9fy2CPHORdvhNJPDny/hwfv+h5ZL8ESPbWQ8h9m8eUR3FN+4OOY24kg iR2Z6aft LQBHTgtaF8PdYq3oY2Vp1juqyJbuEPMlZ9fCyo15eaf24vuuoo/h4qtBU10gkPyOVJd+8QUEN618JB7CxPWsPUiYCFGDcKkC2/CFBOMi7/X+jcp1CbtpQ1b/spoBVvCuiagxcttm0OFCE/fPvAnIX1TAZsSdVmOrppKOw0x9deP4pGIGMUKhygz13nSGLHlfhQ11K/9pgEuGi3WhOzvqXnfNvV8X8ttDeP6pP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that is_device_private_page() does not apply to hugetlb. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/mm.h | 12 +++++++++--- include/linux/rmap.h | 15 +++++++++++++++ mm/hugetlb.c | 3 +-- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 418d26608ece7..24c1c7c5a99c0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1953,15 +1953,21 @@ static inline bool page_maybe_dma_pinned(struct page *page) * * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) +static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, + struct folio *folio) { VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1)); if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)) return false; - return page_maybe_dma_pinned(page); + return folio_maybe_dma_pinned(folio); +} + +static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, + struct page *page) +{ + return folio_needs_cow_for_dma(vma, page_folio(page)); } /** diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0a81e8420a961..8068c332e2ce5 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,6 +208,21 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +/* See page_try_dup_anon_rmap() */ +static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, + struct vm_area_struct *vma) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + + if (PageAnonExclusive(&folio->page)) { + if (unlikely(folio_needs_cow_for_dma(vma, folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + } + atomic_inc(&folio->_entire_mapcount); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 541a8f38cfdc7..d927f8b2893c0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5402,8 +5402,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ if (!folio_test_anon(pte_folio)) { hugetlb_add_file_rmap(pte_folio); - } else if (page_try_dup_anon_rmap(&pte_folio->page, - true, src_vma)) { + } else if (hugetlb_try_dup_anon_rmap(pte_folio, src_vma)) { pte_t src_pte_old = entry; struct folio *new_folio; From patchwork Mon Dec 4 14:21:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 278D0C10F07 for ; Mon, 4 Dec 2023 14:22:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC63F6B00F3; Mon, 4 Dec 2023 09:22:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD6976B00F5; Mon, 4 Dec 2023 09:22:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD71F6B00F9; Mon, 4 Dec 2023 09:22:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 97A856B00F3 for ; Mon, 4 Dec 2023 09:22:05 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 683F11C04E3 for ; Mon, 4 Dec 2023 14:22:05 +0000 (UTC) X-FDA: 81529350210.22.4177CF6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 9FE87C0009 for ; Mon, 4 Dec 2023 14:22:03 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EU4P7Nc4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699723; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4nUQXJdzLDxn5x/X9wL3ukpxzJ+76ZeroIzXMEVl8GM=; b=rNq7WgSAhwJv+HDWi2bKPlnLlsn1Fi0ZwcLih6aVsYjNGLOQpJln5XBQfFQYJOleGdBwiZ AgPs8erwVhJugy2tSeHX4oB13OWhLdWMUq+YB4lr7xRNnsToTYmD35439SAvrojJaMcs9M gS/LUMzgvrdusZc4Ol+DgkZuYYy36AU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EU4P7Nc4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699723; a=rsa-sha256; cv=none; b=VEeClG+938+FYDLAzD93izMVLvnhXUHdPE28obFU/YLDw0P5WMZoHhamcsCHR2P/1FbR97 J6KyLEOS0Q3NozzXC7xTCTf4/ZdNKDPtolUFIru6aCPt7l1qfHmsBAuqB1fqAOgjb0y19s j/r3Zv6Xvd/8HGTPmZQVCXaH7usfCq8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nUQXJdzLDxn5x/X9wL3ukpxzJ+76ZeroIzXMEVl8GM=; b=EU4P7Nc48r1Q/s7gIAIcakDFW5eCGHdgSDQTiToa7KJQXevbAYDD0ms31fu+rE4g5i/DaA W0vThFJDsvukHGK3/Uj+POzq1z5WI73tge+cgukFgA+K0AucXrYUQwC3q7b5bNIEXTuVsj 5AFr8cHlLMDDRHzb/+Um7TV8rwZCznQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-9-ssjzReYIPVSCXays4yclSw-1; Mon, 04 Dec 2023 09:21:58 -0500 X-MC-Unique: ssjzReYIPVSCXays4yclSw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EA9F3101A590; Mon, 4 Dec 2023 14:21:57 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 572442026D4C; Mon, 4 Dec 2023 14:21:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 05/39] mm/rmap: introduce and use hugetlb_try_share_anon_rmap() Date: Mon, 4 Dec 2023 15:21:12 +0100 Message-ID: <20231204142146.91437-6-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspam-User: X-Stat-Signature: pr6jcxnhfuqntdb67q9s9tp8zcduwhxh X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9FE87C0009 X-HE-Tag: 1701699723-710990 X-HE-Meta: U2FsdGVkX18+OHpmwfg7heNGrU34c8h0I9fCcdZuqOG6mWk5KGAu0QECcGW1axq41LzLGg5oVFQxWeU1Ul80il2uUJJZLAPF57sGbwEza/ikeBPLsUCGdN1qwOPTBA3Ledx2AFBYEqtDumsRlY1WtkdwChzcjt7mNsfgtfm/cW24aPfIhzFHL1Ppb6ceOGUnsH+Yb7XAX1HtfZdhb/T2YqqmWLQa2QluL2SlehUwnPEvJHDAs2B/OvvJ41ZitkILxpiz5dQQHgwugkAatn+2HDpNn005KlqKB3L9Kb4GywhGs7bAB6G4QXiBCJCs6KKxq5ER7TG2kWx6DbIJq4I1O543d3JWKm28Yjhv4+KRB6a+chKvpgtrAXrxCG8mgAkW75Chs2jEA3LeQeniqxAtsXOQhJqKag9VKE3mb619g8p+zXzhGQpjicnplBgFXCT3p6JqrTGiTYD0IH1Wl5tkijvydSH8eA2XQXsdQKLKaJhGZsWMuv8SFgYRH9g938J+JClMsqd2gzVn6YivcyxY/zy6B5qsz689MxgP4WTGVoefIROoO3aFhWpCXQAnQhujI+3nFotILcf6GX+Rc39atlQYCxlMxy7I4+KcEEjPQgBv7+M+iWW08YBhv5FOazFuVRrLlvhWHxwxYgCDHFVymv6QnUoLsXzgsclS391v8OywkaHcXdgeftGMxWDckO8uYlv/L+9RKkctEQRvqmb1ZViDJDtYc/RO4j+WIslBTTd9HSamhT0x1KMsmqA8tNDFojnvX9FUk2U2quOovXT9L6g7hjBMOujAGgNVEKdnly+cDt2HtndG8LI4MGlnE6sj5xi+STSS4hrjMMow/DZaXxC2XGMpuZE8YUSSPZtmPCBd7N8PQD9VAogKLrjZZb4le4BYxukD5PAFwQRS7Rn7pQfspIHLNe+Ryd+5k60Y8juYkbJELeFhPGsthE4k6ThKmUQPtr8Y7Xo+w9DxA2q /kDU9lqb rxWAUpYx6spZfel+ed1LXv+eRP9xRykY0RWxQ3mUTeHskyBDZcGONjXoTnvTOLbiSn77ztS54izecaFyt9Rf6u7N1EavmeoD18HFmWW0jBe8S3af+xnre9nJpr+AnhPva7b4wf4Jr2Fe0PyzWOuCMU8SS/aY/Lppif6Yp8OgDuJAjKwrVeFloG9rcGRrrau3VFPIZKH2Iq6yBtxpFh1tvPRquOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that try_to_unmap_one() does not need care. Easy to spot because among all that nasty hugetlb special-casing in that function, we're not using set_huge_pte_at() on the anon path -- well, and that code assumes that we we would want to swapout. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 20 ++++++++++++++++++++ mm/rmap.c | 15 ++++++++++----- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 8068c332e2ce5..3f38141b53b9d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -223,6 +223,26 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } +/* See page_try_share_anon_rmap() */ +static inline int hugetlb_try_share_anon_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); + + /* See page_try_share_anon_rmap() */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb(); + + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + + /* See page_try_share_anon_rmap() */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb__after_atomic(); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/rmap.c b/mm/rmap.c index 5037581b79ec6..2f1af3958e687 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2105,13 +2105,18 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, !anon_exclusive, subpage); /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { - if (folio_test_hugetlb(folio)) + if (folio_test_hugetlb(folio)) { + if (anon_exclusive && + hugetlb_try_share_anon_rmap(folio)) { set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } + } else if (anon_exclusive && + page_try_share_anon_rmap(page)) { + set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); break; From patchwork Mon Dec 4 14:21:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0F93C4167B for ; Mon, 4 Dec 2023 14:24:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3ADDA6B02ED; Mon, 4 Dec 2023 09:24:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35D916B02EF; Mon, 4 Dec 2023 09:24:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FE426B02F0; Mon, 4 Dec 2023 09:24:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0AEF96B02ED for ; Mon, 4 Dec 2023 09:24:37 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D7883A0A87 for ; Mon, 4 Dec 2023 14:24:36 +0000 (UTC) X-FDA: 81529356552.13.430E3E7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 1C51E1C0021 for ; Mon, 4 Dec 2023 14:24:34 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=El2BJwuz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699875; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Py+BBg1WdhAyTKIO2vh7V9agn/F6NhuWPF4QbRDWhlI=; b=n9uYn2W2ON8Trjf6CM/Yh2HDKLXZkIP6HngK0SKizOfgvHpItpj1558uHzdaXKx8JvtZQv 1sRwe7RB3PxXdMXwzxgT875hHyqIAGMQgKcKt2AsKYjn7m83B6vSjj9ImZyFf8Xwnl0I7k TWWfHewceZJ3xiQpOX/AEb9iDQg+gQE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=El2BJwuz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699875; a=rsa-sha256; cv=none; b=R+p24wUiMYA9Z8tyjvWNLFWrnn3vEi1/KVB5VpU9EjuJ44HeXFrFX19ExNrzYuZyx84540 QA5DCT1mFyGn4O93jrOCMJbte1CmxW0d0qf0wE5MwmHw4UOS6eGEa0m7Uhly4tiIaZKfp7 I8ZNRaW8tylzx6gVifj9HG6EbFsoKvM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Py+BBg1WdhAyTKIO2vh7V9agn/F6NhuWPF4QbRDWhlI=; b=El2BJwuzhel6fih45CrkdJZJxTQP5q3c8H5CMhvGfmNg/k2P5PYVXBoG7T2f8QlVXv5dvb 2P1s5cCp/LiV+yrGyWlJfw7RMTua6YOTa5BOiF215gpL6Eo5X9UDqVTkcFg38NdHNDuOJm c4WgsAy0GBIETfITBVe3GX73pCPd7Jk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-110-KmE7Bv2xNWyBuu4ga5J7Uw-1; Mon, 04 Dec 2023 09:24:16 -0500 X-MC-Unique: KmE7Bv2xNWyBuu4ga5J7Uw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CCFA88F9205; Mon, 4 Dec 2023 14:21:59 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 308C02026D68; Mon, 4 Dec 2023 14:21:58 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 06/39] mm/rmap: add hugetlb sanity checks Date: Mon, 4 Dec 2023 15:21:13 +0100 Message-ID: <20231204142146.91437-7-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 1C51E1C0021 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 9d7sprq3oj133sqdqbbugzgu4q7p4ydi X-HE-Tag: 1701699874-747155 X-HE-Meta: U2FsdGVkX1/sKmUHry4M5HQHWSgMt1JCNKMaS9u433J5REEgFpss0YsfrCXNZFk722VglyP1Gplzx0O6aD7sXEx5F0O3zWwSuJqZITMA3qAFxRmHvRpN6v+ECYuMvN5odYyVmsU63SdwRw73KiqDkfdI6cunqq2za0GF7bdHeWJpktZuVW1iDZQha0NScMfwwqmzU7/x8Tl0i/6qsCqSrkLawyziVuw95dLq4oWtjPDCjq87KHxko48wORYjYyVJJSZYyLuKyAxAl0BlWCHuoWNiHl0CRUkpZZpve9mOrdtHZG4CScjmM+RPQv4bx7j0tLumLH4E3gx+7Rwh01wRNv8svoWVyhfwd/4IR0EgXg3yMWPJOkmD+CwyOhHAo9NpjHJZAhUY6f64jYT6H1KjbCSIslDzxliMSwUyifhNqR14RpaBYi+ZvdjP4NERHYWiXNy+pZHJeJgfKZVNDjbNJD7pqxcdMBz4w+9USuoC07l4mGfzTMpxp6g85NvFPnXsJtccAYa25fwUL8Axcri8+yzc8FCVDmrUXj9zaFrJD7kpBModrrBoS8xiNn8Ip99RJpX2HyDRbKRm4M59wiB+9dIhTSpANUWgbWN6PLDzlyySW345MVb+FSOOE0VjSeiYdocklBV9zjbEV+B7F00qvhZScs+Q0fCuzkCrV5oEcHrOXJL5Q3kdecJVrpB+fhJTLi19LguJGipg8yzRwNKTO/ea9oQ+H61zcX7xvmx1F+AluusBV6U/9GlvIeVAj8/olPwBtr8EIwx+EaXXghTb7Emq3Ubs4GOLMkeMoo8mYI33N8H6QzHzd5h0FEYVH/CNTehqtox02J9aCsqLrgyOeUck7Vv6Wi9k/Z0bJZ279a3JVmem3MpH45xyRo7T8HEjOZmD8R/jm8TNuKsSEyKeh20Gbkn6wd/z24L3k9dT4bFF6Nj1DYPGdhV82ve9phhk7auB9kvtBXxgv/5i7lh t5ba1t/I z0g4y/dwwHAsBQTDuCqS8gPaPN6qM9I9vXiBD9UBxuxHk9nXftJftgKSyMe4ETIRUnq8JRuMb5PWwgS+kopo6Z2Db3UwRt/XvHcJAsEXbI0qNO0r7dIn9sjJ6S92aiEf/P7snr7VMg0aruhEmFBwLSUPjOuALMaqOEQs7Ha0m6qWQxPg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's make sure we end up with the right folios in the right functions. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 7 +++++++ mm/rmap.c | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3f38141b53b9d..77e336f86c72d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -212,6 +212,7 @@ void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); if (PageAnonExclusive(&folio->page)) { @@ -226,6 +227,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, /* See page_try_share_anon_rmap() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); @@ -245,6 +247,7 @@ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) static inline void hugetlb_add_file_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -252,11 +255,15 @@ static inline void hugetlb_add_file_rmap(struct folio *folio) static inline void hugetlb_remove_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + atomic_dec(&folio->_entire_mapcount); } static inline void __page_dup_rmap(struct page *page, bool compound) { + VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + if (compound) { struct folio *folio = (struct folio *)page; diff --git a/mm/rmap.c b/mm/rmap.c index 2f1af3958e687..a735ecca47a81 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1313,6 +1313,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, { int nr; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); __folio_set_swapbacked(folio); @@ -1353,6 +1354,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, unsigned int nr_pmdmapped = 0, first; int nr = 0; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ @@ -1438,6 +1440,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool last; enum node_stat_item idx; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_PAGE(compound && !PageHead(page), page); /* Is page being unmapped by PTE? Is this its last map to be removed? */ @@ -2590,6 +2593,7 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -2602,6 +2606,8 @@ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, void hugetlb_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); From patchwork Mon Dec 4 14:21:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06595C10DC1 for ; Mon, 4 Dec 2023 14:22:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E6416B00FB; Mon, 4 Dec 2023 09:22:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 46DA36B0119; Mon, 4 Dec 2023 09:22:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 298916B0109; Mon, 4 Dec 2023 09:22:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1A4236B00FB for ; Mon, 4 Dec 2023 09:22:08 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EC0191201FD for ; Mon, 4 Dec 2023 14:22:07 +0000 (UTC) X-FDA: 81529350294.24.B7FB929 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 1C433160007 for ; Mon, 4 Dec 2023 14:22:05 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BmVJAIfP; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699726; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y8nY91srGGOwHNmHB2IEECEkScJep9NeXgCbzBUOTxo=; b=QxvL/u9k06vS+rJSEldH9d76EXGLn9kPXD3DStMr/NyfSfgDY+V8q/l/zzv5I/CUlkylI4 R534fOSjyukspE1qDau+RZm8Ptq9A1Io63Y1fmwsfXZekj+vFARuWeIvSpnF7PBaH4M1OZ 4nT8uCksBsyvOEdT/5x6nvFuOOHtgts= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699726; a=rsa-sha256; cv=none; b=E1f+l3X0Vrc+7zUBwUJB3KkBRYG57TvOBpyNGk3Ias7gQOby4arrzMegeHojOmKty20DJG EV4em33ruoxfK37M8Eou7D+jYuRZV4BA15ltE6msYufWL5EWE88HfgXlUnTqBrNIamQw/z 5DNq2Nlf8nwRmkVot3B75qlm1uhiL6M= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BmVJAIfP; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y8nY91srGGOwHNmHB2IEECEkScJep9NeXgCbzBUOTxo=; b=BmVJAIfPcw8rV5BGoidQnofxE4E/A0/J/jJhmDh52n2XVTFrMcDXqQMaUiz+gzvkj3P51+ Ft+xOoDjefgMiIbOBsB4lC1IYY3fC0DcHmrjzAgQ2+4sonKBJ7z/y9dTILCBd38EQNlOSE DHFD2dZhkaEo8VS2dGUEVo7LMumlQpw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-201-g2weQkhwMp-9jFxobmepBA-1; Mon, 04 Dec 2023 09:22:02 -0500 X-MC-Unique: g2weQkhwMp-9jFxobmepBA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE9EB1C0514B; Mon, 4 Dec 2023 14:22:01 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 118622026D68; Mon, 4 Dec 2023 14:21:59 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 07/39] mm/rmap: convert folio_add_file_rmap_range() into folio_add_file_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:14 +0100 Message-ID: <20231204142146.91437-8-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 1C433160007 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: x8xf7c1esdcrrjj1q4734bc945x64mgo X-HE-Tag: 1701699725-531321 X-HE-Meta: U2FsdGVkX1/XXi9aLZaI55tSL3xW8SeKJIiLcvL2aNOHTbs0LzY6D9MO4FHqqYsMHJoZNKoeVxwO2md8P594h7VK9Kl2xr6f3dZ+0s5HQLcNZS8zRqLAyCFGXzv+JAGARH1Y/Zk+BuULVR3NsywoWuthg7mLpMeDH4RyhRnuoS7taGQFRENMxFAZAHqhQntBRccA2jnLJ4VUgWOVZ4i1mghjoivQ0WgK98KJncXfqv9Q8YihG1XuuZLLhvts4eDbI6P0VJAFRVt8OWPDEeqXx+7LmRc2sdvW7rPEaWhVnFHReEYaKT9fkzWeAm3hXk6n8qGH1MHnUduEUFc1jY+6m8TJ0P4VH2GpokxzQMZ4SUUfVIpDGSu/J21SODOa/7jpr5X+lUc1xwFo9tqAQoPZpySV0NKFD0EenaMBu2g+riUO43whf4CLNE5g0v/0RXt8feSAFaZP21Jzn+GMqYwndxx2uiLKd2V4eJagwNhZMG9MtJOSWGHZPTEsiwquY+Pq8Dlscrl7Ljd1PwnuNvAMvueQUEsGWmNSodnaYTn5elZc4W8FS8Jqcp8LDl90DmV9EYZf4c8nOcxjdXQ6kDRPUPF/rABGBMQC2f0IPpV4coFyyYgYnUs1OZRVhrsLChjDxD7LtghGsu+QlfRElZ6AloL5Y6Q6WFfyny8dexBXOSiaR+NGjX5IKo8ApEtM9TiGbBodD9qlDnRdVlJecs4iVlfoxCGrQnAK8zz0qYc3kFkA8Ve7WEUM4ljgXlq1lP+St+cqKJ2Xyi2Tj3xKKuAzDG1Kl2zRunDzh+KA2pbXelZ1iuOPkRPr7h6UjPdueRqgZn/ChwNaH1aZQEgY8Ji85PLnJOGFELVkIye1xMx6JFxyzWpLTPWvLsjzT7JCpyyb3Jv28zVl0KTa8lSZBtVEHpd5nvQuP29bRJfQ0Ls/MJ7z2H59K6RUzpkADOcDwVJSwnpLwH5p63phw9/tXFW ekmRsO+Q pKvuK3Es7qJLF5TWDyy6SUA5jAUeQCv5TbZDMXsmDHG+zpS1bj+/Eaum3hJOnPSbJrm10ohb2n7jMrNABilQn67Vp5LA7RSgE9tJx7yzIWaN3Ipa5mUEW3cvYNLZyTFtO2thA6qr0YF4aJXDAsmTsJrDAmZStMJV/MCsGEDiGiJB19kJYJbgu6kbXFd2X0HcD6q9VIxUGfX8IIX8mCQIOZMJJM+gg5nMknW1+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's get rid of the compound parameter and instead define implicitly which mappings we're adding. That is more future proof, easier to read and harder to mess up. Use an enum to express the granularity internally. Make the compiler always special-case on the granularity by using __always_inline. Add plenty of sanity checks with CONFIG_DEBUG_VM. Replace the folio_test_pmd_mappable() check by a config check in the caller and sanity checks. Convert the single user of folio_add_file_rmap_range(). This function design can later easily be extended to PUDs and to batch PMDs. Note that for now we don't support anything bigger than PMD-sized folios (as we cleanly separated hugetlb handling). Sanity checks will catch if that ever changes. Next up is removing page_remove_rmap() along with its "compound" parameter and smilarly converting all other rmap functions. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 47 +++++++++++++++++++++++++++-- mm/memory.c | 2 +- mm/rmap.c | 72 ++++++++++++++++++++++++++++---------------- 3 files changed, 92 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 77e336f86c72d..a4a30c361ac50 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -186,6 +186,45 @@ typedef int __bitwise rmap_t; */ #define RMAP_COMPOUND ((__force rmap_t)BIT(1)) +/* + * Internally, we're using an enum to specify the granularity. Usually, + * we make the compiler create specialized variants for the different + * granularity. + */ +enum rmap_mode { + RMAP_MODE_PTE = 0, + RMAP_MODE_PMD, +}; + +static inline void __folio_rmap_sanity_checks(struct folio *folio, + struct page *page, unsigned int nr_pages, enum rmap_mode mode) +{ + /* hugetlb folios are handled separately. */ + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + !folio_test_large_rmappable(folio), folio); + + VM_WARN_ON_ONCE(!nr_pages || nr_pages > folio_nr_pages(folio)); + VM_WARN_ON_FOLIO(page_folio(page) != folio, folio); + VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) != folio, folio); + + switch (mode) { + case RMAP_MODE_PTE: + break; + case RMAP_MODE_PMD: + /* + * We don't support folios larger than a single PMD yet. So + * when RMAP_MODE_PMD is set, we assume that we are creating + * a single "entire" mapping of the folio. + */ + VM_WARN_ON_FOLIO(folio_nr_pages(folio) != HPAGE_PMD_NR, folio); + VM_WARN_ON_FOLIO(nr_pages != HPAGE_PMD_NR, folio); + break; + default: + VM_WARN_ON_ONCE(true); + } +} + /* * rmap interfaces called when adding or removing pte of page */ @@ -198,8 +237,12 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); -void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, - struct vm_area_struct *, bool compound); +void folio_add_file_rmap_ptes(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *); +#define folio_add_file_rmap_pte(folio, page, vma) \ + folio_add_file_rmap_ptes(folio, page, 1, vma) +void folio_add_file_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/memory.c b/mm/memory.c index 1f18ed4a54971..15325587cff01 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4414,7 +4414,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, folio_add_lru_vma(folio, vma); } else { add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); - folio_add_file_rmap_range(folio, page, nr, vma, false); + folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); diff --git a/mm/rmap.c b/mm/rmap.c index a735ecca47a81..1614d98062948 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1334,31 +1334,19 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, SetPageAnonExclusive(&folio->page); } -/** - * folio_add_file_rmap_range - add pte mapping to page range of a folio - * @folio: The folio to add the mapping to - * @page: The first page to add - * @nr_pages: The number of pages which will be mapped - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The page range of folio is defined by [first_page, first_page + nr_pages) - * - * The caller needs to hold the pte lock. - */ -void folio_add_file_rmap_range(struct folio *folio, struct page *page, - unsigned int nr_pages, struct vm_area_struct *vma, - bool compound) +static __always_inline void __folio_add_file_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *vma, enum rmap_mode mode) { atomic_t *mapped = &folio->_nr_pages_mapped; unsigned int nr_pmdmapped = 0, first; int nr = 0; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { + if (likely(mode == RMAP_MODE_PTE)) { do { first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { @@ -1369,9 +1357,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, if (first) nr++; } while (page++, --nr_pages > 0); - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ - + } else if (mode == RMAP_MODE_PMD) { first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1399,6 +1385,43 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, mlock_vma_folio(folio, vma); } +/** + * folio_add_file_rmap_ptes - add PTE mappings to a page range of a folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages that will be mapped using PTEs + * @vma: The vm area in which the mappings are added + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_ptes(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma) +{ + __folio_add_file_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_add_file_rmap_pmd - add a PMD mapping to a page range of a folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to @@ -1411,16 +1434,13 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); - unsigned int nr_pages; VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); if (likely(!compound)) - nr_pages = 1; + folio_add_file_rmap_pte(folio, page, vma); else - nr_pages = folio_nr_pages(folio); - - folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); + folio_add_file_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 4 14:21:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D723AC10F05 for ; Mon, 4 Dec 2023 14:22:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADB156B0121; Mon, 4 Dec 2023 09:22:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A87896B0128; Mon, 4 Dec 2023 09:22:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DAAC6B0125; Mon, 4 Dec 2023 09:22:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 79C386B0121 for ; Mon, 4 Dec 2023 09:22:11 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 46E01C02B5 for ; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) X-FDA: 81529350462.18.B963083 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 725E640002 for ; Mon, 4 Dec 2023 14:22:09 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LAlUzxoT; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rWanwqicBfazF+1iBEpyOQZ25LJEujBQqHyMDlWvhqA=; b=MzWpbBj9bAMr88VzSTNnrne517137S0NmQS/Yxrs+M31zKnwPgJ2hqbRwQxfQzBQGzNz17 C4JXZEG9RWndkemFy5Xfw59BJpxAWcx+p7U1lZvSnoiFh4u2FDuB56R9VGxPdGoxWXAWfi CXj17GkDFBgKBGRYNyTRbx3kMYbsaFQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LAlUzxoT; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699729; a=rsa-sha256; cv=none; b=GmaFwoVG0SC6wR0EvEiDuIscgAg/1Qmy7RqoX/KA/exs3goOiQZbqcrH05D6HVcn6hGaGb z4ci2iRMbarKBRwFcFdcTAtGgNkSs3L8jcaRqgU0y8gbChYv2r0zemc8m/CdzOvEG/vHx0 5qKsfGc3/Bp3s9ejBkkzVcayT6zdYcQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699728; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rWanwqicBfazF+1iBEpyOQZ25LJEujBQqHyMDlWvhqA=; b=LAlUzxoTyYLO0ukIM9henASMEmCRTRSh4aGJPKaCytwf4HCDnHzrRhaJXTryPDhxp3+QVN CJoyKXp8xctxkOMMePl2kFlSCCcxNMug8X7APeLBZPJcKg63S1G4ETcgewRLEwuIoFpP07 jF89SvjseSB7Ksw4MsjUQ4mliVFAx8s= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-31-zbIm8uHQNrW7J5QBtxdQDA-1; Mon, 04 Dec 2023 09:22:04 -0500 X-MC-Unique: zbIm8uHQNrW7J5QBtxdQDA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8C48C2825E93; Mon, 4 Dec 2023 14:22:03 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF5FB2026D68; Mon, 4 Dec 2023 14:22:01 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 08/39] mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]() Date: Mon, 4 Dec 2023 15:21:15 +0100 Message-ID: <20231204142146.91437-9-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 725E640002 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ry8t6eap5oz7phhdtnxuyj5h1t7kmy4j X-HE-Tag: 1701699729-462337 X-HE-Meta: U2FsdGVkX18ZAvIyyUvF1TCoYn2wAKjDDtue3xt45DnUqyc8k2Efq1n+zjNHcYXWoK6oz/OOgoBOsZcYWeKp4hhlH4GEBs7rew+pLi3JBxJOQHSZpa5AZEQ8JgEtkiwgdion3qXYpv/c42sCYCYtd8bpAdtt86DlzwK8eftcbG+NCAF9sHz3XunNC59uMYiPyBVN3Fl7UFFDbUuku+Kp28kY3aOtYL4WxaeyMuWErCQzHMNSwikSPYAJfSqtdmIk6ufIyzRBtOZWRdGai9SHi/ZgdUAqFwabEVNZq1ruvHM2V6qYiUo7BUJYBiWTlklu5mwWrhv/H6kNiZ+/SCVt0N3j26rDN6LdXv2p1W+kw3J9I3mW1dvUmOBXFh9Zy23+GyY5hAwVUNJg6TfixtCP+nf0dBUhgiLVPNepfDQsuOqigFIQi8VV8wBWUx23Hm5QkMIyEgEhJdAXBOeUX2LiCmKdZwfPXTF595R6rkYzYIM592Uoq7jU46Le+PxhiS2L6CNyhZWZMJaq3bbSTmoXohnF1tPPYjJ7zn13XQ1xtj+Kp3/hO5ZVvWQaEQBOA+YL4Jn0Dp8I6ZREbk/bqji086YMWRJfQyT+lGojtqpZsdZ+02vAisCk9tek/o9toYDtABkRln+L3VBHTEDuMkwqk9TrnY+UT4qMtunuk0os1dYaxTYM9rrt9bu3IWOK6vQCIEmoYujvF1JmVrcScTOzqfa20SoHKhlkcEaXMQdaXr3fxiRxd/2JPjjNaMiYwxTCwZM3ebPh62hNUaf11vDV97gYPfCgxS7MF8TroRcP4MOfhv93eQ/yv44XhXJJe9FHJ8IsHxFzQZr5XySJnkYucGMEIQ+9D8/Jwo33SYkVz15T6UfntKLuyncBnkWJYOP+lQpoC51spJvsAVO5L3/+74J0MYk9Jj3bB8EjJAbB80At+oHpCLZOxRab7tvVslKsJm06vaT4Ex4YVkxvUoZ 3VsOVk1m yo5PdErG2FGaX/0GlLa5iX5DdMp/f39/1KKJ+Ul0r0ya78nwco5bL417FaFKhxXaF2tyLu8zA1+j6iz8S3Q+C5adJdmw24w7QnBGSG4jOkjJX3RJmYpXJ4f05YQBmaSPq72kBYgrXEN4KyIZGxnib5oJhKDFtVvwZiSOVTzvMiDBeSYZhn7f7spt5zA9699PzJ4nVP6gNPsXiCKWTpT7v8b5SEw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert insert_page_into_pte_locked() and do_set_pmd(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/memory.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 15325587cff01..be7fe58f7c297 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1845,12 +1845,14 @@ static int validate_page_before_insert(struct page *page) static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, unsigned long addr, struct page *page, pgprot_t prot) { + struct folio *folio = page_folio(page); + if (!pte_none(ptep_get(pte))) return -EBUSY; /* Ok, finally just insert the thing.. */ - get_page(page); + folio_get(folio); inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); + folio_add_file_rmap_pte(folio, page, vma); set_pte_at(vma->vm_mm, addr, pte, mk_pte(page, prot)); return 0; } @@ -4308,6 +4310,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf) vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = vmf->vma; bool write = vmf->flags & FAULT_FLAG_WRITE; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; @@ -4317,8 +4320,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) if (!transhuge_vma_suitable(vma, haddr)) return ret; - page = compound_head(page); - if (compound_order(page) != HPAGE_PMD_ORDER) + if (page != &folio->page || folio_order(folio) != HPAGE_PMD_ORDER) return ret; /* @@ -4327,7 +4329,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * check. This kind of THP just can be PTE mapped. Access to * the corrupted subpage should trigger SIGBUS as expected. */ - if (unlikely(PageHasHWPoisoned(page))) + if (unlikely(folio_test_has_hwpoisoned(folio))) return ret; /* @@ -4351,7 +4353,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); - page_add_file_rmap(page, vma, true); + folio_add_file_rmap_pmd(folio, page, vma); /* * deposit and withdraw with pmd lock held From patchwork Mon Dec 4 14:21:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66F70C4167B for ; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18E6D6B0128; Mon, 4 Dec 2023 09:22:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 13E0F6B012E; Mon, 4 Dec 2023 09:22:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB0F06B012C; Mon, 4 Dec 2023 09:22:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D1FDE6B0125 for ; Mon, 4 Dec 2023 09:22:11 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 93609C02B5 for ; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) X-FDA: 81529350462.13.F7189E9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id CF6EF1C001E for ; Mon, 4 Dec 2023 14:22:09 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GsBjlqWdTtqgOxvGGH4v93t8i0Uklxk4bjGdGt4vg9o=; b=O7W00iLM+1uLHjHJLWEhdfl1gy5Fl3LpqElgf0w4Ic164NxEq7GqTjeIrmBENSdgbcjKT5 I+jvtQuHSzmzvPoOnQ58lcYUEzd3ftujOpSgS220PqOrYiYaU5HxsOyvubdrKU/dnqeXAs f0Ein3bBRfU7ctKBu4yWXgjLCRuuPLg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699729; a=rsa-sha256; cv=none; b=fH/vbkXvJ8WvnfD7293my0VwO3euTTiFC/c6OSVnX6PswGSD/rFH1j8BVei7Yk7/bkop8M xHB/5jodhYmQX0cVR0qfJfOpe+II8oKBLVoTstfUQJecHy6TgoPOS1uXJRuf8UBq9EsbUo tnTZNr8XLEs9C0239Y9WvdvGaeJDm/I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-60-Bad9lET6NLK8wnUArWRxyg-1; Mon, 04 Dec 2023 09:22:06 -0500 X-MC-Unique: Bad9lET6NLK8wnUArWRxyg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7602885A5B5; Mon, 4 Dec 2023 14:22:05 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C73B42026D4C; Mon, 4 Dec 2023 14:22:03 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 09/39] mm/huge_memory: page_add_file_rmap() -> folio_add_file_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:16 +0100 Message-ID: <20231204142146.91437-10-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: CF6EF1C001E X-Stat-Signature: e86muraxap8eu7wh6afz4i37ws5ctzq5 X-HE-Tag: 1701699729-734515 X-HE-Meta: U2FsdGVkX1/1SduIWi3ng0y8EY0tfSDrDAJi2tYbM4RclSqbImgkxxqZJxi5w2xpFip4lHXDJX70G/PFD0j8f2UGmnic+7gJCSxe7AgqgvJqkyJ9RJbF9WjxFvcxTmQEBOZ4KIbkafc+qnbjMmfR69NXe+/j6PgP/yGJIS0HR1JBOCzayj+KmaeEss/hmOr1pIP+DjaAWx5YeIqZM3Y/RM5/prQKnjed8SGUsolwILKJigqDMLJ3EppDKywntAZaNLSoRZ5CwXezfWGuoBe/0XSGy/P2O/Y9HnMKDpvrIrt8Cn431vmL+ND8Orf3lsgYNs1w5isKFPdfNHIaEb8dYhXrbEc83CkPJFrPJkAbSjnbRoKXXkehz1r8xLcVI84Ph9EPEgHtHVxMGZCqTm+0UFhmHmravNW/PoPUjfNUo2BoGKlUOz0gkBPWQcNUoF4r6OfQV3JrNXKmQ6h8XsSh2Rn45mif+tG4uvgUqOpdwcRpBFiluKPWcOuEAAJuzAoaEOY0VensE9SEWju8AwXclbM+pwgH0nupoqutPHSAmh2KJXcpWTUYzicaj1890RpkKJUSXTi7BIM3AdPES4iCIRQ9Ks27/XwfT0y7w8jn9TPf9VfmUSoeMjSHETap9xfYZ6BDt16yr/fLLOQu+awbDslDHc/fRznNUDqc0A1DetFmhxTp0sBsW1qfbfMSqlvVgT5CjxA9dpQvf6WC9mDtPo8srqor386e1qDk8R6dH+AJ06J9yla+INcAxr6/JFNI+I8hs7CEtUPuew8LMNwlNxT8Td4eHfEjEUXBwqKWJBFyEujUYILeRAok5TOl4KnO5Jh9HyhuCXc4iFjIBytWHNw5IC1ReQNzfaw+ICBU+ax9bP2FRPq0vKEiCIsCGROGQa+FvbWjvcSrEFeqMYHMIwHjHUeAHW2Nosx1H2c0vt4OpKIS01ICshnn8KaVzxPPS1rUP8Pe/rDZmDc5kvr zLF0naqm 7kqVcf/7D8hK8jeI+Q7eWy3DPv9atuOBbwSk9dr0DA4AxVeuAexNfYGbaR22w/pmHJ2tzD0myxnGi65FL5OjIsc7X4glIC9jSpARwSYsLbd6sMq+Rzv8FdeWGK8CA9kOgLVCs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pmd() and while at it, perform some folio conversion. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/huge_memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4f542444a91f2..cb33c6e0404cf 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3276,6 +3276,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) { + struct folio *folio = page_folio(new); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3287,7 +3288,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) return; entry = pmd_to_swp_entry(*pvmw->pmd); - get_page(new); + folio_get(folio); pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); @@ -3298,10 +3299,10 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) if (!is_migration_entry_young(entry)) pmde = pmd_mkold(pmde); /* NOTE: this may contain setting soft-dirty on some archs */ - if (PageDirty(new) && is_migration_entry_dirty(entry)) + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) pmde = pmd_mkdirty(pmde); - if (PageAnon(new)) { + if (folio_test_anon(folio)) { rmap_t rmap_flags = RMAP_COMPOUND; if (!is_readable_migration_entry(entry)) @@ -3309,9 +3310,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) page_add_anon_rmap(new, vma, haddr, rmap_flags); } else { - page_add_file_rmap(new, vma, true); + folio_add_file_rmap_pmd(folio, new, vma); } - VM_BUG_ON(pmd_write(pmde) && PageAnon(new) && !PageAnonExclusive(new)); + VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new)); set_pmd_at(mm, haddr, pvmw->pmd, pmde); /* No need to invalidate - it was non-present before */ From patchwork Mon Dec 4 14:21:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478500 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC3CAC10DC1 for ; Mon, 4 Dec 2023 14:22:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50BC26B013D; Mon, 4 Dec 2023 09:22:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4BB8F6B013F; Mon, 4 Dec 2023 09:22:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3398D6B0140; Mon, 4 Dec 2023 09:22:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1E7EC6B013D for ; Mon, 4 Dec 2023 09:22:14 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E5784120253 for ; Mon, 4 Dec 2023 14:22:13 +0000 (UTC) X-FDA: 81529350546.17.5ADF0DF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 33FC920014 for ; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YzDKJ7vK; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699732; a=rsa-sha256; cv=none; b=weELUZCd0qmlJPCP//RDgxIlPGwA8CuLRSkuFhDpYb5oSYzh8bdF/4MOK5gE1wOyFmiybH 4/uBmskyqyqk3+ppiEYhwzz50aoAR4oIzgHmqZ1b9I1U4ikqgbuR2ETur+tBhRkMmEOSEL w5f5Yo6pw49vNCNBY+nZ/3fspVug7mE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YzDKJ7vK; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699732; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HFIwQ5rYAD0bMXQeuRR1LkqvEC8kZB0MG/s3u0E9e4Q=; b=xjbUMNtfkoIHsNtmCtxhXDGjo21BtZ/ibBay/dMIIMb6gSq3fmUQatgwxtV+gfwHeZYNgM M0UWp08lg3BXiWxZgYtlA/HNgBY4JR8XqEAg6mOhyXOpBxRkZE/y7odFSliO/3JUJ80ozh troEHJHh3Ec0u/b8++lu8Vbfh4Tjtks= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HFIwQ5rYAD0bMXQeuRR1LkqvEC8kZB0MG/s3u0E9e4Q=; b=YzDKJ7vKLDzqjPvT+1Jw2uC45tqKoyi2nQz+2g4ZBja1dOzoFT7tIfPvkDKqhMGwIG5iSo xQ/C83V3DH8pIO8P0dxOz8zKCpImw1Mcn1fcVlNCxRW2oJrZuoqokZw7d10srE3z2z4VK7 ddy+3Qy4xe41gk4DGi/t9xjlDJdzU1M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-253-KmX5I8pcNJiCMDlB3yQUQg-1; Mon, 04 Dec 2023 09:22:08 -0500 X-MC-Unique: KmX5I8pcNJiCMDlB3yQUQg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 524F58F9207; Mon, 4 Dec 2023 14:22:07 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id AE2382026D4C; Mon, 4 Dec 2023 14:22:05 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 10/39] mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 4 Dec 2023 15:21:17 +0100 Message-ID: <20231204142146.91437-11-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 33FC920014 X-Stat-Signature: 46sdoyuekdo7r3y74ogthk3cpcprs34w X-HE-Tag: 1701699731-336547 X-HE-Meta: U2FsdGVkX19xKGT7Ve0LWr7Ob9Y1H4QGs1NxtwMKTHnfyoMemHCItS+uXL5fsLWPKdYBDkMXBuKeIsW66ydzgxSiC1ftEbZeviuT5V0+Hc6ni4YnaT81MxLrjjitqS91gtZoImFVzie22RRa4UzL7Y9m24C8zLgLs+8pRcQxBQ3A0hMg/AXIlPOLvSKfaqSjXEvNvGUNj+xGVJa9QaKKXvcBib78OMkZgoTm777W/k6Bzy4gzXsq3nfSyJ58rMNpkBjzuJ4sFOGDjQQztl/pANdVHNYcjuj6Vqh1lampPfzp55tXrXPhZb9NC236NOXGgtYRLlUayR8TLdyX1fbcvs0IQeFsCfJ+j/bwnzU2YHX9cKIEKK5+pbmVaNPNRTtp4r7mlucKlmJxTdIL2sgu5QWKKMcnwpoRtERbvgoI4PC2CyaTiMiBZi+wG+C9zbKIL1fJBDLUAleXIBW6iZFPGmQQYiNAlVDl6PMUS2BYWqaShlD6mI90xjJWBx/94CgITrxHTczFN1KUygStKnMbsb50DTqax8+nf8NhAEJEkkME6a0jnYyVJj8GYGt0hgBAFms9tjHdFMjlobFWLjwrftyC6gafC7C9DtvM5FQaFTXf3AukqiDsvuI9QIdVcD729VgJts5+3s53NoChlpZDFrA9+8hTcVsIC0shJ0czfdv8EU0zSMQ44vkkAP0C2MWZB+CYTRS1pHMeWS7IOAJUP240vztZoqKLxgSo8bmbpxQgxoG/sVIqVLivt09sR7fFtHDThivuydtLD2wRyz0BRwwVJOTEtL5EybAAA1kQ/FIIaEeQjh1CPCPUpdB5ZtbqByHKJq2r48aeGWlHFxF1hlxH8C+IpKQAm+/wFF8gl5/kGlSF9PT7J1HvHB83e+t4+HIl+9f6JTRYjetKqjTfmRmipnG1kClVHc4BBOoISsM0p35jDy3GrGyZnIqAcuq9DoHclSro6h13g+0V3q0 7ayYY9uL 2Q2mQzXk3cTaw3X24pKNsFXGUxTWT/KxrmOR1B1m6hGmJmJ+3eiYMjs+YqcpgEIuUjKK8GUK1AMkzPdHy01jeEkd8NI6I78pjRd/MyyFb4Rhk1bMG7n8xO7GAm/j0ZVQMJRbleZMfMIfeD2I3DWcwGqi+kZ96dcH3Uf28wKkrLqbaMYCZpAoS6QEIa8WLl8I9R+i9hbvsF7GCxovBVrEXtVhNSjwLa2ZFVWkHSU+a4kHxwXTNh+hELS5l1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index de9d94b99ab78..efc19f53b05e6 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -262,7 +262,7 @@ static bool remove_migration_pte(struct folio *folio, page_add_anon_rmap(new, vma, pvmw.address, rmap_flags); else - page_add_file_rmap(new, vma, false); + folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } if (vma->vm_flags & VM_LOCKED) From patchwork Mon Dec 4 14:21:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CA8BC4167B for ; Mon, 4 Dec 2023 14:22:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5C206B0150; Mon, 4 Dec 2023 09:22:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE60B6B0157; Mon, 4 Dec 2023 09:22:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC2606B0159; Mon, 4 Dec 2023 09:22:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A5E576B0150 for ; Mon, 4 Dec 2023 09:22:15 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 87EBE1A025B for ; Mon, 4 Dec 2023 14:22:15 +0000 (UTC) X-FDA: 81529350630.04.F390B46 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id AE59B4001C for ; Mon, 4 Dec 2023 14:22:13 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ik7hIMgf; spf=pass (imf07.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699733; a=rsa-sha256; cv=none; b=lrAMpI0yRFrXmyYpqzxIpF79QVb+x5+F/HcULO/q5vFJVsv94GrOVvBFluyVz5b3KH42px mqGQKtap7UDmFlJwXl6fZB4eedfIWCjL6iFu4Gq6g9l36f8R0l3MNabMB40E2w/fwfoqKz qiSfMnk9bgbPKQgTPO19Wd+H/X032DQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ik7hIMgf; spf=pass (imf07.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DcozSwix+76TtkWexeYoWqf1En6dAEapeBtTC2bk/5E=; b=FQohPufz1F7o8w6seeblBL4b5avS4lYoq5JnrBf3PcBJgNLAyhwgkUvioM15mU9fuzkxPT luY0fqvr047Z7S5wnG0HUnC84k8fjUgeF1W9BhpxXhZT+8eqQbvEIVMfm23Nts//MR2U6W 6KJ9o+PjaJKt9YNX8KeeogpIgLKBxec= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699733; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DcozSwix+76TtkWexeYoWqf1En6dAEapeBtTC2bk/5E=; b=Ik7hIMgfP2ulxvd0KMg8QH8/0DOwV62IgmkUV2kR2mPnQ/lO+OnYPxxgM/5wfKsrWacWzI AKSBnE3owc78rHYAgPQ/iQtblRFSgXmwYjZuAemD9tIoSwEoBkPZbm5ciX1eAy124w4jcA agMJ280xg8/Xwjvgw3BH4KmWCnnwgcE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-261--TlKEsHfM9OY2J5oZszMwg-1; Mon, 04 Dec 2023 09:22:09 -0500 X-MC-Unique: -TlKEsHfM9OY2J5oZszMwg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 290B08F9206; Mon, 4 Dec 2023 14:22:09 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BF132026D68; Mon, 4 Dec 2023 14:22:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 11/39] mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 4 Dec 2023 15:21:18 +0100 Message-ID: <20231204142146.91437-12-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AE59B4001C X-Stat-Signature: ein865gofp74nej189b184z6exkzs6pz X-Rspam-User: X-HE-Tag: 1701699733-99883 X-HE-Meta: U2FsdGVkX18FbQJZaZ48RGTdVgrI0Cm+iXh0fLBz22SFVc9MPGypa/iu/rLnxZ34RoDmc1uzfa/1ZevjEchS1LKwlC2CUsUfpM6SBncQ6nXQmPdbXCxD0Bc9XYfG8t/3G5qIsPnSPc825yHzg1OYOFpMM4gyAAYdl0zR5zTNAXtr3MMhsqE4DV9IWrygpbFOB0TV+7tIoKypU4HdpB0z6Po81FJgE27da7GrSzz4zuBbEX1zMk+1ohxrVo6097Uy531HWA3zec9ekdIRpLenBQvP4COC8pmrETdgHNodRlj/sjEpzT6igJPGAsBwv3lD7HJfb0XMt8ABEqMf5Q6qoJQCdrs9QUBPcd0Wz3kZi4WSFrv5ME42cpyctxx6KhLqDGaPnYobvzINEVREV4np2M+HU9GjMpDZi+IHhkapFRVKnn6XVjHhZv+h+Py/omvIFFkU9JII41IRcih0mZxUB2UtT/h3v+GVH5vGEPfApD3wBLowosLidFqWwtgv/hacf2VrlecDVaZxYjENfX1XqwC8cOetdt/m8v3ttpIrZnFLXrg0dwDmWkuu2KnE8piyH2pm5TBMTs3Aq/USDQo7HH6Hc+tRCEByKpyMc9IoIJ0xZbv+GQrq4yPoLUa0oW56FWMU8RRgEQG72EfXUbs07s+09RTKAgEdSCIl0nBeTrHmLOBvsCcLfQ9ORDFb/IfPxHOR3el44oRmF5dzoJPNxQdN+s8GKH4k8H4PcGgfO1P1n/wnSwVKQn7WGFTxsGoKiNK/pPwGpd7qUNrZ15/XUKHbJhLC2kdow1lG4HkH/w4YI6oayair/CpZyeqBOgfm8umpn81l5tLq8L9I7XMP2f/+WFTxcJHAbgDElL5AQXzhxCaWOiPJxF5hdEdRreknBoTAxDSqSEfPT2IsOdEe91i9HPZQ9FCbq82SPBODHySgY9qDwxAiEMyiOPJnNj2fDjVzv8dDiznn8MapLaa kImr+sta mJ6PYJI5YRX8qikg5jRRU5v+uXltRp22YXgdZftIqaTGGHmkV10wmfQLUtDxuz+atINZV4rIFef0IAX73KR3KQlpvwGXvWDWr3OHypKKfe2WGaFPGP+O9u5UbR02KOeNv1/MxzjFYwIbS52Tq/dpiU0G9RniHUWCf+ghLJgEVNaqryWErGKY2CTTxwmnTEciZwau+VDpvIyPcMcFSvZJHjezK4g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert mfill_atomic_install_pte(). Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 0b6ca553bebec..abf4c579d328a 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -114,7 +114,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, /* Usually, cache pages are already added to LRU */ if (newly_allocated) folio_add_lru(folio); - page_add_file_rmap(page, dst_vma, false); + folio_add_file_rmap_pte(folio, page, dst_vma); } else { page_add_new_anon_rmap(page, dst_vma, dst_addr); folio_add_lru_vma(folio, dst_vma); From patchwork Mon Dec 4 14:21:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD12EC4167B for ; Mon, 4 Dec 2023 14:22:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CA6E6B0163; Mon, 4 Dec 2023 09:22:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 753A86B0169; Mon, 4 Dec 2023 09:22:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FA636B016A; Mon, 4 Dec 2023 09:22:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4B3996B0163 for ; Mon, 4 Dec 2023 09:22:18 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2CF95160265 for ; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) X-FDA: 81529350756.03.20854AF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 55A32180017 for ; Mon, 4 Dec 2023 14:22:16 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MkC57k5/"; spf=pass (imf06.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699736; a=rsa-sha256; cv=none; b=ddRIvbUu+9n8ftwLOWbY4tFyN+VLi34fw8WFoK1EQBevOkanJOoP0uQrmAd3AKzyZH+9c7 HRbVRsGzeMGf1XmhCK4EBXuwFaB7kD8xaBGajFDApw3ZEoHrgZXSe4fFEe18coNhUBgcHq ReH4xFlYAogg5Vlb/zsK2X+ylgLX3x0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MkC57k5/"; spf=pass (imf06.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O9+7fynWu7ZwbI+2ouLA8wkNah+BDZS2kzyGsyNToAE=; b=0bAOyFBONoDrCA3aaaW/f8nFYfUH4kC8jNqyTnrGdzym5m5jWPP9XezbaWYor6u0GF21eI GGI5hPICVEO6vDv117mtMs7rIIq5bnPTbcqL04xC90VzNfirOCmysPvaV4EsI/QEjQH0e1 dtHElIsQ4Byewmg7wc6FngvxTp+o5tc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699735; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O9+7fynWu7ZwbI+2ouLA8wkNah+BDZS2kzyGsyNToAE=; b=MkC57k5/9apOX9TByBbc8MfBAqnP+ES4/9MKXH3IavpjrNHTMV6GdqMlzYeFmOLis5aGCn uI4Z8d4E4x1O0HAQke5XGN6aUhqKj3vOqSW5NMg671FOwDJ6pYd3xCfl18017vyX0w4Koe LaB0t0AKG/DyLn+Nu0bHCV3qlOoBgRI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-518-ekuHhhc4MEqvFS8uO4ctFw-1; Mon, 04 Dec 2023 09:22:11 -0500 X-MC-Unique: ekuHhhc4MEqvFS8uO4ctFw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E725811E7E; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 616822026D68; Mon, 4 Dec 2023 14:22:09 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 12/39] mm/rmap: remove page_add_file_rmap() Date: Mon, 4 Dec 2023 15:21:19 +0100 Message-ID: <20231204142146.91437-13-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 55A32180017 X-Stat-Signature: g6qk48zi6kgxngqh9779mkq486qfeq3g X-Rspam-User: X-HE-Tag: 1701699736-779105 X-HE-Meta: U2FsdGVkX19aMpzwCKIC6yGJiEjyxRf5KIFN9GeGRFS4tfN6UWhIk4TCVMnpkryiDyrwFIjtniEu481OMsY7BztV3oqwJKN5VHDX5bRc9xGwNaA4W3mKeacmDc1EoO2i3FsBtL61jd6wxKsvqj5g764XaZ3wmBmcIynhd/OxIz2/X+THQtjsTHd+KIYTo2Gpz2c463E7RgxeQoVI5Qbzi5Al6oI8xRw/j4qk0eZ6TZyHeItLnY7AL2acXH/+33B1pd9/RE3xA8ESCO/1N3MCThYdILQ4woVatS6lN6ywqdTUyBFg1pHGjxb0zN062wQEmNc0Y4mOeL8oXegNEETajCnl5BwXsk+TcimPE7FGIM5bVsNh8rcOzocovbH0lbRxzEwRdMeAL3MX5iQ1YhoWpnIcH6C2OxXYUanOSWo7b0DvDMxNdidXUPE3/JEUPqn66avK7eGDwnd+NkEscVXbRCCS7hGMhpLwZ6KpyrKtEjZXjatO4ClDrdDsmoOHXVJFELz1wHZyNpzqWaZRpGREvy2z481gQx5LGjdtdVjb/tUCeU06Nd55Ef9Q9Y58z521CkEumubjb6eX8Jjd7vbaRhbQB8xfY0LIu6AhxaYJqKHf8sRC0L6Lqh2E8rNWkLI3IP2vfkqsjabUPdpXtwwYgaGJEr+AUv1/FeFNxz02ifCJAoEg8ZrWtyhGllR7KG8xUf0OdL5QBtTLSgZdALvHE5zNSiTtqP4OgCF7jsBSsYR++RYaiXouRnQJM6l+adHHnBcbhXyWDYda/OPzvw9eVHtvomwytRxKbgAcszrJ4x6SRJWirAuTBvNqJpM1OqXEztfVVKWtrHhbO504KBwwsC4pyfsTF9SkIavZyMEwdj+lkwZOmgw4JNb2NkY3Md2mpBVTKf6huT2SLy+wvgxz2cXoNB/i6t8tSENrbYSYQ30mz/0ERLRUy2wEc1UbE1Ocqfaz1jH7Bc1C8f7BDVN dOLG967M 9JgnUEkZOl55Gvs+APeyHYgFoxhydE5kaWISd58ZExfXVcXfcoEYNCgzVPHQNtLH0ms5dHYG7hjLQUI2GaXzNFGljpnbfc8oHKf1ItAOKGre1srnVE+xJsrbf5yjhrFYz5Ky5JRvSa6CIiw7nGlGOYsVJBaofqsEGDBndjIntS+IAhUEmaQXFLXe28c32Z+lkZS0EGBWye0hzq9qIv7rEGoKvjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, let's remove it. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 2 -- mm/rmap.c | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a4a30c361ac50..95f7b94a70295 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -235,8 +235,6 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -void page_add_file_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_add_file_rmap_ptes(struct folio *, struct page *, unsigned int nr, struct vm_area_struct *); #define folio_add_file_rmap_pte(folio, page, vma) \ diff --git a/mm/rmap.c b/mm/rmap.c index 1614d98062948..53e2c653be99a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1422,27 +1422,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); - - if (likely(!compound)) - folio_add_file_rmap_pte(folio, page, vma); - else - folio_add_file_rmap_pmd(folio, page, vma); -} - /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Mon Dec 4 14:21:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8461DC4167B for ; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 819756B0188; Mon, 4 Dec 2023 09:22:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CDCC6B018C; Mon, 4 Dec 2023 09:22:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F6616B018D; Mon, 4 Dec 2023 09:22:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 447E16B0188 for ; Mon, 4 Dec 2023 09:22:20 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 19EA7C012E for ; Mon, 4 Dec 2023 14:22:20 +0000 (UTC) X-FDA: 81529350840.13.B1750B3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 65245140003 for ; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cV4zFsAt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HK+wCLAanK8icsaUODtbkemdg+UtHwl4vMlZjDb9LtY=; b=IAQ/I4q1M2YRdsznf598o1DlIeOYao9M5x6IUTuwl2jOjyRxtX9YD8fwARaiDPPqwHWAkb ih1jATaJrfxYzcj23pBdhYuUPD4TnH0ZC1DeuEmpxCg/yfyjU8CRqonK/Ij6ML4MxVL3m9 PDk6jMM3Z2Q8Yvq9UgLJftje/IsyZl0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cV4zFsAt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699738; a=rsa-sha256; cv=none; b=GvZobcG5+cHUMVNk8HoBP23tiqPVIfXiDH0mmHT/OmCYFA5I+Ym4A23Kxk5QgAbjDBWL+m qZxGLlj7NCsArFgMqm95cpHuMnZrmWbA3feWTALuD6HeDkIJOR0fRU3VrQwDdeyLYc9YRG LUPfKT6YjjRNx2uAgT232oUgesUeUTI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699737; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HK+wCLAanK8icsaUODtbkemdg+UtHwl4vMlZjDb9LtY=; b=cV4zFsAtsScxcDDOttLWHSPeaYcjxejUP6jJk20gSBa//4kgKFagntGzja718DD0tP7A0Y 5NgRgQqZOrqXPct3kHZNbYmNnYbS9xEPnlg5aqasND7ky5EsWfEx0dXTxpb93ct2ildDcR pKvA6Q2u6fADregmQJetz05GH0dVMAU= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-441-tU-ty63bN-Gpvd547o2MWA-1; Mon, 04 Dec 2023 09:22:13 -0500 X-MC-Unique: tU-ty63bN-Gpvd547o2MWA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E0D0129AA3AE; Mon, 4 Dec 2023 14:22:12 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 477C72026D6E; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 13/39] mm/rmap: factor out adding folio mappings into __folio_add_rmap() Date: Mon, 4 Dec 2023 15:21:20 +0100 Message-ID: <20231204142146.91437-14-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 65245140003 X-Stat-Signature: ip3jq6iftrnyf45p4ac9qzwz6ez5585x X-HE-Tag: 1701699738-733212 X-HE-Meta: U2FsdGVkX1+hzlQYv8Ml/+QamMYC7szSI55Baa45uZrU9SgD8B8A5k4CNCMlORw6wS2GqV8a5T8RB62H6evYlapY2JPkcvaBqYe52HxJ96AlzZb4QA27LtMz8uuubTPL3AIHBp5oh7iAuYM8KjcqbYen/Kg/5LE5ASneHm31re9bmbFH9zaganx+kCvn8iX4VbrViSx7KZrjlZM6yv/euBc5OZWT3NienCimaEdOyHNcXw27NiILpNuIWeMXQWLntHJwzziDE1HDXydBxC0oPg7jTHHGW/PGcQxr4CF2CKjRgu3fjWzimbgZCddCG/zlMNSyyi1EWqHv6m5rc/SGYSu6OWadkLok6rigp0gSBt1B1Q6rw6In99EUhkq9qBhyhiv1QAwlDWkr8qX59PTf/OwxjTuXNqlnYEu94dTmOULt6T22ZgaP71BWiE0deeiu3r7i+jemak8jgv6xXmimV+50M66TeHeRCpmuEpmj/hpjJCFEbWcBtwwa96CPcf8WN/h1bQyW0kdLABZKhPXfRfuN4Tpv5O0KihLtDETJr9JQPdnd3Fn/vuBpnq8a0Mw/Uzp6hlXIcdT8S196RsWTMUaLSSiGZhUagqUlRGgEF85UhDLdeARVcdfkMIm+rauJslKpejoFBkFWyuVXOAGvG1/aym0btjwgWPXO+ZBxcYMf7/0owm/UqgNpCwA7iq/kFhy/ywh64tfIr8l4g/q9k4xofGXlD8MPdCQjyA1ZBA1fCCGcHXpU1cIX0GYuqq4cWTPJcYDp9mRzm5OYemPnJCeKv3VPITyKtokeZHykODa0qA6SvEoLLYWHA23R6wCXqeZMdTSNv2eqeFMzCLE+HIgFs5/vvnZnqRFPSuxXnpNR6iVF9+90vhybrniRDQgVUyeIFkteg5ZwJd/q64w8+2UlbXYU4CeoYJPxetat771ZpF59n7K7MapD8nvDUL8tCPBPlT5pV09gM5cU0zd Q+90bpAb 1w7n5biUP/N1kHcKhzRfeb8OIie8yHEDRXJcr9IPkb5js8/fKwWrD9muUKyuX9/OyitqNMzN3ZleI8SWBO1Lj4nRyh5l9Pp6uLjWOQ9VZLk0vW2UqSww4shHT0Yj3onLb0U7gjEDxEV79F0bmZJhu+JblLBv8Ur8uend1xd4hGMn8HuDS6Y2o84t1VDBTqNKGfWlxX1msDc65vdZFWh0f3gSO3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out to prepare for reuse as we convert page_add_anon_rmap() to folio_add_anon_rmap_[pte|ptes|pmd](). Make the compiler always special-case on the granularity by using __always_inline. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/rmap.c | 75 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 42 insertions(+), 33 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 53e2c653be99a..c09b360402599 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1127,6 +1127,46 @@ int folio_total_mapcount(struct folio *folio) return mapcount; } +static __always_inline unsigned int __folio_add_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, enum rmap_mode mode, + int *nr_pmdmapped) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int first, nr = 0; + + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + /* Is page being mapped by PTE? Is this its first map to be added? */ + if (likely(mode == RMAP_MODE_PTE)) { + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (first < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); + } else if (mode == RMAP_MODE_PMD) { + first = atomic_inc_and_test(&folio->_entire_mapcount); + if (first) { + nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + *nr_pmdmapped = folio_nr_pages(folio); + nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } + } + } + return nr; +} + /** * folio_move_anon_rmap - move a folio to our anon_vma * @folio: The folio to move to our anon_vma @@ -1338,42 +1378,11 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, unsigned int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) { - atomic_t *mapped = &folio->_nr_pages_mapped; - unsigned int nr_pmdmapped = 0, first; - int nr = 0; + unsigned int nr, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(mode == RMAP_MODE_PTE)) { - do { - first = atomic_inc_and_test(&page->_mapcount); - if (first && folio_test_large(folio)) { - first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); - } - - if (first) - nr++; - } while (page++, --nr_pages > 0); - } else if (mode == RMAP_MODE_PMD) { - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); From patchwork Mon Dec 4 14:21:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6390CC4167B for ; Mon, 4 Dec 2023 14:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B28C76B01A2; Mon, 4 Dec 2023 09:22:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB10E6B01A3; Mon, 4 Dec 2023 09:22:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 951E86B01A7; Mon, 4 Dec 2023 09:22:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7F7336B01A2 for ; Mon, 4 Dec 2023 09:22:23 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5A868A0282 for ; Mon, 4 Dec 2023 14:22:23 +0000 (UTC) X-FDA: 81529350966.06.68F2B91 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 801781C001E for ; Mon, 4 Dec 2023 14:22:21 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="b2m//SDD"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5BTpbXGEFWsHEYKSa5/ZaE1vDQBM0C6WRkCwnDKka/8=; b=T1K3fe8fZYJcqvk+wY/+zPtuMYi3JIag6rEFzpJaQcTUg8B3ToSmMmlSU5WpK5fT+QtCUz UTaHKoz5NPD9ijZbNWBUqoDOHsn1W/TIi41yPn7eC462m6QUJNcsOp2+NRw9R870LyJ25+ jUC8zhz/7tEXssqo7TIZHTDLCaQn+LI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="b2m//SDD"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699741; a=rsa-sha256; cv=none; b=swK1idQQWdcuvxm7RchpQUBipufIO29YMcm7VcfVUK8iIUdmEguKnKbjuQuUEw8co06dYY 4lPElEvtFgiangSN3gPzGadWPgNcXrNii/krP6APgvs0Vo25lv0qto1cUduOAhSuAe4dXU Z3uHXMW4XGrzXv/kGI8JSv0ri1LU7H4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5BTpbXGEFWsHEYKSa5/ZaE1vDQBM0C6WRkCwnDKka/8=; b=b2m//SDDsDdboijMlPseLOLJ0JysTBYh/lfNQLvP+8qnJWH0ZuqhSy0E4CQG29OZg1HAw6 IRO4estwtrf6THmy5U/d2FjbUVInRoN7RsrmIXIMClH0BdIN6acmEfkJyKuofjKA71E05e lxZ7mIY95gC6pgNUZjEDo3av4ZfJkXk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-491-JZokyINqMGSj3FxNN-lSUQ-1; Mon, 04 Dec 2023 09:22:15 -0500 X-MC-Unique: JZokyINqMGSj3FxNN-lSUQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BE6A6185A78A; Mon, 4 Dec 2023 14:22:14 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24C502026D4C; Mon, 4 Dec 2023 14:22:13 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 14/39] mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:21 +0100 Message-ID: <20231204142146.91437-15-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 801781C001E X-Stat-Signature: 5qqm8qx5te4x9kggfp7q5ix6mhyan3ka X-Rspam-User: X-HE-Tag: 1701699741-472166 X-HE-Meta: U2FsdGVkX1/VZyzPxQ1a97jS6D15ZJINRBn5BYQm+p0yePuO7G+ihhLroP7vXBoFK1T9FEJ5wAc7cc5GTJKCPZE7bfXTQEG1Rhald3dPFZnizi5M6YbZNAunD6TOmaRvpOOmEZvUrxSCYgRPMnIg577BcLruIJ7KxfD17zuEEkeG8lQsG+7EzL8pqtDiZhKmvLDVVDwkmFH1x5kKcIBPAeeiiyQaS6hPUlxFQHrGb4epwvntgypx+X7FTM+7aIUDOwRTBMe4h7JwG8ElDhq4UEWJ8rgltumwhS28tJn2Ub3W/akzZLXj0u90VBJPGrsqNQChn3FRiR8TgjePeRdHjiglQutDnl+KaPtsCFw21MqjuYLmo2jBggOyjVYfQs5NoK4bYbxYUbhGBw5sBfqA71IBPlSwMr1aSwki5G9X6sU4V4ZZ1K3LPVjrKUos/XtQUkZpJo82UEy0B7rvgIOK6LwAIVZrDk0rm8jyorf4TxtPbq9yH597Qa+Txpl9jbRuMc2Nog2+BdsiaR3NPqopvNF1Vh8v3d9zvV+3aJeDbtUH6vTkBepsaud2/awcFRy6XTchui/16t9WLOq5UnSzNmosxWYMgDQyOO8dq2g+o2nMJVTNzbaq7/UpCZPvPqAznlmGsLrYGtUXDBDU4zjXT9XP7EUBfhrmfx7MD1KK9HSMhRNhqIn6sDZ5QCN1qy+pbjkGu7WXUQGvsHtBTA+WcKmCGVfbQMW8IHWvM0uxs55Q+ngucZh8+DSOpyqWsmNzns3ltHwIrBnESpIDnvWugWhnmK2lIMEOgnLtVsMnB6ToumtYeqf13ywls20aFYle8fRwE1NiFfC1ET2tcniymeTa5XWlxRm3gaePnKVZFEAau7C0qYNQCarONtCqb0tdZ5Fnr35To/Gk2zVqcGRCQ9KW4/tYsQYGL700eWhmG6fZnhJx8KOJsjgi6jtW7jOAMPprKfGx9PfHi1u7TbC Ph6mKflN gphJIXu2MbpqRr3WsA+mWNI7bCNCYWghn+dBWFIlt6t6FR1FsD9mmTaFScOthBKd86vyQp36fpoMUjiXH+2cVMqf18zX1gSnkVivgA8vUKgqkxCAdfhhcWWj1n+5zXP6rJT6BAxG2PCv4ien4sfyTDh1HSmr2PQIM5fmjUxoraqv8mvgZ22bCRFic5w869WlsbzGpd+C1NGXR0VEcnonH9NNoqfhN/nwdsK16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's mimic what we did with folio_add_file_rmap_*() so we can similarly replace page_add_anon_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. Note that the new functions ignore the RMAP_COMPOUND flag, which we will remove as soon as page_add_anon_rmap() is gone. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 +++ mm/rmap.c | 115 +++++++++++++++++++++++++++++-------------- 2 files changed, 85 insertions(+), 36 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 95f7b94a70295..9e1c197f50199 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -229,6 +229,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, * rmap interfaces called when adding or removing pte of page */ void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); +void folio_add_anon_rmap_ptes(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *, unsigned long address, rmap_t flags); +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ + folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) +void folio_add_anon_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index c09b360402599..85bea11e9266b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1267,38 +1267,21 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { struct folio *folio = page_folio(page); - atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool compound = flags & RMAP_COMPOUND; - bool first; - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(!(flags & RMAP_COMPOUND))) + folio_add_anon_rmap_pte(folio, page, vma, address, flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, address, flags); +} - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } +static __always_inline void __folio_add_anon_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *vma, unsigned long address, rmap_t flags, + enum rmap_mode mode) +{ + unsigned int i, nr, nr_pmdmapped = 0; + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) @@ -1312,18 +1295,30 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * folio->index right when not given the address of the head * page. */ - VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + mode != RMAP_MODE_PMD, folio); __folio_set_anon(folio, vma, address, !!(flags & RMAP_EXCLUSIVE)); } else if (likely(!folio_test_ksm(folio))) { __page_check_anon_rmap(folio, page, vma, address); } - if (flags & RMAP_EXCLUSIVE) - SetPageAnonExclusive(page); - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 || - (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) && - PageAnonExclusive(page), folio); + + if (flags & RMAP_EXCLUSIVE) { + if (likely(nr_pages == 1 || mode != RMAP_MODE_PTE)) + SetPageAnonExclusive(page); + else + for (i = 0; i < nr_pages; i++) + SetPageAnonExclusive(page + i); + } + for (i = 0; i < nr_pages; i++) { + struct page *cur_page = page + i; + + /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ + VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || + (folio_test_large(folio) && + folio_entire_mapcount(folio) > 1)) && + PageAnonExclusive(cur_page), folio); + } /* * For large folio, only mlock it if it's fully mapped to VMA. It's @@ -1335,6 +1330,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma); } +/** + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages which will be mapped + * @vma: The vm area in which the mappings are added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting, + * and to ensure that an anon folio is not being upgraded racily to a KSM folio + * (but KSM folios are never downgraded). + */ +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) +{ + __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, + RMAP_MODE_PTE); +} + +/** + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting. + */ +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long address, rmap_t flags) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. * @folio: The folio to add the mapping to. From patchwork Mon Dec 4 14:21:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E1FAC10DC1 for ; Mon, 4 Dec 2023 14:22:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 840466B01A3; Mon, 4 Dec 2023 09:22:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EFFF6B01A7; Mon, 4 Dec 2023 09:22:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F52C6B01AA; Mon, 4 Dec 2023 09:22:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4B0056B01A3 for ; Mon, 4 Dec 2023 09:22:24 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 262A71C052E for ; Mon, 4 Dec 2023 14:22:24 +0000 (UTC) X-FDA: 81529351008.23.646E456 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 7911F40009 for ; Mon, 4 Dec 2023 14:22:22 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="T/ry4Mv5"; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699742; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0rYG9NNO7tFTRIi4sOXhh6ZZDANHjsXpFzljN91NqwE=; b=KHs4bbOVK41lEg5CPcbtw2Qi2T0DaqppZWqZ0jyW00VVT6hoVGSIjqCMeEz5J/L0/TQJb5 KVCn91IKzujAhOs96U83ys2vpVQk5yruo4Kd8ZyEmIT+AdQZbzIR3bPll/tyJxlZR9RXjf TiD+hbVwjqkRCp6L04k9G1ATHmGu8Jc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699742; a=rsa-sha256; cv=none; b=MljZvwFAYnYQpgiJBJ3KFwhuvRPuA9CREdU4VMihWXR+rWTzex8bgU7V3wjfrFYcWX5pFZ KRSMIryU2F3BXA6t9M1U26uidTSWwjAv4SwdjH550tAn3dvLHN+BueVMucbNtL7XYTm/UC IGqxEL+shvdI7mIlPJWjwTvt8n0MVvU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="T/ry4Mv5"; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699741; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rYG9NNO7tFTRIi4sOXhh6ZZDANHjsXpFzljN91NqwE=; b=T/ry4Mv5V9GFxMR/ne6odO7MpOfSvgVDyILRU2TgAbAg6U6YKhhYCNN2Hp5ybD9vqMvPvt /mrHUwy9CS1ecFCULOJy+6ftcPNcWq7fDQKMAsZPhm0PZXYlxcs8R5MqAcyWbQDeQ6YgfS 34vTWffoa3bVaAmU4cmx2GfBQFwUmys= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-658-L-pBgpvKP7yCqqoQs9ALwA-1; Mon, 04 Dec 2023 09:22:17 -0500 X-MC-Unique: L-pBgpvKP7yCqqoQs9ALwA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EF0598314FC; Mon, 4 Dec 2023 14:22:16 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 052CE2026D4C; Mon, 4 Dec 2023 14:22:14 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 15/39] mm/huge_memory: batch rmap operations in __split_huge_pmd_locked() Date: Mon, 4 Dec 2023 15:21:22 +0100 Message-ID: <20231204142146.91437-16-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 7911F40009 X-Rspam-User: X-Stat-Signature: br916ki1r53ixaoi8q1pxrbh7be6mrmx X-Rspamd-Server: rspam03 X-HE-Tag: 1701699742-61639 X-HE-Meta: U2FsdGVkX1/U6O3nqdbhAo6Z03Cn33ij4SQi7pGU2Dz1+JdyhGQHOzD37hUM6ze7sFA6bXg7KEiI/Ppb5Yz08NXIBfPsU7P4STAVTdKRI7me6sSFXMnxQV9aTZDM3cKrUJOexbVJhniG4xV28UWQS1Sz5Fb87KRDoJ9P4XhlbE9VvLjxGKiESWJubLsvADRwxVN8y6aIwSRQ95suEzc56CG8+2BirNqG0YKPOMVu4C7BxQjDG97l9+m0d3SG312cZF+uyJBmYnYE8iNDzKcdU70IN6LuUKKJXNZlnh4woJOMCriPijv9Az5GEMTnZPwi/Okxy18m/zGWd4us0y5hjF3AStAU3cSV1eaewEB4sk4qhSzAmhflN7hOOcFhSdesqpGgnpIz5j2p6nPPCDBID0JBEt0MCkJDyNFIM4KvVWl5Ig3ds6mYUb2uBnHm5RGGAonCyxvdEOfM8KKaGLqLPdYKNrMO0N7kK1qakZP5aKNULbbFIeiClQvkbV+8hfMg6DUFnj0V6akwNB3MDTZ0P4wcugS+bGnjt5SNLCLModoNNHjYpDe2HrjXq1f66Arh4vj30sX5o9NrxOo9IH0tDZQ+y88GpRZ+bD5sL837sl6OyRTH2exMeKUiBromchQkDE6wTrHosLY38ReYUjtssACj5vfu3io+pB5UAv63fbcUbF6klizCeUo200+tRUpbiEMEoLlTmdcnFHZnxc/HaiIqHBuTBl1maLe+03HsLDliaxVpOKDZxc/UqzQjyvwGfjJxhHwjFTACDtMwxGt4eQhe7+jIVMqqpadsZu4PW8ANviNkAVH2cL2VUH6FIW7Q9y4kCFS0iJVGeKlZJurkTNxX+g/YO9amMVk1ZEcnl6yY3c+zUfCzHMWoMhm0GnkB7F/MMdH44Owq14fus0djIEr/mh0rqXqbb6Abq2ZJqHHwsL+uo3EHDHqlvbjYxeTJPNk0NbbquqhBH5RN2CK IL+rayl1 ziCZBvGdZcJwBcCnUFnzleHQRLiAcaRI8++MEWshqBWD1KDc20uCNSm/EZxOEWzoR5ZYmaTrWRuY1Bift5NLg2AAsi3Ikbz86Ns66CSVLHzsGD1IFp0xC2nQy/0YoTtfrwWaL416JumKOlt8fGMFGJI4kEmXBVzClDhxUe7zHwB/TQ+5UGguUjBSvFvTqzhQNty3RisR3GwhhRBRgUw965DQr3g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's use folio_add_anon_rmap_ptes(), batching the rmap operations. While at it, use more folio operations (but only in the code branch we're touching), use VM_WARN_ON_FOLIO(), and pass RMAP_COMPOUND instead of manually setting PageAnonExclusive. We should never see non-anon pages on that branch: otherwise, the existing page_add_anon_rmap() call would have been flawed already. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cb33c6e0404cf..2c037ab3f4916 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2099,6 +2099,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, unsigned long haddr, bool freeze) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; @@ -2194,16 +2195,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { page = pmd_page(old_pmd); + folio = page_folio(page); if (pmd_dirty(old_pmd)) { dirty = true; - SetPageDirty(page); + folio_set_dirty(folio); } write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); - VM_BUG_ON_PAGE(!page_count(page), page); + VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); /* * Without "freeze", we'll simply split the PMD, propagating the @@ -2220,11 +2223,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * * See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = PageAnonExclusive(page); if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) freeze = false; - if (!freeze) - page_ref_add(page, HPAGE_PMD_NR - 1); + if (!freeze) { + rmap_t rmap_flags = RMAP_NONE; + + folio_ref_add(folio, HPAGE_PMD_NR - 1); + if (anon_exclusive) + rmap_flags = RMAP_EXCLUSIVE; + folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, + vma, haddr, rmap_flags); + } } /* @@ -2267,8 +2277,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); if (write) entry = pte_mkwrite(entry, vma); - if (anon_exclusive) - SetPageAnonExclusive(page + i); if (!young) entry = pte_mkold(entry); /* NOTE: this may set soft-dirty too on some archs */ @@ -2278,7 +2286,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mksoft_dirty(entry); if (uffd_wp) entry = pte_mkuffd_wp(entry); - page_add_anon_rmap(page + i, vma, addr, RMAP_NONE); } VM_BUG_ON(!pte_none(ptep_get(pte))); set_pte_at(mm, addr, pte, entry); From patchwork Mon Dec 4 14:21:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16688C10DCE for ; Mon, 4 Dec 2023 14:22:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B8696B0202; Mon, 4 Dec 2023 09:22:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 146F86B0213; Mon, 4 Dec 2023 09:22:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECE466B0202; Mon, 4 Dec 2023 09:22:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D90E96B01F2 for ; Mon, 4 Dec 2023 09:22:25 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B501816027A for ; Mon, 4 Dec 2023 14:22:25 +0000 (UTC) X-FDA: 81529351050.10.09F92E7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id B270514001C for ; Mon, 4 Dec 2023 14:22:23 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8LZ2Pko; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699743; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SP+gQ2Yl7ptxMmm0EN/M4/z6MAPTYZkvBQ5MoVKoj/U=; b=gSAfl31cCXZJ4vvD0QnCCkb5kzqUEHkVoLVWUm59u+FOATdc/vLYu0gaYeb9Ed69vhnCPD 0ZQHJfLlmCVTiHblR2dGXHmCLY7m18adlaXV7gBwopPPpyM0xezLw19+5Ov9rH8I0FjWRU gcLz+esufoAt9tYgOnyZy2wtKNOY8aw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699743; a=rsa-sha256; cv=none; b=s4CFhC4p14AoCY36VHoO2yFMhA+B17Hi71BpSeFC5xSEbYs3qUvle56UYA/BbF+E5+pOii LnDugf+BixNfQ1J1DBuVLTbHwnEiD8cGUQ2MoIelNZ9d1C8CBMpBG4ADs3bIWuMgeFloh0 qE3E7kHDNNqSO4uzdG5IrBoHbyad7pg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8LZ2Pko; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SP+gQ2Yl7ptxMmm0EN/M4/z6MAPTYZkvBQ5MoVKoj/U=; b=h8LZ2PkoCvkRXgD0f+QYlpD0jMN9DZokLJplP1fBdgbwaHzAo2zXUYM1lZsUaXHDDbwEQI 7zy836OULal4K1TqQDPocKjEI/P1cc8zfPJpWnJFGPEsTozs2THpsDo32iUaJYq1ESbRRo sCXwv62Jrj0x6JkUH1gcH9iDPMW/+Ks= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-629-LlQ3-zByOwSqnbEF8yG6og-1; Mon, 04 Dec 2023 09:22:19 -0500 X-MC-Unique: LlQ3-zByOwSqnbEF8yG6og-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C80C18314FA; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3338B2026D4C; Mon, 4 Dec 2023 14:22:17 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 16/39] mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:23 +0100 Message-ID: <20231204142146.91437-17-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Stat-Signature: emhdwbqu63iayzmtkjzbeob94zjp4551 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B270514001C X-Rspam-User: X-HE-Tag: 1701699743-870615 X-HE-Meta: U2FsdGVkX1+Eq+CF51G9/V2jIQV++u6w+XLgDj5oR1457dqeCIMgxleeD5OYpXgF2i4hcVFMy+VUQlMJCT1MMktyWh7gmBEO0iURZ468OrnljzSe+B3ltPiX10Rl0zu+eQPqRtAuxVb3lQaJPExvrrAOTFZVSspweT8N+VUL+jxUuCX0f9OzGpihty0iapzB2czDWurC+3R8i35+ZfNlHfQTMgdc7YzKQyiG8hnpQr+JLwyROFCcGEhw4JXYTGdaSCud4VntW9MGD8x4QzqAMdEfwLVnUhjWEWMbFejS+V82SUc5S1pPwMRNdgmICOxx4RuWNuk21xmhTmTn/ztuRPfDiL/3P3XdYOO6/RNqtjr5nudkfpBPwAb5up22ruP31SnXWyvbRXfnG3kdJXlChKzvSpPsk6M+c1+ZQWXRI59wawE1gqzMQIwd1WCPE1ikIJ2DocUIECADbAN0YgiqH1YYP+qzEAh6wzXbRoQBx1zaHk+196VdVHbYCkC3+nljd/DzmzFjXvZFlfRaP2L1WW8MGpxvkxAqZ8SH8MfDhPZ+D80kYqIr0ENvLdp4YEfcVP9TjWuN1y474S+M4LDW6w+pPGdOpK5jhE+/QN1zJlZyF4NJG/oIM+9X1HJDRPRkPBnoYg7whGxGwRFFt0c5jo2rMwIg3m6MQskMZZLV7g8mMsV4c7QnSk3PLcWyBr0F1Mz+JxUqcKBCe3XIz/1zyWzK3T+LUAHwpmsfo3FVv9wgNYBK8mqsWwqcCa/8vxfoxpaYgKftU5Sb++8dtuZEjiU0utxJhJgzdopwnbbPI/JB4DdUWVbNgJzBLXs/T6DO4BxGwxvnhUJJLsCEQCKziivf7ZZBBliSFLIegMI1bHoYsMs6gxzYvIIzf9hX858IxSgIT8sQ75/cL6C2FSt9WR6Jwyyow25M8XC8ZfuWkyYAVpojuTL2qUuoTSMMKT7YpNRzDD8iQoFunqyqxaX AErPXw8V tGT6DBZ60GthRSpi6r0yZ0W4jzSbw/K8/MVFlfxXoSJ0I7sQkvV01iYJbHEt/werKsZp+cZggHj2eshGfCtEIlFpnXCXscHA7wHrZarvRJRV+g4fYPqyUWpQXQw00dUj4dsmSn+fdkPoByK3HWbAoetQFH1H4u8TZotqReW1Bx/ScTooINXIBJzqQSjekZ78OqNnYrfFLrZzlm31dnWH//1dhgw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pmd(). No need to set RMAP_COMPOUND, that we will remove soon. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2c037ab3f4916..332cb6cf99f38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3310,12 +3310,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde = pmd_mkdirty(pmde); if (folio_test_anon(folio)) { - rmap_t rmap_flags = RMAP_COMPOUND; + rmap_t rmap_flags = RMAP_NONE; if (!is_readable_migration_entry(entry)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); } else { folio_add_file_rmap_pmd(folio, new, vma); } From patchwork Mon Dec 4 14:21:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39F26C10DCE for ; Mon, 4 Dec 2023 14:22:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0550E6B024B; Mon, 4 Dec 2023 09:22:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 001AF6B024E; Mon, 4 Dec 2023 09:22:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6EF26B024F; Mon, 4 Dec 2023 09:22:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C558D6B024B for ; Mon, 4 Dec 2023 09:22:28 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A13E1A025B for ; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) X-FDA: 81529351176.08.20BBF85 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id DC03B180022 for ; Mon, 4 Dec 2023 14:22:26 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bCtGVC6Y; spf=pass (imf06.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vhcry2rMTXBwyY+r1UEd/G15BeM6OBHeZJqknbajPVE=; b=YddS+JXyDEDHpQEJSSdiykYcDPTHkyC+LXnYtdPzqknedguyAhePQnHtlmCRXv9wdKO5NK AsGOVMjrJ2FAs9iLwplZcBMf1nQQe9XUS2qMDBCiMnDMfvAj9BudBP/KcZ0DJoKQaSQOuU W5Tl13WCBSIqO1MFbEArZwdg+Ogjh1Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699746; a=rsa-sha256; cv=none; b=BRDDEYUG/ykEkaFkVG0YGvQNNg86OyzeTV0blUXnBvh7/3pCoDk1W1yJzcWXR/YANaQtR7 AxugfIY8IafQRGeA0Pc3IXrbEAlKg4CqTDI8Ls9VMgFIEU7Yjg8xDu8u3Tmx0yEynLv1QK iCLazEo7EneWJLqZohNGvm4jqICVbiY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bCtGVC6Y; spf=pass (imf06.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699746; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vhcry2rMTXBwyY+r1UEd/G15BeM6OBHeZJqknbajPVE=; b=bCtGVC6YX9mOreAnG8tJEPhgpyUIz4tCQNgBqZlpZWN00fa+FZLswUwIBMSJ1FvCZJcsa5 ML2sppEAD2b5FLexdfI2sLPruiO4vbqm2a8dTGeI9RwLPqNN0Xy3va7Gt0hHtDEB60esXN gQQXSJJYzHVmQqrGqKRq1DyQzhp7BRc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-688-ZWpJIdMyMpmAHNLnESmGHw-1; Mon, 04 Dec 2023 09:22:21 -0500 X-MC-Unique: ZWpJIdMyMpmAHNLnESmGHw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A76AF85A58B; Mon, 4 Dec 2023 14:22:20 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0C38F2026D4C; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 17/39] mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:24 +0100 Message-ID: <20231204142146.91437-18-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: DC03B180022 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: neetr5oii7uy8dsg36qutrybgrifse3f X-HE-Tag: 1701699746-664340 X-HE-Meta: U2FsdGVkX1+ATjEDwtQTx3B8JWoqGffVSxs1+KlVIyaZG9YheNdsPxbYHLr1QTWsHvvA8qSZ18qtJfgNJ2tfSk11/14MyARAycOH2LpnOhu1lkswbJixBGaaSYzCGLvsoLPctzUfiZZ9FXcAtiUqdbfTJ9UVAbvn6ijHJnNQsvi4+XuEwS+23u4fm0zJqY8OdlZHUXdHm32+D2sFOmgHTjZ8RDHLH7tJyW3OoGQPp51SdYZug5AE/cGyC2ob38KYS/PlEDnzNNfORs5xaicwRvfOMCmewz8RzDrUElIxJja1cPYFSrMZcqPRh3kJat1pw8N1GLjKWE1Q/Dd5ETcZwzJwOy6ho1YLe0SF1RncwWTipR2QUaxa35TfRIJzsadOu18a/anAgBScAtu4y61zOFagoPLrvHQxs6ifR+z3lmrhXntHwC4WuymWj3QwXjoHAZ0BysxZY1r+BnUzeZ70OiNLoNyjnGVTle6oI0qpeLOBGrnONhOZTfBnmYzFrZPsbOxAfDpAdsPXNd5qh+j09G/sPBNLu57TVhugq4aSugOnVUsmo587Huzc3+S0KYXWX5sdzoDB/8TWaA/D4DFilexfWWyMmOGQVclZ3s2jfLU5ycIaADmriXC5kqYEq5/C7++VeTepq4nPd0IEXM5Tk/hb/KzwnYj7fV1q4SlgGA4FSsgshyRYmhH/JYcczGF0226B6haTmh6pg4VVBlkq0UnHfmh3T7KdyYgIV/F+DdmonDciR99AzcSwBqD4esFRGy3pmIgL+h0226UPiFLKAE8gviNpAYy3m4EqyPjJhrnuNxrMSn7Ajfp2+fslcOFmHOGbHyaljUrZ0DI/ekiHFWovMIxs9WrvkSaxOp+L8rRC1ezoRCPTTFw+INoH4N/rPjsfIHa2ppAM6/BttzuOKkwx0OeRSt86yamFWUCU5inC5i7iXyWL8BHVlZzvOyuIus7Q6qtg7sWAbxQ9r4d juN459kM o3/OMvEZS8lQt+OMAbUwz5SUBo1In5RTmCyJA1VKSpxXHQFhd+E+mq5OcncrEs0Z/SF7ix5/2UPcxvEa8GFgksMgmCQtu5RSv+14csdmg+tBrpvVVl7HjBEYbX/vpKL8QFt2D4TzBFIIrClCIurqRzyHDIXt6C8YVMTs4n6Suv8UBxbqqpdITHXrXos2PCOIf3Joo4e9AiBCg4RkMwTwQ/qT2fQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand --- mm/migrate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index efc19f53b05e6..0e78680589bcc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -259,8 +259,8 @@ static bool remove_migration_pte(struct folio *folio, #endif { if (folio_test_anon(folio)) - page_add_anon_rmap(new, vma, pvmw.address, - rmap_flags); + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); else folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); From patchwork Mon Dec 4 14:21:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6398C4167B for ; Mon, 4 Dec 2023 14:22:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 337FC6B0293; Mon, 4 Dec 2023 09:22:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BF916B0294; Mon, 4 Dec 2023 09:22:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13B116B0298; Mon, 4 Dec 2023 09:22:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F30FE6B0293 for ; Mon, 4 Dec 2023 09:22:30 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CA90D1A026A for ; Mon, 4 Dec 2023 14:22:30 +0000 (UTC) X-FDA: 81529351260.09.52E68C8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 16282A0025 for ; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="W6/d5Sg6"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699749; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MP2FAIOx2jsvMIhG8WePR/5GNOuXZxiB4Uil9+VePO8=; b=F6kdxPPaS8raMR7asZTQ1G8trK8JuuaRV5YkwEZW70kU4iMmlyNwg7RuB8E87EBRuSlram 3ubwfZITdw7lKMdhWiDNt3vXy4p4NL2oyKhKsCKr5DCvLsb2+3UpGHHpt8iW7VE+SqNZyl Lwdd1ZhVpHXZctJcIucDsXZ1jvgNrwA= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="W6/d5Sg6"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699749; a=rsa-sha256; cv=none; b=BAg1XGqrhIDhiWtAjQtLGQtlOAUfyKpFf//6F9shznZ4D7IRt1Vb3kphlLyccd+mOmpG1o EIzrDBOe8vlAAJ7bgQJ1o/U8slhY3kYK8nT+d058VnhfXcij/vffrnvhStFK5cb/qbF6yd xAgaYoH/RDNJGK+EIM5EHApilGidsLs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699748; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP2FAIOx2jsvMIhG8WePR/5GNOuXZxiB4Uil9+VePO8=; b=W6/d5Sg6AE7Kq96Q+vn08Ly0rsw/Ju/G+T+TWQr9Veqk71ZZuPBCzHWZvWrTfUpyd4eY9R 3nQvKj0K9XitmP/5mJvTa5FJfv4+TaKhtuipv0q9jpS92AuFlGO7DfcOUT2J4b8zmgMReN 6HS0iP4nTaIAjiBsE2101zu1uj/3gkc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-81-b962mDT3OiSU6I-NXCof5g-1; Mon, 04 Dec 2023 09:22:23 -0500 X-MC-Unique: b962mDT3OiSU6I-NXCof5g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8B19389CA0A; Mon, 4 Dec 2023 14:22:22 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0BAE2026D4C; Mon, 4 Dec 2023 14:22:20 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 18/39] mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:25 +0100 Message-ID: <20231204142146.91437-19-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 16282A0025 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3ayg3cm7xt3dui39z6khn93fgfa6op99 X-HE-Tag: 1701699748-466526 X-HE-Meta: U2FsdGVkX1/mcsf0mh5lR56QrFYiOArE67A9+saJM84WLZ127rW3hW2WB4lEXRI3Yb/yEhr8994Qq86Z4bz5owG4AmtEMZaAsqIOydV5o1euljNTrM588tC4KakNoVQmJcnazC+xn9DNDfoqC7+Iz7Kj6Qu+43ESLpy5uG9pc/Lf+Pbg5KOwTD+TKskIv+M0vJQ8TNWljL76ZFcwplQ6myGGR0mIQ3C2smVXcjY2gG0RrV6V8momzyfCCdfcQl61wPat384hD31EYzdOkxeOS32F1/vEtUS7S6LwaQ8zfsrl8h+gvWrTjhUi5z1xHrfPIyQiv9+wKpQTFLotoEiNBNNQ7gLQz130SMbhaqKYL6zoaaqKSs/JbNUpEt1EBTZpvaQq3YhDRSMvFMwoja2MpPw+I3LMlLgblJy1YQIi79S6+hOq4G05kbRdhL0ZsC1uTDNcrSvLYwHl34dTUDu2oT0j0L3UgKuCk8Rz5qW8ayKSlXsPBXUjOo+sk+hIWONWM3YDjomlw3MYUz199jbNMqiuuqkJvZq09NB8tsslYeJ7hE32P4ZASQHEtbswdwzZ7OJUbi6MmaOsj44spaCv/HjQUrWMs1pNyp56gphqC8K0KkSFbEBqgCLvTv81Guc8GOu5bA/lOaJ7KE7Y2dYdt/fgR7IpEbFfrxkmtfoi/OjP8E3Pl1nBQKfHrkuYG2nyrdUx5hdiTx+PP29SZyg3n4em3yV6gKezcYzbd1uk9vREn+q4MnuZnR83O95D1l20p/uhSHNkQB9iCd9RLM7oBrPqk7aW0XrKq637f6g1vddXVSj2oQHuRMt3RrIH1mRxewk6IbuXWnnvoSGSsLoSbyN9lah4FM4TyANFdCX63fQAyKKcJ0I9BTMBX/rku/TMEJZNfzEJYupgYqlE2MD9t1zh4aHjI9IZNV8A2iETxX1JZ6b1rkdZsEi9HTrHBLRAwLOCRmBfys4pbSQ6zm/ MfvYwdlx TMhJj2t9JgVRrqdWIQ3EnTYgzxbc6WKLzTG6UuNomsyfSMGxSLu871hnjX0+0EgAZxAWBe2HlShl0PMK56eBr1dv6HJS4AGtNO+aZzdp3h4TOK+EKH69FdeMsu+65Sgu17dq3m7JO4os8/Im7a+eFXUfgE1/03fikoQ3seQ0Vfqw+/sKHAAfn83tguePnCXakY4SqSmWNbPthQz4ehHfV1+Mi4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert replace_page(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 6a831009b4cbf..357944588a9a9 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1199,6 +1199,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { + struct folio *kfolio = page_folio(kpage); struct mm_struct *mm = vma->vm_mm; struct folio *folio; pmd_t *pmd; @@ -1238,15 +1239,16 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out_mn; } VM_BUG_ON_PAGE(PageAnonExclusive(page), page); - VM_BUG_ON_PAGE(PageAnon(kpage) && PageAnonExclusive(kpage), kpage); + VM_BUG_ON_FOLIO(folio_test_anon(kfolio) && PageAnonExclusive(kpage), + kfolio); /* * No need to check ksm_use_zero_pages here: we can only have a * zero_page here if ksm_use_zero_pages was enabled already. */ if (!is_zero_pfn(page_to_pfn(kpage))) { - get_page(kpage); - page_add_anon_rmap(kpage, vma, addr, RMAP_NONE); + folio_get(kfolio); + folio_add_anon_rmap_pte(kfolio, kpage, vma, addr, RMAP_NONE); newpte = mk_pte(kpage, vma->vm_page_prot); } else { /* From patchwork Mon Dec 4 14:21:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D27E8C46CA0 for ; Mon, 4 Dec 2023 14:22:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EE876B029A; Mon, 4 Dec 2023 09:22:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 083BB6B0298; Mon, 4 Dec 2023 09:22:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5A6A6B029F; Mon, 4 Dec 2023 09:22:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CC4366B0298 for ; Mon, 4 Dec 2023 09:22:33 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A1A881A022D for ; Mon, 4 Dec 2023 14:22:33 +0000 (UTC) X-FDA: 81529351386.08.FDB73DD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id CEBD2C0011 for ; Mon, 4 Dec 2023 14:22:31 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KAYptAo9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+HQj3RQXy3rfd5XJOTyn9grirPLdYd+avDtLGVmo9F4=; b=A809ywu+1ZSS6xc3I+rcuA0aK1w/MGL9cA1YQJf0Ewl3W0GVLnrbyAIvzRqw9RytQonSkj O053Dpubqu+m6CDb48BGyjA86VXgfN/X/cVTl3S9FgskeOiCtVMsV2ngYuSSIG+1SKBxRB vZFUQoSiMfmg5xrSoGyQEMTxBBS+bdA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KAYptAo9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699751; a=rsa-sha256; cv=none; b=2u30HJSQxMs6Oi913HXcOfy72Qcdx/BhhaKCjOxCxin9P6HmADLueVpU+Gbq8zaVW02PjH JkcSQSvehdIEua3pCu58ruoTJYRNwYLoE5XoIhuVpGQ+E1ozlPeYhqd6/t5XsKMfI3RmVN DHHEcM3cAa3HrPQ+3m7VCaZb5fBS/0Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699751; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+HQj3RQXy3rfd5XJOTyn9grirPLdYd+avDtLGVmo9F4=; b=KAYptAo9eGkLtLQyRbB+7gBcmIFGYohKx5wwoOK+2uiJsuD1mXkUjRhM/R95Tr+wbNb/TV Psoae6fHfqVtf/94YXmVweaIWWYJurdgNwadeuE8vLenYxOsHcI/Zxosalk0J+qpEXrIC0 r9xyUXpTJEUB7ZCSy3nnmieBo4+ED2o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-695-AG-JYXFCP3KppGdqheHrmQ-1; Mon, 04 Dec 2023 09:22:24 -0500 X-MC-Unique: AG-JYXFCP3KppGdqheHrmQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6C7CF831506; Mon, 4 Dec 2023 14:22:24 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C88092026D68; Mon, 4 Dec 2023 14:22:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 19/39] mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:26 +0100 Message-ID: <20231204142146.91437-20-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: CEBD2C0011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: yhqb6zwtqzco77jshzm6a56e6q99e4n3 X-HE-Tag: 1701699751-131738 X-HE-Meta: U2FsdGVkX1+sHpCzEGUyYYUKdipRlt06gXfB2wqe1+76efsjBwQ6QbWlgspygHvRrTvo+KwfkJvV4mBDAgj0yeKIoOt8tN6Nee7cNTCFIr5/+HcqB9KHrYBBIIpQQ7RcH17tVdJA9NFV+acEAsFcxEsilv4LtQbCH05qvQs/4G9IPPINz+eZcPijGzJJaE8jBTy0swkfl9L2EPeAwRCDQ2RgKb3SJRvb5kHF7J7km2bAOcjnugqVhpPfNUnZcHLrarGGgQKQF004H6u0ao1sZ5JxerjxNOBiuQ92DlMjXzYsDbDCVpbNP2u1jBcvKm5E/vz2A8Vx7xzQfkQDfBqRdgb+sP7oJ1MH0ny6/YSpZF5224FQ8RUEbn0gDwgsFRwCe38npm4VMR+k1OfxULwxGqnVG5UBN6uso0ZcXBg+9bmJEAxwc1u1YGhQCqOVrO9rDKTgACfu4srWYBLvlOPMKhy6UZK3yr+bFX2S5veo54VbhkynuI+CVS6efZ23qLHDw3LsFb+qRU3UN76MEFCWe/GRLmEnpWya5a4S1eCB+QYdUjvoRJkYBUPhWsvVL/+pV714P43kLbG8/JvkoopTKVt9WBFm0GfP0g2ZVBmFMr1npPjBfEK7Yj8bfQ6io8NtczXI3178lkaN/nulvxOxHuwaH/XrNPEaK7rHf2x5U3X99HkR+V5Ffkw+T7BiRvMK7Rg0b5Ul+RdSyTG0GPfWOZ3NmXn6rrZ231JeWnLGDDY5yzafUdElwQ/3i7d0GB5xFFR+h5YcyLkbK8dQhKsEj3JgzShYS8PyVky2q2/l/pfd8+7caIFA6VL2dqvjNG5Vfu2QHejsW2ulZzAH1GIq0BsWHgTSR7w8Sp6ua8mtSHgc1PSqxd7p8pPpt9VD2ZD8FkzARM1zj4L3pLD6bKjNfxIWCih7Bmavx70ldcAjArNUKYNZ7Y7343H9dj2E/2FxOMq/+xbwzQ0fE/e6AM/ 2b2kaSfl BWofqOqpLnuYUuTFxAAPsMt3LK5FwayACAmmgbf4mHJ/IgruSnXTDSIHZMVbIqNPxukuYGc50w4Bpfz40yS6+WowQi0AbVFDje1g3kt8IaVntu0Hm7fqPbNFhXWFEaTqoOzZ+wkkuPHUW0i1+nypp5oxgP4DmWepc/vY36yx1E7ekeqbD0/yUPTm1ktC0u9XLhs1YjDMLq5Acj63dd3ZaHhXt4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert unuse_pte(). Signed-off-by: David Hildenbrand --- mm/swapfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 4bc70f4591641..1ded3c150df95 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1805,7 +1805,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, if (pte_swp_exclusive(old_pte)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(page, vma, addr, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, addr, rmap_flags); } else { /* ksm created a completely new copy */ page_add_new_anon_rmap(page, vma, addr); lru_cache_add_inactive_or_unevictable(page, vma); From patchwork Mon Dec 4 14:21:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E37DEC10DCE for ; Mon, 4 Dec 2023 14:22:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A3BA6B0298; Mon, 4 Dec 2023 09:22:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 24B836B02A1; Mon, 4 Dec 2023 09:22:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1C426B029D; Mon, 4 Dec 2023 09:22:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D377F6B029A for ; Mon, 4 Dec 2023 09:22:33 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B5CB2A0216 for ; Mon, 4 Dec 2023 14:22:33 +0000 (UTC) X-FDA: 81529351386.09.ECDDC82 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 16EE91A000A for ; Mon, 4 Dec 2023 14:22:31 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BBNBL3QW; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YcgEyKtjLabsj9Cw4VckjmMQxcp5kDVs71EjXlBqC+E=; b=zeB5IZU1G9mXK+/8H+DtaP0ZrBOy4sOxIXzPDr30Ou+rYZVkO7gY4vzQDowGaXfXQ0tbno cJta8OcqmTZxVgUHD8aOI1uVp+ZYU0HhjcZOr2zFf+Oji3mdyHV14WEw2obXGeEMlrJCbn ZLfCLbtHZjIawSSTrkWFJ+OflD6W4JQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BBNBL3QW; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699752; a=rsa-sha256; cv=none; b=Rb7dtR54n9QPk04WXMRNPOo/UToTrlhOf291CL95MV5p4qYJtJvOcgBCGA+seX7hT/7c+u yzS3ZW45rg8qzg8S1Qc3j3eAe/C2MjTCubdzeH5j1ZRJVmJq+t+wMvLeaeYwu/NfWOngpy 8LvjBtKeVqUmengTn8oihyF1F+Ebra8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699751; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YcgEyKtjLabsj9Cw4VckjmMQxcp5kDVs71EjXlBqC+E=; b=BBNBL3QWhxrNegllyxURZlMhK5hkYA9nbW2PMgkNBJ69gpjZm7Id24NOwODrJlj0ga+TRT UiF1OcKkhs12NSjlred/OFhWPPu3S5r0kiiVfp1bew20KTLlIJ8Ww9CN9X4jw4Yd+RkeH4 tGpjx8JxkVptn988OGeuOo1Yle7tEVA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-600-YeNg8xp2Pg69b3ArcI-xHw-1; Mon, 04 Dec 2023 09:22:27 -0500 X-MC-Unique: YeNg8xp2Pg69b3ArcI-xHw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 464973C025D8; Mon, 4 Dec 2023 14:22:26 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id A5C672026D6E; Mon, 4 Dec 2023 14:22:24 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 20/39] mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:27 +0100 Message-ID: <20231204142146.91437-21-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 16EE91A000A X-Rspam-User: X-Stat-Signature: 485taysijeb35ah37gt59kdrojecyw7u X-Rspamd-Server: rspam01 X-HE-Tag: 1701699751-232367 X-HE-Meta: U2FsdGVkX1+nRJ4aJ0o/p5pu2kjS8oLCN5+Tt6miFtvN52q9WG6KpaoeqprfUBpipbyn/lSfqmrPXuT5tZ5ykIv6roO1HOERWN+MEevy2mvlqAtl3iDvW4C8r+PWAVWyB8FevBs1oA4erQqmfOgkjHpGvRPrbMPnIIzg3riu9ESyyVRGmEwmQO4/2lUyaJBeMKVAcpBk2Bx8avHNLwTYjkXQz3WtBmV5/LtosR/eZwQTBsBYBPhbYI4f4qCbiEQVnk33TcLAX3qMAEafP+j6Lr9gCpF3qrfkuGTE86PIV0TJCRzks4ULCT9DqA8aU07QejhburW30xUHdz2bDjz0sUZudmAIDyiEabJAXoco71T7Z4RB5UyOz9soRjuA36fLL5s7D1b+knB1loRrEnTEtaqLjU6doEHmqj6O6YJoIgxmfqFKwPSI5oTWs6dx+ZgnN/nip05T/x+6KPi2yt3QhRs5gatRtG6eUvTZQJQQyJbXUfDqq2iH2LUuILgSnWpVW5RnQ2kqrvP4EI7CAt1CLH3IECtfqEl7l/pVwEQwvMo5naVs/ydyKhkRtinTBPqkQdHruVl//4qVZiB6n9OA/XU4q/PVQatbaInY50YG5AY1zpqSXuLOyB1aB5C0cpo9wCWjKJwHW/9GqNleL5Mh3AFJwLupvGFVLT9zvfLpNbMBe80boNmG3poeSigTZ2pNX3OgqUrK7u4bZC9v2nmtg0WWu/NW3nXM0E7CVfAVIkXOysYv28dVzNiSSeJpO82oR8Xe6rZONj+VZ7fquY9kxYY2E7NfktNr86sGkMhJTGfUX70kr5UTB4YoQlqhRfwqL1GSJxcP+zptcwd/BvEkggPMgbqpqVxvQY1dYp3Re5/rhrSsV4m0QNTv2OhK6SBaPlA96w6ESMyL62TAK8vU2L5orOPQJS4m9lE3agnPGKq27g2xtNG0FwfKcZ7K6+f5VsqBGXoGWgGbvo6bdFb uMI3SrzU 38bhIYlKIi8gRrFPjCQLwWh6Z+ZBPxpcT1+gZ3OnHCgUvKNJv+nJtS0XNI0aXLtW0EXjY90P2PcsbvQxTR21w+6Qzy+Dz7jiYcB/Jbe/Wjb+l3VqL8XiAOGLxjQO3uoky3Q96BwfjA8kuIgoE67Hje3DqKp+2GVb2V1OdYLbLBRpHh39K9n9g27DN/bA4IYkxvjGx9VAkWqJ1Q8FN0zwiuUTn2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert restore_exclusive_pte() and do_swap_page(). While at it, perform some folio conversion in restore_exclusive_pte(). Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index be7fe58f7c297..9543b6e2b749b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -706,6 +706,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct page *page, unsigned long address, pte_t *ptep) { + struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; swp_entry_t entry; @@ -721,14 +722,15 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, else if (is_writable_device_exclusive_entry(entry)) pte = maybe_mkwrite(pte_mkdirty(pte), vma); - VM_BUG_ON(pte_write(pte) && !(PageAnon(page) && PageAnonExclusive(page))); + VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && + PageAnonExclusive(page)), folio); /* * No need to take a page reference as one was already * created when the swap entry was made. */ - if (PageAnon(page)) - page_add_anon_rmap(page, vma, address, RMAP_NONE); + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); else /* * Currently device exclusive access only supports anonymous @@ -4065,7 +4067,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_add_new_anon_rmap(page, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { - page_add_anon_rmap(page, vma, vmf->address, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || From patchwork Mon Dec 4 14:21:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4345EC4167B for ; Mon, 4 Dec 2023 14:22:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CDBC6B029F; Mon, 4 Dec 2023 09:22:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 731C56B02A1; Mon, 4 Dec 2023 09:22:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 559B26B02A3; Mon, 4 Dec 2023 09:22:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 322F36B029F for ; Mon, 4 Dec 2023 09:22:34 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 14112160274 for ; Mon, 4 Dec 2023 14:22:34 +0000 (UTC) X-FDA: 81529351428.07.19BE43D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 56EAD1C000C for ; Mon, 4 Dec 2023 14:22:32 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Tbua3Ili; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NQM7ysAYyclyPUNvOCqkfLGmI+LHJSTCFqaARoijbyI=; b=txsgnt8eS3pWxyoiVnOdYxeLha/2TCP0gI5hrDxRh5MiNFX4umYRJqz12VEa+2yVzYxw4A 2Zra2XTLzwWNJT9ILB9kk5XFDSTTZ+RymqXRlbwwgk4G7MvYNqzYJPgaTprQxnseR5o1lx pDMLO1lAhOeLox6CtEdCMdObSUHuIxw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Tbua3Ili; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699752; a=rsa-sha256; cv=none; b=jA5/gF3JqNZC2Lz9Xh7Ftcw3G7X43vwcymEkOrHjYJAg2kmlfvf4w2b2xxU6OAhMZ5gBIv VLX+DzTIETK6+GQMHtHsSUdpPSgxbMb2TTSSMNW1b8x8hhgQho07CtFJuBDh6ZUwZ0oDAa SKQhdWdI66FOR8+ismrLt50sJKSO/wQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699751; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NQM7ysAYyclyPUNvOCqkfLGmI+LHJSTCFqaARoijbyI=; b=Tbua3IlireOWwvEkBVav9Z/hoQVf5+r1XCThRMXfs1mU0UN7+/rSaa7CGf08ksObjF38+e UIX656ed4+Z2UVnfXk/CtsOeXgBMJfow+P/5KvGGDNrnlnEABRqpG1XQ6IOyMADImB10oZ PrSvktTmLQTvL3+U5mqvFl/pYAYHk9s= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-220-6HiWY7_OOd6JGbhIhUP00g-1; Mon, 04 Dec 2023 09:22:28 -0500 X-MC-Unique: 6HiWY7_OOd6JGbhIhUP00g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 207872825E9C; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EADB2026D4C; Mon, 4 Dec 2023 14:22:26 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 21/39] mm/rmap: remove page_add_anon_rmap() Date: Mon, 4 Dec 2023 15:21:28 +0100 Message-ID: <20231204142146.91437-22-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 56EAD1C000C X-Stat-Signature: yjoyyt8g89st73n44m84terzbxfyj1kn X-Rspam-User: X-HE-Tag: 1701699752-259152 X-HE-Meta: U2FsdGVkX18Hg7drYUp5CcDE7QhAG7OWuEQ02ch6FKmxGRZLNEjhZovE+Tr9An/xWUblAQjDMKMp0ZP1q8z7zNUx7WIERmcM4wUXY5GyXE2wMSfpGA9l+Zu3ygCyzWFVSFVj04q4vc5mIdqC+ZWhN/tkgCocN8LoY+SvQS/PYv8/1A7FFSkSbMOc5feyz3OzFUQPTrABrEj/n3ijGiaCppzvBTLG5O/QjKwCl4knmQ470A8TvXf+H5r7dc6g8ClnGJSxEq2nCa5uSoxYjbkhPFUI+yNYbGh+xRd0YcxXpt6No156wjyWxDrAjRsGmd1kRdcrj1rN/6PB8zW4/t/o3PtPd6Jy1UqD3GUAwHv8RdI7wVuVOTvVlmehmmP779RdKK04Ydso3jFxJR/wxgkiAjsWkwlyMS6+c7k5WehbnKyWz9IBT8UCvwWVMLr20zhTdafzCDcLUsvTO7cjLh8CgLeJQiPKj5uoJofZq9wNiSeeljrFoevXGGvAIS7KIvVVVLljL26XgAtRypfc4a+VzF3oGNX6YPW02wQfP++lGbT0U1sBWGmy+vHKfvQqo2zZO4MVTeWDtDxSwY8OlPiGWuIo6SZdVSBxKBvGXQOHziIPjMUG5R4dyHQLdbayF66H5LP9OZk2D3pHHcKaAc/ZAXvvNMWP6mJbgBLQ3a8MTJ0XvOIYbHZgggIB8DAvHhgpcWIKWeOFxxLUInBR0ksgEOL+C3oKCv/E06etmWYJTfIBFAmJJ89UOFI8IZBdQiORk26Ope2schv/AjQe42SmMD0j4UhXSbiE145sQDajQnq/JOiwwms3dvLkgkDoHt7fuJB2u89et+3KhinKW8+C0q9CR2Fsy6DHWsPIOD6wP3Vflgdnk7fmPLR0c98dXs2AWLhzSBHE2kCeLKYOnGoTI8Fmi0f/Wyt72nCsfmkPSLrqm41zak4AEDuu6IRrTyBU2aqtNAx7NmGK3IYaYcm nALcbi2Z TzHCJSxUjsryIyQLqaznE3warhW3hjT5Bf9a/ONqJ9BmcMpLPtM7ZNeWI7eTLMDhATqUvoT+2UdoAgop8pF9XVO2XayT+fvxgfij36ESepdXrHbUc62XYJaCK1iS2KsD6Kbzz1Rq/bNx6DTubtOC+zVAgoaa1D9g52Wf7jPCICX3ZisvEUkWNtNKEkNeaFULQSVYD+c3b8DEXoKzrTR4/2wrH2Ip1j56gt10K X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, remove it and all traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 2 -- mm/rmap.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 9e1c197f50199..865d83148852d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -235,8 +235,6 @@ void folio_add_anon_rmap_ptes(struct folio *, struct page *, unsigned int nr, folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void page_add_anon_rmap(struct page *, struct vm_area_struct *, - unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 85bea11e9266b..4cb9d8b7d1d65 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1238,7 +1238,7 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, * The page's anon-rmap details (mapping and index) are guaranteed to * be set up correctly at this point. * - * We have exclusion against page_add_anon_rmap because the caller + * We have exclusion against folio_add_anon_rmap_*() because the caller * always holds the page locked. * * We have exclusion against page_add_new_anon_rmap because those pages @@ -1251,29 +1251,6 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, page); } -/** - * page_add_anon_rmap - add pte mapping to an anonymous page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @address: the user virtual address mapped - * @flags: the rmap flags - * - * The caller needs to hold the pte lock, and the page must be locked in - * the anon_vma case: to serialize mapping,index checking after setting, - * and to ensure that PageAnon is not being upgraded racily to PageKsm - * (but PageKsm is never downgraded to PageAnon). - */ -void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) -{ - struct folio *folio = page_folio(page); - - if (likely(!(flags & RMAP_COMPOUND))) - folio_add_anon_rmap_pte(folio, page, vma, address, flags); - else - folio_add_anon_rmap_pmd(folio, page, vma, address, flags); -} - static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, unsigned int nr_pages, struct vm_area_struct *vma, unsigned long address, rmap_t flags, @@ -1384,7 +1361,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped * - * Like page_add_anon_rmap() but must only be called on *new* folios. + * Like folio_add_anon_rmap_*() but must only be called on *new* folios. * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * @@ -1432,7 +1409,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - /* See comments in page_add_anon_rmap() */ + /* See comments in folio_add_anon_rmap_*() */ if (!folio_test_large(folio)) mlock_vma_folio(folio, vma); } @@ -1546,7 +1523,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* * It would be tidy to reset folio_test_anon mapping when fully - * unmapped, but that might overwrite a racing page_add_anon_rmap + * unmapped, but that might overwrite a racing folio_add_anon_rmap_*() * which increments mapcount after us but sets mapping before us: * so leave the reset to free_pages_prepare, and remember that * it's only reliable while mapped. From patchwork Mon Dec 4 14:21:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4954C4167B for ; Mon, 4 Dec 2023 14:22:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C10606B02A5; Mon, 4 Dec 2023 09:22:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC0446B02A7; Mon, 4 Dec 2023 09:22:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C5086B02A9; Mon, 4 Dec 2023 09:22:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 897966B02A5 for ; Mon, 4 Dec 2023 09:22:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 623C0120291 for ; Mon, 4 Dec 2023 14:22:37 +0000 (UTC) X-FDA: 81529351554.30.DF199AF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 9566EA000C for ; Mon, 4 Dec 2023 14:22:35 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U3t4+JZw; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699755; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yM8m6ebsRHp16rDmw00I1U0rPyCAW3pmLtjnA6MDqao=; b=S1eolryEjBflUtQDZIo0ZySkac4PsFUdFw2vM8UhvgJUu3tJnv97O6d2+49JbtzV65b5Za /98/QM9fzJN7p6aMKa/YtsubSKUjAkeTTJXii1SottxSK2ckfjgptiuKEcX7+isC5AbqY9 nzguTJ9KWMJlItIDNKRS3AGm8NqmS2c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699755; a=rsa-sha256; cv=none; b=zvjzcB7B+nfxx7LHZ9bv8gNIKUB0Qxs7yqeC5I7BpdO/Vfrn3sqvfy91nL6zlSGxwG4LP2 MF6DgKPkDJiCMy+Bbhd6jxdVGkJS2EuMjJzMsrVu0IwDnpwqQXjbjeRYEKcEaVHcylXrXp 2VG6EoRa9THC/m3M0DAKiPerltgbzco= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U3t4+JZw; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699755; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yM8m6ebsRHp16rDmw00I1U0rPyCAW3pmLtjnA6MDqao=; b=U3t4+JZw4O4YBB4IEcn9PDva0xC2v4LPlTXYOq6w2X0r1dgrlBXue3N+jviGYALg6dc7pH NWfsq9XBrlrux3cNScmzGAuMhHC8NPumBJqLsoSj2zhzt/ccVu1a2YXaukOePJb3HYVlyx g0Ymcm7vh902mZVzkoE9X2ZfrsM5WK4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-504-EoSRB73-NTeakXqzF60bBg-1; Mon, 04 Dec 2023 09:22:31 -0500 X-MC-Unique: EoSRB73-NTeakXqzF60bBg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F2B7C101A550; Mon, 4 Dec 2023 14:22:29 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 599D82026D4C; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 22/39] mm/rmap: remove RMAP_COMPOUND Date: Mon, 4 Dec 2023 15:21:29 +0100 Message-ID: <20231204142146.91437-23-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 9566EA000C X-Rspam-User: X-Stat-Signature: 6fd84qtyscbida4j88qykgyu4edrd51t X-Rspamd-Server: rspam03 X-HE-Tag: 1701699755-310712 X-HE-Meta: U2FsdGVkX187Av+A3er/49JSPrgb9dRENIHUcJqsVPV+OmvQbpoh72gcjGhi4pikX7GVK5Lh0rdzV+WvwkN5YfbtqaNJbmcUY+YQd7NDh/fxfAD6no5KdeJrfb+bg6Yx41AZ1viBOAJlv+bmzRm6za43Ax5TAvyf4YiNqoiUHHnbwhlacu/SWlB2P3U2/3Z28GrM89QX9qZdYJf+VuAzq1TvmdmnTLHnK9+TfDJVyFQzri2255nI22W6UT61gkDrgnEOALGn0pMQADOVvdAZ/g6aJgWBFGbnKBRDnSj/ivTBqWEHetysJRu5tut2jzSfx7DhbRDx0QWjgkg6Lbubv4AKZfJXJOI9jW9Z8+LvRWMaLrd5++PzJDRoT+auT01gvCWgXYR5fGwBTUnv5NywvUeqhtbfoNaN8bU77Km2TNqS6zT4dIpjCd4AtsCWVIpIM9quO01IC7ZQn7xZkPImtd7AfOlqIOH4bOwc+P9YTI/mRnDA2OtzjqjUdkI4g9DWvp+OEOq0Zotd64vH2naLfmO+lCXpvrW1UEPoMhSx9OGEqF98ZB5twczNhtMwLS7wPXJAEqjli+aEzmB5OyTdM1of2+pdg6xrXU6YSw+1QkIqOVwtU1RZcDu8RYPnaGZWRxC8auPArjr8eAKBgluFYCHxeJTAT/RyVm4ukfzA7iAdR9nu6YJ/u2X2wW1qAO5hR9UJ+53xe/QQh24lMzpNuLpU9mf3Qc2Ewy8Alecf7cJXNoegbJ3HJ6eQ0eyKItYcENzaUUrtBAMC7BRn5RXrWVOzmAcje63bPX4ZSRN77nnL+0B1HU1EVrvU2c7q+YkZyoENSssspDlH/xNDSRPgfeRk7M8WW0ZvcyjQhxg9NGRj8OWHtiavVUlJruftz5VGjIdkQWr2b8ID0j7QlscIZdv3fuiSvUUyt/K0mA7W8TAYEkKYZ6MODxT9XWWkK64VzNI1gjlXTKw9GKIc2fA k7ZoI8dK lXB0InmFN6kvgYydXuBl/OPbf/cciM0xp8erVk0gMjoMlTSCSEs7YD0zBSBR/bzMRufDsXNLkeWjlhszwUzGduNeYeT6LkzwiQQFQFk5YuFRIgsmDU3C3feriTPW5DTZQSLzDuhuXeaqz3caOtTLXjd3LvY+THB1qTU2AKgFOxDzVJG64fRAPQSRuyCkrINlbEyj6WxBBiS3KCmRtO2wx/Yn50A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: No longer used, let's remove it and clarify RMAP_NONE/RMAP_EXCLUSIVE a bit. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 12 +++--------- mm/rmap.c | 2 -- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 865d83148852d..017b216915f19 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -172,20 +172,14 @@ struct anon_vma *folio_get_anon_vma(struct folio *folio); typedef int __bitwise rmap_t; /* - * No special request: if the page is a subpage of a compound page, it is - * mapped via a PTE. The mapped (sub)page is possibly shared between processes. + * No special request: A mapped anonymous (sub)page is possibly shared between + * processes. */ #define RMAP_NONE ((__force rmap_t)0) -/* The (sub)page is exclusive to a single process. */ +/* The anonymous (sub)page is exclusive to a single process. */ #define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) -/* - * The compound page is not mapped via PTEs, but instead via a single PMD and - * should be accounted accordingly. - */ -#define RMAP_COMPOUND ((__force rmap_t)BIT(1)) - /* * Internally, we're using an enum to specify the granularity. Usually, * we make the compiler create specialized variants for the different diff --git a/mm/rmap.c b/mm/rmap.c index 4cb9d8b7d1d65..3587225055c5e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2615,8 +2615,6 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. - * - * RMAP_COMPOUND is ignored. */ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) From patchwork Mon Dec 4 14:21:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1843DC4167B for ; Mon, 4 Dec 2023 14:23:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AEA46B02A9; Mon, 4 Dec 2023 09:22:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 335396B02AC; Mon, 4 Dec 2023 09:22:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AFD26B02AE; Mon, 4 Dec 2023 09:22:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 05D116B02A9 for ; Mon, 4 Dec 2023 09:22:39 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C1C351A0295 for ; Mon, 4 Dec 2023 14:22:38 +0000 (UTC) X-FDA: 81529351596.18.DE60C60 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id E10B240028 for ; Mon, 4 Dec 2023 14:22:36 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c2B1P9nH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MMp/rsTScJnGvEHSBNzvW50JXqbYQ0nspFlZU7tqJ/c=; b=KEeqvjUxxPZVNGQPYD5Yxfgkhz3VosNmwM+pja0ulkarPB57nX6fi1/JJptJfEA3DIKnYc Vj+TUHudkCaFVn53163hCYeQR97u04c4t7Q1LBW2IRSKcDIz2t1gAoS2Dq1ISBNJRI6twm fcY0QT1npADV3R/e//MNIbZeejQEgxU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c2B1P9nH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699757; a=rsa-sha256; cv=none; b=a4Fz46b9Yq5EcPWF4LkkuyT0blOlEOsCkFqDc6g824tvec2kM6hDiOmFXjILzshWgWNH+I 4sp/Wjoq+snfzuDQKnwRUT+a/ucQB3zLDh92JSSLzeejKwOehjKA5qu5U1aaOESBRVAil6 7jFzVRGi01dkQl9ZkUmeG78i8beIMHg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699756; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MMp/rsTScJnGvEHSBNzvW50JXqbYQ0nspFlZU7tqJ/c=; b=c2B1P9nHHGBI70/0yW1U9DrZ7wOGe8mXF2DLbxr7u6v8r/tmsLff2FILxkbrFOmPBabMxO RoxUE8bLgLHdpdz9CysHpbB/jdl8i4IBhcT4Ie5tI7kbJM8wKDzCzc6BeJlLFKVsxcebQR QrGY4awItnH2yBT/VROehMVfpG6Gf0E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-494-KFAuLerwMeGWU5ALc3K4CQ-1; Mon, 04 Dec 2023 09:22:32 -0500 X-MC-Unique: KFAuLerwMeGWU5ALc3K4CQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4F04185A58C; Mon, 4 Dec 2023 14:22:32 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3A5802026D4C; Mon, 4 Dec 2023 14:22:30 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 23/39] mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:30 +0100 Message-ID: <20231204142146.91437-24-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: E10B240028 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: mszawxrhrjg6ppa4yty1hzqd5365e9j9 X-HE-Tag: 1701699756-139959 X-HE-Meta: U2FsdGVkX1+n/1+qY2fh1Ef/Y6mNUcG7vWmNoGPohsm9rYgJB78HZh4oixQh/clRIalZYSm+pZlMbnCFgc23G3VpjdDNffmUlvC0i/4izirLvgRsZOkbkBXUdDRm3sZc/gtQRpK3MJfV9qEJjD9xwWAQoUzG+mA1MRD+HE+CIBxvMN7cdqlBnzRKCFsSDoMDOcplCpb/tjReb9A/jX2SC/16w603PajVpAZCOUdG7p19y6Wa8z4zlJM44QpcnsyopRBAE7eKZGQBgCk8Mh1ZUNqjRNJIO4yK2xhEZtdg7ZlPpmY6+pydOy50jQ+5M1TeUJ88vRyKWMCb3MTqsgkA2PYJfFHXsY+kTbdQUH+EPtLoc8koGYuqBN67oiDwmO21ZX5GmIWa7TaxZ82Dane7ZpUcarlNVMQ9NlkplGrD2OizbJTEFRWfimiusOOAutl7H0c9BqhovSNW6yLyEDrQCx4TRYIldjEywp8kCE+eO2JVt2GqqUUuiow9G/Gnhmmn/seMbMXP8QHzLywAhTgyrWIi29LaC0eb7PVEi7vnJx/P5MYQu+E6V/RCNpa1igJ1MGYzysViH5L+sqYpqiFJS0RD39CYHTD070LApgDnu2dyh/Gj2lqKPVdGSK1pbWGGVh4Cx0eDUJc9zYh9XD4IEmH0bwTQuiC69UY7L6mGQ3RzzHJPPg4KjrSnjOBf9NRUD1OSAQSB4FVyIy7XyjkrhDzoKbwTTQ8zydku78sgSVC/kIGqiACQbZ5DSl5haOtFZGQy52mN+wO1LtAOsFkO4qyL9dWCPcy7bLDwv82UFy/tBIWwS5XGfrdZeFHQO1QHZ+c9fc3Ed52ncQnsxKq3127i/DR4wvgro0KdgLYwGIwXqVVDZ8Qb7uwljWLRKfsNR5D8zgiy5SEz81jH4yFXJejfQgnM0+rNZK9rxwEZEYmhQu0gdIB6bGJj6Wwf/slihrUT4NNwqTF4fD2KIWG r7exV8ep PTIVlx2WGKJf4SBNK9dPUPhNpm6lorzHrbDmbAufeJSFlnHLTkx0j24B+KDM71nKgIueq/wIv/CiukFhPDeNuLulRY1Hr0ho93IyTaRKKW88uen0k2aFkqP5YaKAS7INlU835CQBhfvHgd7cxVKWOuhysGVYFeUe+ryQ5a1H7n4j7lsPtBuSJBCAHms/mqmYwfRTBigjh/02s5w6ybo/px2gsvYScPJ9NxTwf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's mimic what we did with folio_add_file_rmap_*() and folio_add_anon_rmap_*() so we can similarly replace page_remove_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. We're adding folio_remove_rmap_ptes() handling right away, as we want to use that soon for batching rmap operations when unmapping PTE-mapped large folios. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 ++++ mm/rmap.c | 76 ++++++++++++++++++++++++++++++++++++-------- 2 files changed, 68 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 017b216915f19..dd4ffb1d8ae04 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -241,6 +241,12 @@ void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_ptes(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *); +#define folio_remove_rmap_pte(folio, page, vma) \ + folio_remove_rmap_ptes(folio, page, 1, vma) +void folio_remove_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 3587225055c5e..50b6909157ac1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1463,25 +1463,36 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); + + if (likely(!compound)) + folio_remove_rmap_pte(folio, page, vma); + else + folio_remove_rmap_pmd(folio, page, vma); +} + +static __always_inline void __folio_remove_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *vma, enum rmap_mode mode) +{ atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int last, nr = 0, nr_pmdmapped = 0; enum node_stat_item idx; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being unmapped by PTE? Is this its last map to be removed? */ - if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(mode == RMAP_MODE_PTE)) { + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + if (last) + nr++; + } while (page++, --nr_pages > 0); + } else if (mode == RMAP_MODE_PMD) { last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1517,7 +1528,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * is still mapped. */ if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) - if (!compound || nr < nr_pmdmapped) + if (mode == RMAP_MODE_PTE || nr < nr_pmdmapped) deferred_split_folio(folio); } @@ -1532,6 +1543,43 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma); } +/** + * folio_remove_rmap_ptes - remove PTE mappings from a page range of a folio + * @folio: The folio to remove the mappings from + * @page: The first page to remove + * @nr_pages: The number of pages that will be removed from the mapping + * @vma: The vm area from which the mappings are removed + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_ptes(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma) +{ + __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_remove_rmap_pmd - remove a PMD mapping from a page range of a folio + * @folio: The folio to remove the mapping from + * @page: The first page to remove + * @vma: The vm area from which the mapping is removed + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * @arg: enum ttu_flags will be passed to this argument */ From patchwork Mon Dec 4 14:21:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68614C46CA3 for ; Mon, 4 Dec 2023 14:23:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77CFB6B02AE; Mon, 4 Dec 2023 09:22:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7050D6B02B0; Mon, 4 Dec 2023 09:22:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57ECE6B02B2; Mon, 4 Dec 2023 09:22:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 44C8A6B02AE for ; Mon, 4 Dec 2023 09:22:42 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1185D12022A for ; Mon, 4 Dec 2023 14:22:42 +0000 (UTC) X-FDA: 81529351764.17.B1973DB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 63B5640017 for ; Mon, 4 Dec 2023 14:22:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HDd3xvCh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nwrEIToy0DTNusea81H7rc7vo5MP+ocGWlNPVi4s7cw=; b=FVNMiG90//uqtr14tZINWF6Z7HWLJYxU5wICV8U5MZnsvDRPA3ybuwoc212FfiBFd5b97b 4MMePsLQPUwle9FliO0m/e1i+T6HksXcDNybs4z+CoVa8SdBoG8n5AfL/JkCNel5d23rnB XaNCiZ5DFMajU0CB6Qu2tHKBWp6uUEc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HDd3xvCh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699760; a=rsa-sha256; cv=none; b=2Kg+2fk+x8tBIjHq0D7CwrSzpO8rRBMCWjo1AOQmcp2YFxFJSRGjqpvdjwu5QgGcFNrgzU KV5SwPZwPOpgEh4UCZvfkSLDKjBz9LaLjrNstuHOsfWHS3qe6LdsNFcrOXJttL1Ql7yZWf J6yLb4QXpDDGo9VA+x78LTNilppq5aQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nwrEIToy0DTNusea81H7rc7vo5MP+ocGWlNPVi4s7cw=; b=HDd3xvChATM5xm8cy7xtV6/7AE4wrbW/0+2AnUs4zNIxKDe7QoMVmRXqJHw2DkOkh8S39e X+VAo7lKFg6IW3RH235fsgaXGOkC/jLxBPDQD8T3Rx5y1s7SALWuyyMXO05YSBhyFDFn2z A0WWj4xMhrTNVJrFiQtITHOtjAZn2rA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-135-DF69DVHUPTOZMJ2zvuPJPA-1; Mon, 04 Dec 2023 09:22:34 -0500 X-MC-Unique: DF69DVHUPTOZMJ2zvuPJPA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 290303813BC9; Mon, 4 Dec 2023 14:22:34 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 87D722026D4C; Mon, 4 Dec 2023 14:22:32 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 24/39] kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:31 +0100 Message-ID: <20231204142146.91437-25-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 63B5640017 X-Stat-Signature: j5xp5mi68asjwuin7iuozfhgrqxx1hrb X-Rspam-User: X-HE-Tag: 1701699760-88946 X-HE-Meta: U2FsdGVkX1+kiiGdcdkXwfUHXpDySaLafyqh+pCfEaTzRHRfYBKo60wPJV+LwDH9LkT8JTUI3Rnl6hNTbxhdpVriPecwm5XUbnTFypUdlHXBuH1Y+DR9OvypMqpMXL/OTImAK3MADni2Y3yEr6TlAVdhe/Yld4wqIYC+iRtgQ9G1SxckF3seSg+B4EhzLLWlY/fFzjn9AJxteUXq/2u/4yp83skjTjpXBWE0ydWbvKTZRJwpPN1MDPsIOoWqhfSzDeQYc93lNQ6HCiEGUnZum4WdTk5ajHg1blmsz7fTiQBgyeiWjGjLf2yDp9hN7UbJTqZT9TE7M3mtmVilsxjwg3Nz9ocgxUXLO3doBosv9MhXMLsVc+RDsyE7MCw5HBl2VQiX0fZ0ecEaCouJIyRop0YJqCe809HpwDad7odv+IXjOxWbvxmbw+jtKGX2ib3Zg/dsvNkZliEMY9vIKF/EhUr67ETvmI0lJc9BQ8bpPS8MNQhZgDHZ5zed9ahlxasTc7N02jBlCrJR1WkEL71GV6Nn9+31pD2K0wrf46rSKMcvYU7XDdV4aDHo7rBJ+7srPAPBt8s15+lmkRKWfsdZwaHSaAayUdY15y1/65gGtattUkph9VvvnMzlOgOqfN1zH6rs0G56i1FfzCkygW5KKy+VOMU/d3PIGsw2WaTwQ0KNubnO6xdLFCVG61d8aUrMDbBGfhHiBw5Vzcp+VCAUtdf6vP90yYg5afT2Plsk8cFQ6y0zRoqjQe4O65E14tL5WkBDiMJ6617lk4pCRrySBjgGr4tezdWsQrZKnp2PlZU7lWoCKjbpLvGYczwQ58vkpi4wq7B3PDzhvtbqw5PdbzWELNB6WAp4MYhg9unF+YHFJD8aAhnhYnYYElqNA0NNOV1a26ul0xYEoa1abTZoprjgNXSs6EhffFw6lJXV9oqIcl++O9GOj6Ww1I8XdQoWkX886Fw8n/mPZFuqZNf AXtHF4S7 AKLh1OVcVQUIWNANAeWiN3KR9NHBYRfRSx8b6Uuz1kOMzOoH3KAEedFhtvzqyKGYYddo6OHe2+vqH/iNFpQxUJPJsOvI5XXbcC8SGHfu5fnZOXEhgvpZSCRT6GMDHZWQZJeRZSGjHnYSD8j88hwXBbzH7r4BGPes36rIhI9+lflSBWL/+9j8PlOO5oNf+pbXHc7Aw5Miy2ap5po8Aj4ILcB304A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert __replace_page(). Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 435aac1d8c272..16731d240e169 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -198,7 +198,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); - page_remove_rmap(old_page, vma, false); + folio_remove_rmap_pte(old_folio, old_page, vma); if (!folio_mapped(old_folio)) folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 4 14:21:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B024C4167B for ; Mon, 4 Dec 2023 14:25:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 163346B02F2; Mon, 4 Dec 2023 09:25:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 114106B02F3; Mon, 4 Dec 2023 09:25:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF67F6B02F4; Mon, 4 Dec 2023 09:25:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DDA3E6B02F2 for ; Mon, 4 Dec 2023 09:25:08 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B1B42160132 for ; Mon, 4 Dec 2023 14:25:08 +0000 (UTC) X-FDA: 81529357896.27.D0ADA24 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 145551C0017 for ; Mon, 4 Dec 2023 14:25:06 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CxHZWx4t; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699907; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K8HS/aZd56HSzREOp5VgtDWJgCULJ/ieYaN/iS/Tw6M=; b=jjreNu+UbSmjQt2RE8NQICsCcgJfvrRUGE+AB2jz8T6C66d5bsjd0lGeoVlQfBtH9kB3uZ tfbuaUrDEqPKlGV9CeLJe39KZ4PRHOOSyRnEjf/iTJ398p6BGNONTuGArNL3ppDqK+PaiN GvFJw53QXYXrjAtzVk+dC9jDZof6BWs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699907; a=rsa-sha256; cv=none; b=OS+aR52HMdLOLyNNT+h2dBfph8tq+U90Hm6Kt3uYziIG45eeS8+YTjikqZfZhTgsLb9Ljd p8gqFtqZqbCYw7KAnl0giuyyR0zuJJ0TJiISD+rZ8pxRAtMFCdgbd/FnuXTlJqMrRLxXH1 kRkQHP7Vex5FJrwzSru3mhbQgh+JC6k= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CxHZWx4t; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699906; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K8HS/aZd56HSzREOp5VgtDWJgCULJ/ieYaN/iS/Tw6M=; b=CxHZWx4thUKY0/DwVJgoltL1Lq5jhXsw3U+bwp8VASGVlLRKkuJqgyJyfrjiX0A3D6t+5N ChY5vTV8IMAUvk0tuO1Jf0n23aQlcnHiYG2VaUE7Xgj/9FJs7uyyE9MVe0XsmgsT/UrWo6 RLu38Pa+5ympNgxfGDTZFcJ742VLbqs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-lIncdVQIMkCtbPQ0fZWxhw-1; Mon, 04 Dec 2023 09:23:56 -0500 X-MC-Unique: lIncdVQIMkCtbPQ0fZWxhw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E0B41C06369; Mon, 4 Dec 2023 14:22:36 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 661A82026D4C; Mon, 4 Dec 2023 14:22:34 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 25/39] mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:32 +0100 Message-ID: <20231204142146.91437-26-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Stat-Signature: bd3nzixtbc7q47q1gjiqdsxkh68xen34 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 145551C0017 X-Rspam-User: X-HE-Tag: 1701699906-404429 X-HE-Meta: U2FsdGVkX1+23+Yb5PRXPDT3Jrc6xIqbjTKxU580T0Ic4jBb8wYFqztqREKo1QijeUFqTvJtSvVh9rSrJ3lpGnQaFu506byJQBU1latQhLk0oS7EfoPmUXpNdGQbr3AK3zhuve8Htg7QzIKT3eAJUuuajWorTZcT77Fy+Rsw5/NtpY/WgRc/YRa5sdtRO5OEzBb8y5oWyTtZoK/gUSCjFlJa/Vu7B9oBeAIsHkFtfS87agkTNTBs81wYSzFXEtSGMb6gcUYQsKsr7F2iFmdJTIVzpZhlQpcYgEVL3TpqkHY8vnr0810Zn1ywrfg4STgKuLDtVCorEvY1xD0mWih2A+iljpoVSlajDqOzt1oCDnjWOjbj8xIpgxbHHFQAYUPhhBY7GnTMZscn/RvXdQRGet26vsyKe8/bfKFSmtpJ8bZRuo8EBXx0PUONTrvhwlWpQr12nuGmOeIX4SSufe5dSEVTSNmE5EknGpZsSuYcFKTWimLeSaUBDX14UzAdtS22mPJkgvTC1Z8U6WXljUj/j/mjoEst7k5Bc9gAEMXfY7KtYLrfwgVuZnxVsCYTR9CpQ+GmBUU61YZQCItjPjovDKGOY2uRJPJMrm/4LXPUSRQtrVCTWx1v0nPzP0o3SMi7TihTDLtIv5bWT8GwdM5JAVoTJ7csj58T2zZNFK0c17sCxfZiGjhKf9Yn+M1F1omnDXyONTcD7rCli6Mtf5L6Ao/1cHq1NlE3xQAPs80p9x6LHZRHin1/3Pk1oYWixjy1TV6b2qcaXPYPYuWVCpqfN3Bp28J4p0TeV8mpFqgon2AFP96fzLzIbzXTx3fI8tbZ2rmRfu99jPyrG5M6NTf2OdLTmZt89fETdo8GWPNLfjcWpkkDCezcJAU3rm6D+qVZn2XOCDfgUH5fn2aDCXk82v+qxpu4SQQDrzHIugOEuxLOM2ia/okY1Gx0N4A9INcWPshey6vp7geszuGFp32 c+6VdoWN 7CACx1CaCs3/AY7atJdoawd7pOPlEZmohNzko9J+ZISLBCT1ZEgucbxshL2uaUDBqqXZCUC5TwD6Nqgcmt1lW2VdyTi4tjvXvbnP0sEKz9/EYrjmUu3wnJWv4eYr1nIhKKnS/CPGADkQRpcmFfr98VOT7E4/COiNaKgKncXlhqbfl+mr191pAtkhScMgn9r+ursfE+gJSrRsK9d8G5Uu4paSG3aD8SmDK66pIQ51A63cNzM9jhl8MIWl7tJjTSl1BPDPry4G/qQwphXA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert zap_huge_pmd() and set_pmd_migration_entry(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 332cb6cf99f38..9376c28b0ad29 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1721,7 +1721,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (pmd_present(orig_pmd)) { page = pmd_page(orig_pmd); - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(page_folio(page), page, vma); VM_BUG_ON_PAGE(page_mapcount(page) < 0, page); VM_BUG_ON_PAGE(!PageHead(page), page); } else if (thp_migration_supported()) { @@ -2134,12 +2134,13 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pfn_swap_entry_to_page(entry); } else { page = pmd_page(old_pmd); - if (!PageDirty(page) && pmd_dirty(old_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(old_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, vma, true); - put_page(page); + folio = page_folio(page); + if (!folio_test_dirty(folio) && pmd_dirty(old_pmd)) + folio_set_dirty(folio); + if (!folio_test_referenced(folio) && pmd_young(old_pmd)) + folio_set_referenced(folio); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; @@ -2294,7 +2295,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte - 1); if (!pmd_migration) - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(folio, page, vma); if (freeze) put_page(page); @@ -3235,6 +3236,7 @@ late_initcall(split_huge_pages_debugfs); int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3250,14 +3252,14 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdval = pmdp_invalidate(vma, address, pvmw->pmd); /* See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); if (anon_exclusive && page_try_share_anon_rmap(page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } if (pmd_dirty(pmdval)) - set_page_dirty(page); + folio_set_dirty(folio); if (pmd_write(pmdval)) entry = make_writable_migration_entry(page_to_pfn(page)); else if (anon_exclusive) @@ -3274,8 +3276,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, if (pmd_uffd_wp(pmdval)) pmdswp = pmd_swp_mkuffd_wp(pmdswp); set_pmd_at(mm, address, pvmw->pmd, pmdswp); - page_remove_rmap(page, vma, true); - put_page(page); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); trace_set_migration_pmd(address, pmd_val(pmdswp)); return 0; From patchwork Mon Dec 4 14:21:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54100C4167B for ; Mon, 4 Dec 2023 14:23:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 756706B02B0; Mon, 4 Dec 2023 09:22:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 705D36B02B2; Mon, 4 Dec 2023 09:22:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57FF66B02B4; Mon, 4 Dec 2023 09:22:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4252B6B02B0 for ; Mon, 4 Dec 2023 09:22:43 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 71DB340291 for ; Mon, 4 Dec 2023 14:22:42 +0000 (UTC) X-FDA: 81529351764.16.51D8B83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 9F7EC40018 for ; Mon, 4 Dec 2023 14:22:40 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WLketFYV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KbWS1WI33hCQDyuuf4OMcK1JcfvJiYhRWhxjjruqKZQ=; b=4AGNw1DO31PkSFZ0ToW4DO2fUiYcVr0ZKs/xx4yhG/ulDzKQpfM+pfYGZenTsqL6X5V7Ol 3lIKGeapIAA6CDjiwSteb3BlewywPD/PCnSArW5yepusr2kQp4xud68f50XK5nnLpxbAM+ pm/GhAR4qVeP7bg2sEj/Cq3ssMNczdk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WLketFYV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699760; a=rsa-sha256; cv=none; b=NiLkMGEoEe+sQG46KaD0x66/O4xyyTsYTcybkX6+sVjGfrC0oPTLhU80zUomWRJoPdkGWF defD2Kv3QNB6A/sBU1nwmcxpQxdXw/Rsei51jFv9hY8JU7HpY5NjxELbo11MjqCG6bd6hH Xs3IqXA5/cRHxRQm9hicSL0yJM5gbAQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KbWS1WI33hCQDyuuf4OMcK1JcfvJiYhRWhxjjruqKZQ=; b=WLketFYVLW2bNZi/GxwNRn92cCxCNk/K/BzSZpQNWXKKhph/Zn0LQmX4RZ1Xfmi1sX9Ikc 8hYOffPuVWKeTkK4JN1Fw+y6qUbPExfzHFBOE1/egTf/V1fFItqnmukuMbhyyMp1plRKSB pPHIXx6NE7leVie3PV6ZS7WOH5Q2Nvs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-d-d9o59fPPm40XM48Es9-g-1; Mon, 04 Dec 2023 09:22:38 -0500 X-MC-Unique: d-d9o59fPPm40XM48Es9-g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D93A9101A529; Mon, 4 Dec 2023 14:22:37 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 460F42026D4C; Mon, 4 Dec 2023 14:22:36 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 26/39] mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:33 +0100 Message-ID: <20231204142146.91437-27-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 9F7EC40018 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: quto9fc39h36kogimd5is5um17kxyqw5 X-HE-Tag: 1701699760-554089 X-HE-Meta: U2FsdGVkX18ZaZiD1KP9XsU0DxXPsQHfncURjOxKA+Rt1qeNKZiyTqj2iM/d4119Fz7eRmWiot3SnLt5cOykTqtZF7sx+ssuTS4w0MyV1UQ155Yz9c+rrYCAxlXV5YBh+JkxwtTxjedTkIs7zTofP6if7ggm17Vv0NP6l5Tqbo0cXSqszbq/Cbjd9SPs3T1I0NV6t11bsEFXHIrI+9higvBji1oUuvzf1mhyeoLfYX9zjhNKJW+dJhYDTLNj5IAeeM1atcrzlK3my6es4kHXSB9Auh0AMEZ9lza5WclI7Dt77a7xcODxcQeSAfIZtYeS9H+sC4sk1mWf66Fn3sHIRTPetrwd/V6G8fGJsd56wPhVj/jdmnP5G7GLWlaGaBJem0F0NKVFk4wWj7Z33G05mOMHNOlBfmbZpkvcsV9RwuDUURYzBzzjbO9fKqHuvwyBgNIaX7kUJrcQTfc5HpwOIyU4kg+q6Db27t+3afbdMNoqTqDuhwH309BbkNDxl/E3eidJBwWXuD/0v2z3bqGKpb1nn9Lgu4L6o1c97FdW/Wck1WOCUK5X7kmbQ5xyXerh6neILVHp7xQOAjYwGyCnVplCQdM8ORz47/khHd0u1HpuyfwziqzU2gh3t2fqWuSseXeuea+I7weBV/FpDYwhaPfIhITCmvX6dgT6uCL0kDhzWPj+M3rmnymz9jK+ExeiNPBV+dJXvhxlSeBgZJTCWQiCkt9iemEf5amkReEZPxWrCCl/Fp4YpDcwEajU6cB6JrEKnWq6nwvL3AvwU5YzmSyZZgxRSGTeamuOavN8SFwpdMMdFX4r6FGqGLvw18wKLFzG9rQwvBU4ytkUD72UvyLTvgpZ0yxXbKNFRwh9/1KUDVSYuIRTaEc49fu/yo0QEVzsA5UMaGp8PB3rdMXyxP5qBhzrhYcRSsgsYrqztUhP0CDeyfGXMP9inkiSx4v7cH/4kxIoPodzIUKqQHq bQ+5jIuU XuE9XuOuiQlA+KUOiLx2zuG8FUuaUmspR/zU/IKqeAtwcUrjVQ/FTcvcNaT3aM2yQRxm0LRZwf5mYl6NEcSgkNvnXd7xmwgPIC7lGi4G2Psxvr4h7lCW05URzae9esTJdZ28Lm9ddZYd1k6ZsHr9RZuY7XPejPJiNItJ6AU166AJy2Vrj097z5YdxBucfJ/hjP6f51Ro5+Vk5DmKm33EiJanmTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert __collapse_huge_page_copy_succeeded() and collapse_pte_mapped_thp(). While at it, perform some more folio conversion in __collapse_huge_page_copy_succeeded(). We can get rid of release_pte_page(). Signed-off-by: David Hildenbrand --- mm/khugepaged.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 064654717843e..c2d7438fac22d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -493,11 +493,6 @@ static void release_pte_folio(struct folio *folio) folio_putback_lru(folio); } -static void release_pte_page(struct page *page) -{ - release_pte_folio(page_folio(page)); -} - static void release_pte_pages(pte_t *pte, pte_t *_pte, struct list_head *compound_pagelist) { @@ -686,6 +681,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + struct folio *src_folio; struct page *src_page; struct page *tmp; pte_t *_pte; @@ -707,16 +703,17 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, } } else { src_page = pte_page(pteval); - if (!PageCompound(src_page)) - release_pte_page(src_page); + src_folio = page_folio(src_page); + if (!folio_test_large(src_folio)) + release_pte_folio(src_folio); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats - * inside page_remove_rmap(). + * inside folio_remove_rmap_pte(). */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - page_remove_rmap(src_page, vma, false); + folio_remove_rmap_pte(src_folio, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } @@ -1619,7 +1616,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * PTE dirty? Shmem page is already dirty; file is read-only. */ ptep_clear(mm, addr, pte); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); nr_ptes++; } From patchwork Mon Dec 4 14:21:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 026F9C4167B for ; Mon, 4 Dec 2023 14:23:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A63C26B02B2; Mon, 4 Dec 2023 09:22:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A134A6B02B4; Mon, 4 Dec 2023 09:22:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8663D6B02B6; Mon, 4 Dec 2023 09:22:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6E9D36B02B2 for ; Mon, 4 Dec 2023 09:22:44 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4E1DA4024F for ; Mon, 4 Dec 2023 14:22:44 +0000 (UTC) X-FDA: 81529351848.22.CF74379 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 8A803C0023 for ; Mon, 4 Dec 2023 14:22:42 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DwPcIE8x; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IO1sF7V8N+lGKeYvOYG0Xi/4CGC4wNOlZZ1+xeF2jmE=; b=PBCZmlaemxyhzIEQIY9bTcAtqL+r2tyMJAvgnW4I610YnlauB29M/jDK4sFAtKMX+Ejvky caXL+MbA9K5XVHbI4eRjgZz6Xy1fh8MfNj4el2BW2D0OdmVCbduufCEYPPvnF5fQbUSt0g o39JrBSTqPLnRXAjDp+nuPcPeAelMkM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DwPcIE8x; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699762; a=rsa-sha256; cv=none; b=0dQ/NJ8ctgS7akIOKBGOCNWJ9OGDFK0OofEAUBFAZRlySR855xWoMrG8o5SILUT45I0maJ 4s4jnFzXtCNotUTZKVE4Vh20KyTYeK65xTxwOdh/JMr/ucVklrqd5YbCK+uebTuoJuNSlm ILHeJ9K3khtNA72xYCPgrybVtyEbT08= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699761; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IO1sF7V8N+lGKeYvOYG0Xi/4CGC4wNOlZZ1+xeF2jmE=; b=DwPcIE8xsjcy/DbfLWnubn4gYRGzNx4n+LHPKQEV097WX2cFmuQX8k/eCdNqdnpwkfBYeb sFOevmEqE+T/1gYOPvKLxHFZGFXakzZQvv1l7RdS5uwQR2cPeZQf2X/iMZe/XjJvkksmAm nJgX/1tpRVqNToRFp/T63a1PO8fHwkw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-EsHpcshiPLKnjsrxEa5ujA-1; Mon, 04 Dec 2023 09:22:40 -0500 X-MC-Unique: EsHpcshiPLKnjsrxEa5ujA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C3E941C03D95; Mon, 4 Dec 2023 14:22:39 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2055C2026D68; Mon, 4 Dec 2023 14:22:38 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 27/39] mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:34 +0100 Message-ID: <20231204142146.91437-28-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 8A803C0023 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: g3j5wxpuzjicxpac4h5e5o8ymeqmj7pn X-HE-Tag: 1701699762-840435 X-HE-Meta: U2FsdGVkX1/6RjsIuxOJg0fI7uBWuxnt0CSqfA5Qgs3jVckXC2xbfA5hz/iM9yas27LPBf7ZdBi12sMR0XjDPYBxafHBxNb1kgq9qzYsSs8CkaHb/jAKI1eGamK6jZAph88MUA9VFboADkKon+8pUDMh5oRvo9Ely7D2dTaBVr0eRQ9gxhpyTO/1FVPDT3U/OKHx2N0c7rV0f//xFmiL4nF3yk/zRcyZUX0EjsDnjXoxknpAmZK8z0kwlEKmEqyGkY9g1qCfOB4/iem62gfY914vVGLTZ1cXofLghmA6+xAxCYcwJW8ThiCdbwwaGOClA1+D3m1xwZ+SL48+3/MFI+1bNmHHC4GUtpidu5LV401VuEY741DnhOi4HEhTrggKy1lwcDERxNk3ojJHQrA8bIwBJ6ghAhnsKfQbxQyG4iomHewfCQY7/DjLFakk2HKw6HZfiahGfXe1LA1fmvcmYcGKejzElp/EQQP0SbhNZ0M5REd8lYCmaaJZrqVnVkvrKv7FFygUyVrvuds7ZhOEe1IDhHRj/hyq6fCzLbWaroOSK6jX1g8PGeUFvtgbe/51KEDCzhhKaatxIbQ9iHMijbxelOTd4b2NMUE6Y6eYBSlgwLKZTSG/aHQZp02zzKf+lzg9P57tWwxOflVuB0isT/wxTguVQHuEfmchWoheKgxGBGhPAQ/FYm96ahj1t31OGZKsj17LL5mrViTvgqf/FHDXZlevnqAiwqaTOhtS2KkLL4YD+TPkLnS9PN+7wPEmSHN0mK9NEK+X2HOF2QjopjwkzcuNn/LYLf/yp1i84ry0EDOD11vnFIk5Nh+8o9q7ZB3gog9PN/SGMjVaS+Ja+wFWn6FYNUYkIEIqHxhceLbMepYcSsDtl8BTSYkOqVX8eknHh8j1dvWwjkZ7S75mFdVjKqgQuxFHMXIorBSQyyycy/TzgqVMQ5/19oPusqlGOf76mufPj/iYOytUZka YcYVT62R Pi823Bi0xJ3tdRLOwhjM3iPyn6F226XGpVORtMEHtg6O3ovF0yZ7IUzzjb55PL/zahI7JdZ6H+U3JA9ktY/rR//GwHGf52D3HKYF+SeKQGm1Z3ZW0i/I0yYw0qQnjj2JLCdzRoftFqgDj6HvXPCSS8na1vwvCBcLF+FcOqK9K7gMIedcoazf2UtBMaTm2XlOCXp0qP/KEt54JKweF24ypt8vZcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert replace_page(). Signed-off-by: David Hildenbrand --- mm/ksm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/ksm.c b/mm/ksm.c index 357944588a9a9..c23aed4f1a344 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1279,7 +1279,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, set_pte_at_notify(mm, addr, ptep, newpte); folio = page_folio(page); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (!folio_mapped(folio)) folio_free_swap(folio); folio_put(folio); From patchwork Mon Dec 4 14:21:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8F9FC10DCE for ; Mon, 4 Dec 2023 14:23:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CFDC6B02B8; Mon, 4 Dec 2023 09:22:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 57FB86B02BB; Mon, 4 Dec 2023 09:22:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FA0E6B02BD; Mon, 4 Dec 2023 09:22:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 29DC66B02B8 for ; Mon, 4 Dec 2023 09:22:48 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F3C3EA0270 for ; Mon, 4 Dec 2023 14:22:47 +0000 (UTC) X-FDA: 81529352016.06.5D33599 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 23533100023 for ; Mon, 4 Dec 2023 14:22:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BOYbJkWM; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nWXNFAD1MN3p2bcOLf/LK1onshg4c1syDBwJJ/+GHnY=; b=ujhm51D48NpYa3GdxGtdSOtRF0YgYje5Q9T75E6ENI/9O1GoqsFfbXVEpi9BgjWwhGktMq SBIWgCQkBIf+xfBkgNaenG+gANKDWNF1apMmF5zKssUK4krTGqc4YvW8PHWSv2ildnZkPM hqvdHrSNbR0JXdwL/YVC4vYMd+kXXQc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BOYbJkWM; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699766; a=rsa-sha256; cv=none; b=4WdHWe7pSQkclGiSmy8IRieDfAvARhRmwt49Qqf1TfEJCek1HNxOtB4trL7jDE04YAZj5Y vVpTiIZxTA4hRcOUNfy2wsA+5SjScM3sSZunalcaqLo/lseGfQ9h0iI/5WrbVSi/Z8tGa6 F02QCVsw6/ou3cSS08qtANeZ6tilYvw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nWXNFAD1MN3p2bcOLf/LK1onshg4c1syDBwJJ/+GHnY=; b=BOYbJkWM10cm4fzwbjyWIohDCTHyuIA+gkhO36XgRnr6NgHkt7KORSHzdiNcTaUHCxip0Z ueD1fDpmIRS9ONmkG4BJCINf5ysrlsdKp96rE2eZNVlnaUkEDh2djQcaoaWm+spbfWspzw dflgIlE7zVQqKQWrkonj6r3AwHioz/0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-208-G1DqHr0HMdSb4ua9mh5Iag-1; Mon, 04 Dec 2023 09:22:42 -0500 X-MC-Unique: G1DqHr0HMdSb4ua9mh5Iag-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A7F1A29AA3AE; Mon, 4 Dec 2023 14:22:41 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0C88C2026D4C; Mon, 4 Dec 2023 14:22:39 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 28/39] mm/memory: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:35 +0100 Message-ID: <20231204142146.91437-29-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 23533100023 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 65jqeghw5zehqckabos8waug7a9f8oow X-HE-Tag: 1701699765-820932 X-HE-Meta: U2FsdGVkX19ihyldMl8CNEfTZzSBYWLTCo/L20nZSSmA5SPaxooBruzzJDZgoiAz1UegVL1T3Difv98pHD1abKMI/yIxaPuM37tpse8c0od49QUSMniBfwutXj92aFO6Hd4WWU3kEV/dunm1nqj966TUrzEVWX1HrjUN19qlWdn/9A7Y6o1HIKXsbe2xX+vcCwDUm7Ix8rZQOpwjvQXKmE9dlnVEEq9As0uZ3S2Jj6XW0H5fZo4Ae1ywN8BEiLwwQThSr07OXL4ZNa/duaQ6c0O+XBueohO8sNhONslvcHZuyAJjCouWdIKFwSnfHG1z0b7dDMNk6il47kL4t0O43WW0fVLzSlKYJgcty+TYK10GCdlLy6Rb3ncX2Mst+JQNQX/IsJ1ikFeZWcQe0U4oRB1C7tKi7lJq9EJ7qaya3Qdvtmt1C7Ui1vdBCROJweSBzZRUwmIMvlNKTUmISUm2pHp6OzrowwP/2zwYyinE3VzcS+0IGXxzbSKIEWQAAyvfcqrUzBcuJg3+DwCcOZTv9BG8SDvXEMAf3lDu+AZWpVRiHIceNkSrjZIrbfAMLdQ6RMHLvqqxwQFR/zMSn/u/nRxTw51ArGbi89zWSAYAVDmwm/oQ+5kGQIAGABIFoUyW56lhPV344Pt/PCqiYTK2ZAypYExZ9CFajmOKGvbmr4iL2syb65YhmPQy1EEG6zVKyEBvY7FZ/sUfr3SRNcABj1XHJOSLz26dYLOKh2UCe7D49swAOaLSHw4Y1aqMtce8yaVYQNH4Wo9jPb4rcE+9jLWhASrwOfcBNIEM2hsPa5PX8V9ehnFtT8aZdbbN11tY2uXn9exI4l2zMFy/jFPd/qS+Xpv0SFd9spCIfEYkmG9o6alwS3akvcGqYstrrQhDggXbsGhehZHgGCF+SaN5lj8JXZLKHg3RTbLQEeidcnZs4wZzJi+InqPD6TKMc33WX/R6QA4zaj/rwveBWSB qY1IsNMZ bNMMYkUSr1p5p8VMieH0P8uN6KwzQKdtQkJQ++0EyDy/D8ARQYsrG9qGczfcM0UB+nt5RYioUnqtFAw9hUnBbwpuHMukmIkHlQ/x11a/nIZ+zrbkwd6SbfD/0xSOs2EP8yzkH5dGhdvHWkgokscsnymixqMOkPSIUqoTkAWycrjDxErfmrhOyR7jZwTPVrNWeilbyfdKUKhpYriYe23EVhps3Eg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert zap_pte_range() and closely-related tlb_flush_rmap_batch(). While at it, perform some more folio conversion in zap_pte_range(). Signed-off-by: David Hildenbrand --- mm/memory.c | 23 +++++++++++++---------- mm/mmu_gather.c | 2 +- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 9543b6e2b749b..8c4f98bb617fa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1425,6 +1425,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1450,21 +1451,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + folio = page_folio(page); delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } rss[mm_counter(page)]--; if (!delay_rmap) { - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } @@ -1480,6 +1482,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); if (unlikely(!should_zap_page(details, page))) continue; /* @@ -1491,8 +1494,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(page)]--; if (is_device_private_entry(entry)) - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -3210,10 +3213,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * threads. * * The critical issue is to order this - * page_remove_rmap with the ptp_clear_flush above. - * Those stores are ordered by (if nothing else,) + * folio_remove_rmap_pte() with the ptp_clear_flush + * above. Those stores are ordered by (if nothing else,) * the barrier present in the atomic_add_negative - * in page_remove_rmap. + * in folio_remove_rmap_pte(); * * Then the TLB flush in ptep_clear_flush ensures that * no process can access the old page before the @@ -3222,7 +3225,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(vmf->page, vma, false); + folio_remove_rmap_pte(old_folio, vmf->page, vma); } /* Free the old page.. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd217..604ddf08affed 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ if (encoded_page_flags(enc)) { struct page *page = encoded_page_ptr(enc); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(page_folio(page), page, vma); } } } From patchwork Mon Dec 4 14:21:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2254C10DC1 for ; Mon, 4 Dec 2023 14:23:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50C606B02BB; Mon, 4 Dec 2023 09:22:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4945A6B02BF; Mon, 4 Dec 2023 09:22:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30E9D6B02C3; Mon, 4 Dec 2023 09:22:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1BBA36B02BB for ; Mon, 4 Dec 2023 09:22:50 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B975D1A029B for ; Mon, 4 Dec 2023 14:22:49 +0000 (UTC) X-FDA: 81529352058.17.CEA66E9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id EE61B40014 for ; Mon, 4 Dec 2023 14:22:47 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ePpb0KTb; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699768; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nn9t1WIxRvDYJbnLiLQ2NH5amWMopIWzrUXghX6hcqw=; b=Cm2tgdhq0WFhOenJ2X4DNxoVh4lTlHDeFnSkMRuKUb0P9bhkZOcAjtxeYbPEYpjfRVqhJ1 SqLPCvsUU6oC+jHNcf0iEx6yAjFn0BfV1HQgG9Ox+q52HJIlwxI5Aj1GBBFCuxfJrBhFej 9GGqn3H1idBl+Eojeb+sNmQ04kcwdAw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ePpb0KTb; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699768; a=rsa-sha256; cv=none; b=R75/6TWFlTACNbPxghJt3fM7HajWkvZpE2tTkSkPIL9o0Y7tOK+j17f0pl6jcZx8I9R1p9 k/kBp+iMFF1JjOGbTOv1WIG7F/kgwaqoeeoaFnfS9HM8t4AaB8yBehTLJ5Hv5ceCBxwj+B E2dJVBjW/1okJUWHBSfUncRp8hLKzNI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699767; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nn9t1WIxRvDYJbnLiLQ2NH5amWMopIWzrUXghX6hcqw=; b=ePpb0KTb00uQxCgqJ9Xa9PwyovrddRkdSxpZAJiIKAL4zlWSayH6IdbdFH22xdSRP/56HE NfUqpwbqwnpctcrer9EMIJfl+tH/MN+a74tK/lWAu714uYFL4pLpkNWjGMwrIWZ97L/gT8 dDqP71JTcDjUYDiY29WMxNNGnYFuRwc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-657-UIUYp5yyP5WVSRDqwEYaRg-1; Mon, 04 Dec 2023 09:22:44 -0500 X-MC-Unique: UIUYp5yyP5WVSRDqwEYaRg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7C4E68F9204; Mon, 4 Dec 2023 14:22:43 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id E1A552026D68; Mon, 4 Dec 2023 14:22:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 29/39] mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:36 +0100 Message-ID: <20231204142146.91437-30-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: EE61B40014 X-Stat-Signature: u8w5wz9q4bw1w667yruwennzhqfefkuh X-HE-Tag: 1701699767-191119 X-HE-Meta: U2FsdGVkX1+uzbWeoEIWmzU56duZa1Xw0JYXTc4r/y/wF4M5Ztte0BQp4ZJ+FZKYj7QUDhwEh+BvYDO3w3Cnypq8i0FWJhSJs0k/FxPCwQLmuL868/jzQoASdT13/X8yWgFeeOOTJirWxj/JOibMrasJbTEQhgCZbTWLeNj5edbHSKboxmRXfPrceTUfE8fsWCk8PkzODSo64CXeYDj822j0zQT1j0jiFQXCAGSoalbxztxolb5cp6lwuN9jtBNwRnmQuMyPkm2YG/8et4Kl40p4pVrC/i9++97sAL6KsJRKa50tW8R8j8BmgloOlJZGXpe26RpITI1zA1dkNYG4ffxOJZBMTVcVDajrX0qfUlZADxU+RenifnK5VOX4y4NHhXZVi19GS7xNA23xo4lnMlCmNeRMak/yzH7VNMeM7BxhfPYemJ+wby34tVIuOETw1WjcyYUQBLdOW8LWnVKbPicuaD4yErQILofog3IZgEDpyP/VF+V1UqoYEBPavmQKvJYqF862vfPzJi3hwZYguHkE+P/dEBDGUzf0tGYX2gcPTNDF5zDJCh6Xs/Q3gqP2WTtodil+MRuwskcA89iIg4ygQWhXMekavY2KZumR0pN7tN3X40moVspF4ZWEzCf0AyQjpXd8mlNb6ckEyVxuBjmlPXIjcOHqu/XfDpNM6AixO6U9mJRoZUpuXkGU0pzBKS93WjLeZjI8FgMYn+9G4ELU5gz90H5CmVnyIkOYAWhIwCET9GLdYBl2cYHiHcsg6eENDWq+0gArHEd/iVvgWzQXGRLxzudz9dwgrMoM3df9NlZnueQImbOAddvaDQA8UnNz417OX395K3vH8jYRJvQv9ew4xpvl3nZPIE4j3gCSajKSrRvDnh9aqAlNFBCjZgB1cWQvp+NyoLw3s9v0a0hk2stASVpHFZs3ZvF8Yi5XMH3R2Yu7lVVbnc8DECdohlyrkCsPw4XsBPfNizN /2SWtKyA fbo0jJJzpH+oLpz4mf2Xvr7yRvAk4vxA9+23n0ChLwnO/UwV8H48BqDcVdOAIrDxHu0Mi3FhD2qkwhTNarqzqZkVSeEsPcGITR/2fpwWIi2c0VIozdHLtti1R0+XWxnQu4pfsopbnvZw2X424ironO08qz7yFii7TVzHypYCyZsLN/bOP9YMR82dcTLI13JYdh0riAqLsZUeJTpWRNGf6oVkxtg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert migrate_vma_collect_pmd(). While at it, perform more folio conversion. Signed-off-by: David Hildenbrand --- mm/migrate_device.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8ac1f79f754a2..c51c99151ebb5 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -107,6 +107,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, for (; addr < end; addr += PAGE_SIZE, ptep++) { unsigned long mpfn = 0, pfn; + struct folio *folio; struct page *page; swp_entry_t entry; pte_t pte; @@ -168,41 +169,43 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, } /* - * By getting a reference on the page we pin it and that blocks + * By getting a reference on the folio we pin it and that blocks * any kind of migration. Side effect is that it "freezes" the * pte. * - * We drop this reference after isolating the page from the lru - * for non device page (device page are not on the lru and thus + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus * can't be dropped from it). */ - get_page(page); + folio = page_folio(page); + folio_get(folio); /* - * We rely on trylock_page() to avoid deadlock between + * We rely on folio_trylock() to avoid deadlock between * concurrent migrations where each is waiting on the others - * page lock. If we can't immediately lock the page we fail this + * folio lock. If we can't immediately lock the folio we fail this * migration as it is only best effort anyway. * - * If we can lock the page it's safe to set up a migration entry - * now. In the common case where the page is mapped once in a + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a * single process setting up the migration entry now is an * optimisation to avoid walking the rmap later with * try_to_migrate(). */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { bool anon_exclusive; pte_t swp_pte; flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && + PageAnonExclusive(page); if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) { set_pte_at(mm, addr, ptep, pte); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); mpfn = 0; goto next; } @@ -214,7 +217,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty(pte)) - folio_mark_dirty(page_folio(page)); + folio_mark_dirty(folio); /* Setup special migration page table entry */ if (mpfn & MIGRATE_PFN_WRITE) @@ -248,16 +251,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* * This is like regular unmap: we remove the rmap and - * drop page refcount. Page won't be freed, as we took - * a reference just above. + * drop the folio refcount. The folio won't be freed, as + * we took a reference just above. */ - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); if (pte_present(pte)) unmapped++; } else { - put_page(page); + folio_put(folio); mpfn = 0; } From patchwork Mon Dec 4 14:21:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1FE8C4167B for ; Mon, 4 Dec 2023 14:23:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 005D86B02C3; Mon, 4 Dec 2023 09:22:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA6F56B02C5; Mon, 4 Dec 2023 09:22:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CABAF6B02C7; Mon, 4 Dec 2023 09:22:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B815E6B02C3 for ; Mon, 4 Dec 2023 09:22:50 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8937C120284 for ; Mon, 4 Dec 2023 14:22:50 +0000 (UTC) X-FDA: 81529352100.10.8451841 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id BDF13120018 for ; Mon, 4 Dec 2023 14:22:48 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=akUpp2j5; spf=pass (imf29.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699768; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wmC0HbMgxf2wn+iz2jDhVN1M/dlanF/535x+xcS2yyw=; b=n5EHgss0H8aWqqIh82aFTu7WZLug8qQgd6VRfFgtY22EoOOzYUtLoSFSn/C/XA0+uDqwaD C7ZorAJDpiNiSVqL3Zv7J2+8gRnoN3hAHgODnHhZGnmOhRFIs7JRBKsb8e8VPW9aqb2F6Y 0KLoxcsuFRVSgeBfiy5AgzoI1yxGK2o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699768; a=rsa-sha256; cv=none; b=v8d40Spz3DMxvn8jVfomZciQUhxCxr3rEuOU0yKGWdxzLye+n4Vza3PYFP8MCcPmj0C+9W eiM371sYxHJbOc+zyMgIwgbqtCrksBkjiOETniMGNFAH6AGvgoB8bFjAbzwHFS1rRYtM/c sck4YVVCaeLqifwmuZUwx2+/BpuS/oE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=akUpp2j5; spf=pass (imf29.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699768; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wmC0HbMgxf2wn+iz2jDhVN1M/dlanF/535x+xcS2yyw=; b=akUpp2j522F1OCy/HmXEqMcwJWIKBk0306uZlbgs1hHKI4lW+ktEg+MljnMGPXJ3OBz/HV y/V56cMacxfWisAwgP+Err59q2ewOIyADDofgyAuJRCj4kVN6J6pDk+GsgvHE07OmAGB1T +/fFMoGxQSi7sD+Ge9qLbQVmb4kzqo0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-668-0wmHc9idP7KPtoNc59ZDjA-1; Mon, 04 Dec 2023 09:22:47 -0500 X-MC-Unique: 0wmHc9idP7KPtoNc59ZDjA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8FEA085A5B5; Mon, 4 Dec 2023 14:22:45 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA6A22026D4C; Mon, 4 Dec 2023 14:22:43 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 30/39] mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:37 +0100 Message-ID: <20231204142146.91437-31-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Stat-Signature: o65ncnudwee1rf5spitrqfzojbkgber7 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BDF13120018 X-Rspam-User: X-HE-Tag: 1701699768-441948 X-HE-Meta: U2FsdGVkX1+tHz1oMlum93lf/U93n4IuO7JVZbD5GCVSvTaDB6WcBf70+OgQ1OIFLBZOhPFMntu5oAd3pLNByKr32w753D16zaBsvesmWOOJVcAHHpg/pidq3yTtB0x8EKIyoG2zPQfDerjI8F7O5cg6CogxZZPtjUe+c1rrmBRLiwGzzQxnOf0xFLClieFtFIijZ3TMBg9I+WNQ7WJlG+lViXgzLQlmqzmfm4fKCnl3Yf4oKA3xEBqd5nlELnoVpuFSL0dPvh0wiSzZ3wXxzxNVExK1p6m1mosQdqn3KWzWqlHSBXqcAWmi7olKl/7KYfMHHiL+ZAYvnu/9aZmvAvJhSaGDHsfyvLhIOO6qW+totQgVDyNU+fbMtWG5hFDjSrdBs7nw765FCIk4wcXvYanZ+gZh4k/KfxdqKn/pzSeDSyh4oGH8Ze6mR1+ktoe63yQsZIgq+ub/TxTZJaDdqbhspRa6WMkAzyv80lLPvvY6ypj0ocrRPoFFAYyvMmz6o+O6PNCuxxU174oa5bmukmTYZ0I8P6uk+ub2yvzUg+hasVhlhxFajnE0EpdONpwFcOFY6F3tOqSSqUMx9eMZjqFo54AYsAagnMSUlXZdSSAGsuAf/vaeG4ILVAeDjKIv7UljJpGqBIfXfT7KpvAMK7prVHOXUzzt0M5rY+vfcJk1pW7/+hp4QLPcoV5yHYj2IaYGFrNZfiUVQiTmGngbGInYah95Y40uJVGlxXbNabSJxQ2l5aCOMRMKo525M8zsLo4NtTpXv4gErU/ug4R+hnsVC1fAGHkfyD2NJnDSJIaYxqKByYvrBN/ZAd130CemGWz0rQxsvu/NpuoC1AdUnWG0oh/sn0si3kJs1b5O41GzJKiXdeJdXWBBIahQuHp3J4NpwQ7laMmK3qXqltbN47roLqFroQLJGU/X4aRk92oPKN7WPfJxZYCjJWqpfBZ6nu7uiL2xbcV4sh4e3Kr jfAzP9aX RzTdP050ZflTXS+39nmZg86jYM0kRdLg2X318jxysVd2kUs+G5IfoPh+KLVxAI+ble9Y9CBa3mTawnldebknhg0n7vw1h/vDkCrTTFqPfFr80NItTXdxUSncWJXh9nbx6SxVgViRuPpikGoYkkLnsBiQpcqNCOvP8349kz+2O08zuJKRz//9BJvs5Pgfd8aoGHl1WSo3K+PYQ7TvH6Op45bZ5yA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert try_to_unmap_one() and try_to_migrate_one(). Signed-off-by: David Hildenbrand --- mm/rmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 50b6909157ac1..4a0114d04ab48 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1598,7 +1598,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_unmap() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -1879,7 +1879,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -1947,7 +1947,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_migrate() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -2240,7 +2240,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2379,7 +2379,7 @@ static bool page_make_device_exclusive_one(struct folio *folio, * There is a reference on the page for the swap entry which has * been removed, so shouldn't take another. */ - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); } mmu_notifier_invalidate_range_end(&range); From patchwork Mon Dec 4 14:21:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B73C3C4167B for ; Mon, 4 Dec 2023 14:23:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6924E6B02CB; Mon, 4 Dec 2023 09:22:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 643526B02CC; Mon, 4 Dec 2023 09:22:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BBC86B02CD; Mon, 4 Dec 2023 09:22:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 398DF6B02CB for ; Mon, 4 Dec 2023 09:22:54 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 18FB1A0242 for ; Mon, 4 Dec 2023 14:22:54 +0000 (UTC) X-FDA: 81529352268.28.4637AED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 5265E100017 for ; Mon, 4 Dec 2023 14:22:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="ES/Ov3Lx"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hz7uaUsseOxwoTHd/q5qxZ6Zu8cM7Yp8tdIii7iwU6Q=; b=c2xR7sg0lgjy0WIfioLofEepSfc9HXJS/AH+JqXR3+56u1ZQRTDCGBMXMEYK5tm5sCaakw ZR83qJnlKUs2MQtSawAvY/PidQL7hjvdDeiivvavaH/2VyJ/6OE2oWfx+Er5qX5xYgthZv ulMpBy6Fb5rmMBVlO9C5ZUUfClIQepY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="ES/Ov3Lx"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699772; a=rsa-sha256; cv=none; b=zWjmxyJbA+zxilEMX4hnNVudc6RQVB2toHQSDB4GwohdwjpKlHWk8mTpljbTi9QKNuHGab ce8zt9l3pAB9kp9l7NwamXqtTxNmYu8Ng4kfvJVMu7nlvCxQ+8MvjX+FrlANCdnHwLdHE2 X241vPzEXH7CLce7KZNWTn8lFRd1gVc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699771; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hz7uaUsseOxwoTHd/q5qxZ6Zu8cM7Yp8tdIii7iwU6Q=; b=ES/Ov3Lxxfm6j78JXZlTQKv1vRjfegdqLWn1TcsHqibuRFVPlSv8DKz9Rw+73eQ2430Ff9 YLqMD9jyay+DambW3iEp+uyrK7hiuwBTEZniMTZMFgYUqy6jlzryWvcQpVZUngWFEKCE+1 2eJER4RmvooJfPkzonsalwineF6iD3M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-270-2PY8J7EQOv2OTifGixg-kQ-1; Mon, 04 Dec 2023 09:22:48 -0500 X-MC-Unique: 2PY8J7EQOv2OTifGixg-kQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9F0CD8314FB; Mon, 4 Dec 2023 14:22:47 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 01FC72026D68; Mon, 4 Dec 2023 14:22:45 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 31/39] Documentation: stop referring to page_remove_rmap() Date: Mon, 4 Dec 2023 15:21:38 +0100 Message-ID: <20231204142146.91437-32-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 5265E100017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qnp647uqumctthji8wuotuw3g1wtqgt8 X-HE-Tag: 1701699772-988790 X-HE-Meta: U2FsdGVkX19YHOEfxfveNayq9WvBELNsveRiZNjJGg6a+PC9kdcnB7w1KKvBp+WDd9vjxcwXCgKmSuklZUjBLGuDh/V1MWj6pgaGxX+SRggM5t9n42AT3Y7VBsQHBQhY9+QQMFB4QxpAC8DqbUokxZnAOxRR+EWZ0trtM6Oemd/hWdKuB4KtdE3icXKaWXZsbeJ704p4F8xzD2nQpgvrn362iXCos9jdwn6yFZWEwphlFwWaYV2OZqvq/KeAmMz+X5GSKvpSU2SqoVi8Ibq5/XHrmZk9AN2ysLSsPXKCY+SQfiRc2XGi1VAFNeSLhWk8HO0fMCyy6/pEr8KvZ2eG+QHDNH0RU+fTPP6fBqLMaAdPg5QLmMx9Kd0N/QDi1o2OVYGVH/LX6xi+Piwjm4tEo4UaQ/bHLFLNK7ng6B01nGqf9n17cGY/yH7UrUsbNnftaJ0buzcJRdn2/Ix+/XD6h+aICgztTBHpcP6m1181DQl2STRU/NNDGIg2Q0Qu58Yc7EJWUKrCZx8G3OeuoXfTUqHskI56ks9UX2lijVoQR17HILhYpQRrcW0Hk5ayCq1YI9vrDYMkQ3IXpJCJYclBZ1lKJ0NuGbbe+9q5c9/P4yAfo1BbLJ6L4Y3giVAiMPBEgYKme3D4q/Kl1mFZOELoxcnHlTWAzorCVxxWu3pegTQVSAsqD3B4iLZ+KWAILdbiZwXwuCizNME49Pt1yOKa+339qD2TBfckgBJW/ABAIwuKH5eIFaZY8V0tFHqd4vS7l8pQ2bidAUWbwoT7VmiOF1K83bHKAvrC0F9e5fFUqnOCfbzjmzztC9Lh15mLHyHnks9++DI53YJHHv51WO11yvNs1KiJcfOjET09kIi8hnscSru5EOJpEygH4dPvhFMZ1M4K+nqEKl9Iaj5qdt2Fi74ocWiMRoWMgbtQFEWzD01dR7eI8L16IEiQg5xwdNmHGx0b30zr85FcU3X4z3W vt/W/xab 6Bm5hGusEMI2pIjBl6zOa86BGnDSE2htgsbdzOoR7/43Dad3MoWdE5zbTNKppgaQQvo9cLcqyifp7XLiUrvtNkNR6eNgI24Q3biqJlNWlGnDV+k5nVqoJcP0QLDZKCxMST56DwAGJasQccITEIrcxyBZk6qs3dprOozMxQr0fMueOwLPN0maVlVirJzDfQtrR6RTZkBSZNz+IpT99AAKJ9Qqb/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refer to folio_remove_rmap_*() instaed. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- Documentation/mm/unevictable-lru.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index 9a607059ea11c..cf81272a6b8b6 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -156,7 +156,7 @@ Partial unmap and deferred_split_folio() Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use -in page_remove_rmap() and queue the THP for splitting if memory pressure +in folio_remove_rmap_*() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages. Splitting the page right away is not an option due to locking context in diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevictable-lru.rst index 67f1338440a50..b6a07a26b10d5 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -486,7 +486,7 @@ munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. Before the unevictable/mlock changes, mlocking did not mark the pages in any way, so unmapping them required no processing. -For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). @@ -511,7 +511,7 @@ userspace; truncation even unmaps and deletes any private anonymous pages which had been Copied-On-Write from the file pages now being truncated. Mlocked pages can be munlocked and deleted in this way: like with munmap(), -for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). From patchwork Mon Dec 4 14:21:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC98BC4167B for ; Mon, 4 Dec 2023 14:23:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3ACB86B02CE; Mon, 4 Dec 2023 09:22:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35C126B02CF; Mon, 4 Dec 2023 09:22:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 187D26B02D0; Mon, 4 Dec 2023 09:22:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 03B7D6B02CE for ; Mon, 4 Dec 2023 09:22:56 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CBD9514016A for ; Mon, 4 Dec 2023 14:22:55 +0000 (UTC) X-FDA: 81529352310.10.B35F0A3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id D9CC6160017 for ; Mon, 4 Dec 2023 14:22:53 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="N9/377Wt"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699773; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nxWJm4ToHrNF66jwBix2CbkD6YpwBU1ELf60FTyZZ+4=; b=kXAGDHUpZod9giezCywCygS+Ft05Z6C+tfPLXA/lbxOLFdafgL2eJGDC1jnLijYIGFebLm iwxgiFfpGeaaC2KxvqSwU+5h/safb3MPCZ+/b9Z+xVRuroPyXtOmA7Ywubifkrpjj4CSos lHaB4sT9N0iECdO9ZZIaw28JtnFFsuU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="N9/377Wt"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699773; a=rsa-sha256; cv=none; b=WCbGNdy98pQz6rR3Ma9d37WotXFS95i9BFwzYv2rgeBCiZECPu98L9xvotn+4yx/kEeF77 3+WTRyCd4giZnXyEo9zm4Sv3iVe93oeDJe4UKt1T63hUP+27N/vwJzRUeQY8WsyMxirWP9 FMDXnJ29SDUwmLsyb+8W0diHBB+i1PY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699773; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nxWJm4ToHrNF66jwBix2CbkD6YpwBU1ELf60FTyZZ+4=; b=N9/377Wtz6O7yFFkBQ3UGVtyN0BD9K/5Qtj8oROrbjKqFkEkD8ez0dhaa60ixp5VJA7yzb P9jkmwtys4vNPnNDySCbjdCqJm0N3+dsptomFCCUfhrHIjtqbIFcTcJLSGoCwPZ8OE4EAB e6MOO8pqVaKCwViIryYsMukjWK4xUgA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-T-_XZXA9MVC5F6_MoTvAnQ-1; Mon, 04 Dec 2023 09:22:50 -0500 X-MC-Unique: T-_XZXA9MVC5F6_MoTvAnQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7448B3C100C8; Mon, 4 Dec 2023 14:22:49 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6C602026D4C; Mon, 4 Dec 2023 14:22:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 32/39] mm/rmap: remove page_remove_rmap() Date: Mon, 4 Dec 2023 15:21:39 +0100 Message-ID: <20231204142146.91437-33-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: D9CC6160017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: nse3mrbip59im915rkrtjj9yubuisyuw X-HE-Tag: 1701699773-669368 X-HE-Meta: U2FsdGVkX188esZnu85vdDwC7qpKgflnGt2YhT+8Iw/z96fRQaz36JKyx8oCKYL9L+9W1Es8blMXe17bMSuBhrbvxIpXLL5CB+zJHX5mCTWTBX7RE3QJD7HXUz1L/6CQo+k8Zi2JTMwpJifTldgdyI6u060bkkvsHzCO3GsDh6IFXV7JP/+VkEWMtaVS94yk5UwtiI2lUcH07BMOLsKZsrhGROYOkCyqC92hl2jGdr7mLd596YKAfY6ZBR0MIXXdchQp8xxcg525F7yj/MyS0Sf+Wdh84KPSl6BywjFrcKIJ0BcW8/7D1Ovh2xjQf6KFnGQVDkH5wrIhUummh6sG+X72LaEw5+SP5w45Ta1ZSfLPvkzQoDPThVEbMbrN4IYZ3eyphRdnrSohtC7Q96GfSejDIQeLEIbRtXta0RD6JzPguSWCrP4+zyKAIwAk4a1HYuzBeHH1OtvjWoOsDhCVntv0rODmRaZNxkqv7pYtvZsqlA85W0747u0TxuYK093/+EDLypPD09niuj7JU/Pp+GYlV3yC+o4/lOdOb+wo/izqeMpU9rhwymu1NsPGrM2EiPwlo6mJBKLy1TONNgsbHmmsj+dNlYPAxWKX2yVKnXNHAuBxIrCVIvJ2XERQ5FAfSPG8A0lUF1hdYr7Ar3TwagK/fGk4JzrBoGfx4x6I1GB9ieBDw9vUVJXHHPBQcDZvzAjf1yz/VDXwxjklOiFl85dWwaGtnk9kvmV2k43HoOH0/dTd8GTrvDr42C2h4H6LFH2elZNiPycu47VztQYF8Hg6+2sT7UQBgmtVrR9jqqyDvZFPI07+sfPcWxUr1wkF0NMI2k5OJ2S/KiLH0UuNfQ8f50sq/KUy26IMmQZm1tQ4hyggDCYjhoRL0I6IIbKrEWfibMZ10YnkPsX0//c0uG4H3K3ywYTrd9ZufBGwB6x8eZ8Lpq7Q+eo5aTlpe3HolBuURMMZBo3lXIkXsZz St8bVDNl rwHWswV94yCu2N7b/b1AWVEAKOf0019YjizmT9LgF6PUrThSkUR+bk1wCg19iRo+hX8B6rVT58FoUFGA+oOUmj53QZ1N/S3BGThcXykj8zKwGimjxXF+Vxns6cD1ukR8lYAt4eiD/NmBjpg9sa0rtLLs4KMYcFqb6aTyUUie/IecEBtdNYuvckniNAjPDaxQ2oL6IdNa4eZbhEAcn4ohKsYiNr89I0rXElYTJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers are gone, let's remove it and some leftover traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 +--- mm/internal.h | 2 +- mm/memory-failure.c | 4 ++-- mm/rmap.c | 23 ++--------------------- 4 files changed, 6 insertions(+), 27 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index dd4ffb1d8ae04..bdbfb11638e4f 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -239,8 +239,6 @@ void folio_add_file_rmap_ptes(struct folio *, struct page *, unsigned int nr, folio_add_file_rmap_ptes(folio, page, 1, vma) void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); -void page_remove_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_remove_rmap_ptes(struct folio *, struct page *, unsigned int nr, struct vm_area_struct *); #define folio_remove_rmap_pte(folio, page, vma) \ @@ -384,7 +382,7 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * * This is similar to page_try_dup_anon_rmap(), however, not used during fork() * to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via page_remove_rmap(). + * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device * private pages cannot get pinned and consequently this function cannot fail. diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5f..43dca750c5afc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -651,7 +651,7 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma) * under page table lock for the pte/pmd being added or removed. * * mlock is usually called at the end of page_add_*_rmap(), munlock at - * the end of page_remove_rmap(); but new anon folios are managed by + * the end of folio_remove_rmap_*(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). */ void mlock_folio(struct folio *folio); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 660c21859118e..d0251cba84795 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2317,8 +2317,8 @@ int memory_failure(unsigned long pfn, int flags) * We use page flags to determine what action should be taken, but * the flags can be modified by the error containment action. One * example is an mlocked page, where PG_mlocked is cleared by - * page_remove_rmap() in try_to_unmap_one(). So to determine page status - * correctly, we save a copy of the page flags at this time. + * folio_remove_rmap_*() in try_to_unmap_one(). So to determine page + * status correctly, we save a copy of the page flags at this time. */ page_flags = p->flags; diff --git a/mm/rmap.c b/mm/rmap.c index 4a0114d04ab48..8e86024953c03 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -470,7 +470,7 @@ void __init anon_vma_init(void) /* * Getting a lock on a stable anon_vma from a page off the LRU is tricky! * - * Since there is no serialization what so ever against page_remove_rmap() + * Since there is no serialization what so ever against folio_remove_rmap_*() * the best this function can do is return a refcount increased anon_vma * that might have been relevant to this page. * @@ -487,7 +487,7 @@ void __init anon_vma_init(void) * [ something equivalent to page_mapped_in_vma() ]. * * Since anon_vma's slab is SLAB_TYPESAFE_BY_RCU and we know from - * page_remove_rmap() that the anon_vma pointer from page->mapping is valid + * folio_remove_rmap_*() that the anon_vma pointer from page->mapping is valid * if there is a mapcount, we can dereference the anon_vma after observing * those. */ @@ -1451,25 +1451,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from - * @vma: the vm area from which the mapping is removed - * @compound: uncharge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - folio_remove_rmap_pte(folio, page, vma); - else - folio_remove_rmap_pmd(folio, page, vma); -} - static __always_inline void __folio_remove_rmap(struct folio *folio, struct page *page, unsigned int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) From patchwork Mon Dec 4 14:21:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18484C4167B for ; Mon, 4 Dec 2023 14:23:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D3B86B02E3; Mon, 4 Dec 2023 09:23:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 384E86B02E4; Mon, 4 Dec 2023 09:23:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B2276B02E6; Mon, 4 Dec 2023 09:23:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EA3A66B02E3 for ; Mon, 4 Dec 2023 09:23:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BE0E51201E2 for ; Mon, 4 Dec 2023 14:23:37 +0000 (UTC) X-FDA: 81529354074.30.7A6813B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id CF73A1C0025 for ; Mon, 4 Dec 2023 14:23:35 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TwbH7afj; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fiJf6xaTCn9hTJUSpG0inAWeRQ8SaM/ZLU1i30jXpOI=; b=Noqtlffydx7zPp/iZHA0PjpAPy8PlLqPHSImg8QuRROm/z25wjcWbjtV2QxQFdj3byxZoC xXrJkBu0m1VVAi1u4RdZzWSLZCuz5m/xzPSknczTurZVKN1SjRsjjYSBv73jLQ4c/9tXxT fHAtK5G9fOG9cIZsMDP4hwpU/IYlV98= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699815; a=rsa-sha256; cv=none; b=m7PBWrDzsBUDWAAihfnNU4WACR3aU/gMGi8CvYUX1KQ2ByfflUTK7f7kI1nAthedaGcwOk tdi4OZOL+yBny+puvxUPm3usCg3wL0/jS/pQtdnAftXxO3ikhx73y7L93a11V5mhw+zJ8M C6JEVyDQunA/SGmxqSSaDVDXZ++scBk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TwbH7afj; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699815; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fiJf6xaTCn9hTJUSpG0inAWeRQ8SaM/ZLU1i30jXpOI=; b=TwbH7afjQKQEnyqLEGtIb9t5bXh3U5QEDCJWgVHdQtBaeZebGIrAxoQTCzZQqmeWYyDfXQ EBTqGPsfxOjKnwLSxLUp/2FxdxnNGxB+5Xgs47yzYBJDiiloKy4mWUY2JVoEHH6KoYrcgl rX+gF2/UnIhaVoE2qkza9sX14fhE3RI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-164-gWsSkqhfMSOvilQ5o3bMzQ-1; Mon, 04 Dec 2023 09:23:25 -0500 X-MC-Unique: gWsSkqhfMSOvilQ5o3bMzQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 53AF1811E82; Mon, 4 Dec 2023 14:22:51 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id AD95F2026D4C; Mon, 4 Dec 2023 14:22:49 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 33/39] mm/rmap: convert page_dup_file_rmap() to folio_dup_file_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:40 +0100 Message-ID: <20231204142146.91437-34-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: CF73A1C0025 X-Rspam-User: X-Stat-Signature: 9y3y6wrfq9cuoond1npjch4abjd3j6ub X-Rspamd-Server: rspam03 X-HE-Tag: 1701699815-554988 X-HE-Meta: U2FsdGVkX18gq1fazmpehLDkfeB2zIXPfdy+YRuhKw7Y9HZtwusCJG3dFObT1BbXQY5Pbd5Cgc9lpxgYgoDjO88gK1pGKbefwIX1ytFNQFzfDLg06UcjPB4jaogGHPQsq5ZUZU72o+fzTTaedtMVGiOK72sS6r1PaBN4d/dYJRqSyB4OMekHQ2JdJbUwtlbG6UHvlZLpIIlbQb+ShUV2f8sShuTe8c89TCy8upNuCc+tks8ryngSZkWiiW+ALz/E6v4EY0uc4GoA3Fh8sxrubZkrZlva9pn2ofMq5yM3ywCgWxwKkI+zy1AeNsJTiIxsOnU0vZbhjb8aUOP3cBZ2XZudqZeJM9dS4+sEuzfck2/myd2WyEAJThELRCrLa60EW0jtflnO4xvVJEU8EK6Tcn+EVEUWxZ8ayoWlu9yaIIPwyC3SF03yoTsHvZjtfQI0VNUFwejwLtygbG/N2rGoEx1IeMlYpdTtS6XpQW9UCyxbPCRR2yLQoKkIch321hO6QUc5SDLTrYTWJX74YtIcDvGMjJEK6Bhslyy8n0Ikh0GL7EAZhOqlbbjjm558zps6oopqtZeKJdETqDasd1MQU/mUSVLoFaa9TkkfI2Bsq5d/faLBKS/GvwLqVwIMsn0oz6eSyshytCiB4++WldKehst0oRXs+5lm6hZMtK6cFCrpVmF/MZOYah2QHQgk3dBLrwaSLRUFiU+rFu6fyLNDnedBhMVOTrAC4DJcZYQ1nouC/Qi2/7QaNx0pJsEfOS0gJY7lveHvQtg+MllIZbF7VtZGU5wRy/hq8GA4DXjplncsBZrPe1vnmRRRo+ofWQKXnlmErJ3gK03wBPAzG86f7EAOLsq0sTXm/uVlifGERysuL7SO8LlabHAi1mycVmRVkPBDBOmVu01JQlo5wvKwDqOBILvtDFHLXxCDIV7OLVJeb7puLMoHAeAsZBC1UxjLgaTgI63qHO5NEhHuWbC oqOGWKLW 6R6b49hqqi4u9Pv8eXZ4tO2QOHkau1/pgY72W9GEIeQlS8EQwm+294YypgPBvlVVvKmntP98PaIkStLZPc2N64ChxO0wq0XAbZV4OYg9QopQAH/TdCdVyExhx5j1yUYj5dH1zwPWE2H+QFZe6/z1RzHhe+F+9dgAcRSRqQB4nskYH3p2qyo2Y+CF/ABiP6gJUR9gb1r6HjWyWzy+2GfgpF01k7D8XPnJjS+PD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert page_dup_file_rmap() like the other rmap functions. As there is only a single caller, convert that single caller right away and remove page_dup_file_rmap(). Add folio_dup_file_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 56 ++++++++++++++++++++++++++++++++++++++++---- mm/memory.c | 2 +- 2 files changed, 52 insertions(+), 6 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bdbfb11638e4f..21d72cc602adc 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -303,6 +303,57 @@ static inline void hugetlb_remove_rmap(struct folio *folio) atomic_dec(&folio->_entire_mapcount); } +static inline void __folio_dup_rmap(struct folio *folio, struct page *page, + unsigned int nr_pages, enum rmap_mode mode) +{ + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + if (likely(mode == RMAP_MODE_PTE)) { + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + } else if (mode == RMAP_MODE_PMD) { + atomic_inc(&folio->_entire_mapcount); + } +} + +/** + * folio_dup_file_rmap_ptes - duplicate PTE mappings of a page range of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_ptes(struct folio *folio, + struct page *page, unsigned int nr_pages) +{ + __folio_dup_rmap(folio, page, nr_pages, RMAP_MODE_PTE); +} +#define folio_dup_file_rmap_pte(folio, page) \ + folio_dup_file_rmap_ptes(folio, page, 1) + +/** + * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_dup_rmap(folio, page, HPAGE_PMD_NR, RMAP_MODE_PTE); +#else + WARN_ON_ONCE(true); +#endif +} + static inline void __page_dup_rmap(struct page *page, bool compound) { VM_WARN_ON(folio_test_hugetlb(page_folio(page))); @@ -317,11 +368,6 @@ static inline void __page_dup_rmap(struct page *page, bool compound) } } -static inline void page_dup_file_rmap(struct page *page, bool compound) -{ - __page_dup_rmap(page, compound); -} - /** * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped * anonymous page diff --git a/mm/memory.c b/mm/memory.c index 8c4f98bb617fa..eaab6a2e14eba 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -961,7 +961,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, rss[MM_ANONPAGES]++; } else if (page) { folio_get(folio); - page_dup_file_rmap(page, false); + folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(page)]++; } From patchwork Mon Dec 4 14:21:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7304BC4167B for ; Mon, 4 Dec 2023 14:23:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AC0A6B02D3; Mon, 4 Dec 2023 09:23:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 337496B02D4; Mon, 4 Dec 2023 09:23:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13A536B02D5; Mon, 4 Dec 2023 09:23:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F15C76B02D3 for ; Mon, 4 Dec 2023 09:23:04 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C977216027C for ; Mon, 4 Dec 2023 14:23:04 +0000 (UTC) X-FDA: 81529352688.28.FEE5BC7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 2B4A0120024 for ; Mon, 4 Dec 2023 14:23:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DPOP+wUP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8XTtIQTcL5nOGoGd5kvk/Mh9+sDV+bVrih5Rph4+Al0=; b=m6e7o63pz5htyqumsNUNVCtS5i6OzW/Ok0xsC4Ayz8S1Nn1BSzyl5iXM/Iql+S+KUrINYx 0sm7QlAVCLNWs59iHdsAmm8cN8nZfRWkeNb9Yh8IDxW5ozIz5IjpX5B7TwCRzNLESEs4J0 THjU24zL0pxEmqt1jMqtH9WgMmceEaU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DPOP+wUP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699783; a=rsa-sha256; cv=none; b=u1Qi2ee6L1PUFH15krkSyPS5G4YA6Agz9JqsoaT7Eg5BfkRjNjN/oO7LqwWwA7s+t8NU9t X87pu6m/Dbd6NDyVx5eMBdCEgxrSCDzY1CvJ/TF6h/htTqUnnwIUzp1ON/axqb+behOkMI 1oloLScx9oTrF9FdoFVUC0nxLd+3AY0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699782; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8XTtIQTcL5nOGoGd5kvk/Mh9+sDV+bVrih5Rph4+Al0=; b=DPOP+wUPJvPB2XzAesIEGHGQLLceYdLdqdabiNlg5HDQ/ZCA+2GmtLcVK6g7D7sAH+NitT GuJ0rBmo5K3rX9mGqprde7nxBWIoH3HNplpmpfMi+iJypMfpkGx4igmadyP/+pY4CrFiB9 060hkhxL+iClV5/OECmmIZaHaqzbF+Q= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-353-hT6yNEVjPHOWv0XESwmOZA-1; Mon, 04 Dec 2023 09:22:53 -0500 X-MC-Unique: hT6yNEVjPHOWv0XESwmOZA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B9A53813BCC; Mon, 4 Dec 2023 14:22:53 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8CD092026D4C; Mon, 4 Dec 2023 14:22:51 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 34/39] mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:41 +0100 Message-ID: <20231204142146.91437-35-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 2B4A0120024 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: k56ftiq1fu57qr99aqjb7mzpr399aq7z X-HE-Tag: 1701699782-204562 X-HE-Meta: U2FsdGVkX1+/s5lXtlHMxyKEozXAHLZFgUOCoI8WKx19hET70CM9MRvoNANEIFI4vSWF4Lr9MekK7MrPemB0eM3A4WJa2tyAqJgAXXFwMNb0Mgwjm6ZgiFebx/16vm4yScnenaeE5zPqsMv/QuhyLF1YHSmC5LgX1z6OnOzAuwZydT0A9uDNMA4lUVCyCMc3PT50g1dH2W6/PFMx9Afa933FpmV6HhFRuycGv2g7bJN1+qB4Z0/v044ydQScnIunJY9quhM7lLGkcMOBSpiEOZNOQltO3OLFA3Tf1jJanBnzvmcqvyVGUlQVSeogYu1jNqomrpEgl/5C7H6p+4LGwPnSsHFVIYgbfY5Roy2CWoMh+TkE2MzEPtUw8kbWcv5aO3ghrT+jadkKYx7ujxdpxo+wZkaPy7UlFTcePcMq00aVsuSvK4wXFL1kEN5uc6bBzKhzp2d6B5WG+c3xqqKmEb9OrSqqjSvCYDJLp+2FLYGAtrBZG0S3hov1jPL0sRBqzYwQ1b69IRz4BUcUk8Y881rfmQ5pOZFmA8RMe6/5LYHzPzR4Ze2NyGTlFQNWF4W4OyrymPHvlPIjHMAxBOFbod9oj0Cvt6Ecu0/rLn55/zo9BN9Ou64Mi24CuktKMy6LcXE1fQHxOsmbBuetWTpZNt8x+c9jwjJQoxqHUmrSz84VF8RCyGylvK/MPJu/gf892rljPh+FoJGuFQ3ruNcaEE/o109TJipDfAEN2AHy5QMG/st0zJOWHb7SmlymlVlsliJ2OXtho5xVOVuCjZAp6H0nEqgzKZojHmgk7qgwo8BPDvCPuozRP1QrdkxCZKRcmQRMR9mFScZ8vpWXRQmz9F5roc+fdJbNkDOafX/EyhWO/vQTeIsOb9L3TdPyMOUzF9ZVPVl2btKaiEwVFo6NKbi/dwiJcTewtEv73B4dUlcQKlN8G9drbBTKE6m0RPxpTD3YJ7WqVsJdb8YWmIO SOLGrFVf Yh/f0OKFKv/WFOfpOhXrBUGy5HZJS/iO4A0tb/g7lc6+luQ0AhOsZBjb0Ugy5uoab9PVbKmarTrJ263Q+OjiFgQEtAf9qur/4QxM4X3nWaILccmuhTy2IJCVntC3fufEWcNsCkeS5C1lZg6i9ELGDdycpuTqWFIzCG9SMcj3FmCtqr4rie3V0ka5QrgEs5L1lATyOqtb+drnW2Ec/zk2ry4uqAytjNW3R7mgW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The last user of page_needs_cow_for_dma() and __page_dup_rmap() are gone, remove them. Add folio_try_dup_anon_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand Signed-off-by: David Hildenbrand --- include/linux/mm.h | 6 -- include/linux/rmap.h | 145 +++++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 51 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 24c1c7c5a99c0..f7565b35ae931 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1964,12 +1964,6 @@ static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, return folio_maybe_dma_pinned(folio); } -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) -{ - return folio_needs_cow_for_dma(vma, page_folio(page)); -} - /** * is_zero_page - Query if a page is a zero page * @page: The page to query diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 21d72cc602adc..84439f7720c62 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -354,68 +354,123 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio, #endif } -static inline void __page_dup_rmap(struct page *page, bool compound) +static inline int __folio_try_dup_anon_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *src_vma, enum rmap_mode mode) { - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + int i; - if (compound) { - struct folio *folio = (struct folio *)page; + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); - atomic_inc(&folio->_entire_mapcount); - } else { - atomic_inc(&page->_mapcount); + /* + * No need to check+clear for already shared PTEs/PMDs of the folio. + * This includes PTE mappings of (order-0) KSM folios. + */ + if (likely(mode == RMAP_MODE_PTE)) { + for (i = 0; i < nr_pages; i++) { + if (PageAnonExclusive(page + i)) + goto clear; + } + } else if (mode == RMAP_MODE_PMD) { + if (PageAnonExclusive(page)) + goto clear; } + goto dup; + +clear: + /* + * If this folio may have been pinned by the parent process, + * don't allow to duplicate the mappings but instead require to e.g., + * copy the subpage immediately for the child so that we'll always + * guarantee the pinned folio won't be randomly replaced in the + * future on write faults. + */ + if (likely(!folio_is_device_private(folio) && + unlikely(folio_needs_cow_for_dma(src_vma, folio)))) + return -EBUSY; + + if (likely(mode == RMAP_MODE_PTE)) { + for (i = 0; i < nr_pages; i++) + ClearPageAnonExclusive(page + i); + } else if (mode == RMAP_MODE_PMD) { + ClearPageAnonExclusive(page); + } + +dup: + __folio_dup_rmap(folio, page, nr_pages, mode); + return 0; } /** - * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped - * anonymous page - * @page: the page to duplicate the mapping for - * @compound: the page is mapped as compound or as a small page - * @vma: the source vma + * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range + * of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * @src_vma: The vm area from which the mappings are duplicated * - * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq. + * The page range of the folio is defined by [page, page + nr_pages) * - * Duplicating the mapping can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. + * + * Duplicating the mappings can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail. + * + * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in + * the parent and the child. They must *not* be writable after this call. + * + * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. + */ +static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *src_vma) +{ + return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, + RMAP_MODE_PTE); +} +#define folio_try_dup_anon_rmap_pte(folio, page, vma) \ + folio_try_dup_anon_rmap_ptes(folio, page, 1, vma) + +/** + * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range + * of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * @src_vma: The vm area from which the mapping is duplicated + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. * - * If duplicating the mapping succeeds, the page has to be mapped R/O into - * the parent and the child. It must *not* get mapped writable after this call. + * Duplicating the mapping can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail. + * + * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in + * the parent and the child. They must *not* be writable after this call. * * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ +static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, + struct page *page, struct vm_area_struct *src_vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + static inline int page_try_dup_anon_rmap(struct page *page, bool compound, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(!PageAnon(page), page); - - /* - * No need to check+clear for already shared pages, including KSM - * pages. - */ - if (!PageAnonExclusive(page)) - goto dup; - - /* - * If this page may have been pinned by the parent process, - * don't allow to duplicate the mapping but instead require to e.g., - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future on write faults. - */ - if (likely(!is_device_private_page(page) && - unlikely(page_needs_cow_for_dma(vma, page)))) - return -EBUSY; + struct folio *folio = page_folio(page); - ClearPageAnonExclusive(page); - /* - * It's okay to share the anon page between both processes, mapping - * the page R/O into both processes. - */ -dup: - __page_dup_rmap(page, compound); - return 0; + if (likely(!compound)) + return folio_try_dup_anon_rmap_pte(folio, page, vma); + return folio_try_dup_anon_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 4 14:21:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 728DDC4167B for ; Mon, 4 Dec 2023 14:23:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A1386B02D2; Mon, 4 Dec 2023 09:23:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 028356B02D3; Mon, 4 Dec 2023 09:23:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE5326B02D4; Mon, 4 Dec 2023 09:23:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C988C6B02D2 for ; Mon, 4 Dec 2023 09:23:01 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A87A7802AA for ; Mon, 4 Dec 2023 14:23:01 +0000 (UTC) X-FDA: 81529352562.08.E64F839 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id AA3DC80017 for ; Mon, 4 Dec 2023 14:22:59 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cYJIXjrH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699779; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QtlsU4Oj5pegnlPsv+IBcEoRKm4An7qodS6gdDA58Hg=; b=IrTmm9hTseVRgc2B/TguCorU2Edy1SPfLKEDBMksAuUnBRul9gj8ADVJYCiBdjPjaYLoR1 20XpAtHVFrW7eumKXhYQ1eVRddTHY/14A4ApzlVnmBhxO4UzhboRfAOp5Ur2JSswMj1gtx W9sCq7+LenhzMM1yE9OJR4MnPRIw8kY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cYJIXjrH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699779; a=rsa-sha256; cv=none; b=xOLMXb2/Fv9GD7Ax2/cP3nuCIUSA8ldefQJbaSg9egr2Kr2wUAL9I4sOdKemxObvc6gWO9 PvJV810THB81DPlSUd7OMcNkeEkpGUGvx3Rqj6r0ZKNqWm2qCJj1BM/Hvng8arG4ReknSe +2FPBMuAy5idx9Icghw1b2AUnHBmgTk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QtlsU4Oj5pegnlPsv+IBcEoRKm4An7qodS6gdDA58Hg=; b=cYJIXjrHSW88QPY+VPCYkDv3yAVyyv/HZGn3gWiR+XXJyfc8tOwoUES9YlsWzZEGtIY7mp XvN/g00743Uomd5q+rWiLOAeXkyE+tPMvpBYRkbUWT6wboP5G8h4ubYTYEH8z4bNzNj3g0 smyVAEhycuY19q5d8sD1vmVPzhgheAw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-451-GtUO8C-8PWedYvC2iAxb4A-1; Mon, 04 Dec 2023 09:22:55 -0500 X-MC-Unique: GtUO8C-8PWedYvC2iAxb4A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 483868314FC; Mon, 4 Dec 2023 14:22:55 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6440D2026D4C; Mon, 4 Dec 2023 14:22:53 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 35/39] mm/huge_memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:42 +0100 Message-ID: <20231204142146.91437-36-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: AA3DC80017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: d56j3nuqdgrshg18ibrxqq4rdkpr9k9w X-HE-Tag: 1701699779-831233 X-HE-Meta: U2FsdGVkX19QM+dCd+nriEl2/o4YIcejreW9v01KHB3XEZlqiIS2iQC00siPGvtyOnH3OLgpum3NR9rfjQwPqOh5tZNEUEblCxO1JaPxuW3/ZZM0X8otiyZ3/ul8ixlPo4lomvQY/LZ9nk823T5UdZzKKbiy7e8InJ/7jH8V+IaEIVFNSAm85Aa/J+nsqx3ctYX09gmdPYHs3Hp03Vxvllzy+dw5FYMyLg8HmF/s06OkfYMiQVoIqnSgGYQKwzcOToPPjbj8p85kax4xVKGn5gsWS04+eBEmm8p8aoveLpLmMy7zyQoXWcGD36wUxDplDlXnc3IC9ivZTFXqwwFSLZeG9DXoUWswP1iHsVjHfrpm7sVuLhAzmYfrKghhCcY2fp42VEE+A64On3QPWQgsWHf2b0K411FP5x08SSI7peXuijiUHJbc1uv4iR7SCo5KlpNVFTMn3QpPEK2jFNpyqOMbfhlbEaSNrc6W9piGbppwwF+AnG4W/LMsIvwdgXJILwq1s7q78wbw+u1KYOyZeVPQyA6PPwaGMYvDCp1xClhiIX8peVrfW88YizJaV4HFqz1zonKf5ee8sq/kPe2EcvNz5vmbXUHE4SpQyJvQRlUOU1Sj8RRDDzjNrbBn/h0mRez7hOBTUxMv8SK6VH4kPLM/AwiZIN0yAjYaSDUV53LCS+N1S0/Hz3AmSIDKA/0xxULNzHtRYJoRXuES42qT19GdXYtyGKwABs7ipfVMpL0qhCv57vapy8XQZrJQ0ye02WE0DHVH49a+ooDveNoRMHAVysjM9hBmgQ5dUbLIK2c9e4x3M6kyec7C1w9gkFwM+aXRo6+IQQglcUB3cU2d25f5bxIqSkCtJaIy/z6mgm94kfXgVEtOZ8lE08COAe+TfcElxMZ3OnK5To3AvLd61INuO5vrf7VgfFVGL0FwoP0ALN6IB988WAeHDAyPD5v4AWkr4a8WpXguyuOav4g vq7pfTiq 9KCpBwsam+x1boGrPJt1hRlh5TkT4wtDGG/kaDr3DvrFFMfEx//4HkQkhWOXFkM6dW1ICHbn4VOmP9r1Fm3lhm1zcFN45DqhLPhaeyvSfiGH/H+oxDf8M5hFTi40rlsrMD7HFtHC5e1Nsq3wAyd4uvRzHxcCL0CxKi97VjlkPK6whsHu+y8RWGBkqscXuvAcJNXS1I4/+u9g0BdibA43PmJ/Brg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert copy_huge_pmd() and fixup the comment in copy_huge_pud(). While at it, perform more folio conversion in copy_huge_pmd(). Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9376c28b0ad29..138e1e62790be 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1098,6 +1098,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, { spinlock_t *dst_ptl, *src_ptl; struct page *src_page; + struct folio *src_folio; pmd_t pmd; pgtable_t pgtable = NULL; int ret = -ENOMEM; @@ -1164,11 +1165,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_page = pmd_page(pmd); VM_BUG_ON_PAGE(!PageHead(src_page), src_page); + src_folio = page_folio(src_page); - get_page(src_page); - if (unlikely(page_try_dup_anon_rmap(src_page, true, src_vma))) { + folio_get(src_folio); + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ - put_page(src_page); + folio_put(src_folio); pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); @@ -1277,8 +1279,8 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * TODO: once we support anonymous pages, use page_try_dup_anon_rmap() - * and split if duplicating fails. + * TODO: once we support anonymous pages, use + * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_mkold(pud_wrprotect(pud)); From patchwork Mon Dec 4 14:21:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3507C4167B for ; Mon, 4 Dec 2023 14:23:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C85B6B02D4; Mon, 4 Dec 2023 09:23:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 92C236B02D5; Mon, 4 Dec 2023 09:23:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A1BF6B02D6; Mon, 4 Dec 2023 09:23:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 67B366B02D4 for ; Mon, 4 Dec 2023 09:23:05 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3984F16027C for ; Mon, 4 Dec 2023 14:23:05 +0000 (UTC) X-FDA: 81529352730.27.A4B93DD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 8A22214000C for ; Mon, 4 Dec 2023 14:23:03 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EONB3l3o; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eN0JAnAR2NtaQAP/A7ZBLwE/xmH/RpNbwyJWy6g296A=; b=K99/wGpRE4zT6tI/rAgKSqmZCVOGQWaBsc5G6RyUNFeljX5k64A/ROTE8yjI1AJudEKsL0 18lXEwJ1ISP6oMmW4P0x6oo+QFGBXc0H1YXqzl6PQFtIeNJ2BzNanZagWXj8c0+rk/4dXl AODIfDvwsrbVTLKvMjWWq8U12jyGgkk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EONB3l3o; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699783; a=rsa-sha256; cv=none; b=qNOpLAaWoOspHzndNWyKHonA9OVLojfU+ld/OVj/oRYVHyYaJUzECnyOHEfblqs16LHkpg /vOzwHsf02na/EzzRHm0Ty+q6Kx2d2DePtF8No63Eoz9ZzHqLLmEF3cq4TtRxa3yz5yqtq sdhA499KvLoWM74PA5TJHgba0Ven1Dw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699782; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eN0JAnAR2NtaQAP/A7ZBLwE/xmH/RpNbwyJWy6g296A=; b=EONB3l3ojxov4Ve3dgw7ma24nawN1SGb46mepEH/CdviEWJr7ixqypo1uFjfj7fx1sjLim PZqfecHzlD/ly68F/8EPNFazlaWZ0IZCc5k5MYpD/J9nLrl0Wg7QUusOD4a4taRRnwwE+T spSYrzsmQ9/WVy7k5vuE04yjyS6wroc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-340-G29icLgwOhGGlLJZvf5FDw-1; Mon, 04 Dec 2023 09:22:58 -0500 X-MC-Unique: G29icLgwOhGGlLJZvf5FDw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2DD81101AA6F; Mon, 4 Dec 2023 14:22:57 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EE922026D4C; Mon, 4 Dec 2023 14:22:55 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 36/39] mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:43 +0100 Message-ID: <20231204142146.91437-37-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 8A22214000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: za8wj56fqfepk4xxcpjfq1xxd5zk8d1f X-HE-Tag: 1701699783-527154 X-HE-Meta: U2FsdGVkX1+qA8ikfdJV/42iCtA7jckfx+8VTxfXEmIM4qOa6BnO7sRhqq795uLgpoPr4DRQFJwm6CN2zOVbkxMITXov8v/0wc6VJ0nB7PtRQGMTpL4xTO8QePPuh1qWo3+zuATeO0cwijhAI/PS7EQmULlRZ/lYjp4xOLzUfJnFJj1TWFbPcfaSJB/fJtrzaBnxEyR8o+n203eJLdzly+eRvltlTyBdfVvL31TGHqV0nOXBL8eAc5wTAYZLIm0PrazfN4oUiAxmG0M5ee/QbLYNUoLCTViVIXWAQhI2GGMpgi2JNxLgzvWyhMeE5H4vQYME91J6t+UTGaz87ueAuz6N60OEBkunj+cAuKoznctWVpQZL+IG64kI9arZG7oTCsRglwHufxoO3kGsvGOQj0K7YDSRFrupiOMOgRxSkr1NnZCh9aPp/U+rLnjRz4xa/WR1C9dX6KwQgZsyo7z6reGpwvAKYjZo0El+hdmFj33tleadAlcnllC7fFklXIN8ruIIeyeSQ3hfUqcrLLDyWyViPCpO0TCGXu/ztDx0jM5+d5j1Ok0FLl1kVMo6QP/bxpJIW+rcNB/2x6Cda1RVpBWFUA0lyafTCtIdQflZgoxCuTpKWdy+cqlp3Iae9BpneDr8iymSXDm2B9Vnaboa32XQASnMZdlWhJ3roEaDCujCY9RZ/XQnuVM63VV00WGrtyqP8MFligojucHky8ycg2AmHTkDL5LpVCEQgkx6O9WCaYo0TBzOAUpB4uEo6QFL3i29+CGeP5f0TS+eOBjCPyOgzR+jFVZ3SuEFu6FgVumLma8YRpxxwoVJwHfbuZanpUiV+iy6JsLzDIrVKv1mF1X7P2JJmaSMgiZS20h5I++xQXMSFrftz23cUcmDQeR9i81AU/x5hDkLFHMvCLaMHkhs/7LLsZEGUdl/VJ1VwFmcKFUwa92C/0x9IijI0bqYlotcW7/qtbJCn8VckSw W1VvWZ9w 1BY7Xp9prpenq7YaL4ZV4AOd84c8KlRx6Ldl2YIScHI052INjJh56NGo+B5jJm8Py8P+u5J7JpoF5rBNWw5oGix6H5O69nLkZgAG+0Sqa0tlbZB5q2DTbAqLhtwZnzoIY6ooU8ZDljcyIYJ4xxriFfgDu9653vk/40ix6WxB7Xj9znK57ZspUa8eEqb5ShaPAd+r2ZU+gkqrfX6djgffSPOXS7A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert copy_nonpresent_pte(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index eaab6a2e14eba..ad6da8168e461 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -781,6 +781,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); pte_t pte = orig_pte; + struct folio *folio; struct page *page; swp_entry_t entry = pte_to_swp_entry(orig_pte); @@ -825,6 +826,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } } else if (is_device_private_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); /* * Update rss count even for unaddressable pages, as @@ -835,10 +837,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * for unaddressable pages, at some point. But for now * keep things as they are. */ - get_page(page); + folio_get(folio); rss[mm_counter(page)]++; /* Cannot fail as these pages cannot get pinned. */ - BUG_ON(page_try_dup_anon_rmap(page, false, src_vma)); + folio_try_dup_anon_rmap_pte(folio, page, src_vma); /* * We do not preserve soft-dirty information, because so @@ -952,7 +954,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * future. */ folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, From patchwork Mon Dec 4 14:21:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31FDDC10DCE for ; Mon, 4 Dec 2023 14:23:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 953BE6B02D6; Mon, 4 Dec 2023 09:23:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 900DA6B02D8; Mon, 4 Dec 2023 09:23:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77CDD6B02D9; Mon, 4 Dec 2023 09:23:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 63DC96B02D6 for ; Mon, 4 Dec 2023 09:23:08 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 40550120119 for ; Mon, 4 Dec 2023 14:23:08 +0000 (UTC) X-FDA: 81529352856.22.8123244 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 8140FA001A for ; Mon, 4 Dec 2023 14:23:06 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QfDiIXM2; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v4HAiul36EO+vclRiaij572sHtWWM/DwfWC2VtVH9HE=; b=SfH2ZvD+rC2THCv6wM9sFYE3w9nILVTLopFYxIeGcKbSBx/pUTmunQ3n445A+cObbZiIiN Accp8CQAUSBViRqd6ZVbwAHxQ978RDOSQH8XhRzcjgV8aBeCCoBbTq4eNezqABlXIqB9JJ ypl1Y33CmUpM0xg5OeD1toXbbJ6n1aA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QfDiIXM2; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699786; a=rsa-sha256; cv=none; b=XpFNLTqJhZbiARuNg7+8MZhiqYZiSFEVDp97BCx63LgHEjfymnA1L+LsF37MJGB7809iSh X/LzhYeRp6rIOlI6UHN6rAlTjmEy7QvBnCDGs0v8MFxTGK8DK2cpvmoIrUYPac1FsZ4NXr J8QFWWqFw7km46vWSvpyp4X4PYmO6HQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699785; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v4HAiul36EO+vclRiaij572sHtWWM/DwfWC2VtVH9HE=; b=QfDiIXM2sIT/8GXDUL0iPJL9v03bFOpbc9kPI6UMfotLps2FKvcvgAHAvtQMrxz394tAll zzjgkrx8gF21DqrRePvRFitfjGuNJ/R1VISBMPRUI5flqteafTFsCSaK099M59lJgj+b8C xVC3Jy0mbY0ovwXMv/Nu3enYFBwERhc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-38-f9IMTYTNM3mqQeNw2EDP9A-1; Mon, 04 Dec 2023 09:22:59 -0500 X-MC-Unique: f9IMTYTNM3mqQeNw2EDP9A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0378589CA00; Mon, 4 Dec 2023 14:22:59 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 661192026D4C; Mon, 4 Dec 2023 14:22:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 37/39] mm/rmap: remove page_try_dup_anon_rmap() Date: Mon, 4 Dec 2023 15:21:44 +0100 Message-ID: <20231204142146.91437-38-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 8140FA001A X-Rspam-User: X-Stat-Signature: ap14bmxwynpxqtff4owpob9nbjjicjum X-Rspamd-Server: rspam01 X-HE-Tag: 1701699786-526951 X-HE-Meta: U2FsdGVkX1+cGXJtFVVVXb9Cmv7rQyRvzmAMxjCJ6bNNcR160HLurBbzdf8Y27DgFdj0+tLcCAPrMTOZJzwrboQ54KbN1NhHQOT8KsOJwTaeIpIoBF/hKV8u7Gj58LuDTN1yGsM0NIC9O0PjtzGQTpultep8iSffR+5PqqSTAYR70aS6OOBy71da/ktY12i5x1cBr7yEIdjQgIhRc9ODxvvWNNdg9JwdpkJUZN4EJJpYt2B23Ofj6p2haWqxLWHZVesMO2cp/QydvCneQAr3WGV0Ua1nTnmlo/HFBIZ40Sg9FQBOeu3+xoH+ksrrJeSSUvR4MA4eP0/w3V4FItaVzkCKr3z6gohalbP9io1kUL2gAdlcS9wC+X87jBJgPMbG2SerBvBMQpMMGsw3pICuhqN/7Lwa1bNowZuylxi4UJ4dTXLKNvUa3mkUzA5LdxDklDYuEkoJ2AhjfUuWgAj0iTSMHDjKS4ewpkyc7/6UB6NaWjz7GfPIDUy7EvWyDyah8a3PEGRtbjAX3WZuYVstk6fBUaxDe4conYtGmiWCM9YpYn4CAMUOYYvILg8RdK9vJWJGNw0+hJ9JLXp4li4SjBxBeg5zOiCs1lHbhrPgQac1GczdOZnY+sK8mjxfxm5JyqnnxMUI2uiEJL76muHvM4oRXD6/uLZn++SY6lWuHgxKXBz0Rx8pez/vIill9BbehuDgBpcL8yNR/PoD0CYpmt8tfCcIj7IvZzNN3vk5xQ1JvjYI1sp/6k9feqGYk8MCMj+Si1MDdglcYR872Dea2A5pxAy1mx8HtPbrqirsn5366j86Zp16ntUeu4Slj6k0c57xbMdXPSE7xWUWJ+XwwFKMZP7or3Wk0gOzpZuOCyrqf8MA9BelXH7v9iFiPuSpHN0PMzrZE3dXSNNCAQa1vsg4t1H9U1itJNKSGIWvZWkIjjN9utQU6StrG08IvY8BksI55pn1Bspu1aOnKc+ 5QfvbTVk IK7g0Zv5JlYqqigQantK6kMsUY4BhxOTG+HBgnvBH1FDl1donBY4uoDub7+bkQF+V3JPySTsUi/bqkcEVRX9fmOapi3M+3EOAHSOSv17cRstihlr5qKw7bxVpu1ANIYHCfzp8tYzd1ieptZaWKOcgvcXtgaYnB113S4mUq8PNS0xesMn3+zoTgFiZSO6aNovfm04/wiHwdOqD+csrwQv64X8YPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, remove page_try_dup_anon_rmap() and any remaining traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 84439f7720c62..3c1df8e020188 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -251,7 +251,7 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -/* See page_try_dup_anon_rmap() */ +/* See folio_try_dup_anon_rmap_*() */ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { @@ -463,16 +463,6 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -static inline int page_try_dup_anon_rmap(struct page *page, bool compound, - struct vm_area_struct *vma) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - return folio_try_dup_anon_rmap_pte(folio, page, vma); - return folio_try_dup_anon_rmap_pmd(folio, page, vma); -} - /** * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly * shared to prepare for KSM or temporary unmapping @@ -481,8 +471,8 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * The caller needs to hold the PT lock and has to have the page table entry * cleared/invalidated. * - * This is similar to page_try_dup_anon_rmap(), however, not used during fork() - * to duplicate a mapping, but instead to prepare for KSM or temporarily + * This is similar to folio_try_dup_anon_rmap_*(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device From patchwork Mon Dec 4 14:21:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D315C4167B for ; Mon, 4 Dec 2023 14:23:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 002FC6B02D5; Mon, 4 Dec 2023 09:23:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EF6576B02D6; Mon, 4 Dec 2023 09:23:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D21156B02D8; Mon, 4 Dec 2023 09:23:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B63B36B02D6 for ; Mon, 4 Dec 2023 09:23:05 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8ACE24011E for ; Mon, 4 Dec 2023 14:23:05 +0000 (UTC) X-FDA: 81529352730.24.1A30C74 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 95AC0100009 for ; Mon, 4 Dec 2023 14:23:03 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYjMbKJC; spf=pass (imf14.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TrXZDuex4z9dO0cUmH/YoAM+lWjEkvkLNiyxPRViLRQ=; b=MicxTWejrnxJQ0hlnJIULSyNHZZ2zNIlRST9RzZM67+nRYDwe8JY8/MFp+wCp81bbewhPm I0petzbYwn296VC9djv9hB9cNYxB1qfbTNbrcUIC+SMO7cW+N5iDYuN67/KozIWT97FuvM /RkK/skGA6IuQmkqKerMCyKuUx3ckHg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYjMbKJC; spf=pass (imf14.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699783; a=rsa-sha256; cv=none; b=hT1iUYOBxBURVBPReI6CdJUZYYhQue4ydkskOnPeAYHstYh8Iz5adSqORoDY+sxuX/nZPq LDYQppP6gwdnJDpSHQL32okzz6znLw+kbC5cpwONsVaKoiuGGOAIo4dKP8/Y0vBugl3NLw 5zSpIqZ2BHcD3eEkAx+7HdXs681EJ08= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TrXZDuex4z9dO0cUmH/YoAM+lWjEkvkLNiyxPRViLRQ=; b=TYjMbKJCgRhqfaNUE5FfJJ+qigEYSVuqvskfmyswBzIr8yvH+ILxmF8UFhj5QYLhIJIQX3 SrPVsdc7xQMbx/LpUMSmCBH0h6YYkTq/6NVJvYI3RwhatW4L6vLkMdjTJJFM8M8RRPoMVc epWTqcN6FiA/NG9azlRm5wfimThfqyQ= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-295-Ef5PYdNvM-qrNNOKUuEk5A-1; Mon, 04 Dec 2023 09:23:01 -0500 X-MC-Unique: Ef5PYdNvM-qrNNOKUuEk5A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D721E3813BC8; Mon, 4 Dec 2023 14:23:00 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BB132026D4C; Mon, 4 Dec 2023 14:22:59 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 38/39] mm: convert page_try_share_anon_rmap() to folio_try_share_anon_rmap_[pte|pmd]() Date: Mon, 4 Dec 2023 15:21:45 +0100 Message-ID: <20231204142146.91437-39-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 95AC0100009 X-Rspam-User: X-Stat-Signature: 76d9x5dzctbn4wx1e7wfrzwxoxc5jnf9 X-Rspamd-Server: rspam01 X-HE-Tag: 1701699783-590066 X-HE-Meta: U2FsdGVkX19yFDBkl9eHkQuvYxMTIg9YJWpw76/cQKjXmscq8ZL7c4CsQaTEl66T7W2qXcYTM+ym0k1kNBzIlNodeCJgtp5onpw8jTTcww6Wvp0xXIqYZTgMFl1hW8Mj+yteYv2cd5PNegvoBMNoSS03YfvZ+9oHgrGwIgEQkUK2xsyRNPm8xBiJxzeIlR1q22XhSp+kIza3rpNCDA2JHUnZaSFH/M10ZcbDNVjEU0XOHu1yQPvE6mNAKnrsdLVpAQczm828o/RjxET63BPyrRsxTKZcLqHa+sYd+qn/lTODNiaYVrNlk7eRQqCXDmTA6w9QDNQljlFzjbgkkXkqrqH++0PcXNFx6dxTDg6iFBnQTVYdNpvvuFhY238AlhpgQI+5YMMZR/gXD8ih1il4w5VaQmIEkn27uH+gSgmfBlmC48I5+9B7FwPa0H/CN0oyDuESFgxFLGrquO6VVDLFMwb2/Og8ehY1rRL35RCRcz2p3oCjYKsSGYa/askZL2tvh9WS/vfehkFWlxF28pOb/ftQnChe3UWMuQuHR3/Y7IsPIHjYf8/XXz266r8xoD1EJQ5DVCcYlzCwjMgaApbPdYTjE6JIhiN7mOdNJifMB970XwkUmsuPKBQdnvvtK1+Qfp7kXWJvJzhcrEpRMoh/frkhC6ES/57F94LS7YYq0vPVHL+5PTmPRXwTShyYgpOnElcooFbTcxlL9ygvPqEG/g8KFYLIqqBg9LNJ87BMtpeAtrWTTCopSVDWXhXas6sckDXEkwpmQ4xUQK2VCeHwha+LXqL6ToMwfF5fnRa7AElhl8P1vJHVUirMSndNo0nbSWCAVehpyB6kUM/5hWAW/R/XSr95VPmU+Kx662BHix8+1HbgJ3C2Yb+jaHhL1jbennA5Veeh8ZAPIbsQPytZNuLhBhtU9R6Y4EU0z2RviBP1+1qfR5Mbk4W8ApFFsp6hGLI3J/uY+/GYd5/Z6np f5i5Hehk 02DR83h6XmQ8uwA62BYktQstdVhi9g6Kc2qyxQvmnC+LPNmx0kqKrCOTGhS1+Ropd1bCvhOL16cmukRjT2T2d3hZ5nBlz7qi5omKW5SWJGfDSoynQl3KfebYoIVmH7Sb0u/rMQiQwNunjsXXySci1BGUvyL+JEnQ9oLCGocDftS5E2XQiBRf2oQfRNZZWcpAcMD8PazvechX6XQwoFqfbqe8wbveMtnenNxk7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert it like we converted all the other rmap functions. Don't introduce folio_try_share_anon_rmap_ptes() for now, as we don't have a user that wants rmap batching in sight. Pretty easy to add later. All users are easy to convert -- only ksm.c doesn't use folios yet but that is left for future work -- so let's just do it in a single shot. While at it, turn the BUG_ON into a WARN_ON_ONCE. Note that page_try_share_anon_rmap() so far didn't care about pte/pmd mappings (no compound parameter). We're changing that so we can perform better sanity checks and make the code actually more readable/consistent. For example, __folio_rmap_sanity_checks() will make sure that a PMD range actually falls completely into the folio. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 98 ++++++++++++++++++++++++++++++++------------ mm/gup.c | 2 +- mm/huge_memory.c | 9 ++-- mm/internal.h | 4 +- mm/ksm.c | 5 ++- mm/migrate_device.c | 2 +- mm/rmap.c | 9 ++-- 7 files changed, 89 insertions(+), 40 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3c1df8e020188..ab3ea4583d502 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -267,14 +267,14 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } -/* See page_try_share_anon_rmap() */ +/* See folio_try_share_anon_rmap_*() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); - /* See page_try_share_anon_rmap() */ + /* See folio_try_share_anon_rmap_*() */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); @@ -282,7 +282,7 @@ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) return -EBUSY; ClearPageAnonExclusive(&folio->page); - /* See page_try_share_anon_rmap() */ + /* See folio_try_share_anon_rmap_*() */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb__after_atomic(); return 0; @@ -463,30 +463,15 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -/** - * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly - * shared to prepare for KSM or temporary unmapping - * @page: the exclusive anonymous page to try marking possibly shared - * - * The caller needs to hold the PT lock and has to have the page table entry - * cleared/invalidated. - * - * This is similar to folio_try_dup_anon_rmap_*(), however, not used during - * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via folio_remove_rmap_*(). - * - * Marking the page shared can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. - * - * Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY - * otherwise. - */ -static inline int page_try_share_anon_rmap(struct page *page) +static inline int __folio_try_share_anon_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, enum rmap_mode mode) { - VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - /* device private pages cannot get pinned via GUP. */ - if (unlikely(is_device_private_page(page))) { + /* device private folios cannot get pinned via GUP. */ + if (unlikely(folio_is_device_private(folio))) { ClearPageAnonExclusive(page); return 0; } @@ -537,7 +522,7 @@ static inline int page_try_share_anon_rmap(struct page *page) if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); - if (unlikely(page_maybe_dma_pinned(page))) + if (unlikely(folio_maybe_dma_pinned(folio))) return -EBUSY; ClearPageAnonExclusive(page); @@ -550,6 +535,67 @@ static inline int page_try_share_anon_rmap(struct page *page) return 0; } +/** + * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page + * mapped by a PTE possibly shared to prepare + * for KSM or temporary unmapping + * @folio: The folio to share a mapping of + * @page: The mapped exclusive page + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pte(), however, not used during + * fork() to duplicate mappings, but instead to prepare for KSM or temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pte(). + * + * Marking the mapped page shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped page possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pte(struct folio *folio, + struct page *page) +{ + return __folio_try_share_anon_rmap(folio, page, 1, RMAP_MODE_PTE); +} + +/** + * folio_try_share_anon_rmap_pmd - try marking an exclusive anonymous page + * range mapped by a PMD possibly shared to + * prepare for temporary unmapping + * @folio: The folio to share the mapping of + * @page: The first page to share the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pmd(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pmd(). + * + * Marking the mapped pages shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped pages possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * Called from mm/vmscan.c to handle paging out */ diff --git a/mm/gup.c b/mm/gup.c index 231711efa390d..49f32411c68da 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -177,7 +177,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) /* * Adjust the pincount before re-checking the PTE for changes. * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). + * barrier in folio_try_share_anon_rmap_*(). */ smp_mb__after_atomic(); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 138e1e62790be..ebbf5ee6192e7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2224,10 +2224,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * In case we cannot clear PageAnonExclusive(), split the PMD * only and let try_to_migrate_one() fail later. * - * See page_try_share_anon_rmap(): invalidate PMD first. + * See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = PageAnonExclusive(page); - if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) + if (freeze && anon_exclusive && + folio_try_share_anon_rmap_pmd(folio, page)) freeze = false; if (!freeze) { rmap_t rmap_flags = RMAP_NONE; @@ -3253,9 +3254,9 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); pmdval = pmdp_invalidate(vma, address, pvmw->pmd); - /* See page_try_share_anon_rmap(): invalidate PMD first. */ + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); - if (anon_exclusive && page_try_share_anon_rmap(page)) { + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } diff --git a/mm/internal.h b/mm/internal.h index 43dca750c5afc..b9b630717b9b2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1047,7 +1047,7 @@ enum { * * Ordinary GUP: Using the PT lock * * GUP-fast and fork(): mm->write_protect_seq * * GUP-fast and KSM or temporary unmapping (swap, migration): see - * page_try_share_anon_rmap() + * folio_try_share_anon_rmap_*() * * Must be called with the (sub)page that's actually referenced via the * page table entry, which might not necessarily be the head page for a @@ -1090,7 +1090,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, return is_cow_mapping(vma->vm_flags); } - /* Paired with a memory barrier in page_try_share_anon_rmap(). */ + /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_rmb(); diff --git a/mm/ksm.c b/mm/ksm.c index c23aed4f1a344..51f2d989be2be 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1161,8 +1161,9 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; } - /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && page_try_share_anon_rmap(page)) { + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && + folio_try_share_anon_rmap_pte(page_folio(page), page)) { set_pte_at(mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index c51c99151ebb5..9d0c1ad737225 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -202,7 +202,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); - if (page_try_share_anon_rmap(page)) { + if (folio_try_share_anon_rmap_pte(folio, page)) { set_pte_at(mm, addr, ptep, pte); folio_unlock(folio); folio_put(folio); diff --git a/mm/rmap.c b/mm/rmap.c index 8e86024953c03..7bb3a174efc8d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1817,9 +1817,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); ret = false; @@ -2093,7 +2093,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, pte_t swp_pte; if (anon_exclusive) - BUG_ON(page_try_share_anon_rmap(subpage)); + WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, + subpage)); /* * Store the pfn of the page in a special migration @@ -2175,7 +2176,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } } else if (anon_exclusive && - page_try_share_anon_rmap(page)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 4 14:21:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13478540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F009AC4167B for ; Mon, 4 Dec 2023 14:23:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 819B16B02D9; Mon, 4 Dec 2023 09:23:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A2F26B02DA; Mon, 4 Dec 2023 09:23:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61BD56B02DB; Mon, 4 Dec 2023 09:23:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4E25C6B02D9 for ; Mon, 4 Dec 2023 09:23:09 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2B2321C048A for ; Mon, 4 Dec 2023 14:23:09 +0000 (UTC) X-FDA: 81529352898.11.8D0714C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 7D06A2002B for ; Mon, 4 Dec 2023 14:23:07 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JW7zfRuh; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701699787; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VzMmgnK/Xwl+ZoGTv7dPX8K+DyrKez8QvfXwl9HQj10=; b=4RzG2L2szdZElFYvS3A0Uc4jEzwkiWCSS+xaHv3wVU7AA/ovf37GgCm7JUqAxMX8Y17mPS lNE5xqncHzJqWv05lmwzlr9nQCJFMteKYledH6iR9FIHb/P4gHarAzWkAPeKahBG317o6C rH+W/cep7tBPbB94AzR8UtqNKkRbFNw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JW7zfRuh; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701699787; a=rsa-sha256; cv=none; b=JW1/L62MeqChYzWKJDoa8MbKscKz16fjQ9TzExzA1eqZaUYvnOtgCmSs51elFjM2wb0S/1 nZHqJp3ZOf0L68TauGr/MG+fefJI0+UdmLgohNlPr/EO/5zr8jYk+E8nNZo13M5YRYj9rl H8BPnpzD5bhARlyum6pfujnpvlSFrdk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VzMmgnK/Xwl+ZoGTv7dPX8K+DyrKez8QvfXwl9HQj10=; b=JW7zfRuhQleyyOXWtszJ3PmHWZzacXu6JSMwHe+8iR0gy9TRtj6YcQwFeU2mf/IEMCw16G ncy5sdbkFwEy8fLeKX+XjRsHzqFuPVCUYct0PBPbyD+K8MWOba+wo9PVpa0+ghUlHO3FOA LNYYyq2uXf43CKtMhiYifectAaa/gX0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-489-6zp6WxYFOY2-2yvUwM7PyQ-1; Mon, 04 Dec 2023 09:23:03 -0500 X-MC-Unique: 6zp6WxYFOY2-2yvUwM7PyQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AFCFD89CA05; Mon, 4 Dec 2023 14:23:02 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1B80C2026D4C; Mon, 4 Dec 2023 14:23:00 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 39/39] mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED Date: Mon, 4 Dec 2023 15:21:46 +0100 Message-ID: <20231204142146.91437-40-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 7D06A2002B X-Rspam-User: X-Stat-Signature: aytibofx6hia9coi1p814prg4kmq8nzp X-Rspamd-Server: rspam01 X-HE-Tag: 1701699787-604103 X-HE-Meta: U2FsdGVkX19i4dVJUnAeZGXkagxPDVwk1G9fQ5tgWWQ9jn+BOBwtbwz9y47RErPLs73qPt8PkqJ3nxh+Q2u7MpKFATY4WEhujzLLpauFvju9TTifp5lacSaO4IInS82XW8JABTEsFBMejsdMq3q/balnphundN+xNYNGMvYzowLoWWOu2hI0QXQH++90EJa+jjYxTFcz/gWlDw5Q05p5B5n+87W4VecHhOP/XGQj10dpDRaSdES3F/joMy5lyX7nvSz80EX/rO5TNZZuuKtgmXv+9xVOPy8Dszrkh3xlb5Vz3ADuv+EpWtZwsBWRqOXxbwG8kAKn/iPsEygmycSBknoSs64SITs/yJtl49Rha9MQTL+k2n7MhVqrTNvv9/9NFnA8Cflz1DS6dXkHZv9G5F7F0b6FaJH1RdWanjWswM6yJ+2kY2fpAvxfO3Ij8lEo8P008gX7c92L6bhKYViof6PLf2+kgekz6WSf0fRuppaIU7qu+lDd8GfRdsA3/JXHz15ZXOsLpdETVPDtGMZhbUgWYyQO/DyRsHZcEjjOpn6IvOc7W5+yaQBaAlUsLo211FpW3OaIhm59no5PlXBAoU6CaXcfP5n+49vdGH1cV9MwjQ8RINM36ZnnBK5l2R/pfRA+y2qEWCDm0BHm7FB/+Gm40zpuVFM+qw4x306atcTC5FNDKT2DERLComw1aNu+R2XzxJ1odlSj3uWb74pJN45XhrdgSUC2O4MPFwJVlMWX3HtbNk3yDEncTm6UprG84/zhA3skbCMRri6brMkcYrkupK/QzYlUitBWwO7fmsOQUkZix6TrvNg7IyForD/++20Mgw13cCWDO2pu5oQFk5GdnWeWz9XBQDjvFazmq6sE7Tnm3zAwa697Wrn034T9WgfTMVhVu4nhGZ8C/hICZONahhBmZfCO4WQAHGQIWd8sQ5WszMw64/SJEVWtHycQTmAdvX2H8ux0VlATy/M wFU7eH1y GVyE0PpMCvc/IPQF+grtJJLuZ0q6Bq4CYerRGdzTrnKdYl+JydJ4Be/drF9b8Ai8xVGCxiarOnRTAoRgM+HqMHRdQgGjnFymBILrjK+DUnflfpPGYyd6McaGAiRFNbZ9A6vFV1LeT4Ptdmr7hzRYtsMcrEA/0W7r+wXtRirPYXCRlTK5WIP26BQAkpkwz0DT4yJHYMR0Of5ehiivAS5t5rPSJaynHATn1zjVF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We removed all "bool compound" and RMAP_COMPOUND parameters. Let's remove the remaining "compound" terminology by making COMPOUND_MAPPED match the "folio->_entire_mapcount" terminology, renaming it to ENTIRELY_MAPPED. ENTIRELY_MAPPED is only used when the whole folio is mapped using a single page table entry (e.g., a single PMD mapping a PMD-sized THP). For now, we don't support mapping any THP bigger than that, so ENTIRELY_MAPPED only applies to PMD-mapped PMD-sized THP only. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- mm/internal.h | 6 +++--- mm/rmap.c | 18 +++++++++--------- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index cf81272a6b8b6..93c9239b9ebe2 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -117,7 +117,7 @@ pages: - map/unmap of a PMD entry for the whole THP increment/decrement folio->_entire_mapcount and also increment/decrement - folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. - map/unmap of individual pages with PTE entry increment/decrement diff --git a/mm/internal.h b/mm/internal.h index b9b630717b9b2..700b230666f87 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -54,12 +54,12 @@ void page_writeback_init(void); /* * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, - * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * its nr_pages_mapped would be 0x400000: choose the ENTIRELY_MAPPED bit * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. */ -#define COMPOUND_MAPPED 0x800000 -#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) +#define ENTIRELY_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (ENTIRELY_MAPPED - 1) /* * Flags passed to __show_mem() and show_free_areas() to suppress output in diff --git a/mm/rmap.c b/mm/rmap.c index 7bb3a174efc8d..a8e3563182103 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1142,7 +1142,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); + first = (first < ENTIRELY_MAPPED); } if (first) @@ -1151,15 +1151,15 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, } else if (mode == RMAP_MODE_PMD) { first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { *nr_pmdmapped = folio_nr_pages(folio); nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ + /* Raced ahead of a remove of ENTIRELY_MAPPED */ nr = 0; } } @@ -1384,7 +1384,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); nr = folio_nr_pages(folio); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } @@ -1467,7 +1467,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); - last = (last < COMPOUND_MAPPED); + last = (last < ENTIRELY_MAPPED); } if (last) @@ -1476,15 +1476,15 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, } else if (mode == RMAP_MODE_PMD) { last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED)) { + nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED)) { nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* An add of COMPOUND_MAPPED raced ahead */ + /* An add of ENTIRELY_MAPPED raced ahead */ nr = 0; } }