From patchwork Wed Dec 20 22:44:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FFA2C46CCD for ; Wed, 20 Dec 2023 22:45:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D97726B0082; Wed, 20 Dec 2023 17:45:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D48266B0083; Wed, 20 Dec 2023 17:45:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC3096B0085; Wed, 20 Dec 2023 17:45:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A7FA86B0082 for ; Wed, 20 Dec 2023 17:45:16 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 78368160482 for ; Wed, 20 Dec 2023 22:45:16 +0000 (UTC) X-FDA: 81588679032.23.A15BBBB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id D9F5314000E for ; Wed, 20 Dec 2023 22:45:14 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KdQDlHyZ; spf=pass (imf09.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112314; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G+bCjcGey/V7RsjKz554fGR3BU3YAAJC2JoClW1JlMc=; b=wro+UoX5jgFsJS3WE9ZtF0a5gMd7zCceSU/RGUdamHd4miENFnRc9FbNrf6ok01iuuVSXO z69YwWinAzScrM0pmMIe4XyntaGwaj4ZxSlhlsy+tiLTTqTshoNwEaESIP3bRMQEpsKUAA 3g338C13NT7W4OoeyHtzuKr+jOUd5zU= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KdQDlHyZ; spf=pass (imf09.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112314; a=rsa-sha256; cv=none; b=yfB7KwG6eDAhuxCFa8UcOjdd4cHRiUCAZuxI6WiyS3RSDjDj3df8Kl0m8ynO8FHf3rQ85u cKoJG5VOmmDA1dr0XL4hMPM1+at0lSkjXNIZcpgPq0ZZqlzJrRobijw1J7S+wcXD6AgIsU XS4fTwGJp/nX65hwSU0JOOs4yiKHLgI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G+bCjcGey/V7RsjKz554fGR3BU3YAAJC2JoClW1JlMc=; b=KdQDlHyZ8qNzDTRIaDD42fYM3wf5ZK6u2BvQtht7cJ+J+NxhxWLHnpP1g0YnZjCLjEAOQa dfRIG+uwUw9U8TsK6kHtqtyvXPe+kP2vKXQIomXU45tv+v7YPwnHsay+6VdJUEdpD8hLet jNnScDUUheJAjw72plxEpnSR73ZTFJY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-425-Wt5IrIHnNeaZg63pnEl0Gw-1; Wed, 20 Dec 2023 17:45:11 -0500 X-MC-Unique: Wt5IrIHnNeaZg63pnEl0Gw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BB553848C06; Wed, 20 Dec 2023 22:45:10 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 608DF40C6EB9; Wed, 20 Dec 2023 22:45:08 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu , Muchun Song Subject: [PATCH v2 01/40] mm/rmap: rename hugepage_add* to hugetlb_add* Date: Wed, 20 Dec 2023 23:44:25 +0100 Message-ID: <20231220224504.646757-2-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: D9F5314000E X-Rspam-User: X-Stat-Signature: f6jizeu4ps4ynsjmzd331cusncf561hq X-Rspamd-Server: rspam01 X-HE-Tag: 1703112314-27633 X-HE-Meta: U2FsdGVkX18bPLxKJJa6fVKgLyDW/ERiOu8IBveiTawTbc6Ww16NIpBSL1oJdMXdJTuE59IW9cXwWjO9KXMdY7E2XbVu0zcTxnlCKr1Uy+6rn8h0Ar2fS1LHI2PXxvfbM2rCV9u6AVWEoLynDwt9HdX+1+kFAGFMgp6T/lUgUzid+L+ZTL1nVcJZo/RgMQofuIcTiEowNnRgiq0IF4TZofEBfqP+Tg4VPdBumtCbDUYvDzP7P3W4dkA4bMDYz9p7ONrBCQSRnFyXlvubP856DeFFniYdxDiLL70T5SSLlopHBjC/WXzlkcOCHzb+JMSPvh028WkJ0FUQ97/EfluctAkGiLNXOJddkAxD2i2DhaNow2vUfXfGYv4CWGy1+TJQzNDJgXzeo1qz1tFchcjetUr4DF4M/mkVIRmFccpPEgT2VnCVdcl12V00vlqwnB4iMIVC5g2j8XR6oGR7P+v914ricLuPM4OeyTZf99eyUHXrqs5KgZ85v37xmCSHstJABzvD+pdaLLAMcimOqyFRqxymN1M7Xhk53SHvYGSrFqEX77gYHuJrY8VTRJXY5R8/sk7fcb9Q9pl/bcmCAAeMymLUvAH8R82tWvTFrh/hfria/K9v8o4fkGqU7aP4p6jKVIE/zjVfRnpBsBlwDbMeAKxjdLype82uOdSi4nM3vmgTREEvBm6yPcIQKcLNCUMQexV05F5MlHggv6MYe4xTFCajRUTT25Dh4H2YUxFlB8CAek40WeBAAmyAMytWrI7f9iJcZftzKl79c1XjZ6M3uU/OI+ErEV9cjGEpK4yv+LMU3oemg/lMnLGDPSDsDunsO2b625fhgRltx2XrFBmxmyX0peo0d/vpHyngNn8qVpfHYdoGeA4kPqSfv+k70Cc0vJ6rmzfD5JFVcPqu/JSWWkH8daB3QZw1VX6SKqUap7MUTX2Lfsejqeb3Qau53TRe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just call it "hugetlb_". Yes, it's all already inconsistent and confusing because we have a lot of "hugepage_" functions for legacy reasons. But "hugetlb" cannot possibly be confused with transparent huge pages, and it matches "hugetlb.c" and "folio_test_hugetlb()". So let's minimize confusion in rmap code. Reviewed-by: Muchun Song Signed-off-by: David Hildenbrand Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 4 ++-- mm/hugetlb.c | 8 ++++---- mm/migrate.c | 4 ++-- mm/rmap.c | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0ae2bb0e77f5d..36096ba69bdcd 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -206,9 +206,9 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); -void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); static inline void __page_dup_rmap(struct page *page, bool compound) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6feb3e0630d18..305f3ca1dee62 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5285,7 +5285,7 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); __folio_mark_uptodate(new_folio); - hugepage_add_new_anon_rmap(new_folio, vma, addr); + hugetlb_add_new_anon_rmap(new_folio, vma, addr); if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(vma->vm_mm, addr, ptep, newpte, sz); @@ -5988,7 +5988,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); page_remove_rmap(&old_folio->page, vma, true); - hugepage_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); @@ -6277,7 +6277,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) - hugepage_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, haddr); else page_dup_file_rmap(&folio->page, true); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6732,7 +6732,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (folio_in_pagecache) page_dup_file_rmap(&folio->page, true); else - hugepage_add_new_anon_rmap(folio, dst_vma, dst_addr); + hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); /* * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY diff --git a/mm/migrate.c b/mm/migrate.c index bad3039d165e6..7d1c3f292d24d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -249,8 +249,8 @@ static bool remove_migration_pte(struct folio *folio, pte = arch_make_huge_pte(pte, shift, vma->vm_flags); if (folio_test_anon(folio)) - hugepage_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); + hugetlb_add_anon_rmap(folio, vma, pvmw.address, + rmap_flags); else page_dup_file_rmap(new, true); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, diff --git a/mm/rmap.c b/mm/rmap.c index 23da5b1ac33b4..9845499b22f8f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2625,8 +2625,8 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * * RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) +void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); @@ -2637,8 +2637,8 @@ void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, PageAnonExclusive(&folio->page), folio); } -void hugepage_add_new_anon_rmap(struct folio *folio, - struct vm_area_struct *vma, unsigned long address) +void hugetlb_add_new_anon_rmap(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) { BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ From patchwork Wed Dec 20 22:44:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 967E7C3DA6E for ; Wed, 20 Dec 2023 22:45:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 228956B0087; Wed, 20 Dec 2023 17:45:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B21A6B0088; Wed, 20 Dec 2023 17:45:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 051776B0089; Wed, 20 Dec 2023 17:45:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E4B4E6B0087 for ; Wed, 20 Dec 2023 17:45:19 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BDB29140430 for ; Wed, 20 Dec 2023 22:45:19 +0000 (UTC) X-FDA: 81588679158.26.0EE05FF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 04C6A180002 for ; Wed, 20 Dec 2023 22:45:17 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bawJfze1; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zb5u0xx47K41JJXfyNoHDQEukBbwbPO9Q+a2q/uOjFE=; b=3PIsPr12mnnG7oICqjzmckvGOt+JMYCuL6GwcO+p1C5P/gX8FU/Ilo3D6V5xBzot/CWiai l+6FDZWCNLC+lXIPZYdnvTxUUNbbPERozATFUuZh0PO4T35+wNNc8gx0ou+/rCNkeWSDam dYtO0HOgVgZp+KD3bujRtw7Q/uB/s9Q= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bawJfze1; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112318; a=rsa-sha256; cv=none; b=M07iJKnX4daCSchSnfbn0hsV+URO2Y3Nps/K/QkCn2SWTZWAbOgnXUU2ql8B1LDMsSwqAA hvhdtawMs4nIPer54QMOFubG+G2fcQMu77wwcAHdpWapQkDWi4W9x7bX1jxbrOoTZLOIAw yjmlj/9LzLMAT3Le6kJ4KQuqzZBjAgY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zb5u0xx47K41JJXfyNoHDQEukBbwbPO9Q+a2q/uOjFE=; b=bawJfze1Y2ktY/x/244LN098CEHo87dSfBPJCqsLD/AUTFI2Y1AELzCMvpt8gi5fJt+cju xbaPsjKuo4gYVH1ZclNUsUVdEG0gDxjQictsCBP9Aq8DwkWgmx5w5R0STfrBEz/Y5HVr/1 T0gSoxZd0/w/ksH+sp1dxfD0/eX3xBE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-1-JUWNMJ1_NEm_REskD0WbEQ-1; Wed, 20 Dec 2023 17:45:13 -0500 X-MC-Unique: JUWNMJ1_NEm_REskD0WbEQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3710986EB20; Wed, 20 Dec 2023 22:45:13 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 273FD40C6EB9; Wed, 20 Dec 2023 22:45:10 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 02/40] mm/rmap: introduce and use hugetlb_remove_rmap() Date: Wed, 20 Dec 2023 23:44:26 +0100 Message-ID: <20231220224504.646757-3-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 04C6A180002 X-Rspam-User: X-Stat-Signature: 83edw69qxwrjrm43pphoidoart9wixku X-Rspamd-Server: rspam01 X-HE-Tag: 1703112317-15527 X-HE-Meta: U2FsdGVkX1+OW3tE1wMI3nA4LKhJM4n6lOMrWpZFYTBP4RhkJOvN2KtJ0kRZkz2VAwZpj1M5EMr3vyT8lh0zSvbKpHK7ShxSJctJRkKA4hU8aSUd6FnGqhYgw8jhdPGWZLwIpsuQTrs72sgsvZS1LFYcCECMtMYmNjcqS+qqnxC4OGn2MJHCbJwKQj6RghYfbNO+DKD9DK4Nl0xk6HYbSy4Uhhha18TbLa+3RfgDbdqcFCDP6TqhdBLowHqR1XHYxTbchlmHuO9P+N1OAcQhZOPnLBTCnKvrgO3avXd9bAB+oUADziXLgbJbkl/JYelrZiVwKIttrx+Tuhd1XuYAQwEzRuoWUfB9IfIBKH//ZQTrVhsh74PSVSK7RUhBBJyjLsXEZ1XwlEC0jO7w9EwpdboL7YixAc5iEg2GVLxTP2EyjDXIz6J2CajZFcMjEpaKCGW6pI9KaQjAj1zbpkKf2e3rNIea7kGDMy1OxomUpMNmZ0b2oJyt833Vb1GRU/8wTKqToKkSlS2xWHQL77K6H86zNT+XfuE5fEIZKOMRtyNWfq7rvUrlmqzKZQS6QgE3vEooRwUDUTy7lL0Z0by/sWgr0s9SMmekjXTVm+dtvK56M2AZ2Mi6c0lST4U4l395Fb0hfJTf33Bv5uXsBNsOfdAWOUBMElrvtRwoPiCQ+4yAlMeu0rYqIrHiAUucRusbrxdUZYNYS7P3oCxtEHpmEHxoCH/D1jr/aeGF0y2mcf3p97rcnCKRcHAcHz4oV/4pqOS7f1k8+aQYvPPAaouSX5z+vPurZcqttTuOMKzvsZon8RFapmZw+FMfo9D0xAQoZinlc/r5QCVI8Std/KI3v3O8kATYR++vvcv2JsKScNvYbOLmUrsZLk1N19BvmcnRIQjk3Yx1Wy8Hy9ZGW7UOcdtxFEsqSa3zNJdv1AMoOlkFUxzrxV0Jh6EKTkZbnZOJE7kf7rwoVYEM74EeLfp fFBV48B0 1gmy3W3zJeLeO7Zr1ZBJ0TuIWEtmo5mFhGgaubg0dULnAa8yFcoz82KCcb5M8gCnMDARNS+WNSDu48iPE/0mY4D47F0BvpyaQP0ju3EIPkNSgwzKA6bLYe9efgC1SWsdmBn5nS9UK/fe2fDqrYvA22Zy/JCMQEjuF9XyD4EJflkDc9R2d/8DfzneNzwYvSwdZbetdpcRNcXyF7LsV1XedUhVk+BEFratvl7RVHi+mKdzsS9MTxZVjYmsBj7GmZ/VN+3foZmEJLrf212M+tCUxKBjWDGK21/HRiLgXR+F+Vi8BXUnFHYTZ44PcF2uIVRc01hNOvM9e5CL23/j/uVgTfOGavwWdPHEAOhsE2ErqsqmcT14= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Let's introduce and use hugetlb_remove_rmap() and remove the hugetlb code from page_remove_rmap(). This effectively removes one check on the small-folio path as well. Add sanity checks that we end up with the right folios in the right functions. Note: all possible candidates that need care are page_remove_rmap() that pass compound=true. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Reviewed-by: Matthew Wilcox (Oracle) Signed-off-by: David Hildenbrand Reviewed-by: Muchun Song --- include/linux/rmap.h | 7 +++++++ mm/hugetlb.c | 4 ++-- mm/rmap.c | 18 +++++++++--------- 3 files changed, 18 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 36096ba69bdcd..64ae6c4d72720 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -211,6 +211,13 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_remove_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + + atomic_dec(&folio->_entire_mapcount); +} + static inline void __page_dup_rmap(struct page *page, bool compound) { if (compound) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 305f3ca1dee62..ef48ae6738909 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5676,7 +5676,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP), sz); hugetlb_count_sub(pages_per_huge_page(h), mm); - page_remove_rmap(page, vma, true); + hugetlb_remove_rmap(page_folio(page)); spin_unlock(ptl); tlb_remove_page_size(tlb, page, huge_page_size(h)); @@ -5987,7 +5987,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); - page_remove_rmap(&old_folio->page, vma, true); + hugetlb_remove_rmap(old_folio); hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); diff --git a/mm/rmap.c b/mm/rmap.c index 9845499b22f8f..261e1af0d254f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1480,15 +1480,9 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool last; enum node_stat_item idx; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_PAGE(compound && !PageHead(page), page); - /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(folio_test_hugetlb(folio))) { - /* hugetlb pages are always mapped with pmds */ - atomic_dec(&folio->_entire_mapcount); - return; - } - /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); @@ -1846,7 +1840,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2199,7 +2196,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); From patchwork Wed Dec 20 22:44:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1A91C46CD4 for ; Wed, 20 Dec 2023 22:45:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E3286B0088; Wed, 20 Dec 2023 17:45:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61C876B0089; Wed, 20 Dec 2023 17:45:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47A6F6B008A; Wed, 20 Dec 2023 17:45:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2F49A6B0088 for ; Wed, 20 Dec 2023 17:45:21 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 031C0A05D7 for ; Wed, 20 Dec 2023 22:45:20 +0000 (UTC) X-FDA: 81588679242.11.C27F55E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 685E51C0028 for ; Wed, 20 Dec 2023 22:45:19 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=glVxp5dg; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112319; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MXqiJu2pun8vKoCf/nv4oO60M9YSAaLT+Ujt3IpxQcI=; b=8HHydAleIDEXz7Pa4QT5+k2kPAoPz9kZ3HN0Et8hKcSroq5qmFoX/aUi8ykMSU8fCO8gDO 191+5sfgZj9j1Jg5Mo2nRhmPfliRO2gs6ylXVJc1HHY1ZukyG/ChJLkfUfo4oyebHBgBos zYaE4EH6N1k2mdJHkT1q7UsnFNT0y1Q= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=glVxp5dg; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112319; a=rsa-sha256; cv=none; b=4srYIt39mo5nsdXdcyTMbytckJnc1vN+XBHogdLZqE5y1c505vP2aJ0i5jXJcC/kpgOrpF Pu6NwhuZhAYCG++k5JAqCxXKBdEkwl4KghwVX9zBsmS44RFA33Z/HjhEGmPLXdcjUlfh/k TQAQbPmT00JmdTIMm/DndIo7G5RojNE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MXqiJu2pun8vKoCf/nv4oO60M9YSAaLT+Ujt3IpxQcI=; b=glVxp5dg4JhN9zWpgtzVPuPlrg8UUr6k9kMzEch47LaHVpdMcOGr8OrStE5R8kOy0sk4xE 6UjfIeQg2xEXJsTsa2C7Vp9q1DmXrXJsX/WNHyfxH8LAlsQQnu6oOgfePXuKsmN0GoolQ7 /dwr8hw7jLegI4Pegvu2tO2HW5urWrA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-615-YB96pkyOPFqEud271oQ8LQ-1; Wed, 20 Dec 2023 17:45:15 -0500 X-MC-Unique: YB96pkyOPFqEud271oQ8LQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 36C35101A551; Wed, 20 Dec 2023 22:45:15 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7185940C6EB9; Wed, 20 Dec 2023 22:45:13 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 03/40] mm/rmap: introduce and use hugetlb_add_file_rmap() Date: Wed, 20 Dec 2023 23:44:27 +0100 Message-ID: <20231220224504.646757-4-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 685E51C0028 X-Rspam-User: X-Stat-Signature: 447eirt8mp9pi53hh81doj638ifckn1q X-Rspamd-Server: rspam01 X-HE-Tag: 1703112319-65726 X-HE-Meta: U2FsdGVkX1/fMhwmqxsr7tnbKmxw8dqRslMRDiC1bkOiflA31+X+kiHa807k6L7hzJELuKySScKZEnU9NEXspkEmrBwiNL7yNm/AykbSCIGHBIySL8chV6pquKiP732qmPM2E7JZ8mD7sZiC+xvCWGVM/bmJ0iIcjtEiAm8cQ8zdNw3GBleO9hII10ZrkWBKcb5cYpDA1BrI9l5Hiwv/8SWaGS0rgSkLREHgHFrshF6WPPYjeo6MZVedOFYesLAH26rTlt6Tn/ihTkrZhr7T8wHBBsZxCmceixRwWhH7iof3F9TSrzQGj7HzQHcnowluT9Zo1LpI4v6q0uT45Xb/Sn2hRAjgfTA7T3biL368lZ8whFEsv7v6AXzN8J0wYYbobMtUmdGXDvMIQF3jZm++EB70qsYMilbPwp7iUHbnjwJDEDmiBpyZxoVoaH1nTk6OPLbyGZzLo/rV1TD7J2gtyLJxJIgBbhSG64J1fc4ytGlxvt9vCRE/rNLIMVe3d1IjwCp478mpdKDvWnIKv38Af+5f4Nw9EUE2FiUhqjEOQInNTC2x2lm3VFkZV/5FH69gTnHzuRC8l31/VV3YKbl8AfCveSyvSyC1mMqzJLIc///JB/WeO3oDyG7WsWqbkhkcTGShV6iw4VS9Jybs0aj0gJVhCB4ERCZrwZws99H+g9wjB6BjVEcJfA8Jc9Xfh1sgtFPLzeppaifRPPJNOrvEUJAFxCCiD42+yIiSJNJ/pOMoH36iKVC3sdru/3Wf+2gR6Jaf1i+5ExBXZZ8761qJrZj+j/opwTA7fe8vXLq8h0C5fM2NGJJCKNnGodzCu9wZmdbPp6+yCYs5OjtF4T3NZXhRbdIb9t5pjSmZ+bFUActv/LOUnOEFkuwftuov/UFC+S22cMzr/yiCTMshEL0R4ivnYqTYg+FYkipFksqC2X2izw66GLAI78YBB3JHWy9r X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Right now we're using page_dup_file_rmap() in some cases where "ordinary" rmap code would have used page_add_file_rmap(). So let's introduce and use hugetlb_add_file_rmap() instead. We won't be adding a "hugetlb_dup_file_rmap()" functon for the fork() case, as it would be doing the same: "dup" is just an optimization for "add". What remains is a single page_dup_file_rmap() call in fork() code. Add sanity checks that we end up with the right folios in the right functions. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand Reviewed-by: Muchun Song Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 8 ++++++++ mm/hugetlb.c | 6 +++--- mm/migrate.c | 2 +- mm/rmap.c | 1 + 4 files changed, 13 insertions(+), 4 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 64ae6c4d72720..56900a16f41a6 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -211,6 +211,14 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_add_file_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + + atomic_inc(&folio->_entire_mapcount); +} + static inline void hugetlb_remove_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ef48ae6738909..57e8981879314 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5408,7 +5408,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * sleep during the process. */ if (!folio_test_anon(pte_folio)) { - page_dup_file_rmap(&pte_folio->page, true); + hugetlb_add_file_rmap(pte_folio); } else if (page_try_dup_anon_rmap(&pte_folio->page, true, src_vma)) { pte_t src_pte_old = entry; @@ -6279,7 +6279,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, if (anon_rmap) hugetlb_add_new_anon_rmap(folio, vma, haddr); else - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); /* @@ -6730,7 +6730,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, goto out_release_unlock; if (folio_in_pagecache) - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); else hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); diff --git a/mm/migrate.c b/mm/migrate.c index 7d1c3f292d24d..0e912443a18c3 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -252,7 +252,7 @@ static bool remove_migration_pte(struct folio *folio, hugetlb_add_anon_rmap(folio, vma, pvmw.address, rmap_flags); else - page_dup_file_rmap(new, true); + hugetlb_add_file_rmap(folio); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); } else diff --git a/mm/rmap.c b/mm/rmap.c index 261e1af0d254f..a57ec926daf0c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1395,6 +1395,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, unsigned int nr_pmdmapped = 0, first; int nr = 0; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ From patchwork Wed Dec 20 22:44:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E50C3DA6E for ; Wed, 20 Dec 2023 22:45:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E36456B008A; Wed, 20 Dec 2023 17:45:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBF6C6B008C; Wed, 20 Dec 2023 17:45:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C102E6B0092; Wed, 20 Dec 2023 17:45:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AAB736B008A for ; Wed, 20 Dec 2023 17:45:22 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8630D1C0A72 for ; Wed, 20 Dec 2023 22:45:22 +0000 (UTC) X-FDA: 81588679284.02.EFD6A11 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id AF75A14001D for ; Wed, 20 Dec 2023 22:45:20 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gphkSQWU; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TnwuPzh6vqXFM/JzpFan02MNs+lZlbvUqepTeKYzv68=; b=rzCb9AJuLxlOsDrI6T+0782iGK93XmKlhjOyZXAHXSK2beFbElHM/f7c15vq32YwqnxjYn nXpjmcDyq9WjQy/nKCYAlvaK9r/+CpuyCz3yjx2vEd2oeuj43kBeF0J4FNXfaR0xiiBDSv JOv2sOcx9vqJPqPsfA/zg8wYK3iVWRU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gphkSQWU; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112320; a=rsa-sha256; cv=none; b=4mFQSYdxNvDAfyso6GFMdbOawITU07sfl3Ho+BL2I43Ae1BTLSbLbbxp0oWyXq6NiFaA9J nu9Z8iBTM6Vgq5aybW5J1cPU3Cn6UEfXHpzZzA8akZXZBq8e+J5wIN4VpoWFPLkrlBKOr+ BlouNr1DJccNajGcD/HPvrplolUddAM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112319; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TnwuPzh6vqXFM/JzpFan02MNs+lZlbvUqepTeKYzv68=; b=gphkSQWUYkovBVebkr5btzrlqXMfIGtKKKr9E2vJnarZfa8RwejrQSU/flCHtjob2fN/HA YxYRIcSh+erVqATnJ1vcjc6Aqln33Pq+ZAGfJJQ/hvUm3F7M/OauUqrqbAkMarBEoMUdk4 kN6ZjiAyeo7eh2p3/FCUP/qNfLEpZ5w= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-493-jRqZUihhOO6Rc9G1XGUnEA-1; Wed, 20 Dec 2023 17:45:18 -0500 X-MC-Unique: jRqZUihhOO6Rc9G1XGUnEA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B24DC1C41B26; Wed, 20 Dec 2023 22:45:17 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 899DD40C6EB9; Wed, 20 Dec 2023 22:45:15 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 04/40] mm/rmap: introduce and use hugetlb_try_dup_anon_rmap() Date: Wed, 20 Dec 2023 23:44:28 +0100 Message-ID: <20231220224504.646757-5-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: AF75A14001D X-Rspam-User: X-Stat-Signature: zmdopazhza7ihjdqgjknsymhhri4gzyx X-Rspamd-Server: rspam01 X-HE-Tag: 1703112320-920855 X-HE-Meta: U2FsdGVkX18ON7arkRw1jQ53cRNWTuhpkTXuquDsuccu35pzh2mdHV25QaENvVwjht+TyFBXBNa3ewITRnosXjxB2ZdvJrMl985uVQS/eHW9HL6i9RiFGkhqvdpn0UiychfojggryESBcz6tuQbMw5J3pNCvsFRuEZ29zTthLVuU8iwDSaqNkA3gj1E8EH9bb/ZEno7lDok+UUNVWj/n4RmdtbNa+EkhlFZsegv4EI9CpnjSVTndU+sMnQP3L8pOFwJlIKq5x7JKdxDq+fzG5PfG1KZ01xMiyREK4rVxljjNfv/KxzNeRcDq0IjWw2NCGP7Iab5bUnfZyQwttsBrQJdyQTNotV5qPCkTB8oZflFeApG6OxxcMnF7b9SCS+7mX/AtXCmA9y1qqy+RxxcsQfx03cOC6O9I6850LliUQMygFxGRlVCYRTKJXgQAF7DftQVXB6QpQfegcYKtBYskn4poNGD1PTWqPyh0GsouqcRWDVZR+Xy6plJVfdzOG7fv/1FzWLjxxOMpq80DOEBXsSF0kpMomEACjB/IT2AaQ1PhySxQXH1kOObklq2VhG1iY20FbYV6o+iDWU7gn/q/BngxIoKBVKn9iic8MDiztpexjsG6xRKNbMlM5aoYZymAuPHBoAtr533B6RkdjE6slTYYtwGSTjbn6pHSyEyFG/iI5cYz+jacfKAAHNS5rfl2Wx2yrZEpuj6kgReuHrf2is9s8KRCPl4LAbMHqRp/noeyjA0QoaeNnMnzBv+lRSnTojYFP0aZ+Pn8QzCpc9w1FzqWra4TwMuyvC7K5x4GVbPP5vZ15YqeWTYjRgDsrL3Fz+KXPmMjtb09WfqPtfNJlnFxsl7v8KX+XObE8rtsvTwJnD+YCdcWb/zgzTcjBSugUdBKS6nx1EHQOFvnDpy/LUYc33nn0/1CpbAfLrrCPbGMlVCYAJE3uIQKWNfYeijz9xOiAFtuhbYlhUwCCWS DiUqn+nu qUY5ua6eqMy06I1azvylw7b9F+1TkfMxWQRh4dd5Z5mGyGqQ6Uykw1mS1EFZxCnSU8CZbm14o4NTyWggQYXpO0E817I1i8F9sqBxuAJkLYUEwjiLlcWheuK8+b0IrF0MjcGbeJWoqrsXDdSE2GwbI18DWt24VGLT76CwiBa5NPPmR1h/VqTjYvVgLJZ8uPBppDTvSBkYF1GY/YdGv2ufxsQhZ8JYb9d8yotCEf6K/dYLNGcaf6HhE54KuavM0YM03lQW2gli2r2mYN2Fg9wQs8gDgz7IhZ3lq8WJjKCMJCi9hQBA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Add sanity checks that we end up with the right folios in the right functions. Note that is_device_private_page() does not apply to hugetlb. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Muchun Song --- include/linux/mm.h | 12 +++++++++--- include/linux/rmap.h | 18 ++++++++++++++++++ mm/hugetlb.c | 3 +-- 3 files changed, 28 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b72bf25a45cfd..ae547b62f3252 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1964,15 +1964,21 @@ static inline bool page_maybe_dma_pinned(struct page *page) * * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) +static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, + struct folio *folio) { VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1)); if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)) return false; - return page_maybe_dma_pinned(page); + return folio_maybe_dma_pinned(folio); +} + +static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, + struct page *page) +{ + return folio_needs_cow_for_dma(vma, page_folio(page)); } /** diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 56900a16f41a6..5f26752de945c 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -211,6 +211,22 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +/* See page_try_dup_anon_rmap() */ +static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, + struct vm_area_struct *vma) +{ + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + + if (PageAnonExclusive(&folio->page)) { + if (unlikely(folio_needs_cow_for_dma(vma, folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + } + atomic_inc(&folio->_entire_mapcount); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); @@ -228,6 +244,8 @@ static inline void hugetlb_remove_rmap(struct folio *folio) static inline void __page_dup_rmap(struct page *page, bool compound) { + VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + if (compound) { struct folio *folio = (struct folio *)page; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 57e8981879314..378e460a6ab41 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5409,8 +5409,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ if (!folio_test_anon(pte_folio)) { hugetlb_add_file_rmap(pte_folio); - } else if (page_try_dup_anon_rmap(&pte_folio->page, - true, src_vma)) { + } else if (hugetlb_try_dup_anon_rmap(pte_folio, src_vma)) { pte_t src_pte_old = entry; struct folio *new_folio; From patchwork Wed Dec 20 22:44:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D187BC3DA6E for ; Wed, 20 Dec 2023 22:45:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B40F8D0008; Wed, 20 Dec 2023 17:45:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 165438D0002; Wed, 20 Dec 2023 17:45:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E84BB8D0008; Wed, 20 Dec 2023 17:45:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C79FE8D0002 for ; Wed, 20 Dec 2023 17:45:26 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A79FAA0375 for ; Wed, 20 Dec 2023 22:45:26 +0000 (UTC) X-FDA: 81588679452.27.4A625B7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 008B88000F for ; Wed, 20 Dec 2023 22:45:24 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OMguaNlZ; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112325; a=rsa-sha256; cv=none; b=zJZ3oeDI7uxct6l0o8kZILsyw+/riQi145PzT0Y2xpn6RHEiTrM55fivqM6qGqC2rPbO0e a5/JY0s/q4UtjtChp9LG0B8E11YqOWrv0Mw/whjdL7tjilbLp9cD+s2rEf2XgRUaWrV0gN Gwq2BAJwNdMCQIWflZU4m2wz9OIf9Zs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OMguaNlZ; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mTgg+xMXQdacmBIURVO4NXuqVVzQK4rkW7J9Pz1Od1g=; b=jPsKEEKiv/eW9R+KcY35c7hDg5yTzk7r6C7Aze0+4RHsMGVjzOi6PzVXpa1KVe+jLmdDQM K2ueUBKTgrDEphOru2+Zsf+Dt7yZixzupkbv3JRoUEOY3+9xhmJ+72QKJAXSglrZdmfgeF nFGE+mfLRkx/tvEtZIe7X6TMNNyGouE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mTgg+xMXQdacmBIURVO4NXuqVVzQK4rkW7J9Pz1Od1g=; b=OMguaNlZPxU0XYBLdbuBPw1r7WH+hbxY08SPeSPIDoFz2NQ+UBsrQOIUWQ+hcKf37aZrVp T3LH8urMgbir08uwJz7ZNrVwVCW6fJwQQjyhWSyLoVb2cnxDFWfwDo+Cxuze+TKYImrULD QS46K8v99PPdNlps6XqXBn0j3P3I2ag= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-131-WcUjTUD5MXKhfmT8SK5HuA-1; Wed, 20 Dec 2023 17:45:21 -0500 X-MC-Unique: WcUjTUD5MXKhfmT8SK5HuA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 793E986EB22; Wed, 20 Dec 2023 22:45:20 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1E03C40C6EB9; Wed, 20 Dec 2023 22:45:17 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 05/40] mm/rmap: introduce and use hugetlb_try_share_anon_rmap() Date: Wed, 20 Dec 2023 23:44:29 +0100 Message-ID: <20231220224504.646757-6-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 008B88000F X-Stat-Signature: o7dpwq69xjiypj7gi6k9u3gqfdff76e3 X-Rspam-User: X-HE-Tag: 1703112324-439744 X-HE-Meta: U2FsdGVkX181kreEwU8O/diQBRhJ5dKsdSf7CmTFP2dfEyDp8Q6M9eDl67X0CwRLl4vLy4R0hlmn01BeKdBAQBUHJQhwtfKxFgu2xHSsm0EZctIPt5Q4KsUxnRe9MXzqRi0aYOVP9ygJqhABTebq3UkBxhI4gHhWhRf6l8y7xClK9BQ+MGfPveTXZ8gvwAdAZFb9kMYOJRuKdBym+sGmUYwFuq4Yu48iJaVp2PG1WBY4Eh/VGHGC84ubpaiBl1sYBN7pu/HIUuYZYeWNny+bpVoCIenzqx1j/0LuHYcfYaWE8gmb4SrH/E1TzdSOvdqMTDNcH1G3EWAKSJhWwG0TZRpBYWANtkqih5W7uUwxz6yNhOy5FDA0Av7nOH2LDxyQYaTY8Ol+6JkpXB7remgsKBPCYXso8lAeYrHvUmvzFp+CwV8J+xp7JB9Dnf0FNohWt+NSwHDz/tg7lxMjnexfhYZSh2PzExZ4TMIWYwdz5offsdoXprkb1AcwhX4XT9SX1K2jg2T+y9XPOVLug6YHErUQ8cB2JmR1VdY7zbFtfInZqXIr3ayuJH8+9Ot2Xp3kzelUDa6jNiBefXky7lvwDNpFRBQp9jHmmswcQx3RJOuqeCmSQAFpg5s/nAxZA4KBgYwCvZc/alj0s/5wNq3MfhXBiRolNgYZW5qD8cGT9f2UO6t7gaEmZv3kRMHLCyl58JQOVXBVHmKtuGbI0NeZUp9+NoKzM7iwxA3ek4qKK6ilmSEpfBhxvMKvi/P5ywxgGdNh/UqdDdGz/+FV1io0X/KlniCzlmp7O1m2yTuoJVo2niYYtcWJN0FbBzjRqb586m/Eh5TGnldYjUrYt4AOOXYpevFBtvp7NfkI2highZG86ZN85S8fX870y2SMyYDtQ6kjoGH9cq7ogicCKxjXRrMppCETwJKyQmxppUHmukMPWbXIbgaq+ED0imF3VDbnlFtTujBS3dm7TQvjQqc 2JOgOpJY YOFHhcbtJ3qoVtGAQvHdMAUp/nV6u4j6TB+hFhR5T+mgTlqjlleS+GzI/a06q6s23WtThRTP8HzKbBslC2wLwOCWCaZa05LiOmf4im1r323I8mT6FC0tPdN428UaZeUZEFexKDwJVScRGXA2m3JdvWbljqIn4RCFvMEdpOS5ksKPot25V2GV+sNzCBNrMTJn3h6aq+mOvLHj7lansLmQVr1C/bAvtLAtJHSHMX8wwog9vRjGbR6yxwnTA1wHDPiHhG4q+9PXMtxlEiXaSjiggRX/i7Ex1FPuWkpv9Ayzt76aBPwU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Add sanity checks that we end up with the right folios in the right functions. Note that try_to_unmap_one() does not need care. Easy to spot because among all that nasty hugetlb special-casing in that function, we're not using set_huge_pte_at() on the anon path -- well, and that code assumes that we would want to swapout. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 25 +++++++++++++++++++++++++ mm/rmap.c | 15 ++++++++++----- 2 files changed, 35 insertions(+), 5 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 5f26752de945c..d6fefa0f04105 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -227,6 +227,30 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } +/* See page_try_share_anon_rmap() */ +static inline int hugetlb_try_share_anon_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); + + /* Paired with the memory barrier in try_grab_folio(). */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb(); + + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + + /* + * This is conceptually a smp_wmb() paired with the smp_rmb() in + * gup_must_unshare(). + */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb__after_atomic(); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); @@ -331,6 +355,7 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, */ static inline int page_try_share_anon_rmap(struct page *page) { + VM_WARN_ON(folio_test_hugetlb(page_folio(page))); VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); /* device private pages cannot get pinned via GUP. */ diff --git a/mm/rmap.c b/mm/rmap.c index a57ec926daf0c..c229e48cf5a9e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2149,13 +2149,18 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, !anon_exclusive, subpage); /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { - if (folio_test_hugetlb(folio)) + if (folio_test_hugetlb(folio)) { + if (anon_exclusive && + hugetlb_try_share_anon_rmap(folio)) { set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } + } else if (anon_exclusive && + page_try_share_anon_rmap(subpage)) { + set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); break; From patchwork Wed Dec 20 22:44:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AC16C3DA6E for ; Wed, 20 Dec 2023 22:45:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C38048D0009; Wed, 20 Dec 2023 17:45:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE8618D0002; Wed, 20 Dec 2023 17:45:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A89F08D0009; Wed, 20 Dec 2023 17:45:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 92F468D0002 for ; Wed, 20 Dec 2023 17:45:33 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6B5B6405F1 for ; Wed, 20 Dec 2023 22:45:33 +0000 (UTC) X-FDA: 81588679746.03.67B5BEA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id BC04FC000B for ; Wed, 20 Dec 2023 22:45:31 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dxdtD0aa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112331; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w6z2XPcaQaFkvw9bopV7cZh6Qr6jUgqju9ezjmrboLY=; b=4LIuW3azXFeSucUUkQYYCimdFuLlfBhzYvaNcyKYu0Y8V0+WvpgAFiOkl22vhpL7GO1nk3 q0rl9YcOs8f2XHlNAM5UTSLCfTt1UBM7u+FlgOmC0H9J3eGDIefP6lRFg/GGKQXHpexhPJ gSsdwa4MSwFHAEfX5lM0MnaXpEvfsV8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dxdtD0aa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112331; a=rsa-sha256; cv=none; b=EfCfXTus2qvKZHC1fFVsqd9IbDbIC8kMp5kJdoeamNwD8G49DiHgiBgbW/q/AEBiTT5GEQ oSS6QPX/7K6ygAm1ZVHY3rGgBfUVvIuG7QOCU5FedwqIvrDSnimuEIeXTNZ0ZRs3pZuTsz 5N54VMeZK3iDUByRi5M7BVQJeqRQdj8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112331; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w6z2XPcaQaFkvw9bopV7cZh6Qr6jUgqju9ezjmrboLY=; b=dxdtD0aaP0Gyfa5debhNBaH4AfI33zUXBYlDRxJJU9zYhN9chLUQ8iRZaEOrFJYtJtnvm4 oicsogUf/B6v3WQCK9UGTB3R2/kp+rHd2VhYajuZNwQV++q8eZzHcgBJlGRMT6l1c3aYBE XbGWAOnUPUahE4lvjjHfvme3GtoP1r0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-266-ykSjz5bzOj2Bw6ZSVoP9RQ-1; Wed, 20 Dec 2023 17:45:23 -0500 X-MC-Unique: ykSjz5bzOj2Bw6ZSVoP9RQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 122D5848C03; Wed, 20 Dec 2023 22:45:23 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA27740C6EB9; Wed, 20 Dec 2023 22:45:20 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 06/40] mm/rmap: add hugetlb sanity checks for anon rmap handling Date: Wed, 20 Dec 2023 23:44:30 +0100 Message-ID: <20231220224504.646757-7-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Stat-Signature: k7yc4hrainh5gsbxy13jcmgncdogwde3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BC04FC000B X-HE-Tag: 1703112331-454551 X-HE-Meta: U2FsdGVkX1+OOAT3WKYz14psZ3aXybHJy8REdEEK8KYFQebRf7aSqkFTozYDbi6Z2HD2hlNWOvahybNAVV0uqX8n6bHlCsI1T9NKCtg/apOaarYHnZCmhrm8SpSFd5WE0C5Yilq7P2bqgrDZp+viMvSN2WfN+ja/jy9zmyj++I828VUfzHxmI8XidtOfhn84jNCv1PfR+y17IR66/UdauzYDJy7e8Hyxv3glgYOYFvNQX2hYSyQyT5Mj7sZlaseI/88GPILzRIwdj24DaZ8QqMY7XmFfUqB/Gwg9syMJaHbf4HgEACq5hn6oQrBXZV4468LQJK0dOQ2nGHz+/hULhvtRcv9DK+8mNNLhzDb2SnAvpBgGKkHSoHAJKXvEgNVWuugoHiaIWrcmsJYOI0x9nwfiWyot7XkhhkyIcwa1Lqh5DY0KiduarUHtvBqmef9gQhLq+Z6yg2zPQEzWDNRcUuhEV71CCwS+zL/U467flFycGK3G4N0MULD28VerEtH3p4QStcxJaWAG2s+B7319F9l+SWAJ9Tb92u+x+BK7ObEm3isAvsucbcyTHHgddT/3TBzJyPcqkeYmEmB6TtfROOAc20C+N8ebYz4cVdA1Ul5FPdx3pFTY958a3HWRWg9wU9J8jO8ZNPyChmsP7QFlj+Lo/TWK/JVCFX3xNPUn/rqy/QXmUINcMYW0voMVwYcTc2xDoqxNcEW4PLgAijwcLyKmjPdxMWCJceUjMk/nU+nymRwjIwZVobfNoF2aXa+jKvpFq2fkOXGbEjQQUwobuEALt70JNgaL3BMVRtUd8p77hejo+3er5TFX+16zDescR972GoXZCGrp6E2sEIjgxPXGbYMnQa0ZJIkJSUEHrIpgPZo4rVyY97E1hIZadfMyCaFHAU6dn2uuHmKFrGnWwxt/wNEBMTaxgVW84U77Ml8RZvAbgiekitCLKVOtpk4QkOaTJYSt6Yw5aTTdkjk a0rOLqtr koKkxGj2GjaNfGdowACYNOY1uRoFFRn9doo2ecQEluUyNzOWszL/3MmWVuXIQcLoA1nA/b7L2/7LlbGhdWlTGbPwQkIELRd0sjD07Ld/dJ8CzN/WA9IrvnGQDaNAUfzX9ke06lBxOm4+MBmRe3Mz3Q3Q5ANa6z3CEgADMxMwAr6yW3kNMeP8oSTkl/e3ZqBf/D00WSER/IO6Y56sJ0j8wbg7uUaMTqsI0GS+lljKE/ZtkRqI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's make sure we end up with the right folios in the right functions when adding an anon rmap, just like we already do in the other rmap functions. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/rmap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index c229e48cf5a9e..6a1829324053e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1262,6 +1262,8 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, bool compound = flags & RMAP_COMPOUND; bool first; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + /* Is page being mapped by PTE? Is this its first map to be added? */ if (likely(!compound)) { first = atomic_inc_and_test(&page->_mapcount); @@ -1343,6 +1345,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, { int nr = folio_nr_pages(folio); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_VMA(address < vma->vm_start || address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); @@ -2634,6 +2637,7 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -2646,6 +2650,8 @@ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, void hugetlb_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); From patchwork Wed Dec 20 22:44:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E73E5C3DA6E for ; Wed, 20 Dec 2023 22:45:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D03D6B0080; Wed, 20 Dec 2023 17:45:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 754A48D0009; Wed, 20 Dec 2023 17:45:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CE1F8D0002; Wed, 20 Dec 2023 17:45:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 44D276B0080 for ; Wed, 20 Dec 2023 17:45:30 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 269B01C0BE9 for ; Wed, 20 Dec 2023 22:45:30 +0000 (UTC) X-FDA: 81588679620.28.8D26825 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 882D640005 for ; Wed, 20 Dec 2023 22:45:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RkSxryl5; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112328; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KK8FwY1ijiDBkQheEy+hwW/ONUEObbiexE1hMATnzWI=; b=BmsNLgoxPE5Hdp1E4jxbgYEw3sNGW1EWAkG7gxU69otUVjcfxJFPX6EBv89nyakLZ3X15H GhH+UpCW7lnBMei2rkggevDy8+XHzMNzqn0xmvItfN/R3+q6g60aaiO8qmf5NeIC3yiwH5 9gZei5A7K0xTJFb8VwJ07bbbvKZcJC4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RkSxryl5; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112328; a=rsa-sha256; cv=none; b=wf5cHlVdV2MO1PgTfSapMOpdLwbhrIUa/PeVbbhlROU0901W+9slemdzVJZpb8AfDYBL/C ZBkdTz3R/PSauqvvw+UnJMMyxHBmPUTTwdfJ6j8txvQZA7w3816hTaY+KgALrZrx7RJg1A Pnysbk91+wfvZLCXBHxh5fgi+X5+xRM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112327; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KK8FwY1ijiDBkQheEy+hwW/ONUEObbiexE1hMATnzWI=; b=RkSxryl5WLdpyX9cPbNQLLGvdWnt8ZJWtaMuYuShOZ+KVi+wtABI7QsmuZJR2cIq6MoYTU zbDL8Qv5ysa6WDnd1FDUWnLpIQ2ra7mUdTc0/lvxagDug4b2tsMJ7kvkcBIAugMoJiOUmq sddFHzqp7L9Ul5IVWvsbtF6j8Uj2pZE= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-438-rNvmpqCaPpiEFQBUirp3Jg-1; Wed, 20 Dec 2023 17:45:26 -0500 X-MC-Unique: rNvmpqCaPpiEFQBUirp3Jg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7D9911C41B25; Wed, 20 Dec 2023 22:45:25 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5448740C6EB9; Wed, 20 Dec 2023 22:45:23 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 07/40] mm/rmap: convert folio_add_file_rmap_range() into folio_add_file_rmap_[pte|ptes|pmd]() Date: Wed, 20 Dec 2023 23:44:31 +0100 Message-ID: <20231220224504.646757-8-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 882D640005 X-Rspam-User: X-Stat-Signature: w3mpy1jo5hgdbf95bxttbdx51hd36xab X-Rspamd-Server: rspam01 X-HE-Tag: 1703112328-537331 X-HE-Meta: U2FsdGVkX1/wPYymLd0MCk9fkq9YJ6XKHZrWBuBGkGBRdu0JrmbQnBGtRcIYRAbhYNC6ZCY/wQ4MQaR2glFSYOXWs5/aBlt8DX+iHuY+JkKtI0CHyAMkLcZK9OFHq9DbKoCa8KuFpHiCtO2XT0AXbbT4vX34nTL12WARiiVSJ7g5N6F4djLCkyvUh2Aomrp1DiDYLcC2DS539f/KZI4T6+ud89L68mE6fU7Y9XSV9hqKbVDZJMgK0bNlpgVdcFSCNQTIoQrbIxbFVoXvxPPTX/ibks7bPR6ZbPDCAzksnHjTw++imtuKFrn4tXAfvJ9xOpZlCIoBKXCj0moY/jbCDGjmGDcoi/0ELz7GMLu/1Md0pHf7kOJ63jO0DJc9/t2JadrWakexBDa2ZDrr71FD8kjrpfv9OcIGk2mi0pR52NVeNxuSB2/y49kJMQtXUB554DZqJvXh6ZHFoQDQTlggFiWZilWA00yk4d6O3bMgI5jCrs85hsvnqOhLsAtc32ZEve3tXh3geuZDqlAP4wMFojWMW1HRK4w5jYgEoJ2ah9ceSRt2iMLCS0nCkv3Q2/D5w/BPZNYEqVVQG9px010oQJykuzGau6EKJTma8Z0g/NV0lGX+sXPVCI6bmPdTRgHahNjGnUm2uv2/KTaYzbU6+tWf2qFzNhp4x56ommNbDlD6YzAOCiM7w+fchNw63Zn62O8YzavFTzp/eceDAdWqnoBROm0ytMf+9ozESXEf0F4X5xB7xJj8ErTxDlB1Y8es01CG0yFvh9l5RCC/XPAjJnV8nXei4VfnoD/HnTIClS46iz/RVUH0FOmyqDpaiaH3jn9CGPw5pFw776/7RK3PUpNpUKYJybYENX/TOsvLwAGRkKDkMZsmCj6LR9zP+0QSJS3fG8Jx8hk/2JqKBnMVLkOgZWROnsMaTvwAQKevv1BYpcwCmvm/kr35UNIxSDs1cxmHezbUySt0xeW05g5 wKmzR4FF GYQAjnkbKQpAEh1k2dTScVJCQQLZtNLja50kbSNpQYWohgnrP0h7hrm9JNMtQAD1/11lBXevTk2aiwv9Vv/2JRZKDui4HM25yRBgwc2EfmlIgoNWtK0TO9qfvp+tMEArAMcM9mZo6QU44ino1sMce+UWZUfctleI2lbOM+dL3OYj44Taal5SfgfCAX4ozlvTTUtA/ZU/mVVBtx5nd6Bi6BHp/sMeKBGJ8ixSAPBrLjl4P+2iZsI0dXmuir9w6hs3ONfHxuYL7xgk4HQlXx67TCjipuhVNMXOvkaZSqXiA6DA8EAs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's get rid of the compound parameter and instead define explicitly which mappings we're adding. That is more future proof, easier to read and harder to mess up. Use an enum to express the granularity internally. Make the compiler always special-case on the granularity by using __always_inline. Replace the "compound" check by a switch-case that will be removed by the compiler completely. Add plenty of sanity checks with CONFIG_DEBUG_VM. Replace the folio_test_pmd_mappable() check by a config check in the caller and sanity checks. Convert the single user of folio_add_file_rmap_range(). While at it, consistently use "int" instead of "unisgned int" in rmap code when dealing with mapcounts and the number of pages. This function design can later easily be extended to PUDs and to batch PMDs. Note that for now we don't support anything bigger than PMD-sized folios (as we cleanly separated hugetlb handling). Sanity checks will catch if that ever changes. Next up is removing page_remove_rmap() along with its "compound" parameter and smilarly converting all other rmap functions. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 46 ++++++++++++++++++++++++-- mm/memory.c | 2 +- mm/rmap.c | 79 ++++++++++++++++++++++++++++---------------- 3 files changed, 95 insertions(+), 32 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index d6fefa0f04105..3d86a76b28368 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -191,6 +191,44 @@ typedef int __bitwise rmap_t; */ #define RMAP_COMPOUND ((__force rmap_t)BIT(1)) +/* + * Internally, we're using an enum to specify the granularity. We make the + * compiler emit specialized code for each granularity. + */ +enum rmap_level { + RMAP_LEVEL_PTE = 0, + RMAP_LEVEL_PMD, +}; + +static inline void __folio_rmap_sanity_checks(struct folio *folio, + struct page *page, int nr_pages, enum rmap_level level) +{ + /* hugetlb folios are handled separately. */ + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + !folio_test_large_rmappable(folio), folio); + + VM_WARN_ON_ONCE(nr_pages <= 0); + VM_WARN_ON_FOLIO(page_folio(page) != folio, folio); + VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) != folio, folio); + + switch (level) { + case RMAP_LEVEL_PTE: + break; + case RMAP_LEVEL_PMD: + /* + * We don't support folios larger than a single PMD yet. So + * when RMAP_LEVEL_PMD is set, we assume that we are creating + * a single "entire" mapping of the folio. + */ + VM_WARN_ON_FOLIO(folio_nr_pages(folio) != HPAGE_PMD_NR, folio); + VM_WARN_ON_FOLIO(nr_pages != HPAGE_PMD_NR, folio); + break; + default: + VM_WARN_ON_ONCE(true); + } +} + /* * rmap interfaces called when adding or removing pte of page */ @@ -201,8 +239,12 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); -void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, - struct vm_area_struct *, bool compound); +void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); +#define folio_add_file_rmap_pte(folio, page, vma) \ + folio_add_file_rmap_ptes(folio, page, 1, vma) +void folio_add_file_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/memory.c b/mm/memory.c index 149f779910fd5..7f957e5a84311 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4515,7 +4515,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, folio_add_lru_vma(folio, vma); } else { add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); - folio_add_file_rmap_range(folio, page, nr, vma, false); + folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); diff --git a/mm/rmap.c b/mm/rmap.c index 6a1829324053e..cc1fc2d570f0d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1378,31 +1378,18 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); } -/** - * folio_add_file_rmap_range - add pte mapping to page range of a folio - * @folio: The folio to add the mapping to - * @page: The first page to add - * @nr_pages: The number of pages which will be mapped - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The page range of folio is defined by [first_page, first_page + nr_pages) - * - * The caller needs to hold the pte lock. - */ -void folio_add_file_rmap_range(struct folio *folio, struct page *page, - unsigned int nr_pages, struct vm_area_struct *vma, - bool compound) +static __always_inline void __folio_add_file_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_level level) { atomic_t *mapped = &folio->_nr_pages_mapped; - unsigned int nr_pmdmapped = 0, first; - int nr = 0; + int nr = 0, nr_pmdmapped = 0, first; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, level); - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { + switch (level) { + case RMAP_LEVEL_PTE: do { first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { @@ -1413,9 +1400,8 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, if (first) nr++; } while (page++, --nr_pages > 0); - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ - + break; + case RMAP_LEVEL_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1430,6 +1416,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, nr = 0; } } + break; } if (nr_pmdmapped) @@ -1443,6 +1430,43 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, mlock_vma_folio(folio, vma); } +/** + * folio_add_file_rmap_ptes - add PTE mappings to a page range of a folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages that will be mapped using PTEs + * @vma: The vm area in which the mappings are added + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + __folio_add_file_rmap(folio, page, nr_pages, vma, RMAP_LEVEL_PTE); +} + +/** + * folio_add_file_rmap_pmd - add a PMD mapping to a page range of a folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_LEVEL_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to @@ -1455,16 +1479,13 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); - unsigned int nr_pages; VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); if (likely(!compound)) - nr_pages = 1; + folio_add_file_rmap_pte(folio, page, vma); else - nr_pages = folio_nr_pages(folio); - - folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); + folio_add_file_rmap_pmd(folio, page, vma); } /** From patchwork Wed Dec 20 22:44:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14F63C3DA6E for ; Wed, 20 Dec 2023 22:45:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3E718D000A; Wed, 20 Dec 2023 17:45:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F0798D0002; Wed, 20 Dec 2023 17:45:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88FB28D000A; Wed, 20 Dec 2023 17:45:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 74F058D0002 for ; Wed, 20 Dec 2023 17:45:37 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4672716010B for ; Wed, 20 Dec 2023 22:45:37 +0000 (UTC) X-FDA: 81588679914.29.51AC44F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 980D410001D for ; Wed, 20 Dec 2023 22:45:35 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="dHZtPhJ/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/7RNQidug/EwvaArOE9sRzRSaxqKYQznxGvOG5hZ9FI=; b=pC+f7noyrGQSCxpYanIWO7sCdGOjDSdKGAnLgGR4f7XK58pmHx1Gmt6uQ5gTE8Unex376B XIUp7phjWeGEIrDwwWNxE/T6UDFlYP1TWkWfju8abHfuR/XlmxL4qKYaM9pR42iwoVwL0+ of7Z/oa7MjwIq5x/1QK8fBkKxl5IL5E= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="dHZtPhJ/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112335; a=rsa-sha256; cv=none; b=TbEDzhZF8uMI9WHzOW5Y6EJfizGK2dyIGaqZES9otRMhfUOocMIij5AiEI1Du4PjHcmjmN MX0m8Bi7zXmfaT+v2WAwE/5206wx4RPTliiVcmYO77WRqoiFW2XZgLX3c8w6kvY8J6JYM+ ZIdVucYC1cTrnIrB7kkEpeiKgM5rIkE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112334; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/7RNQidug/EwvaArOE9sRzRSaxqKYQznxGvOG5hZ9FI=; b=dHZtPhJ/VlmSGWT5DKbZrqOnyDcWPI+bc9c5cGEGr9kiNtb8E7NAef4Cg+Z9eJC3xpHMyf Pk54gMBMmWMsDNIPm6a98e3vzZoG9tQcuclupuhkLBX9UWL8IokGuqEy776skIulsMbqtt jhv1e5zY0DU4ax0mKRMkQ7feIB1dDog= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-97-pT_9TgFiNeKKCpu7Ia4-ig-1; Wed, 20 Dec 2023 17:45:28 -0500 X-MC-Unique: pT_9TgFiNeKKCpu7Ia4-ig-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 11CCF1C41B24; Wed, 20 Dec 2023 22:45:28 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD23A40C6EB9; Wed, 20 Dec 2023 22:45:25 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 08/40] mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]() Date: Wed, 20 Dec 2023 23:44:32 +0100 Message-ID: <20231220224504.646757-9-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 980D410001D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1i4yjwug4z8xh5a3b8gfaonu1hytqwzj X-HE-Tag: 1703112335-505158 X-HE-Meta: U2FsdGVkX185PGV7d4SzjL/7r9SGk/PIOcfMVQn8bGCK6GWLNBFVvOwWJX2QfHDMqv+EIvRSpehGBlNTruupPu0QN8syJMdvjKQF4mepCfEdmIAHUOG14XXOjrQGKVvo/eu4nKdu/mjO2hYO3QpcSA1DJLWWyVzNCUBFckHE2HsiFamk/5t+v7QVUtbruHk2pNGHEsDZmbl9w1WRNgJdyaGc+xbXx+i31qzB7971HO0VqhXI+Xi++x+hhLwi5hz2hgGGkqBsSvYzge3NyUqv+hh4qYcyZlnQPH0zlk7ibA408ZJWU9UdGgEjEfMXLHfjqoU/qQMPh87TwqmG49Bbdir/EfPHiezXtFkHmpDtpFubcgZIg9osFnmy9D98kJ/Qk4YbvfTKHcXwI1apE5nfI5niJGDiMvhLUXSvZA79Kk/hkH3c4RQUahe8F8DLvoI+H7oB/+H/e7aanb684vRzusGOfSkU7yC93EMDxd0sSIJGw1h0JM/FhE9DWwYM9MP4T97+AiaYKa9325CjpVHCvHhSeMcbWQPl2CZ8Puwd5lzvx5sgzFzSZAkhzvchiIgSiZPXJwIf2L64TmkyXtYkVegI6Vohx7Q/B8nYN7JHmTSk28KhdBW0MxZQgJ12QWDoY5mHFRTTyNEUtPbCSBybqUJZf3bxQ+0hBouAoNBKsVaKx6s3JO4+7PZk0Ky/9kpTj+bNd2wbFRB2Pf8yfEuDZjXFkjGoPKHYCMfztou/bpZ39VFMBKSUZWscjG38BtFeVxHfVTaS7yU5y8/t0lVt8zf0xk0/NN6ysk+ymZ3tyImJF+P2sAwfvnvrQJ8arKei19lJnU2XFEJH/CZXzQ7g8P+8VyZX1x9mLk2Q7yKI12Rf6z97ax6KVwKECDUEg0t+6LD5pS86GlnB7M9vdO8rPAjJZo3uV5TrJIbeVM2/4atyftKTtJO3q8U/YeIqbIi0t6YtlFvHmWZqS6oy3yU Lqw/uZ8n mrjW4OxifKuq4vzqCbhx2hX8gLkmRGfwtRFQ4fLy+HleNR6AFhY+J45E0ffZs3dwIJ4MhkM64K5uWmYS+knLeDrD//TI/Kn8rmIWspOoL6yU4GgBVCxbfKbS+KxSPBgwgLGRFifjWW2fXZ70WhNw0qOdX0ke5UCx3MRnqDgK2dGqCnOspXiaiHmhwtwg9KCgy2cTp8cEzTZQHDcs5viCV9tm/kVGiGsf5h7els/VkR/PD8dy4XqPs3hFh7sjiGcu4njj6pkSn51ongylr+Ma36i+BY4f2D0843r/LIPEaPRhtkOE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert insert_page_into_pte_locked() and do_set_pmd(). While at it, perform some folio conversion. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/memory.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7f957e5a84311..c77d3952d261f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1859,12 +1859,14 @@ static int validate_page_before_insert(struct page *page) static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, unsigned long addr, struct page *page, pgprot_t prot) { + struct folio *folio = page_folio(page); + if (!pte_none(ptep_get(pte))) return -EBUSY; /* Ok, finally just insert the thing.. */ - get_page(page); + folio_get(folio); inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); + folio_add_file_rmap_pte(folio, page, vma); set_pte_at(vma->vm_mm, addr, pte, mk_pte(page, prot)); return 0; } @@ -4409,6 +4411,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf) vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = vmf->vma; bool write = vmf->flags & FAULT_FLAG_WRITE; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; @@ -4418,8 +4421,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) if (!thp_vma_suitable_order(vma, haddr, PMD_ORDER)) return ret; - page = compound_head(page); - if (compound_order(page) != HPAGE_PMD_ORDER) + if (page != &folio->page || folio_order(folio) != HPAGE_PMD_ORDER) return ret; /* @@ -4428,7 +4430,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * check. This kind of THP just can be PTE mapped. Access to * the corrupted subpage should trigger SIGBUS as expected. */ - if (unlikely(PageHasHWPoisoned(page))) + if (unlikely(folio_test_has_hwpoisoned(folio))) return ret; /* @@ -4452,7 +4454,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); - page_add_file_rmap(page, vma, true); + folio_add_file_rmap_pmd(folio, page, vma); /* * deposit and withdraw with pmd lock held From patchwork Wed Dec 20 22:44:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26FEDC46CCD for ; Wed, 20 Dec 2023 22:45:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 122538D000B; Wed, 20 Dec 2023 17:45:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07B638D0002; Wed, 20 Dec 2023 17:45:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E36598D000B; Wed, 20 Dec 2023 17:45:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D13D38D0002 for ; Wed, 20 Dec 2023 17:45:38 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8FCE516093D for ; Wed, 20 Dec 2023 22:45:38 +0000 (UTC) X-FDA: 81588679956.23.1D3E0A6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 142F58001B for ; Wed, 20 Dec 2023 22:45:36 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZGjQPSJ1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112337; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ofk61e3x6rLiqTTNkmXQTXIg/O8hfjxoFIvAzETbtQ8=; b=ppCTBAojAmfciCvzjvOD+3EIrSTymlfilYEZZJoHIYX9KERphoNuOryiqUsemcMO2dIWLa OH7sAWJdmHQiVhHc55OBtMarS8zVOacTPDbeuIxIByO2QCbqXYiZ39LzPxldLhxO8PGGGS DE5pKQLl1Bxz/RKGaBwPF2air61arp8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZGjQPSJ1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112337; a=rsa-sha256; cv=none; b=uSfa2jlJydDr4xboXNWDg1lPeYmk9TtLhO+adJBYnrf2z2hk0cYcCvMpdRGJLPKRHf3XeJ FlgeM1bKgY4N0v0ezxNvkRoJqEH3yvlJE5xULSAhR5M4Z63ijq/tlYKFtE/bJEA3H1IU6n ELpTURquB/+lIvzg4MdJ7VSpEYAzRSo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112336; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ofk61e3x6rLiqTTNkmXQTXIg/O8hfjxoFIvAzETbtQ8=; b=ZGjQPSJ10g4nK5li01+ZvZOUDCVnFJ/kk8xNCgtlOQRndsyOHQU1JE816/1GuVy+pZEVCQ amPg3Sy3qevQ7gH8OoVc27t3Yb169vGJZ4YUmbj/5vjlFyerbl5IIzMPLDxbko4Q0bzqWt ot23GFIeIZ0mkCLxf67CH1UWP363EKk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-179-veEaKfenML2oj5fpTRrRZA-1; Wed, 20 Dec 2023 17:45:31 -0500 X-MC-Unique: veEaKfenML2oj5fpTRrRZA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B99F863B83; Wed, 20 Dec 2023 22:45:30 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 611A040C6EB9; Wed, 20 Dec 2023 22:45:28 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 09/40] mm/huge_memory: page_add_file_rmap() -> folio_add_file_rmap_pmd() Date: Wed, 20 Dec 2023 23:44:33 +0100 Message-ID: <20231220224504.646757-10-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 142F58001B X-Stat-Signature: roynph1bdd6iedk9j8woz8witkf4yitt X-Rspam-User: X-HE-Tag: 1703112336-153536 X-HE-Meta: U2FsdGVkX1/8gF7S+ztUDsY/lUX13CRB/FbLuxBiFltIQ828mdQs7oOS0r+P1qWKS22PZUWTN8MKkcuSIBc/FA7dgNW1z8WDR0aZtyrydt2xS1k7VpkvnzmMgjMVkpjDhpXw+uq7vnclqXNsc9G4VGmIh9zrD2bQpdjyrc94najV+WsmI57a8qGrDk7C8qLI+7koQCAD9DhJfMoYfuecsqY6u34jlkoNhoZnm0SJ+fanpxWUf6PAAFNc8jU7FPu3XhTZohNbhr4cLckKOk0oVnfLtAIDRo6T6zYaSAQA+bXdvdZwmr6oGL73rqwFXtqxmRp4Cw+7BHS0M7E6ZWtm4bEUCqablf4CnBAuHzGX3Ol+B/VnOWBEmPcN8cfiQAUQDL5qsscbFN+E9JEgv5OADkUrkX/6ElMYDEi9H+TXjI9P+YzCha01N4ea3b4xboHxVRnvaeXMuE6QmuACVmrIJcC8uoY9WoiYH/gP1cm6Zt6Ub/onjGhbP3PuiJwHnZGwPCoOSd+4Dj+byEd723syXd/1x/ImKl+5DkSKvYNe56k+pkH1BU62YHIqQs6Hv7ujmvA/I9rypviRJXs96nS6VVH+zP8zSdcv32ry+mjMtYDmiJev/0dd+K6eF+FkLqGF0+Syo1GESAALPF6hLS3IHxBh+7DBDQBPT0pXvQnyJdsWJBsKD13e/+uzMXeI66+EBFwAz3IfjvRQnxTWrmxUBbDYp5MvmbWWLcHumg5ScFpKWrqec8Iyd73iKKKLBt/Y89LiCKklYAr2Yd8xJ4LhB6kY9cBnYfPmWXvhb6tC+7p04omP/gYvjKxPkGhI880G93peOjmE/5A8yJZsq8jmOfASzNH4Y147tw8INJbjjKmymqSKNOYhPpUYCRsqUvlb4wq44vmVT5kp/MbTuZW2j/rwuaG0HjbkZgB8gIQmAst7YcJv6j73W3jEqqKTUeDH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pmd() and while at it, perform some folio conversion. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c50dc2e1483fb..bce6f987f36a3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3577,6 +3577,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) { + struct folio *folio = page_folio(new); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3588,7 +3589,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) return; entry = pmd_to_swp_entry(*pvmw->pmd); - get_page(new); + folio_get(folio); pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); @@ -3599,10 +3600,10 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) if (!is_migration_entry_young(entry)) pmde = pmd_mkold(pmde); /* NOTE: this may contain setting soft-dirty on some archs */ - if (PageDirty(new) && is_migration_entry_dirty(entry)) + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) pmde = pmd_mkdirty(pmde); - if (PageAnon(new)) { + if (folio_test_anon(folio)) { rmap_t rmap_flags = RMAP_COMPOUND; if (!is_readable_migration_entry(entry)) @@ -3610,9 +3611,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) page_add_anon_rmap(new, vma, haddr, rmap_flags); } else { - page_add_file_rmap(new, vma, true); + folio_add_file_rmap_pmd(folio, new, vma); } - VM_BUG_ON(pmd_write(pmde) && PageAnon(new) && !PageAnonExclusive(new)); + VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new)); set_pmd_at(mm, haddr, pvmw->pmd, pmde); /* No need to invalidate - it was non-present before */ From patchwork Wed Dec 20 22:44:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D224C3DA6E for ; Wed, 20 Dec 2023 22:45:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73AEE8D000C; Wed, 20 Dec 2023 17:45:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BB258D0002; Wed, 20 Dec 2023 17:45:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50C348D000C; Wed, 20 Dec 2023 17:45:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3BAB78D0002 for ; Wed, 20 Dec 2023 17:45:40 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1C39C80783 for ; Wed, 20 Dec 2023 22:45:40 +0000 (UTC) X-FDA: 81588680040.11.D6BBF83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 6E35A4000F for ; Wed, 20 Dec 2023 22:45:38 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZY38VdMe; spf=pass (imf17.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XIS0gsmH7QK9fc1k9Ch7Ppycbt3c49uXYdCQJR3Zfvw=; b=Rx+ttLOaryzAhUK8hjfeCkPPS9gtKHy8asajI+q28C/CAnJxiJJD/kiu1URFu5vqqjoa3m uYotrR2Ya0Aufz0Q0UcSk0XHcdBR/clecQ8GMol35ApE2MRemzMH9q59E5B6Vyg4ekCBKF MPVnHjdUwALuIRfdOD7KPtWdDC7AdIA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112338; a=rsa-sha256; cv=none; b=DAc99UDsYEaly37jAMnVo++dvxyRDvjNcFtcCLN545tqRIPLwfFKKfJ2Hx4yHii9qvHXcR XGOpgOZLeO2kl3vcLdmi0u6F/1mSO590SDmEai/5qhfZ13ZEdoUTpR6ApKrfGlZiUD1M3W jdYpVdyTDcQUufflvrx46IRI/dCZIDM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZY38VdMe; spf=pass (imf17.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XIS0gsmH7QK9fc1k9Ch7Ppycbt3c49uXYdCQJR3Zfvw=; b=ZY38VdMe4CXZh+jR+TVGUG53SnRTC6PimuGtrTk5BCu/olYe1hds4IDAwxzq+Dk5ElEtFG s5p/2nBjh/S64o0BhRB7+PADsYcZKslAn4DT1KJuQzwaW84Fx9fkFwsxAV7pbmy9f34jHj utd7xcnREAlfIpKvVzTxXxruo7U0gIA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-686-kA5itlGvPZq2AHOyuzUQEg-1; Wed, 20 Dec 2023 17:45:33 -0500 X-MC-Unique: kA5itlGvPZq2AHOyuzUQEg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6099A1C41B26; Wed, 20 Dec 2023 22:45:33 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0707240C6EB9; Wed, 20 Dec 2023 22:45:30 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 10/40] mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Wed, 20 Dec 2023 23:44:34 +0100 Message-ID: <20231220224504.646757-11-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Stat-Signature: cpjcuyfod8ff86sfy54rpbyzcqhbx39k X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6E35A4000F X-Rspam-User: X-HE-Tag: 1703112338-868482 X-HE-Meta: U2FsdGVkX19kGCn3iTYbpmyj98JgtdOpSpJGi8FvHJMIQ/CUW2JK9WqgdvuEeJyvBg1PeGIHkfHndxPK2KdsJBeWoH2arix4QdZH0hbzeRaSVXo62/ke8BlMbBqFcO+i+Q+dOAIVAeHMdYGH2+Nq12GydeipcUl+kZOF2e3HNVEzjnhy1LmOopK3ORzZnfU4qQKzX9HSpRJW4q5OQUR8i4PlEmR/ejW91TvrSGlp7Lt/67Sb6tcnXuIcmrKqIwuZ4vV1tZCkUc6htAWMEMUY175VqmZV1Pr4BYxeqE9emhuRjR1Wty5B4F4wqG6NV3gO3BUQACQO2pV+ufFo5h2T38XKazVptwfEfLzOaz2UPtROXGp/yh/gkdoo7IdRBfiiq0ZJMPEZTWmqkVpfLidPQhEveSptLhBM7gNIDGrSOHc5n8sDL/AyanA7aAT+FB2YHeG1/LRl27FWWU/pNxuBwR5kWouSKPvTUrrXN3N1+sStfMJQHdc9mORDZSC/8rdpkKk2Z5QiGiw0Uxw79fm6LTwhiyuv717GCiRbIO/jctvkeOMkfdsM2vPzdaq8ytNBNhBiB/uxBoW47xzPDfXuHxCdPL9BkLUmpiUYdAmnFqswTmKkxpISGBKN+7Rk9oPlwWeXF9XF5zPOzqepE42zEO6/ns/6CoExYw5e46PBWeNCO1PE99mzzCi8KAKmSDHRbcUPTRQR+eaV0UTu5oMLxgCiN9Mk/IJ1hbvq/Uz9SHJt206l4+Ay2L/DVVu6fR7r9gTtYgCd6sXkJWwGxGp20KZExubEcdrZL5cKIJ0bDtN4aQNyJqJP89eC38dNznMTokR1POLigko+H/F+woXi0v3uHX6g2V3he8zbchYx5qA/w+K1lPE9wFBPaYTvwh2awlSyod4BNjDWSgBX+rRuOOA0L/B2MasyJZ8q328JATq3Ll0p+AiXGcPm2pN/imOl075I6v03/ztTnZcEtoK SbUtIs8q wPBQGAnHIZuOsfcTdW86q5IXN7oOl/7MjhuKBKIVxcaYSo7VlNDjzrLBjfyAkWZyQ52H1xidDoK+J4KkN/8kgIfzwcBLxOZsSGgOSKpVJWB/8CBazSqu54T/uJDNpXZXjTW1+ty3IT3v4dPZd6EzmEF31aAKXOT103cEILNArZ+c13LDn+GPmlT2UUHz7Y9A0wwNe+0UufKXwFsT4iJGFWJdR5wb4F+wNKYaF9DD/YrRpXNGYYyrke6rZrz+/SGqmsAbj9rDV0M39hVwmnp4gNnadF6Nq3kIOyiF59KXPnXyD4uw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pte(). Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index 0e912443a18c3..65d64a119cabb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -262,7 +262,7 @@ static bool remove_migration_pte(struct folio *folio, page_add_anon_rmap(new, vma, pvmw.address, rmap_flags); else - page_add_file_rmap(new, vma, false); + folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } if (vma->vm_flags & VM_LOCKED) From patchwork Wed Dec 20 22:44:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A445C3DA6E for ; Wed, 20 Dec 2023 22:45:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B507D8D000E; Wed, 20 Dec 2023 17:45:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD8168D000D; Wed, 20 Dec 2023 17:45:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 951568D000E; Wed, 20 Dec 2023 17:45:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6B7AF8D0002 for ; Wed, 20 Dec 2023 17:45:44 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 45358405F5 for ; Wed, 20 Dec 2023 22:45:44 +0000 (UTC) X-FDA: 81588680208.07.1AC0630 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id B65BF10000E for ; Wed, 20 Dec 2023 22:45:42 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Hp7ABkjX; spf=pass (imf14.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8A5dFtRO86PFrUgCRtietWW+Q/OHDectiZ99Zg/AfEE=; b=Pu0TGzKQkWQ4TV7ELeu6SMXmYIW7q3wCAAeXzrqV3jRmWa8xm3h7Z6IdEwa8g1xAHi36iv 588n9g2OMy3pzC422bOmgOXecPQjfxCykEfSdrB/1KxhfdR7kwOhm2iUl2BUA2EM77AqdM RHiNo6lIj5DZGdAEznmb0uxzW5hyT/s= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Hp7ABkjX; spf=pass (imf14.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112342; a=rsa-sha256; cv=none; b=zVsi25fMfRGQeLrR/jVwcu+45qfH74zB6mR2SMsDBobySWMyTkl7mBTKJC7cbnDS64yYR6 22w5B+G456/BlJZpSG5Y+s9TS6Pb272ro70N2Qu+SfyvdRYapw27QHihMwZR11H2te1Ofk 9+49twcOjPNoPlbmivbWrx3ek44hPS8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8A5dFtRO86PFrUgCRtietWW+Q/OHDectiZ99Zg/AfEE=; b=Hp7ABkjXkhs5zwakHqhLg2ZoVMu/iVTAopL+CrHV5XB8ZGwE3I+LJH0xpEqs3JfJB8knLG fuYL9ppAqoEox8ErK8xMjo77SKiFJibpz5vtnF6WNh1TduxOIGIWIugaQZ3GwkKTmQl7dS pN1StMo8+FbIWcU7Qi8yR/xjGKrwT9o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-137-HwopZXaBML6m-AFz0tciBQ-1; Wed, 20 Dec 2023 17:45:36 -0500 X-MC-Unique: HwopZXaBML6m-AFz0tciBQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0F9B8848C06; Wed, 20 Dec 2023 22:45:36 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id C16A240C6EB9; Wed, 20 Dec 2023 22:45:33 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 11/40] mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Wed, 20 Dec 2023 23:44:35 +0100 Message-ID: <20231220224504.646757-12-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: B65BF10000E X-Rspam-User: X-Stat-Signature: e8czfp8y97pz3m5whednb55yd4azzs86 X-Rspamd-Server: rspam01 X-HE-Tag: 1703112342-148040 X-HE-Meta: U2FsdGVkX197jOW3elvsXtElhPAJhjPDRklYhL2GFnM63MKlXbLfbJG46YzhYDeMZ4BatvmN6CJLJIxkJQmEOIe7AQN9eXxl1WH+ACkkrour9XQ7oiJvbf3Lj6+HuzLHpWp5HRprYJQ90LFI8LeZBnkHP6Aut45vuHakGUp4asvzqvmVOBpYPmz3D36f46VHgvc8E87Fd27oF9Xm45cdkUk1mbGvkQJLKBMsB98Gl0+YamKswnmCRj2HI6BUqctMOieTCsJkR5ZPPVGqidgS0Q9w4kOaCojf2c2kXS0zeKDvrXL6Anr0udwbToCxAM2PM2/PtmXz/7Ez4c+t+9ebZtdyamvxIQOVKLoUAVfx8G6yJUXNkrGqA7y0Be18ZCc7LfCqnhdw1YsnBpLXDPvpjTmmbDpmzy2G44+qod+muzdUhpe0snCEQabgyAWb2in3F19akPKh9L6zdybAC3Ic8ub6+oRiGD9P8oIgI/eqm5z9l95QiTGomlvSN3hzn2bX3edWkKyzYK8TMVO3SGEUF6c0pHWqTGa1+2N3JNWeoJwbusz76cyI3hzgnW1ihAspN8LCAuE6McVjOyk7/p8r9c9Hwsm9LmdhVIf7oWdtOWNw/QZedRceyba2RogkzxTDS9Z71BRLdkRrGDI8hYKp01azQ74yUoDLGkOsOdomQm3C4VHSFj1SwLL7xwwk5YbUzB087baB1wAzhcgupwttYdiT14f1RfFqfCwtcjlxzY/zLQc3laRJqvLqAX8PXazNC7PfsWeiEgsYOcrsv30Dnk/9m56d/xRZWz8FTpDykha3xWuFjL4fspo3d2ySNmg6pnTppC+VoCyksxHi2aKjMz6YXySRPj9HB4GxqHMFW7bBWVr9/tewmQi0KQfTLO2mKIpYX/z0hh4eafYdRH5Ud/6RmPSeSVEZzEgr1ODEcG8Bz4Qf1aq9PJDF3HB79SLQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert mfill_atomic_install_pte(). Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 203cda9192c29..5e718014e6713 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -114,7 +114,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, /* Usually, cache pages are already added to LRU */ if (newly_allocated) folio_add_lru(folio); - page_add_file_rmap(page, dst_vma, false); + folio_add_file_rmap_pte(folio, page, dst_vma); } else { folio_add_new_anon_rmap(folio, dst_vma, dst_addr); folio_add_lru_vma(folio, dst_vma); From patchwork Wed Dec 20 22:44:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7676DC3DA6E for ; Wed, 20 Dec 2023 22:45:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2A168D0002; Wed, 20 Dec 2023 17:45:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CADC68D000F; Wed, 20 Dec 2023 17:45:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3C5B8D0002; Wed, 20 Dec 2023 17:45:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7CEF78D000D for ; Wed, 20 Dec 2023 17:45:44 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4FB76405F7 for ; Wed, 20 Dec 2023 22:45:44 +0000 (UTC) X-FDA: 81588680208.05.9272149 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id A0DCC1C0011 for ; Wed, 20 Dec 2023 22:45:42 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=M7pnIPf+; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ylNzjWXs3fCxjx+PVR/JHpztU6Uk8qsi+uFdQ6XNBcU=; b=JTbaLsmbF5S03Gu7VbrlH0o87znxN/10VrQ/TJ3CXJEMrG3lweDKHYuCpjU3XlliWqvzU+ e7xpAZjb6w3RjjnZvbCTjI8mmxg/Le6rBwBMf6SOG/uLdRoqs2MCqG52OgjH2RRLQeUbTx IMkesM76dp9UEJ2mOHGpggqgjN5L+Rg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112342; a=rsa-sha256; cv=none; b=rr4dZR5/E6OsOocSbL4EIY9S9NICHBtVBJ23/oBzy2Agco6Aj8PUZ641DM+N56K3flpp8E jKPiQIC40iPmBjWxx5kh0YxMyLXVHb3tRSNcIf+4ubh4v/AhQbsbyx5dHiqW/ip5y2hQmV ufwFgIOZQBivXVzL9H8BiGGhCI0Tjk8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=M7pnIPf+; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ylNzjWXs3fCxjx+PVR/JHpztU6Uk8qsi+uFdQ6XNBcU=; b=M7pnIPf+Pq6jzqX0nWN6inEKJ76P76R3DR67P4P78RAB7RTnDZ/Y3WOh07CNLb/GblA/4H 8us2O57PmeZSrXQ4xhsFlRv+AyjmAsYOwgMTiMTrEYwxwoH1frlQEsgqsDwl1ncki0VVI8 OUfAxYYxyKH+L5+ZRZVU+MQ9K+5Bexs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-196-VNQ_qnwZMhu9M2ICFtPKHQ-1; Wed, 20 Dec 2023 17:45:38 -0500 X-MC-Unique: VNQ_qnwZMhu9M2ICFtPKHQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 33DEF185A780; Wed, 20 Dec 2023 22:45:38 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5D67340C6EB9; Wed, 20 Dec 2023 22:45:36 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 12/40] mm/rmap: remove page_add_file_rmap() Date: Wed, 20 Dec 2023 23:44:36 +0100 Message-ID: <20231220224504.646757-13-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: A0DCC1C0011 X-Rspam-User: X-Stat-Signature: kz7xxdt41qin74k93w9kzs9eadpt1utg X-Rspamd-Server: rspam03 X-HE-Tag: 1703112342-83037 X-HE-Meta: U2FsdGVkX1+Y7zLdMLgWo8IfJXbKmtDvIIT2UEjNryHxQiWLoelT3Y/dxp/ZV0+38/5UoS/q9ob2Eq1pH3xloMYhjrc05PoFe9XtHSdXW/jQ1CD+RFMEdXRCEyaeb5lm2LnZgpDg1b632Nu298C8eHL3LKDUANPfyfJHNJUfSBSsCRaPfjaemP9rISV4dMOva1Xp0kJATJUImj2VVGSfuZAKz0QsqWSuN52VIHR5Pfv3XDwOz2OKzM/5vrR3k2tilFsaeLjkmiHeD6xXlDlNTMXhs2+y/fDoLoJphV8ARHq7Ua2SUf+b49xO4JWFoOTB8Xk5pEagOCAMeLph1JiR3d/5rgkVfsj7FydNUpJTQirh28IC5nlPZsikL3qNcBc6E1BIxwFKVVvs3Y9xsxoRV1kk4YU/aek39sLtR6vu2ZiuylQCjWipfR65m37QD0zLqgG8w3Z/pdjIeJU6UEbbS1H1yfq8tn7llq13JLIKGHPx7uJLXr6stxgYilnBBJWvTCuSag7DHbgPHTBYXNgGpYMVLyESyR2G0tuMTQsjk+QL4MK1FFQ4DgE8r3Oe5xNEOMcqc2TSX+NVOeWLClAaTa6E5W2TLcdUbgotG4mjf1lfX+FJiJidIIevxGzWR4xQ6tFk/U4cguyOQsDhPkBoqAzFx16iyzF7aZgKDlaFZUPy+g31prL7xh8xmGNVxQUNLb8TzdQB+jtNRBV7AVXaEyXh+UJzmKat7yCZduzocVXF73bZunosH1RyVExWaCt7u5HL4By/FGki416g8OFwHKsRoGwaoC3mWHQQvWICigFMYAHeJgyFhTOz6bLx10Vq7nlAitDSnTCS80OjCExCyXymwx4+wyaT58IJMes3ePL0ddSzbGPuSebiUmnKyLfolyYgRIA++FMFH0G58LBwxc+gpbqDwf5a3OSfuvIFEflbvd3XoRPE1iuPWzju/+ykpRjaeO99eZAc+aTASjx 2gS4+KtT NNO/iQSsxAU4sMdkXNYLjMmryxBgz2suj9R4dE3piFPLkUvh5w1awutiD2k6y0mC+1WG1Bah4RsrYLYiXlMvL1N76eKvoAyGJ6HmCJKdastJBleJ4g690gVdQnayaZxvNYrDVEzY+KoOanUPROE/Lo2QYw7G9BRV8+2wkZujWoN7J9A33A00qKBMJ6rcE+OcnteiAQ3KF03obOi/DUiXBbxHggEe0KvSLU69RntXPHV05aoNfzMzbqy9qQVFw3bqvxxiJ3RVvVxvxhksl8Mp+iEdKdEdWF/5SyNRR2FRAdwDICIM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, let's remove it. Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 2 -- mm/rmap.c | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3d86a76b28368..6a4db6933e7df 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -237,8 +237,6 @@ void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -void page_add_file_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_add_file_rmap_pte(folio, page, vma) \ diff --git a/mm/rmap.c b/mm/rmap.c index cc1fc2d570f0d..5ab5ef10fbf5e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1467,27 +1467,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); - - if (likely(!compound)) - folio_add_file_rmap_pte(folio, page, vma); - else - folio_add_file_rmap_pmd(folio, page, vma); -} - /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Wed Dec 20 22:44:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03B9FC3DA6E for ; Wed, 20 Dec 2023 22:45:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F4D48D0010; Wed, 20 Dec 2023 17:45:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77C358D000D; Wed, 20 Dec 2023 17:45:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6212F8D0010; Wed, 20 Dec 2023 17:45:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4E2868D000D for ; Wed, 20 Dec 2023 17:45:51 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C40EE40A95 for ; Wed, 20 Dec 2023 22:45:47 +0000 (UTC) X-FDA: 81588680334.09.DB4CBF5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 2552B20013 for ; Wed, 20 Dec 2023 22:45:45 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I9TumUH7; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112346; a=rsa-sha256; cv=none; b=ES74ar6oxv5nIk64wN18rXk1mesIDrPVtZ1LRHi4R+a/4WzZUYFBwDs61tL2vVEgUVOmrr y5J/VusOszfvP5kiP+V+JaEKZv4dwhg/2MMycdlScvRDmrCU+AZnz9ZEUQee4PRXRs2TLc WEDZaK6rob3gfrxMvloBtQOefHI3Js4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I9TumUH7; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112346; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Pjqwl2otIfh4GV5tfgKtj45x9lDmjEhBWckBey16zKE=; b=VpGg0wCK4I00sLMDi+5ScpM7trfUJX917brpmRATJYI9l4Hm8b9GYShxr2g9+YpqdjzawW P4BKrbZCO0dc4utvb/e+7JHLEb4k923E/OdNHfY6LD5cU76dfs81Yhj7gU5bMXHcQof6Me ZLAdppuamNlFzEh7tQg9C6HzVPFTnsc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112345; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pjqwl2otIfh4GV5tfgKtj45x9lDmjEhBWckBey16zKE=; b=I9TumUH7Stm1kGcYk4p24BB+kC1+C5foJaXVuwzM5Ud0C/6ujLKKBWE3x5BoBzJ3DibgjG Qok68b8AWn94fOQThXN06GZQnz1rU3TLMtblIJVMBIwZqjyZ+BrO+y2u228Gsh/poLZT38 r2YOlibHR5V5X51Ll+YgM8By0/vaFKE= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-581-jJV6QouwMIO0ofo9VqPsZw-1; Wed, 20 Dec 2023 17:45:42 -0500 X-MC-Unique: jJV6QouwMIO0ofo9VqPsZw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8CCB3C23FD0; Wed, 20 Dec 2023 22:45:41 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7129740C6EB9; Wed, 20 Dec 2023 22:45:39 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 13/40] mm/rmap: factor out adding folio mappings into __folio_add_rmap() Date: Wed, 20 Dec 2023 23:44:37 +0100 Message-ID: <20231220224504.646757-14-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2552B20013 X-Stat-Signature: jzsjrhzoqgdkyfxswk6csuo4zugskpg4 X-Rspam-User: X-HE-Tag: 1703112345-651569 X-HE-Meta: U2FsdGVkX1/k4MfsAVByV4PwlKR5KCQ2ivYge/fqnyOodVWIxYjipb3XfV8SziGKsDl1UjLrVydV3bavwFevvZe1vcVIIAJ3z2yiJ+Yu7zl5G+3CaPtdeFRmwaGy1oQmq+jgh2bWCHF9ZjCCQ4d10xtUqHFvGOjfKBDJvMeGQQtyDdJKqoaALmWHkd+fD/uch1+ZxNUymohcPX84r4okQq+9K4BjvWyryPp3CacNro0Qf/q8UZ12HSTcwBfzTUtKXZoK38+sf26Fq43JL2VmN/YPJmBVW7B5LP64flVqCnGw4ZJJwG5W61TOJ3GEEImzmCKNCcVS1hvAgVZruiJoZcUtpyWJT4nundLmWK54PGscgs+fc1+/9fnsqWfbt+xijt0enSi8wOM1+f/urR4UmCFqe+pJQGGmlz1ret9i7ndJnj0GNHCKfbDkEcpkXX3JVe6EFAwZ31mjGvuIF/NO6yFqyPb7KiziRQF+7mupQMTVZISLTGvPNC8dvYMiXYpAPbo+cqwWEyqRm4fC7s6Aqo8/pc2PI6h4B84ZB6x8ZXRLh7/KTgWmRefq/8gSiL5GI2E+wCRj4d3BbZmylEMiC7ip/T3DeG4X5TDljxzyLeAtaz5ntDgSgow7aX8G3fac9EzDFSvoraTii4UfBUOALOVV86DCUbRl4PLNBV3XwCedlsRxHk6w8Plg1OcWy04GXiIKwDg1Yh2bn53FjQ68zqDFfTQdG7CU7lzsGKa2cM0bVw9PFytFi35xtpNcD3KPCaaTkdYovjJeJKNyDo/px/FOvsrmnWbs0TpV+V4Q1/axjDziextJxVZbkKVrIrCNYykyi7hQVVEL04aFMK07CnXu60ZCpUD72vt0I8gjHpCJ6E2Sk7cGlNlGP7QFz9rLjGJ/YJzF8qF8pv9aZ417t1bAs5YZ7VqclNFgQ5aIaMAedSYuQ+epwafL8t7RrwXg5K+R0IJbe0XWiLVyeWh 2qR2qHNr tKR/kjeONqThCKpSZk9D2RK6QZqq2cHPOLOrDA4AmTohX95HPHJuXUeLa3ZC0unNP+FyNRklkdgcxnmKRTsLlx0ykOmgwduuWMNTBGIejCp5mGtb8lMNcfAZls4r2CG3aDGMPwenm8axfugMQorAqdWlOtXJXtVIehzcwm9G5NSd44fcf4WQ9xsF4oO3jzNf4Xlrv2HP+gSqx7MgTSI6pAwAQ4vTMfD0SdOuW1antjS2zTC8Bu+x5HeT4Fw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out to prepare for reuse as we convert page_add_anon_rmap() to folio_add_anon_rmap_[pte|ptes|pmd](). Make the compiler always special-case on the granularity by using __always_inline. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand --- mm/rmap.c | 78 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 44 insertions(+), 34 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 5ab5ef10fbf5e..895a8534a935d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1157,6 +1157,48 @@ int folio_total_mapcount(struct folio *folio) return mapcount; } +static __always_inline unsigned int __folio_add_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_level level, + int *nr_pmdmapped) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int first, nr = 0; + + __folio_rmap_sanity_checks(folio, page, nr_pages, level); + + switch (level) { + case RMAP_LEVEL_PTE: + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (first < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); + break; + case RMAP_LEVEL_PMD: + first = atomic_inc_and_test(&folio->_entire_mapcount); + if (first) { + nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + *nr_pmdmapped = folio_nr_pages(folio); + nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } + } + break; + } + return nr; +} + /** * folio_move_anon_rmap - move a folio to our anon_vma * @folio: The folio to move to our anon_vma @@ -1382,43 +1424,11 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum rmap_level level) { - atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0, first; + int nr, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - __folio_rmap_sanity_checks(folio, page, nr_pages, level); - - switch (level) { - case RMAP_LEVEL_PTE: - do { - first = atomic_inc_and_test(&page->_mapcount); - if (first && folio_test_large(folio)) { - first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); - } - - if (first) - nr++; - } while (page++, --nr_pages > 0); - break; - case RMAP_LEVEL_PMD: - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - break; - } + nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); From patchwork Wed Dec 20 22:44:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18BF7C3DA6E for ; Wed, 20 Dec 2023 22:45:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9D158D000F; Wed, 20 Dec 2023 17:45:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B25558D000D; Wed, 20 Dec 2023 17:45:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 935928D000F; Wed, 20 Dec 2023 17:45:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 812CC8D000D for ; Wed, 20 Dec 2023 17:45:49 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4FCEC80448 for ; Wed, 20 Dec 2023 22:45:49 +0000 (UTC) X-FDA: 81588680418.26.A8D00B0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 92A7C1C0026 for ; Wed, 20 Dec 2023 22:45:47 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eSTbqv+t; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fT8ITtdjY970fgU9T3ISsA7bFyZzQFNQcv3TmnG2Cz0=; b=jyOH2xKgWloi133ZNiBoEwoTUA1dp8zn7x5UEI5/qqK1+Lfdog1ZjVpGY2guuMlPM8rFSK TLgJNZXDIQ1hh/7uMgx2Ez359NJhOv5MAzhP4AxT90GqqA9dMmZKqGcmay3MRsEj/j6Zbx XfdB5fktgWuB0IPmM9eFXK41aZXNaU4= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eSTbqv+t; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112347; a=rsa-sha256; cv=none; b=KQ5kRV/+cBz0j4b/SJ2VQxljCslJ9RR0sdSAK6yDMmnsUWCLTtfuKjQoDU06bzAEq8eVzY lvnST+sXHG1NL6mrLRbkVyhJSwKALxCAije3gahpfLah1FFiWFSgJNV0ANs3FhbUeDOZy3 /mkxcdpH44GzfVhaKiypuyGcN1T6bbI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fT8ITtdjY970fgU9T3ISsA7bFyZzQFNQcv3TmnG2Cz0=; b=eSTbqv+tbVJWK425GQSU/jjZEqATtAccLXqybxVw/+y94lxnxqPJ5ss+Bkz8HwamKaUxtC kpqzZjWgK8Mp9zkfxHTdTyr/1dF+5sECtPU86ffJE8RaP6iK9XVa96knrEp4VHNJRI6ZW5 fFB9z4WiQlN0COZJYBB12zg8ZZDwztY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-641-OvGVr2QHOyKfAifuTxGNMQ-1; Wed, 20 Dec 2023 17:45:44 -0500 X-MC-Unique: OvGVr2QHOyKfAifuTxGNMQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6FD2F863B83; Wed, 20 Dec 2023 22:45:44 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14DC940C6EB9; Wed, 20 Dec 2023 22:45:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 14/40] mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]() Date: Wed, 20 Dec 2023 23:44:38 +0100 Message-ID: <20231220224504.646757-15-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 92A7C1C0026 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: y75hbsx3rizjkkpz7npbxjx7aws3pb3i X-HE-Tag: 1703112347-399387 X-HE-Meta: U2FsdGVkX19URdlAfIO9japEfHPrgaYdL68UZrCy/oMD/ndTvyFWINbFfR09IDMVpw2YQ7gRFn3te0k5aqMdsiNA3SAEluiXlyNDbPp7WY65IY4PjAUmJAueUu7TJ9gcsHZLQpf7hSwoTTapZwmFHe8PhDYThLWjHfV9utZ6cx7ZwwqmdWPHYjRXt28Jo5kmQbm8xEeGDvqk7mT1AAleDQI5VPqAUZG5zjlF8nbPTqUNw3JLT5a8Q363vBTjkzqjvYqBzGatHGGMMdmce32DIZpxjiUeAmdIXUG/MucX5O/KF6sHpe62cPhUYwZBX0JRm7U6Ej1ydUZpqhjwAuO21QqGJg3/UdS1G7wyuHV/DSEsmQVXCLm7kXSNrfh2nrA3x26ca2QJAPMWY9JpqOp8gdvFcgqWJfGwc0rNOq37kwVgjONXqVBiUsvpIt6oKj60Us/gJSDTyxjgp+wIZnsixkfbl9mbzUmlnJGPEtppkqz69LqAyKICn58EaMECS9IGUcGzpLupTRmtkOzws9ZuiDuq3blDwOpEFt6QU7wnXjHnv9QrcQj47qkKugx/umFvnMc0Ot8XpmlTgJlTC52ll5bQ2QdQLrYXh56NFGrkNsDb2R7oHoqf1rM0Kjzxsi3KoFTbzRoQUqWFEVMDYRRbLpSt+TlRJF+lgy+iLsPp01HWm7vINfvQQmgIlRJCMBDQHKhEQdn/4zNx6Bv3Al0O6UHYi/zhQ8ozOaAxyh5wdsmMj5/uUdXXUZUJrCaBlKU4eZ6DC1Lr97FyoksrxDgasnJ9+UPVqNplY/VT7wzG4uZfmvTgTpOLGB9XOumY++MlOd963CzdJCfnQR20SnoJNgzzw75VzWKCvqlIYmnNX22eZRiCooMz9DZKt3p/nPM2SeI7NJQ8X+QtyvTEE+Bln0MyXhd1MQPyO9v0OoBwbEcW+hHpTqaNH6ytQTPLE9Hpj78CwIeldEsKgWdLJd+ ohQuW7k9 LXzFCOohHccI/j+XMawp9diDfaDN9t0NvhvrxIx5ECL18UXrOEUcv+iEhMEe+M/TqG4cWtfK7q7sUNNcVz+OdXjBw+znh7dl9HWRdQvV1a+dcWwCFwVXfa3GRRttfZPFHHGMQSxau6/4N3PEk+H1jhYzZg+VrgoigbI1S1cH41qhf7qIyczL25D2wIZtpB1eHNAC3UgdsgUyj4Q44KWekTAeHpP+9p7SnU900AffqWrsLMsC2vd2TUy3R9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's mimic what we did with folio_add_file_rmap_*() so we can similarly replace page_add_anon_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. For the PageAnonExclusive sanity checks, when adding a PMD mapping, we're now also checking each individual subpage covered by that PMD, instead of only the head page. Note that the new functions ignore the RMAP_COMPOUND flag, which we will remove as soon as page_add_anon_rmap() is gone. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 +++ mm/rmap.c | 120 +++++++++++++++++++++++++++++-------------- 2 files changed, 88 insertions(+), 38 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6a4db6933e7df..b5da3d86200e4 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -233,6 +233,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, * rmap interfaces called when adding or removing pte of page */ void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); +void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *, unsigned long address, rmap_t flags); +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ + folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) +void folio_add_anon_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 895a8534a935d..7f380f5a34c90 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1299,40 +1299,20 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { struct folio *folio = page_folio(page); - atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool compound = flags & RMAP_COMPOUND; - bool first; - - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(!(flags & RMAP_COMPOUND))) + folio_add_anon_rmap_pte(folio, page, vma, address, flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, address, flags); +} - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } +static __always_inline void __folio_add_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + unsigned long address, rmap_t flags, enum rmap_level level) +{ + int i, nr, nr_pmdmapped = 0; + nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) @@ -1346,18 +1326,34 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * folio->index right when not given the address of the head * page. */ - VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + level != RMAP_LEVEL_PMD, folio); __folio_set_anon(folio, vma, address, !!(flags & RMAP_EXCLUSIVE)); } else if (likely(!folio_test_ksm(folio))) { __page_check_anon_rmap(folio, page, vma, address); } - if (flags & RMAP_EXCLUSIVE) - SetPageAnonExclusive(page); - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 || - (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) && - PageAnonExclusive(page), folio); + + if (flags & RMAP_EXCLUSIVE) { + switch (level) { + case RMAP_LEVEL_PTE: + for (i = 0; i < nr_pages; i++) + SetPageAnonExclusive(page + i); + break; + case RMAP_LEVEL_PMD: + SetPageAnonExclusive(page); + break; + } + } + for (i = 0; i < nr_pages; i++) { + struct page *cur_page = page + i; + + /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ + VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || + (folio_test_large(folio) && + folio_entire_mapcount(folio) > 1)) && + PageAnonExclusive(cur_page), folio); + } /* * For large folio, only mlock it if it's fully mapped to VMA. It's @@ -1369,6 +1365,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma); } +/** + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages which will be mapped + * @vma: The vm area in which the mappings are added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting, + * and to ensure that an anon folio is not being upgraded racily to a KSM folio + * (but KSM folios are never downgraded). + */ +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma, unsigned long address, + rmap_t flags) +{ + __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, + RMAP_LEVEL_PTE); +} + +/** + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting. + */ +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long address, rmap_t flags) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, + RMAP_LEVEL_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. * @folio: The folio to add the mapping to. From patchwork Wed Dec 20 22:44:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500628 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6A81C3DA6E for ; Wed, 20 Dec 2023 22:45:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8AB48D0011; Wed, 20 Dec 2023 17:45:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C0FB68D000D; Wed, 20 Dec 2023 17:45:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB38D8D0011; Wed, 20 Dec 2023 17:45:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 95D348D000D for ; Wed, 20 Dec 2023 17:45:53 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 70F83A1A07 for ; Wed, 20 Dec 2023 22:45:53 +0000 (UTC) X-FDA: 81588680586.25.6299888 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id C14A08001C for ; Wed, 20 Dec 2023 22:45:51 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dR0oP0n9; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mdoP1mKHev+ysUrRTdfAENapfdNdps+V0pIhMC543AQ=; b=C4phyig/Xk9JqgF0VRTMOy5K/uGJtOKbiXaDtDage/7yBSxF6zPt6xAZCR1+Zc4XoUi7yu sHSNBPqk4QnjXalgCBJmqq4z8J8zBC4oy58t60iqlTEPOYdPaJnqJTS96uSjPi4Rrd0eco hNVWZDQ5pkdkS9yyynuQ1CN7QNjLmQY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112351; a=rsa-sha256; cv=none; b=kwC1dOgPXRtJUq9BbkOB5IbSe9kcOFBUfqo17ypPvBoTYdnA+zH+Bq6oWrbgqevVEjb82c 6/blf1u4vs8TCVt/ZX0SCBcx8gS3l3tFwmnaZio/r9mDE1k6Inf7NOhj2efO+UfaiJgIPo zufzxVLz6iJbV5Qt6TxctlNbq4lpPKA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dR0oP0n9; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112351; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mdoP1mKHev+ysUrRTdfAENapfdNdps+V0pIhMC543AQ=; b=dR0oP0n9wJCenQzoLnfbx//zUXivu5QF3fxMwKGiyhIxoPz2FRg8nL4gbmjlQiZV/7Qyyr QtPhOj7BLojhZlPqqyWwAarXTCWH85lfvbTkEtKX+Z3rllCtTMPPu61/ePOforZPMHu6Tj JUUTbAYXYiCOx9wmsxPcOpcmS8jyTaQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-26-BWaSJ04ZMwKSE-3aJaWQ3A-1; Wed, 20 Dec 2023 17:45:47 -0500 X-MC-Unique: BWaSJ04ZMwKSE-3aJaWQ3A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 39804863B84; Wed, 20 Dec 2023 22:45:47 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id B7D4F40C6EB9; Wed, 20 Dec 2023 22:45:44 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 15/40] mm/huge_memory: batch rmap operations in __split_huge_pmd_locked() Date: Wed, 20 Dec 2023 23:44:39 +0100 Message-ID: <20231220224504.646757-16-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Stat-Signature: pwo5oogpekpc1zzdudn7649g8p6pysiz X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C14A08001C X-Rspam-User: X-HE-Tag: 1703112351-776053 X-HE-Meta: U2FsdGVkX1/OBx7QglW6lymqbTY5ES/D2I+V5gZzoejszdFnIwvDdt65p5h13nZiq06W0KoIk6IiXmnrERFaQPz0zVXYG5JK4ELge/mHqg3jOQHHmGNExf1dqL+mNINdd+F4ka6o3VfIkxVppyx6TiqFb6udKEeGiLZepQRofxvo5v51+0Jq4yq0eFC1d2CwEnWGkHNk/JpLDSGK8VuyVK918M0idmBQs5hDdOjwrc8m4fPIc2DR29oIzTK4A3lP+9O8xDb4Fd3KPNWp2/QaiSq65H+a693pA5ph0tFPzw6dMrg0E7H147di+Jw9enzvoeTEKAyTjntsku4fOV34AZ83mwHggUigojwlj673Ex1fCUm32wIjVipl04Qt88YfBUqJzfc3D37VV286e9TxahtVE9rUJOaJmjokSf7Tls9N6m4TKjJoLKmuUS2fb06dId61CwD7uXvHlOELDRXTnPrptCopMAPvJRcTl7DNo3h0CDRnNioy2okokXt8acTjIA78JmegI0+ncphKjub8+xpCtxu9y/3ZytB9wumLXao2bQnyF6ra9XFVWzXz59Av3/q6AtOIr6VKYejX5SHeps+qzkJHTEK9c5fw/EZdnFR8iSjShBT9MgpZc2N89sMgxi2OXWk0Waqvwnvrngybftz3sxA83i+5yOjEKy2k1T59bHHLTr3N3qllkXDGIff44HGl/wM4bAVdUKZlQ+xPeaUlwuuzfGYnF6iL8GPBcpc9JzsHfmxFC0MEUABKfZk7dPsrlTh5V/IVrpJR05Z09o3DNL8b5jq+9SaiGQzk06Rdd/2xqZI/XjtT1lvJ4SfYmqLihO9yfzT9/5j0qiEoBXluvTlmfM0kSts0fbxeiXermrkru7e3K3CLhJKJwyIx99RT+OOFwAZzfXdHUr/JfWKRgRkOlE9bYEp3ZFj3tFRCX9BRVONhOYhvbU7DpY2pei+X/foQvObPM9tK2OE fuvC4rCW R+nAhrzXzrn/eB4KVGvn6Zpak0YZVvukcbn1A299hWa9b+d6MjHQp8vlTRhKHu3nYh4WGGet+n51RJSQZVxTQMoH5QjiicEYHXS/u/zw7HCzGBO+7nSIYFd7/5NEqKx9t1+pj1F6VSwVxMTBj/LMjg+Lo2A6vNMPfiUMgUybsMYZzbeqnPbPuPEvzzYB9TNcQgjCRJZ45QCfTEdLZjfvMRtxtmK7wfm2Q509by/ubJcH6A3qCeSXZtGOrBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's use folio_add_anon_rmap_ptes(), batching the rmap operations. While at it, use more folio operations (but only in the code branch we're touching), use VM_WARN_ON_FOLIO(), and pass RMAP_EXCLUSIVE instead of manually setting PageAnonExclusive. We should never see non-anon pages on that branch: otherwise, the existing page_add_anon_rmap() call would have been flawed already. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bce6f987f36a3..d4c5d22d16117 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2398,6 +2398,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, unsigned long haddr, bool freeze) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; @@ -2493,16 +2494,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { page = pmd_page(old_pmd); + folio = page_folio(page); if (pmd_dirty(old_pmd)) { dirty = true; - SetPageDirty(page); + folio_set_dirty(folio); } write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); - VM_BUG_ON_PAGE(!page_count(page), page); + VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); /* * Without "freeze", we'll simply split the PMD, propagating the @@ -2519,11 +2522,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * * See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = PageAnonExclusive(page); if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) freeze = false; - if (!freeze) - page_ref_add(page, HPAGE_PMD_NR - 1); + if (!freeze) { + rmap_t rmap_flags = RMAP_NONE; + + folio_ref_add(folio, HPAGE_PMD_NR - 1); + if (anon_exclusive) + rmap_flags |= RMAP_EXCLUSIVE; + folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, + vma, haddr, rmap_flags); + } } /* @@ -2566,8 +2576,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); if (write) entry = pte_mkwrite(entry, vma); - if (anon_exclusive) - SetPageAnonExclusive(page + i); if (!young) entry = pte_mkold(entry); /* NOTE: this may set soft-dirty too on some archs */ @@ -2577,7 +2585,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mksoft_dirty(entry); if (uffd_wp) entry = pte_mkuffd_wp(entry); - page_add_anon_rmap(page + i, vma, addr, RMAP_NONE); } VM_BUG_ON(!pte_none(ptep_get(pte))); set_pte_at(mm, addr, pte, entry); From patchwork Wed Dec 20 22:44:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9363DC3DA6E for ; Wed, 20 Dec 2023 22:45:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 278C38D0012; Wed, 20 Dec 2023 17:45:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18D6F8D000D; Wed, 20 Dec 2023 17:45:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECF8C8D0012; Wed, 20 Dec 2023 17:45:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DA10D8D000D for ; Wed, 20 Dec 2023 17:45:58 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B8D8EA05D7 for ; Wed, 20 Dec 2023 22:45:58 +0000 (UTC) X-FDA: 81588680796.08.522AEC5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 213BB140021 for ; Wed, 20 Dec 2023 22:45:56 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMyI3CYm; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112357; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zqDId8uyvnGLvlQWpKrOuAVvsXstgtUNoP8HJVv8bTA=; b=K52YKJYlwaBaT90d8x9cN/WV6CNfS/ikEtkR5pAKjhL9cww+vPO0DhWxAJv3t8/pq4nBg/ kghsUzKJbYjB4x6t0IfJN2cCUCRxUSwYUUF0RJeWWNBG3owQWxNvmrvvnPXK8LO85tus0H U4365gMM9mu+yNo5t6xNb6NEU3lT7S4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112357; a=rsa-sha256; cv=none; b=6no0uBjBBgq53sPMWa2MZU0haOJmnbYNK421y5r3WQixmNZsVGngTT6EQyBmpcmS3tXyw/ U9mw90KZrxUcmsuahh8/DQfBDnC7e//vCUlhn0QSYc7KFQOuGLPPJSx41IZupC/NPSUbi/ yUJh7kUx4U/QMosRDm3q2gvPNmXXBU8= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMyI3CYm; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zqDId8uyvnGLvlQWpKrOuAVvsXstgtUNoP8HJVv8bTA=; b=KMyI3CYmhqsOxPcwg7v1uVmTsd6P1d/1rQvtlm+psZ0FUQ+pakAy4XOrxytgmHhET9a0xI dTcxKCEOSj8T0h1vRRCd8Ksa+Kw7ULC+UONlfJpMg0Slk2YWdtTbmiuOEstwfg5iO3ZPNE LJoNtLhFouN3C1k0yGcUsJAmBZDqd7o= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-483-X02swWYoMrmIknXW0v73xQ-1; Wed, 20 Dec 2023 17:45:50 -0500 X-MC-Unique: X02swWYoMrmIknXW0v73xQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 19FE538425AD; Wed, 20 Dec 2023 22:45:50 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8187140C6EB9; Wed, 20 Dec 2023 22:45:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 16/40] mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd() Date: Wed, 20 Dec 2023 23:44:40 +0100 Message-ID: <20231220224504.646757-17-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 213BB140021 X-Rspam-User: X-Stat-Signature: ffyqpp3mmqbaps7jx11zt3yartwcmqri X-Rspamd-Server: rspam03 X-HE-Tag: 1703112356-157321 X-HE-Meta: U2FsdGVkX18ISk5cKd7nXgoGzukNEpzEBeXIvvtvg89VmQG8MegCUE4Ilgo00fcdro8ZjlG2EIL+r1HP9hMEXVI4kn18TnrCCR3SM1SBvCH2hvqBSLTt5Ge5Uimg2t2xe63tffq44sjsC41MtDLkElL5PJQqy+061W22ZsKjwgNbjG8hh/sOWjSQ3DwwEn/LflPQ43PBvRYaHgJHmpvGfJNLei26sh9Zfu6o0jklJNxUZEM4LT/xVsMBK6YiuqATx1BkNu47iOzFMUZMQ1KsSRSaGgOyfy9wYxR/c5VcSXdwBYYu0fQxosA3DQqFVMTT8el3t0pREroRzldmUDfIARc1E9AxWv59cTPV+fZfxx0VpQPezeKaTUsTUeywhDQaCeb0gbcSKWJ8wjz03ejdR1oNH9QFBUkfWBHvnjNccy052jM57X3tnKmQnEthJCklXCKfUxg7vBbbHSqffKy7ofZlZJs4AzArin7r59/V4xDGXTSkd5biZZmVrBLa+1Lur+Nj3U36i6VfIq5bKm04Dzfdqi3J5lbAdI9rrPej5Fj/IrO6ZBzs6DFSSYGnpjQDphV7IeB/0vpsOpBvij/hGwURJpaC2efXAJVg68/oi9r8nt/VH1xBcmi/iofdJ547ct7PO5/CG58pwzGuYjy+OvvuSYqLoCJrac6F6k3Jk5/xJQ0TLiHY60M3z1Zz3nGjs2r4IkmGBDKwBSzbn/oCw8hXHIN92ij5TFEqw50uFJLgusPI7aB/QhBWMCRKqK8R7JrFNjYUqViTEm94tVL/MYX1EwYhuMoVtq90FSEO230gMSvfVXpSKxC3UKDg7h9//sfflo7iRhITSW6no6JsAHa6y7c3sOIsBsKiXsbRwcYe3ltbS0m/L4yVR3/9YIF6g8X8SVRM6CWqh51ZqGsZEQtPSYclAWBatuWdnOSygehjPOaFYneX0HYBgfOsdNVs2cMQCQQAbSX54Qf8k6n hceqaKOy DQOmQDmJIJSi0yz9xsV3mKywLficqDhwQjTR/nrMEMD0HeyqFeXG8suM4yTcoiJXBTxZqQfivufG5AK7QUHj0x7G3IlSgK0gO0Zz5f15BzHNuf0F008yzsniZWyGtF+yhQ6fuRG3BqIFa6//AkI7hyA0+Bio+ZbvYMwwOgB1v2cgddLDHlIszKW7qA0K03Ixl4ZDbSP9WmhFLRv0sCDSzDcrIOA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pmd(). No need to set RMAP_COMPOUND, that we will remove soon. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d4c5d22d16117..1f438326b69bc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3611,12 +3611,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde = pmd_mkdirty(pmde); if (folio_test_anon(folio)) { - rmap_t rmap_flags = RMAP_COMPOUND; + rmap_t rmap_flags = RMAP_NONE; if (!is_readable_migration_entry(entry)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); } else { folio_add_file_rmap_pmd(folio, new, vma); } From patchwork Wed Dec 20 22:44:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47337C3DA6E for ; Wed, 20 Dec 2023 22:46:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D23F28D0013; Wed, 20 Dec 2023 17:46:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD50B8D000D; Wed, 20 Dec 2023 17:46:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD8E48D0013; Wed, 20 Dec 2023 17:46:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 991D98D000D for ; Wed, 20 Dec 2023 17:46:02 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 71BBD1403EF for ; Wed, 20 Dec 2023 22:46:02 +0000 (UTC) X-FDA: 81588680964.04.961BBF9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id AD76B1C001C for ; Wed, 20 Dec 2023 22:46:00 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AzJDo77E; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112360; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e4j1gZ8J9Me1FxgaIKP15u5MZyCwMx8COQ6zxxJs5nQ=; b=5YWtpiS5cI1HrZdPMLr0DqjKhcW5jSLVduZC0YN5lDd6UDf9n6h5xsGpd3/4ro/oPn3ZGs ml97K13A2GK0JGnBGxeZZd/m0fKVI2h0lO3yUO6rrPnZR2oRntyVZ8/4tsnIfARRcMOTU8 lbtcWdel1X4+neEvOqxPFQnWtQmyItY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AzJDo77E; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112360; a=rsa-sha256; cv=none; b=zTcfq3buHVp7YsOj7ng4gQ/MgrXRgNZKW55qVLmlZvC80gUO2Fe8f5c0Nsd/615DDcyV19 TYPFG6ZNaH54R6iUVP++P6qe6SXpjiMQOcKmhT+628C1UPmd8Kr2VAOy4YhTSsPHnlbTsP 55CPnPSUX5RdftBhmsPxLYozfzlodPc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e4j1gZ8J9Me1FxgaIKP15u5MZyCwMx8COQ6zxxJs5nQ=; b=AzJDo77EYwblsAEEl0yfiQ8ZEaQ0gIwQhMK7eEQRsOLkl7XnPuIjcGkqIA/1AJ1CKCCpVP xF4cOZ97g67neHwbiIX2ktobn2w9hgugCqe6+6NtNlqdzy2Lx9fFeGFe78wNAFN8beFR8A +qCgdy0H0O6v4J/K7A0wrQiR3kwI/fc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-374-rwHNZvE3PyqDqMksZaYNkA-1; Wed, 20 Dec 2023 17:45:52 -0500 X-MC-Unique: rwHNZvE3PyqDqMksZaYNkA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3A84F848C03; Wed, 20 Dec 2023 22:45:52 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 783D440C6EB9; Wed, 20 Dec 2023 22:45:50 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 17/40] mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Wed, 20 Dec 2023 23:44:41 +0100 Message-ID: <20231220224504.646757-18-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Stat-Signature: gua4oy1th4d8aob8w4tnd8dycgghioi5 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AD76B1C001C X-HE-Tag: 1703112360-149097 X-HE-Meta: U2FsdGVkX1/Wbw3lD7Fr8EeAUDXSZBykTRW5K2aQO5aDiuIn4X7E89u1SXsp4uay/I8lEeJUyzqBcYUYvuFEJSrxankkw9/+HHjI6ABSc1kTUshs1jLPhiI3Jur6xtbcgT7JBcqDNj5alK7hbPsAKw/R9yT7cLoWZrRj+6NNDwzJLqJpOf+Ahb2c3HpjEVkgvgCKocoA3hCano73KMyXmiNGJad04naRPtTAUzgUQqTCuawX/dDCsECu0Zq2eeQixPh4ORocBMTu6tlHsfe5nPMRxveBxNXQLqDKUdyKN1QlZSqLrjrtfUUr+OZYoSUaIXLu75et9JqeBrp9j5u5utHfBd4Pl1jelKP2Q/H88ytPjWRZqv9Sr+OGnQPdZrXqhRYD0i1YEdOYgT1nARMh18oWlV0+1oUI1WUA8sGRmetJXrYPAJQqmIj8f6Tr51nORykAP5Q7LFXEuE7v89oWYy8nmbWLLsqqZjbf+WrFSriUGl3GQbU9LoMZIjjrB0IBekLEOkkv46OziPou4DlKvHRvzFYcR9JuV+Z81+2ZTq8sEXeP/JV9G4uUv1NlQAZODtp61QnQk06Ydm1HJKC1Bd0cbk/qfTfaEJOfiIxE2byY6zJaZaTLMorLGWrZ2TLq0zOZFwIZ+HDm5I6juU2UX51QG0HNpIfXwa3bCoQFPxOgVWZRkZ57lciK0XNkBxDNejUnzVIgKj1//oFNj7tNmglFAx9G2ws2CYr+pBRtlA9mT3CUll1gqNSXIR48QwpdAKJuj5RONdkmjCw0ZuMQ1GMn/DM9ueJs2ehGedrAORnHnsqedIDb4DuENt5BUGxU+8Rc7JdbPm1JdO0JdvlJ1ad/CINq7R81EkQPGk7MtHpFVSjwpEPlP+m8hEOLEAtjYtkv2LS78aucCHiVaxRaPn+h/J1TVzIGr6l4sT8PLrE5oeyiGVThhigA4zLXisXE0qXs9r095N/Ns0l6vjA wHr+/hHi 3llLwAuEzherOCCy/S7ir+k6zE9hgd3jzIgj3Y0iDzjv+BHp6/L6phQfb125UQm01wfV+CCFbmKFu9oVExrvyZ5WohFle1gR56yEC/8eQrLYi04N7E+G1YaNthnlXLyBMptBU6Bb4od9BhFpAi/OQmLbIs96O1c4bw4v7ToNUnyhelUuEBwiwHLKUmR0oN3+KC9dfuUl8zy8KKjhupP5tfUrjiw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand --- mm/migrate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 65d64a119cabb..b37dd087da265 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -259,8 +259,8 @@ static bool remove_migration_pte(struct folio *folio, #endif { if (folio_test_anon(folio)) - page_add_anon_rmap(new, vma, pvmw.address, - rmap_flags); + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); else folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); From patchwork Wed Dec 20 22:44:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F263DC3DA6E for ; Wed, 20 Dec 2023 22:46:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2500D8D0019; Wed, 20 Dec 2023 17:46:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D7518D000D; Wed, 20 Dec 2023 17:46:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 003418D0019; Wed, 20 Dec 2023 17:46:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DE25B8D000D for ; Wed, 20 Dec 2023 17:46:16 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B6A57C060E for ; Wed, 20 Dec 2023 22:46:16 +0000 (UTC) X-FDA: 81588681552.30.40F842D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 121C540020 for ; Wed, 20 Dec 2023 22:46:14 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Qzx6VMY9; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I1ebH8KFDZAwQmMcpL7CVfnw0D0yWryPChREMd0TJrI=; b=T/MEKdcZMN+u7Haeg8FN1f35cFD/QhLNOKoxzfo7200qU5U38LpF2rpP2/6kfyKCYeIDtW M2Zz7N/rR3AVerbEQSWEGTKgjXHpkp9z0sdoI3CKJpAf98rnC5+rOd7tRYI4TbKYGftSYr 6HA3JbvpLG98L/1sTb9EUvhbw64AsFg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112375; a=rsa-sha256; cv=none; b=ct6isK95d5CzKPe+N6QItB776YnSL2Wv2/7VRMDXYt8X1qzoVHKwXsTdgIGv4yTZa4/2yk yGihGPQ3p+HWBGmzXOz/h1EgECEPU1LmQDw/+Ryut+nZWDFVQshPexWsYVjcqVxNAgel0m t8e3HqWNqqIaG0Koka3Hb+RbD1NKM/M= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Qzx6VMY9; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I1ebH8KFDZAwQmMcpL7CVfnw0D0yWryPChREMd0TJrI=; b=Qzx6VMY9tescPO2U+WA1J5JTpUj1KyWwlVb/yOuneqg2Gz6TXVSe0eYcJwjIPMdNo/HREX u4UXWCKRyaNJH/V+TBYhcb5yU4u8hcgrVzRqMlbLu2Ul/+4OJRQxVmkmK3/FupCxHsNZcW I/zuC83P557ziqa5pPnaj4qYwjsCZTA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-102-rl_PLPfaOFO5EtohbY1EwQ-1; Wed, 20 Dec 2023 17:46:10 -0500 X-MC-Unique: rl_PLPfaOFO5EtohbY1EwQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AD8D4101A52A; Wed, 20 Dec 2023 22:45:54 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9D45040C6EB9; Wed, 20 Dec 2023 22:45:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 18/40] mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Wed, 20 Dec 2023 23:44:42 +0100 Message-ID: <20231220224504.646757-19-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 121C540020 X-Rspam-User: X-Stat-Signature: 7mdonnjnh7q7zz536pfircnn5s91qtfi X-Rspamd-Server: rspam03 X-HE-Tag: 1703112374-757153 X-HE-Meta: U2FsdGVkX19ly4Fsym8XYaAwWGbtdq5LAEFrAoUv3m7tlBR5RFdbjuDLB7LBZeYCsBzzSpFTeKB/h7Av2+YaXk2mKifPXWXAesJNfbkM5t3WqlfjJjamB3wYpf4cTbja5k26kIGY6QgDLzO2/3BWRNccC1DuL04nZy+0CDaXL1KSvlA7tk/27OFx0gOG000FZx0F+DdbZGQMSchac0WHuljKR7VXogQ0uS5TCtSODMFFWb0z+HV/fhOo3hMIYnkaMTWTF6RHoQgfddNnH8ls58Mt1b+zH+09ReCeTZqMUrAbk8tq3gP31Uaek/dPu4+K/9zHHxjHy6jdyd1DzmbK/bIAFBMqt2faBCkb1PtPEeCVGhkLc4kMDvWtJ9P49tO87RoNZYpfJ03tDSztbg17SGafCgUGXLMoke/BYnLcqYQYY2Qe+gn2m5S/ZO+zUZH9SU+djQP0RqokpTJ0PoN4IJ/weYzEXEuDshgTvDbeHOhq6yFQP/MUzrwp0jOMi9mEjGwqmZaZOI/a5JIbv+r/WxkVuKjsah9dptri6+Nqihaf51xIIC7VCVYSEG2qHOIO35jNc+RcK7GgRlqR0PzowQ6gYEhiATdaeGlIgNOpFBC+u+kZQMitbvzvlmK5XJg35Ydb+l1qrQ5EVmI+27pF8vyga8OPmLjPqbS8Ht3cs/YMIiF6pxIagRWt0FxPBH2/Ufi/8FIppu3vFoWgai3JkstAYaujKTJsKuGhVyImgEx9nWgaR0IOQYSJ+LGjagGq5qk/OpRH1ysU3TBQSwiJ2u2xPyf3FUjJ8BWbWyXx28BmXqza714DEI0ySn1M+kKHwXYgyHAeerz3pUzKq39KA/MNtPmYC063OzfaTj9lOrzahopZQx05ffIuhWbeOHonEati3FKHxxkNRNIn4WCaXRaicN5+dvDxyFJbWYtbq0alcq2cv2Lk/dIoxUOTyrL1jlhEdV4oM/kmJ4tn3o7 DZ5b7IJm kI9oH+rpxfCh1RTaglQsf6CTTU9xUGM3R4nRjtnVzykGccRG/yey5IZDoGagOaNXvIoYzap7AFGsSALSNwFnpvMcGa4OxiDZYK0OUrkL5Ax6rvCzPhRKdkl9Ffre7UlzHHM6ttmOEHuzIlTxbh0wSPoAKMyvCPPhTh8vBHKWss9KBoFeUb0zmKEKyhEIXjxjHjGwOxtl3Qy20Gu72iTRLtTnjGQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert replace_page(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 8fa6053a225d9..146aa75fa6ff7 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1369,6 +1369,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { + struct folio *kfolio = page_folio(kpage); struct mm_struct *mm = vma->vm_mm; struct folio *folio; pmd_t *pmd; @@ -1408,15 +1409,16 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out_mn; } VM_BUG_ON_PAGE(PageAnonExclusive(page), page); - VM_BUG_ON_PAGE(PageAnon(kpage) && PageAnonExclusive(kpage), kpage); + VM_BUG_ON_FOLIO(folio_test_anon(kfolio) && PageAnonExclusive(kpage), + kfolio); /* * No need to check ksm_use_zero_pages here: we can only have a * zero_page here if ksm_use_zero_pages was enabled already. */ if (!is_zero_pfn(page_to_pfn(kpage))) { - get_page(kpage); - page_add_anon_rmap(kpage, vma, addr, RMAP_NONE); + folio_get(kfolio); + folio_add_anon_rmap_pte(kfolio, kpage, vma, addr, RMAP_NONE); newpte = mk_pte(kpage, vma->vm_page_prot); } else { /* From patchwork Wed Dec 20 22:44:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B58FC3DA6E for ; Wed, 20 Dec 2023 22:46:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3628B8D0015; Wed, 20 Dec 2023 17:46:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 310E48D000D; Wed, 20 Dec 2023 17:46:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 191878D0015; Wed, 20 Dec 2023 17:46:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0274B8D000D for ; Wed, 20 Dec 2023 17:46:06 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C88BA1408B2 for ; Wed, 20 Dec 2023 22:46:06 +0000 (UTC) X-FDA: 81588681132.02.A580855 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 19702A001E for ; Wed, 20 Dec 2023 22:46:04 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VYiJaVPa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112365; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QIY4woM6QT5vXKPOcZRD72sCPk41egGwN4b1+ampIqs=; b=3q+AfiEjfXyTt9nJsNYfEtwsrGOYqCtnpY3mktN0bYHDxSYZPCry78CyNGZsT2sYvis+Vx S4NTnO9IHVvizeSNTVlhd1Y6vcZAib1KnKdhX1Bni6DUkzNHEHRon0F76K/rXSvc2lOcZs gPAw/7ZryTfSZGqcSh6vB8JaBBR2qbM= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VYiJaVPa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112365; a=rsa-sha256; cv=none; b=p/hzwoEUI4JRlwaZHRSZwbPGuPm7OnkTlqt0UAqNNNLN01YOfW3ln3jj/r11RQbffD5vIs HnlVacTjyaMNYlATpR2aRTCDl0T0mS4QoZw0HN//su7Xey+UuYBtQIcsSnkx3if6ngPZC+ db8hbL1fgItLxXBgxlpinULHV2ns800= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QIY4woM6QT5vXKPOcZRD72sCPk41egGwN4b1+ampIqs=; b=VYiJaVPaF/JSi4lx1vt/2OVKZW22LcVkW/+gOZlM/FLfq4uKtxbKMNtBkLz89PGjbnLd2I TY2Tgtgnd62R9nVZNYr0SaugquWSYqvB7bR7X03wLgK7LMDiVNCGQC6lkDLyhDWkpOqH/I x5Nnk310/zV1j4cZlDGK6xfPX972Jxg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-340-u30R_eRpM7yzGggrf7qJug-1; Wed, 20 Dec 2023 17:45:58 -0500 X-MC-Unique: u30R_eRpM7yzGggrf7qJug-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 708A1848C02; Wed, 20 Dec 2023 22:45:57 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id F2EAC40C6EB9; Wed, 20 Dec 2023 22:45:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 19/40] mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Wed, 20 Dec 2023 23:44:43 +0100 Message-ID: <20231220224504.646757-20-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 19702A001E X-Stat-Signature: adn19d83okubk1rcjde7kc7wypfn7tkn X-Rspam-User: X-HE-Tag: 1703112364-875111 X-HE-Meta: U2FsdGVkX1/Hq9xPe3k9lUlVrejD0XpBWWqCVTGxlTj8RzSZL6mUVIHRfOiGpaWnyoAqrkw8ezUnekDqIXUrQLOa5OhWCJPsdYsqVJvdpMJnA4T7xskHL7j727uWtoBJ4mrcNxvrgHwsul2peTdYqzxGPTNOYHkPFfI2/lwZ4+RB0DKouL1vIFsLIbzxzFJsk8/1GoorvvqUi2nHzJefqYEhsrbKCg66pm/BunZZVhj2rgqK2iveK/edf1yxqw3/tq++PcCmE09H8c7tewut6jgIRJ3f9XOsv4/JKG4pjz56BN0VvL+y4TI8aMmFE5BehcCQA9782ivs0LeafTlq6mxUSO5CkP7RkGqL04BGL2eVy8U1RagteA6AO6vy9bilfnvtYJlKQvUBQaIZlTPAmX7BNFeZFX4Fw+dGMb/PrcPSGfHDZQB3GlJCtnW/295zWbm9iQXJVV7UwuNUFctz4LMK2JQIkUwi6CCYpZZrqmbUdINJ3PZ6IjSPUq0UcUnk97ZvNzfu1b48cC4R5kVSPsaSDIVP6otXvQkKPTcbX5ptUMt3kMWSd0D5jA/GYvFDOrizv9P6iUxmNURcSWEAJClYafOAIuscjNB97unoE+zaApgi4VmXGEMbsnBBR2t8tlX43urYEV0hDgXxiWayoYpAXDYrZfkooGAdncDahlXKhHSJOq1mwqK8Cp4yd+E1mQ+IkpoERmIuTdGaJXre1CoC8Rc1vURh84fvL1iH0a+D7avmzcBEPXkSFuserxic4UiQ7iQCEI6UOqsmo8PwbBmgP9O1GC7ftlcqQS4SVS0ax4epO2SW6iADMS6qcRBRcVq/e/7SeyS8wvcAtTWmkMyChRu17ZsMVLKlrGTcY01Wt46yXcQRzT+XUOpSLtjquS2wP5hYMXnpZEenfY5OJJ272Z0UdpgUieB7fxskO35FEgqg/RD2vcCTXBquP9dYPkMX+ppKwCiSGtY661t YV7IIy8r isugCi3fM1CillLMUk0PiMNwc50MAyNSnbPDG2u050cRO6QyNLidazXyyidLF4nTkQ++Q+abgIAjt+oCFPY8M68c4lsaAJ2/fGy410ehoVG2jAEihw33V3dkJn95jdWRsDBuJk17Cutlj3kaupGh+Ome2dTQqVDDRoufgWz34sphr6YDBQRagqkqatU1p4rEu+Zpf15bk9kEhBuPLTJsnjDrqtQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert unuse_pte(). Signed-off-by: David Hildenbrand --- mm/swapfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 2f877ca445137..3eec686484ef5 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1806,7 +1806,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, if (pte_swp_exclusive(old_pte)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(page, vma, addr, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, addr, rmap_flags); } else { /* ksm created a completely new copy */ folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); From patchwork Wed Dec 20 22:44:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43B15C3DA6E for ; Wed, 20 Dec 2023 22:46:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDCDA8D0014; Wed, 20 Dec 2023 17:46:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C84BC8D000D; Wed, 20 Dec 2023 17:46:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFF678D0014; Wed, 20 Dec 2023 17:46:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9A81A8D000D for ; Wed, 20 Dec 2023 17:46:05 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7868F1C0BF1 for ; Wed, 20 Dec 2023 22:46:05 +0000 (UTC) X-FDA: 81588681090.16.BB05A7B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id BA9754000D for ; Wed, 20 Dec 2023 22:46:03 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SB34udq1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112363; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9Tzzmpk6zqHgghmJ+i7focJ8fdDU2jQOAW0uh5TyPi4=; b=feZpcJQfUMQo6xy5tN9XSRFfjLTHfTOrGZIxtCLX2Hc5zZfKkr5Ofzbh3Mtu0AxzHUYOtK 6UzyCiPCk+zRQKOmIrAwUTKDiopnP44Y+LJQsUI++5d4aqJG28A+z/kpWzCaII0GmfpUC+ vTfEKNUxYhb5lvrj0rn78XZTWBXwZAc= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SB34udq1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112363; a=rsa-sha256; cv=none; b=5QdBdBogL30LJwzQTp2L4wUfnOxhzr+HnU4xuw4dl/DqtIDL7qpVQkEc1cK5yNkSuhMTUU JCHYv0q7qscD3Y5SHPnlVv9hncZPxKH7TBM/dIMC25QHPD5bc2EtGjcYwKoa0r3miqNFxE tZlifzSdxZnbEs8FwABc0SQYcSpOYqg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112363; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9Tzzmpk6zqHgghmJ+i7focJ8fdDU2jQOAW0uh5TyPi4=; b=SB34udq1VaUYdz2so8nwFIU2I2bhmBDQ5Y/83KIPnAlCF+No+Cszw3ToLKqJ82PgGj1aZk s7p77Apxo9Pp+P0MK+cBsetEnfOseERASUluSEVKDJQGAyeNR4QTKi9xcHXPKVRRhAmI6X dxC4rPQypYIGe3m+dxBbCREwTzGkSBQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-492-LliV0quuP_Oqxcmuv5B2DA-1; Wed, 20 Dec 2023 17:46:00 -0500 X-MC-Unique: LliV0quuP_Oqxcmuv5B2DA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E106486EB20; Wed, 20 Dec 2023 22:45:59 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD18440C6EB9; Wed, 20 Dec 2023 22:45:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 20/40] mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Wed, 20 Dec 2023 23:44:44 +0100 Message-ID: <20231220224504.646757-21-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: BA9754000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 566ogxw45eoz5fyh1mpftwy8nbtftmuj X-HE-Tag: 1703112363-347162 X-HE-Meta: U2FsdGVkX1+OohyGUTezXDBHhyzaWLHB0ZUfW2JSWvxF9F+Mu1x0ZA+vbR9ewEPxZyVXw9V2QaHv6SKiiw6R5TxNVDkSUnY+qIRH7shaG2dEkDhLfmoHiEVVMcOrVxdXMDQhe86qI3lnJG+3HUEdetyeirz63y0TktzIhC+G7PExkXvXjcrHRdEe41SYyFTwASnYz1saafBzaHG3DBNWxTbLLsFaU7kuDDmoVNLj4nxqhoOq6UBE7qmN+iLbYkDBXFI4R0sImqj09ixcrpt5Awnz1RA8SHv/1UxhgA+SHD0lOKi4ZZDomrUmYCrPlsfdd7mUS8RNxO40RkN5KR1lt0cIZ9FJ2YOEZXzdgcQJOuKAQRSQBO9QlO+Q1yrE0ZbFrxhMddC2uf0YhoH7Wi47305paROc+vkbY2Q/FYGQ5QKjviDajrAwmIk4W7Kne8qxFgOEI7qSH2NarPTJ9CBd2/PJMFXDRqKtZ9SOxR8Rj0nYc+rqwagymlyVhme5JIn2u31OPDPZzw1osrukA2mm/cibGg3naE/j7E2JZ0Ss8rptky4lX8TP9g6bl2gboKNjQGYYHqp9GMEnX/5EscZjVYO/qfIi2jNrwM7T2qFSFUC8B7ITiQqTLivv68tUPtN9SVe0awjhUYb4UVpRwliUG2Mn8RMjjmpps3R3vWDgFYZq2bPmZwLEzl6D50NBxfcz1c9M1GJ4bD9nzH4nE9ZHJLkvFR128AZj5RbvNn9Iei7zcm9qBkD307fvXcWwhhMEi+j5P3f9fb5HmhJwQChKwvRHuTVDjG/c9hM852hOW4ChrgGKX3flfmoe7RDviTNYxILsCH7IfH5sP8FmUG9KbMdXm85ONYc4k24X1vkLIt9j23xKBW32f4YS//vaFoJ6n809HSesJRAqg4reM//T3dmJswmUjT63TdCtln4NcZ+Fe2HNPmgqY+498QxaUF2f1ns7BmPEyTzYZvk9n0x gSMgKooX GKFlMNlLn29ce+gojEVpGDKuF91p4w7m3hEDNFoQqZ3zNWrFu5MpsZA0+Xfm5wEslXxEd19t7Xz7/e4/8AWwgikFImHKiW0XInQTLlGi+V0VoC+WafRYIqevmKbOW3njBos2tP1vR0dobDm3kkl3rxMpp/ecbXp5xZ5y2c7MOU8sxMJPYP59Ja6sya4CFiSwNZ8h/z04qNGf7ZTaGnatnw15UXw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert restore_exclusive_pte() and do_swap_page(). While at it, perform some folio conversion in restore_exclusive_pte(). Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c77d3952d261f..6552ea27b0bfa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -710,6 +710,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct page *page, unsigned long address, pte_t *ptep) { + struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; swp_entry_t entry; @@ -725,14 +726,15 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, else if (is_writable_device_exclusive_entry(entry)) pte = maybe_mkwrite(pte_mkdirty(pte), vma); - VM_BUG_ON(pte_write(pte) && !(PageAnon(page) && PageAnonExclusive(page))); + VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && + PageAnonExclusive(page)), folio); /* * No need to take a page reference as one was already * created when the swap entry was made. */ - if (PageAnon(page)) - page_add_anon_rmap(page, vma, address, RMAP_NONE); + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); else /* * Currently device exclusive access only supports anonymous @@ -4075,7 +4077,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_add_new_anon_rmap(folio, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { - page_add_anon_rmap(page, vma, vmf->address, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || From patchwork Wed Dec 20 22:44:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B20CC3DA6E for ; Wed, 20 Dec 2023 22:46:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18C758D0016; Wed, 20 Dec 2023 17:46:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 13D668D000D; Wed, 20 Dec 2023 17:46:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5BD48D0016; Wed, 20 Dec 2023 17:46:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CE2DA8D000D for ; Wed, 20 Dec 2023 17:46:09 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 69197A19FB for ; Wed, 20 Dec 2023 22:46:09 +0000 (UTC) X-FDA: 81588681258.14.723892E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id B0BD1100017 for ; Wed, 20 Dec 2023 22:46:07 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QmEZtwhd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mS0pWaBCqy9h91FWQHLx11u81chGpDs+z9F+Kww2vxc=; b=FMLZUKTpoPwsA+BjaMYFIGclb6qt3rVWEwwKS8Tij74tet+Vj5yNnL8YbXVy8sVwxKeIT7 QaeJwM8vhcxEo8ksdMPnJXoe0qHOTjCIEiyLftvUtXDmdvH9ZKKvCHbroYgAuz8PMEtOEX CqwWZgq31a88Hw63Evcxtm1dLOvgJOo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QmEZtwhd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112367; a=rsa-sha256; cv=none; b=udJQ8+sg22VxWnrnTHJSuWlTZoeAA+AX4x4kzVBUwv9Cwuwh21IpnefhgmE8ykc1YP6ks+ qCEXJAqaQBXGywoC3oTLVNn77MjnuT2mB2TH2MpAYXr3zdvIe/6eYEMECN5d/Mzo6RRR/3 y3skq+AVv6VCTc1gPwIrv9DxaDKWtlE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mS0pWaBCqy9h91FWQHLx11u81chGpDs+z9F+Kww2vxc=; b=QmEZtwhdd+8IYU1t9xWI1tmZWcs3QmqwNJuH9uvOmsaiIwpkkGoFzunnGNhdxQHejt82cw H2qB/b8IusAfBUmd8j1iN6BHYI1Ono/XKOHDvXSHhhos/38ypXmnjTCMlyTFRda86kcEdi E9VDdPVnAXJj8USlXbHE2270E7v4grQ= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-509--oVgBcMTPd2G_zQpLfDtjg-1; Wed, 20 Dec 2023 17:46:02 -0500 X-MC-Unique: -oVgBcMTPd2G_zQpLfDtjg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4742E3C23FCD; Wed, 20 Dec 2023 22:46:02 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3692540C6EB9; Wed, 20 Dec 2023 22:46:00 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 21/40] mm/rmap: remove page_add_anon_rmap() Date: Wed, 20 Dec 2023 23:44:45 +0100 Message-ID: <20231220224504.646757-22-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: B0BD1100017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 51qiiq6qu11epcqbubjeigko8xqbi3am X-HE-Tag: 1703112367-84060 X-HE-Meta: U2FsdGVkX1942P6h9awluVsZIFvMMX5dRVHNaziiuKwJAqfqHtfIBMdMtT51cd8okIeBwG1ubkwXResl/soVgmTnY5DIinrrNoK8M24B0TfBh7X7vXCj7KP5ocTnsSpP2b10ehZVzrTj89/fYMU0FNvEbqfGS7z/Z1g+Hhd/ATPHIZeKHasBkbTFVCSr33EggxtaFMXIlS3SBlJVN76MRJ9hYOQAoV6PbWEnSb0eI/NC3lmPB25yPQ5Ut4/yOZuu6BkqCxFTRiRAIaeIqYK8w7ECcswfj0WJm3mRruum7QVBP9HGpvpZyrm/vKABAOBWzw4fHnsCAirpNuv2wm4ekcuWHEggiNjCv+bjG5OuiC9tBCixFpxspHBzxZmpKyORwKCeCJNfTviJTOu5WkH2fMAGKATGGe380Zv0aVx/7uYRdvvAFe+bIqt8JrUTaubemKpO5GBoZitbxiZPM4TC3xaQNnwd4hp+cmoe3elxJwpbZUn/51QJAelLdqLN5V0iFkKAu7e/JEiCzcEjKDNpXy237jNh6HoT69/VMk3KodoBC+9t7d/dxxjbpnkHbPeqmlia4BfaUZaB3uaP5Hd7SSbrJvy930mlw6vRXT/S41dvpqChJjiVfBbqyM/QzXGwsDw+7/XqY3HxQLaz9uOLAq0bXCHudyXP3pzOHDOl/oI0fKev4feNR7AiIQTt7m2kvK3kCo3XxgbvrTsQDw9A5Fa8Jps4uyNJtGTLAl6LnKeaUu/e3hLy4c0XgqSBrDss9/12ogwHWgnBtwZHtqxmHxCPi1cm11lI7KBlb/RIgl7U7s23lFvKHbg2zmk9qDGfMdXHGaJicYhaxFALCSqnx82BRDARTQkKR9ANfbEZxZe9Qq0d9KjRtixPMHiOE3mr+nVjpIvztK+6t5TodNR+mpvD1AZkBp9Ve8QFx6hAbp7CGfyUeGlecJ7nHLHuHhTAFJKeGYbjAvpVQuIHl87 prXOMFAl 4P27pN2cnravFvDv8Ya6IwXDQg0biOozlrlmYHlzN/2VO9durSpKCOHnLgZYYPqFqc5wEX9xflORMZIBcAIQmzvvsIF+89SvSfXLNQK4b01L8yDZY6/P+bmHLjf+RbE4EO0pFDcbdvxPrPrC1kVhfQhGxn0TTRPlggpCiN5NQelHWppJddc31vI1TuLu1zia6slR0EGL3DWmI7QM1C/9PhmwVtTFliudXDXnV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, remove it and all traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 2 -- mm/rmap.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b5da3d86200e4..fe7b5a8b0e75b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -239,8 +239,6 @@ void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void page_add_anon_rmap(struct page *, struct vm_area_struct *, - unsigned long address, rmap_t flags); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, diff --git a/mm/rmap.c b/mm/rmap.c index 7f380f5a34c90..87415bbf24022 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1270,7 +1270,7 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, * The page's anon-rmap details (mapping and index) are guaranteed to * be set up correctly at this point. * - * We have exclusion against page_add_anon_rmap because the caller + * We have exclusion against folio_add_anon_rmap_*() because the caller * always holds the page locked. * * We have exclusion against folio_add_new_anon_rmap because those pages @@ -1283,29 +1283,6 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, page); } -/** - * page_add_anon_rmap - add pte mapping to an anonymous page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @address: the user virtual address mapped - * @flags: the rmap flags - * - * The caller needs to hold the pte lock, and the page must be locked in - * the anon_vma case: to serialize mapping,index checking after setting, - * and to ensure that PageAnon is not being upgraded racily to PageKsm - * (but PageKsm is never downgraded to PageAnon). - */ -void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) -{ - struct folio *folio = page_folio(page); - - if (likely(!(flags & RMAP_COMPOUND))) - folio_add_anon_rmap_pte(folio, page, vma, address, flags); - else - folio_add_anon_rmap_pmd(folio, page, vma, address, flags); -} - static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, unsigned long address, rmap_t flags, enum rmap_level level) @@ -1419,7 +1396,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped * - * Like page_add_anon_rmap() but must only be called on *new* folios. + * Like folio_add_anon_rmap_*() but must only be called on *new* folios. * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * @@ -1479,7 +1456,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - /* See comments in page_add_anon_rmap() */ + /* See comments in folio_add_anon_rmap_*() */ if (!folio_test_large(folio)) mlock_vma_folio(folio, vma); } @@ -1593,7 +1570,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* * It would be tidy to reset folio_test_anon mapping when fully - * unmapped, but that might overwrite a racing page_add_anon_rmap + * unmapped, but that might overwrite a racing folio_add_anon_rmap_*() * which increments mapcount after us but sets mapping before us: * so leave the reset to free_pages_prepare, and remember that * it's only reliable while mapped. From patchwork Wed Dec 20 22:44:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E5C1C3DA6E for ; Wed, 20 Dec 2023 22:46:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3A9A8D0017; Wed, 20 Dec 2023 17:46:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BEAED8D000D; Wed, 20 Dec 2023 17:46:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3BE68D0017; Wed, 20 Dec 2023 17:46:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8F8B38D000D for ; Wed, 20 Dec 2023 17:46:12 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6447FC060F for ; Wed, 20 Dec 2023 22:46:11 +0000 (UTC) X-FDA: 81588681342.23.A41B027 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id A8B981C002F for ; Wed, 20 Dec 2023 22:46:09 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BQs0h7FH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112369; a=rsa-sha256; cv=none; b=wmgw1OIBwS6wGP4888DdVaNL9onGttIxicOiQdCz5T2CNvXOahof2czdpSs4LFkiJuxbYJ KvUEOc/Ft+Y0hfpdCqJhdYI4sM4AvPq3soMSFQuuwwbYGZsGTPsxVodtdcLSSKA3tPaejn E5oOLOjJVTPtPj1EttfKkyY7ZNePl58= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BQs0h7FH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112369; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H6kyFjjwWNovISb45JpAxGunVS8pf5Q0izk9V4kCoBM=; b=g86Hw0YnLAUgyzbPyLGrsL1OEnk7UWbnY0AJTqCEhfqR/TkME5ccqqLWwhZpL1w4uIuzfP cHGH1bWx7Ywct5QzuDnfrP8NBWsxRCel8i/lGsrTNlrHp2tJSj4ddQiseoRVXUoeHlEdEo hWWSfjZDNfDy/DYDzxJhOiGJM84/1ls= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H6kyFjjwWNovISb45JpAxGunVS8pf5Q0izk9V4kCoBM=; b=BQs0h7FHPEtRwOHfiZyDBykD3sGV2U2FuCjkcULe9ALy5u6qbBhud46omYfbx9WSSwksar 1lTFgfx/RYO80e/DkitT9njFFxPMFaxQqUZHGcIkfwCq4cJUQ+baOWZ3pQlVmC/FN4hOif aPGNMztUwXMwGQqX/kXwZidqBROU4iU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-564-BPRXZUsgNgykeRsAk0KDlQ-1; Wed, 20 Dec 2023 17:46:05 -0500 X-MC-Unique: BPRXZUsgNgykeRsAk0KDlQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 09319101A5DC; Wed, 20 Dec 2023 22:46:05 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 90EFA40C6EB9; Wed, 20 Dec 2023 22:46:02 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 22/40] mm/rmap: remove RMAP_COMPOUND Date: Wed, 20 Dec 2023 23:44:46 +0100 Message-ID: <20231220224504.646757-23-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A8B981C002F X-Stat-Signature: hpsyk6gfnwuos6zdrcuxkep3f4w4n7zj X-HE-Tag: 1703112369-688819 X-HE-Meta: U2FsdGVkX19oR9AJkJwL3cafSwzC0PCucNl6CMLzVgxkfFraPOjugBE2vmxDym7NfBGLe9TwRYlw2w4CMyBhtYiE6H3+O2yzMzB6nwjL31dF+c8adVB1bExjAOWunC+EdA1WzhZUMC5NDeExFfXE1dAYIZFe2T+idXg/HWOrbrkXZAsR81JZ8gENdBy/x7v4D8kFxYsPV0H577cogtCYmB0zqnwAL6ezzAd3FNXdNMG7QWKYfeRaW3tHAarpdAJENV4Z5/pVeUq8clpsmCRb7e5e80mBh5ZdRwECTSoP3yq9riQ+Zo/VR1mOWa3VIrR1QJ7Ege5HecJZd0CSNMxc9GfPmpL/DmBZ4DyNCIilQ2w8ABVQhDCSFK3lIHvD/MYFhxMblBQ08HE2/QDrIo/Nk0AE2GYE6EeHVAB8w8Snkhmfy21YoLBOlbAA/5+4vMus5YbBMv0t6T3z6FNhq/1/v1WEtRZqzV6ungHumIWc9KWg74cnYELMQxDevL5jDekWfR9GUPjyDT4mDfCXuL5CAHDuKgM7EUSFkj9Sb4Ec2LS8hNO9B6FJRZVvF0hFGbl5oMJ+a0sAub9mwv+CK6UwRzvghqeDrTyAgqgBNXDTcBtLPRRXiiX+/oIKeiAIAtvuKBHaRbFXTGZFj1s5Frce0dDcuZaLZ8OfMnT5H4dQIG6U6eXcy0923zK9R+Ws6Iv/E/5b+C0p4I7D2RXpEiZzj3OVk9Zm71syNpsNSXvM4WsccLiQwrQZXSgzF1/gDRsBy+2p7Mkl7EveSBPSs1/jKOAsn0X0ZSN101Efze+3RHIdvsAkW4YS5VLc4YtyzH7583vvKrNFCzXZNdyuQyC+f/m9LOLxPxolTuhv3IQTvOPnIbQoDg+7ulVIJiieVn52QEZnw8Osp7Qyz/fmavw9IYnH37ejYVnWMj/Hu3y0WkSG+EotGMeHM4KEDlyMxMQoO9W5JE0loxBU/dLj16b jE9bvEhx AxPTJZw5Uyuj5cQgK7ppIDgWze2DeNuwfvtznaITImbxkk7SC4VOYwu15hi0EP1ey7nqLu6QGsfQspnvyEHibG+Ytku5C4m/g3ezbOG6xRr8OoPu4XP7veuzH1bKvYeC3Eg2LZhblBK1nXw2uUH2foPpTPEwbFRU4LDLqsxHifYLyCE1PRIRACV+6FK4wLgQFduuDfGGMYhQ5ju3egmgucY023A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: No longer used, let's remove it and clarify RMAP_NONE/RMAP_EXCLUSIVE a bit. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 12 +++--------- mm/rmap.c | 2 -- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index fe7b5a8b0e75b..bf6cb79aa7a0a 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -177,20 +177,14 @@ struct anon_vma *folio_get_anon_vma(struct folio *folio); typedef int __bitwise rmap_t; /* - * No special request: if the page is a subpage of a compound page, it is - * mapped via a PTE. The mapped (sub)page is possibly shared between processes. + * No special request: A mapped anonymous (sub)page is possibly shared between + * processes. */ #define RMAP_NONE ((__force rmap_t)0) -/* The (sub)page is exclusive to a single process. */ +/* The anonymous (sub)page is exclusive to a single process. */ #define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) -/* - * The compound page is not mapped via PTEs, but instead via a single PMD and - * should be accounted accordingly. - */ -#define RMAP_COMPOUND ((__force rmap_t)BIT(1)) - /* * Internally, we're using an enum to specify the granularity. We make the * compiler emit specialized code for each granularity. diff --git a/mm/rmap.c b/mm/rmap.c index 87415bbf24022..2b386b9f6791c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2662,8 +2662,6 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. - * - * RMAP_COMPOUND is ignored. */ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) From patchwork Wed Dec 20 22:44:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13B42C3DA6E for ; Wed, 20 Dec 2023 22:46:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22A038D0018; Wed, 20 Dec 2023 17:46:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B35D8D000D; Wed, 20 Dec 2023 17:46:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02B8E8D0018; Wed, 20 Dec 2023 17:46:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DDC608D000D for ; Wed, 20 Dec 2023 17:46:13 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B6427C015A for ; Wed, 20 Dec 2023 22:46:13 +0000 (UTC) X-FDA: 81588681426.01.4DC1DB0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 10A1F14001A for ; Wed, 20 Dec 2023 22:46:11 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="D7/ssT31"; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112372; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/d9mpve/qdOWVGsQDSuYBEYTclHzlkzYvVo312FD3s0=; b=49xSqMeUBrI+8ycK7FJV355ZgzkpfnB+fgBtymuzRxNyYGSuV7OXCrRK98e2Rv9gVzRgL+ g/aogfU3rvoS/7+dt1acrTAZKwZ1HFKN1aT/2JqDlcDwHd/WzeUVCxUVDQWl5b0TbmoKpn TA4HOmXLa3mlJRFMmdhEJew02h3mad8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112372; a=rsa-sha256; cv=none; b=N90aFYowt+SbyDkC7NTm1e4foYw4PY0DVNHocKNawiobsr96GTSTFp1RNx4KY4oCgNY3bw 6w6MhM2dcm9hUJ8CjOZ93DvnTcvcqJ867R2Qo6AqWtU/Cv4Dd7m54vHm/WReHzZ3PxpRUe xd9k2lvH5hsiLd6f3LQPPTJs802NW6I= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="D7/ssT31"; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112371; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/d9mpve/qdOWVGsQDSuYBEYTclHzlkzYvVo312FD3s0=; b=D7/ssT315aHkEftYfRXpjmo5fAGGtKuPyVTQNzjv9tN6iRHTEt8Zldk2CxjXUe0awd4zIP nT6sSRQ09krktda2Y10JbaQ+k3v0haRa1zRSyQ4GHokFwzPzl+VlY/BdXwHcqZyz7sZP/p aKmxzDIRZOXZ9aXM0Ie2Het8/XC9Th0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-640-h8r1ias0P2CwC8ml7LleWA-1; Wed, 20 Dec 2023 17:46:08 -0500 X-MC-Unique: h8r1ias0P2CwC8ml7LleWA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 923D42803609; Wed, 20 Dec 2023 22:46:07 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 44B9640C6EB9; Wed, 20 Dec 2023 22:46:05 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 23/40] mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]() Date: Wed, 20 Dec 2023 23:44:47 +0100 Message-ID: <20231220224504.646757-24-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 10A1F14001A X-Rspam-User: X-Stat-Signature: fib7cygt5yexwngycijpe9xg9cn1kd3x X-Rspamd-Server: rspam03 X-HE-Tag: 1703112371-751776 X-HE-Meta: U2FsdGVkX1+aw3jKF5tGLGoMgCggod07u1qomNHUADfC0c9Tb+CKwr44S6ELCWJZFEOJpzrRvKdE19wN5lSoBEoWrQGfAw2AR1qeaymDjaSbMCRTXqv6cSUXzherSF7S+ZBLKA4fKNQQLhWiTisT78JAFKVjUKylnlYOQtEPN6afrAFK3J86MeCD2IPZKwpem66Kd83BlCFY9rTQ8Anz0tcW9rOn2jgI/xdBJvdiZb1w/xtiO3F1a/zmTxuwLJZvU+Bt+Rt1Mw9z3AvhH+xAygSuoCeJ8H+GvCNlXQljfZgVysu7eD9TUeesvbwc8Pzh7RTm0Y0EuVeVUMJIsiqDZ91M5y0RQXlfo1UPGXOeG6jSozmSNiyh8GDAn83lfmAe69CNw2TUHXD81zVoebXHujavAjK06QxBNO5o9wD8IWVxae2NMXuC7A8UepH90iPUtIMPPUgur1zRHTqHEzdQ1CxOyqhQC+mxXUueuNi4LGdxa2kt4SGPzSNIzd92Sar5M5QL0SbnnQ7EHnCgMx3JQkEvuSKkm+zPu8m59H9enPtRTmSDhKl1Aknb8aXpgRTrxAYhaPfghQTbaQfyMlLq2Rz5gWQtN+Z7qskNIg2iuxOpJfJzJgG2HGmS4Uz0cPdG+SpBogXbcr7mmIsVOtzwxbi5NRE7BW7GEiHsHCY12Z0nQLFFKN7EZBvABscjme4+4BAyyttZTPaWuxrDC2NlqMz7UCoU8jYJW2418A0Ce3bXN1HVXTtxFMec0As8lB27OHdOFLsR0ZWaeg8tPZGqrbmB4XjXsLAzkxZIpO0D4Tco/DCqNZc/ewu54B7mnsAL2yvxZACpHgqBFgVbV6HVuGCemKXmE5tZZFS9qIXfzYdIKRGbQ0NT3Xf8JTcr0M4UXg31S0w80UdNdCcljDypQbbFLxkTh5dLzyIZmf2ITUTMimCKMkHmDgAfselEtU3Gs3PXTdW+JlmQuuA0Zgr QLv56x6c 3GPWZQxHsnkeUGCEbGPkpmLUoyi+7tee+L9GP6HjdXtId0pEKKJou2GCz10E4Tkat6OUXzRGdMx2BuhhDZ+28yZtzhVZ0KMuzaSPrR71ROtVDlbv3ewAJf02HhZfLytl2OHNu0ZHn56B0oXjQ2KqBnKdN3Z0TmOgvVfVZcOWSxQ6tEplN/ZXI4+9NUIcvexRPz9Fc+GTpiBpwHLFYkx7ZbP0HR98uxY0/YFtm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's mimic what we did with folio_add_file_rmap_*() and folio_add_anon_rmap_*() so we can similarly replace page_remove_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. We're adding folio_remove_rmap_ptes() handling right away, as we want to use that soon for batching rmap operations when unmapping PTE-mapped large folios. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 ++++ mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++--------- 2 files changed, 72 insertions(+), 16 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bf6cb79aa7a0a..57e045093f047 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -243,6 +243,12 @@ void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); +#define folio_remove_rmap_pte(folio, page, vma) \ + folio_remove_rmap_ptes(folio, page, 1, vma) +void folio_remove_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 2b386b9f6791c..1273180753953 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1510,25 +1510,37 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); + + if (likely(!compound)) + folio_remove_rmap_pte(folio, page, vma); + else + folio_remove_rmap_pmd(folio, page, vma); +} + +static __always_inline void __folio_remove_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_level level) +{ atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int last, nr = 0, nr_pmdmapped = 0; enum node_stat_item idx; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); - - /* Is page being unmapped by PTE? Is this its last map to be removed? */ - if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + __folio_rmap_sanity_checks(folio, page, nr_pages, level); + + switch (level) { + case RMAP_LEVEL_PTE: + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + if (last) + nr++; + } while (page++, --nr_pages > 0); + break; + case RMAP_LEVEL_PMD: last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1543,6 +1555,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, nr = 0; } } + break; } if (nr_pmdmapped) { @@ -1564,7 +1577,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * is still mapped. */ if (folio_test_large(folio) && folio_test_anon(folio)) - if (!compound || nr < nr_pmdmapped) + if (level == RMAP_LEVEL_PTE || nr < nr_pmdmapped) deferred_split_folio(folio); } @@ -1579,6 +1592,43 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma); } +/** + * folio_remove_rmap_ptes - remove PTE mappings from a page range of a folio + * @folio: The folio to remove the mappings from + * @page: The first page to remove + * @nr_pages: The number of pages that will be removed from the mapping + * @vma: The vm area from which the mappings are removed + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_LEVEL_PTE); +} + +/** + * folio_remove_rmap_pmd - remove a PMD mapping from a page range of a folio + * @folio: The folio to remove the mapping from + * @page: The first page to remove + * @vma: The vm area from which the mapping is removed + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_LEVEL_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * @arg: enum ttu_flags will be passed to this argument */ From patchwork Wed Dec 20 22:44:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40D01C3DA6E for ; Wed, 20 Dec 2023 22:46:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0042B8D001A; Wed, 20 Dec 2023 17:46:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EF7938D000D; Wed, 20 Dec 2023 17:46:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4BE38D001A; Wed, 20 Dec 2023 17:46:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BBA468D000D for ; Wed, 20 Dec 2023 17:46:17 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9C59314055C for ; Wed, 20 Dec 2023 22:46:17 +0000 (UTC) X-FDA: 81588681594.28.A201948 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id EBFAE10001A for ; Wed, 20 Dec 2023 22:46:15 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PVnQMLjg; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112376; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tVk0EEeLHw3+o76tswuza0N9QDfjbUBpt28VqO6EQjM=; b=THM6SzxIwpgRfj3LkQ29Et7tneLi2gvFi2ARzlz1RzqhZa4l7Qtn7Kf+9oO6264tRc0U+Q y3/IYZaRCQUHTGDH7YJ/fhAeZab1um2OGxnoSeYppYCtmYCJPQH3OmTO/UGI7t00VyIF9V 1y/PDjmhOCk14RF0l0UNrFO6hNZaMZI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112376; a=rsa-sha256; cv=none; b=70lNU6bD/0t2OjDAv1olk2ZBw0k0reKQ9udNS9cbNtBj4nV9fxCFBmU/hiZqutDwEAqdWw 0RG4nn1xO9EUj4PC8GWv3Td97Uf7u9RtsZlQl+XpwIhiUsHwbFvkibPLJFSI9t5PJbb4lu kk722rVCaUWLEtSRVhYXQq7qczedHs0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PVnQMLjg; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tVk0EEeLHw3+o76tswuza0N9QDfjbUBpt28VqO6EQjM=; b=PVnQMLjgR0K2aR9IRiLHuITcihax27LX/imexNqtO1KofshwaJmjevWoybPvfaNdkVRsVT ZdRPLa3xGbmtP2uqoH53WELjMib2/lzv1wZL2VBqbsvCM+5eyoEoSoSGzHD6Bn5SOxw1mV /cXF38ebCU4c2oVfiuFp70+F6qXmUeU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-447-W2MzE-l6OeqzSGd0SCRvqQ-1; Wed, 20 Dec 2023 17:46:10 -0500 X-MC-Unique: W2MzE-l6OeqzSGd0SCRvqQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 437B786EB20; Wed, 20 Dec 2023 22:46:10 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id F294F40C6EB9; Wed, 20 Dec 2023 22:46:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 24/40] kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:48 +0100 Message-ID: <20231220224504.646757-25-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Stat-Signature: gdq1mow6cpn3xirdcw799yd7c6sybk6i X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EBFAE10001A X-Rspam-User: X-HE-Tag: 1703112375-93142 X-HE-Meta: U2FsdGVkX1/9LbYL30iBgUhLeGUBL1WCjXMvl5L11H0TH9E/2ESyxmEmFVj7gRVvSsdEw/tmWJCo9SMIeeSVNnv3Aj9tyWmGJ3MHboJEzsSlju81W8DlB9NvR+hdHzDkhBmaC/Cdj82FRD+kUPIqJjsY7RqdFvJvbtpN2bn2wyxp5fZLhM6vI+PYY1JLulXBQQ9wlMbuGnkOFTydpToRfpZA9ULn4aQhMSCN02PaBsJRr+aFPkjsT0VWooK5h6AA2fn5aHwvc4eO9nkOMveirx/6IuHejeWkjvZqcsQoMPaZeGwfvABQurK4LwsoNjcNA9XJbALgRk7lwooboYL8bZU7nDFhN3SSffe3XSi3rYXrvGPrrmycKrwZXV37szrXdoDhq+eduHOp1CK4oaSATGBGRLRTFrr71+UWWqqksukoK0jO9TT82rNpIeqcLrH56/3JgbalUV/VICfM00bg55jQ1ZG0CD4WGNhCH1GESUTCnmqoiOheqrCYdV4Y6Zn4pdkd1psDBlqiAwUh+OYXQUThKUnLpQUYPjyuhgJXKnkdmVQSIXYfHt8RwaSjZQujUdQwC2Xc1eqXXASl7yqlC/I7pCGkTvqrvUGbX7yrd5D0zhqier0ZeooSNc9a+5YJh6P1ZyANZlbWLxABmHWZUg+x9FAF6SDzFXlZ8YmJX8SjlCDDqBjeKipM15pZXMjPnNWnaX7+VRogvtX/d7vYPRp7CfZbPVp35NFF7uMQQd8ue0qTpucKzXF/EWvKtkRzryfVBEeR9Vi5Ktm8/vCA0W3cCtSbsasKM5lxnRVZQTmJMq02veMC4M3ceUVy+bjPZBTw/diSoHnD7waYqgpl7U5VAm1tZkRkPyMdzyW+1lCM9VWMpd/Iz3W3Iqg7UGpMKYCn3vrhtUEYzP9WDNt5NGPAKJGUMcbkjuLdE4LUY1l7yn3moAT6YYFibgXhRnY71BPKm3LOlgvkiFKVxFA 8BvYDmha IIn4gg6l4e8/9d1CHKHGosdELmMmyo32Qv1iHbL4XppAbMaIhfanP39BViGv5sko0NS4q9eQRq65uiEjBPUXsY/l16EoL4z5w8D2Dpd5VpbQ/Swi2jrsvcUfbIlRkA6+D83oO+0KRmjhlu8QDVt+M8vYTq6reSUTgJbveiIJ/m7t6E7UOkPQx9bMnH4ldMyMjirpp4iO95WiAncAU5k9B8+YBmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert __replace_page(). Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 8b115fc43f041..485bb0389b488 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -198,7 +198,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); - page_remove_rmap(old_page, vma, false); + folio_remove_rmap_pte(old_folio, old_page, vma); if (!folio_mapped(old_folio)) folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); From patchwork Wed Dec 20 22:44:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83741C3DA6E for ; Wed, 20 Dec 2023 22:46:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 471648D001B; Wed, 20 Dec 2023 17:46:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4204C8D000D; Wed, 20 Dec 2023 17:46:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24E1A8D001B; Wed, 20 Dec 2023 17:46:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 099008D000D for ; Wed, 20 Dec 2023 17:46:19 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D6D0B1C0AE8 for ; Wed, 20 Dec 2023 22:46:18 +0000 (UTC) X-FDA: 81588681636.16.2E1114C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 29C8A40004 for ; Wed, 20 Dec 2023 22:46:16 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="F4o/Io7y"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G1DHm1KAW70S05Ll7y3nAT3Y+2rV8NSuluKMu+qR6Lc=; b=IMoWIPkgr2LxjMjc5LvhGfsUeYJGb/zVNm/bPJPwVykJHK/guDiUZtw+G/MVXPnfMttE1R mhQS/rvS+YR4/JLG8IksxT2dEb7708tObE88ddpBfqF/ZMbyaIE8AfaKZNetXOoIw2gtih EJNG/rghJrXDwU7CXD3QxI/8PlEhJYk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="F4o/Io7y"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112377; a=rsa-sha256; cv=none; b=pWEXp9bHGykc/oVmCMw7Qv6O3th6CUQTm8uH1/27cCTvARipTw0LREpKOuM9OIM/69lDM2 h5RnOv1jv3EFH+CTcSeHr/c94pi/PWr9e8gbzftcARXdITAD7mN+fQdabVCGLoXmPCMigV A7k1YgArfLYfwVTLoAOc+AgqBRlUqzk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G1DHm1KAW70S05Ll7y3nAT3Y+2rV8NSuluKMu+qR6Lc=; b=F4o/Io7ye7Rz6FqU2YfceG0cb9irIrqNe77YqJ6SsysI3AnoX8z4X1z9FVE9oHE/yGGrCF tgOo5wWDLSofuzrLeSSxG+MqwS5BFoMRGizxW7pK+e8rInC+KyGcl6jCfQWww9sH32fKiR 7Q7FZFciVxFMiWYVHc64CP6vp8QFgpk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-365-LtmkEP2FNZWZAiW0FRkA1g-1; Wed, 20 Dec 2023 17:46:13 -0500 X-MC-Unique: LtmkEP2FNZWZAiW0FRkA1g-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 97715848C07; Wed, 20 Dec 2023 22:46:12 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8000340C6EB9; Wed, 20 Dec 2023 22:46:10 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 25/40] mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd() Date: Wed, 20 Dec 2023 23:44:49 +0100 Message-ID: <20231220224504.646757-26-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 29C8A40004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ktr4kmwy8h993d9t8qeie85escb19ddo X-HE-Tag: 1703112376-535826 X-HE-Meta: U2FsdGVkX1+/hODDlVje3b6jPMYxvuFSmFsxPpHqnBjx8rZ7MACWx6Kk6hNDUC9CrNA08IdbQu/2m5HOLDU8Ik92CR2xxPNEh56myi9+emfRwUZj0kWwMGwhYFdF3RcqY/XjCmMcSp/XABfZppqBvkAbocQUOzeXuVMQe0SKhIeVPGGM+NGmLih350vJChe7p0jPc8wbjZXGZOkUuNkCrzQJQAvgkfYOXBUrNOCCKPPk2AyyCcJ1TRMz6yCT6FuoXVUFi7+eGiX/1egHp0tErHwNDczmzC63cr710H5+JPhnIh5u7sCPtgmXBDrmomxfjksqWCed4ygtw2GrKmsAVF3lbvS0XEN0LpXKOhaS4l+0ItRNri56WCSJVYaa4TEL/iDpOwtsxHp/oOTQx+PC8hVGBblie4QLWAO8THCch40Sd/NRCkTK/FkOEIZHhC8dc7eILMO+s8OXd3DKXPeH2+bZeVI00P7e3IYpjzW2t7BPOYzYd7vsqe7gJDiCy13GDzaMqeFlaB21hKdw0jsHBBuaO2hsQMvt9n/7Sxbh8oHmFm2uMbS/aPUm8s6tg2uFtdTyzFvZh6Lykarolmr9/+HR+f1rgNAJi06dWVlcJmLJ0RxsoM643PEkhOkzHvgETeJw8grnmH/EusaQEbsjOpNhVTHvNNEaDryB3jsGqnzfNRSxJidM56NX0EWL2XQs92JUOVdvCzpPxHJSPonWYfy2a7XZLitI3sHl9RuToO78Wh51POwG91xGg8nkPT9WwhPo/BYiSULAgl32WeS2kWo5MOpOrWZmxIJ9aL02XYzjYjD5n5w88wugrifOXSNA1u9vslWD52z/3bw1hxaqQYZqV830UVGAVtN4z8SC9w/E8cm2LAqGXKFynKDauBIKCJQxQwZ9II5xpmVUkMwYegaopJfkYV+iKmboBAZzBteyeCakqksMeCHCO9trbuLMb2t/7E4oPPVrMkw+x9b 5H9r8AZj jfgXGQEkhaxSlq3j7CyXM+0d4HaAHOoA1cpX20QvLHw0ZhUj8fvwb3Hyb5u/0yPTFVVrGLdzGSIQrvBxZut6wNoDFPbseepp2gy9Fq0ZzNzETAMbSM/tiGaShR47e/7e2PuHbs1G6JT5uHh7wS4zjWg1lwja0MgFMJGdNohDb1Jm83pKOK+cWbt64fzq+n8hZAux7RkVKwyuFQKi4AbXxHqwQYQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert zap_huge_pmd() and set_pmd_migration_entry(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1f438326b69bc..e7bc0f38ddc53 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1898,7 +1898,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (pmd_present(orig_pmd)) { page = pmd_page(orig_pmd); - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(page_folio(page), page, vma); VM_BUG_ON_PAGE(page_mapcount(page) < 0, page); VM_BUG_ON_PAGE(!PageHead(page), page); } else if (thp_migration_supported()) { @@ -2433,12 +2433,13 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pfn_swap_entry_to_page(entry); } else { page = pmd_page(old_pmd); - if (!PageDirty(page) && pmd_dirty(old_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(old_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, vma, true); - put_page(page); + folio = page_folio(page); + if (!folio_test_dirty(folio) && pmd_dirty(old_pmd)) + folio_set_dirty(folio); + if (!folio_test_referenced(folio) && pmd_young(old_pmd)) + folio_set_referenced(folio); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; @@ -2593,7 +2594,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte - 1); if (!pmd_migration) - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(folio, page, vma); if (freeze) put_page(page); @@ -3536,6 +3537,7 @@ late_initcall(split_huge_pages_debugfs); int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3551,14 +3553,14 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdval = pmdp_invalidate(vma, address, pvmw->pmd); /* See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); if (anon_exclusive && page_try_share_anon_rmap(page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } if (pmd_dirty(pmdval)) - set_page_dirty(page); + folio_set_dirty(folio); if (pmd_write(pmdval)) entry = make_writable_migration_entry(page_to_pfn(page)); else if (anon_exclusive) @@ -3575,8 +3577,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, if (pmd_uffd_wp(pmdval)) pmdswp = pmd_swp_mkuffd_wp(pmdswp); set_pmd_at(mm, address, pvmw->pmd, pmdswp); - page_remove_rmap(page, vma, true); - put_page(page); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); trace_set_migration_pmd(address, pmd_val(pmdswp)); return 0; From patchwork Wed Dec 20 22:44:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1098C3DA6E for ; Wed, 20 Dec 2023 22:46:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E39698D001C; Wed, 20 Dec 2023 17:46:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC03B8D000D; Wed, 20 Dec 2023 17:46:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5DB18D001C; Wed, 20 Dec 2023 17:46:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AEFF88D000D for ; Wed, 20 Dec 2023 17:46:19 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 963EF1C0BF1 for ; Wed, 20 Dec 2023 22:46:19 +0000 (UTC) X-FDA: 81588681678.07.E8E3B3B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id C97EE40013 for ; Wed, 20 Dec 2023 22:46:17 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ByDGAS5p; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112377; a=rsa-sha256; cv=none; b=JCUUsrdmIcSdsKKDFzbke16RPLykSVgn1tTwg5P4av4lHUe/MmW79rzZgCDj0MJ12eeRNT qlTM3klipmfLIP/N3OYOK542UC9g7thBg0/vPPvoAXFZRZK9fwG5d7uTlyDN4d4Gvh55Pi t2Foc4yWmg7mHi1T6W1All2f+ic3C0s= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ByDGAS5p; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hbXwGjN0pCeXqd5W9uMe29zoEexoygHhqKeEUbLF/bA=; b=BIRDTetVzyiukMHE3914KAw2AlJEu5VLxqeamk087disX3HwBCaCAdTw3eBjhXzp2VX6a8 VWRpvzqdSYdtN2g3vjZgJXG/fkLypHyYna8y71586roSsUvLhiyjpsnxcjW8AnshEZCiSF Ff3kVgrGAfBVxdlzDQ3wK/XuQsRpCQo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hbXwGjN0pCeXqd5W9uMe29zoEexoygHhqKeEUbLF/bA=; b=ByDGAS5puFBZwl9BJNlPCWBANHjmmbOEssD9lLxW37krxu+qvItnW9G7wc0v/i5lZmDmt4 znVft8knsWNMODY7YutHojYL72WuzWmMZN538xAPZopk3Ev1GeC0XI0h8528nqrYHepG1g /1RbQyuoPgMJ1K6xvQDCtALj5jYbfUQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-482-jF4_JFJ2M8CdeF3OMNX_YQ-1; Wed, 20 Dec 2023 17:46:15 -0500 X-MC-Unique: jF4_JFJ2M8CdeF3OMNX_YQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 11D821019C82; Wed, 20 Dec 2023 22:46:15 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id E63C540C6EB9; Wed, 20 Dec 2023 22:46:12 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 26/40] mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:50 +0100 Message-ID: <20231220224504.646757-27-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C97EE40013 X-Stat-Signature: 7yhtez7c7xii64677mcdu5aq7m6zye44 X-Rspam-User: X-HE-Tag: 1703112377-345619 X-HE-Meta: U2FsdGVkX1/LwsTM54mmgFScVBtjTNaKn+Mltc8hX+S9l8UUAB6yePaWfAn50/JqbrqXYFnGtO1pSuxT09557/JZSAlHqr4RUaksBMkiehgHt4eIPGWVoul2P+DIRv7wMte2Nr+2a9Q4AgEjlWc+49TjyaUz+PVNjS1aJvE2UbEz7YarDDzQ7/OYe++lMuOy83nt0IFoWHsqWrUZ23EkmfxiJWYpsEyasU2nTMpVfhuafispFjEEd9ecJo524dAXpHqciFgWWQU9lBrIJgS+tuRI/C+jzXw8sZbnn6nZcuc864muv7ytMox2J3w8pGOAJljK+nIb0A1zsC+j6OgRuCNJ5sy1+MoMRs8qXkuKfwmrd0TJJEdRBEXoJDRS8RNdEYOVWoK8lNPadzL8l6wCpQBZkGR6aS3udb0GhD4o5LwQ/QCJqAaVuKlXPk8Ts3MHzcJTk6aVW25HrcSnvG6Z0SlrapKEhMt+/gK3Vntv4etXYawZqGeoIq6AAfJ/nRKtWODe3UyKYAlxZCpyzNlAp9kxFjQ2SPv8ia886EjAn35nK3CGiszWwkK490HFN20fI8Co3rj6ODouKPHNNZnSeUZCnHzSZGTwRsNj3doJSrjsm8caCzgjIfGIFIvn1BqtP23JRUf+qEf990ISbo+GcDd7l1TwbDHEtkChkvw/yrZ9MEKTurGxm0bZoAMhHpl6tYR/xwoXjXGuN2CDIvMRKst+aDDhjJ6dsTwP8PG/5LWjReGP5VNw0TMMPIooJ2PjVy0o47QlbBp/wtDAs7FtAZI+7y+utf+PpMJYmrGhNWdcBUrIopKtYuYFuy0JTlRXDU+zjzqRbeOT/Q46zbEtJx5mdUunU4+0y+DW3MUg2CL9hQKG7IepEmkaa/kOiUig7fXDA2CPytQFQ9019PETNFzWEcXakGGvqDUpG8iuPw2Zb60c2rQwTpwRT7hvPhOa6hWeN94zNP+E6/oMMOB J8EanGFS qgdH3y3WpwE+r7aS1Y+VcnSF1pqkaKe5zKGihnScbD+2kiC2iDQtfqhJkqzrZ2Axz3DMJByLgPdAuUllH4LuCowcExCObDiopHH5dthi9zo74PcTVNeFCS31wZWDXRD9/B/PutGZ9TsWMOfdAVKb/tHVg6UE7D1xka/x9nO0691ZYovNP4gFKzLqhtRfQ36N6Ymrlcw0O+fEkUwrbUXikcmHz2Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert __collapse_huge_page_copy_succeeded() and collapse_pte_mapped_thp(). While at it, perform some more folio conversion in __collapse_huge_page_copy_succeeded(). We can get rid of release_pte_page(). Signed-off-by: David Hildenbrand --- mm/khugepaged.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9cdea59fb4c03..15ec9c729ae58 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -494,11 +494,6 @@ static void release_pte_folio(struct folio *folio) folio_putback_lru(folio); } -static void release_pte_page(struct page *page) -{ - release_pte_folio(page_folio(page)); -} - static void release_pte_pages(pte_t *pte, pte_t *_pte, struct list_head *compound_pagelist) { @@ -687,6 +682,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + struct folio *src_folio; struct page *src_page; struct page *tmp; pte_t *_pte; @@ -708,16 +704,17 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, } } else { src_page = pte_page(pteval); - if (!PageCompound(src_page)) - release_pte_page(src_page); + src_folio = page_folio(src_page); + if (!folio_test_large(src_folio)) + release_pte_folio(src_folio); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats - * inside page_remove_rmap(). + * inside folio_remove_rmap_pte(). */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - page_remove_rmap(src_page, vma, false); + folio_remove_rmap_pte(src_folio, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } @@ -1625,7 +1622,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * PTE dirty? Shmem page is already dirty; file is read-only. */ ptep_clear(mm, addr, pte); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); nr_ptes++; } From patchwork Wed Dec 20 22:44:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08BCEC3DA6E for ; Wed, 20 Dec 2023 22:46:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A1488D001D; Wed, 20 Dec 2023 17:46:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 051BF8D000D; Wed, 20 Dec 2023 17:46:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D98148D001D; Wed, 20 Dec 2023 17:46:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C705E8D000D for ; Wed, 20 Dec 2023 17:46:24 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A31FEA18A3 for ; Wed, 20 Dec 2023 22:46:24 +0000 (UTC) X-FDA: 81588681888.25.B72280D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id B5A31A000E for ; Wed, 20 Dec 2023 22:46:22 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CvypeodO; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112382; a=rsa-sha256; cv=none; b=DwJrawsq4rXGXn+cMh3Ct1MfSHDN9swsHhp2u/IjzqHPUTderkWKUXIBQpMOsjiQRtaQ1G JJPKk2SLDyTb5XGNsZJdhQBwAPthWIya0y26eMjcQuuRis+65BTI3EPMuv25EGU/FO4G2+ BfmVyrta8MHFCPubAidYNmNvWZNQQtA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CvypeodO; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112382; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XNLi/aVY5jwoALWpPAveyLrvuNqr0potAPprV7oOzTs=; b=w2laWPm8BDe/QBlLleZyjpWAxaYDvKgVfhf/R9wzIt60Ea1uKY3D/WZ4Q0NnvTWCfXBKBq Ry2J+3G9rtlRC+V0A9ghQIv09GGvQXMu9gumqO473yuir1N3TrRwgiqV+P1xQihukch4cG vcxEYdZDYNwLyg9E4AKxr0eZl3P39kE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112382; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XNLi/aVY5jwoALWpPAveyLrvuNqr0potAPprV7oOzTs=; b=CvypeodOrgL3GQl8dPsatuxmVTlbX2OTWkfmOazKzBkHu1kAQrHLgrRW+Ra6kqEK1YYnYg VRDbsDUP78x8JWpG8JHtw/ltyLZlV7Vb2N7wKew7NrlSwNhf9lzXyTF7PtLw7Hnj0izBjb xH3jAyFx1+xoo754BHp2GkTHKmpAJVs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-122-JReApqYgOjavD14OGWY4tg-1; Wed, 20 Dec 2023 17:46:17 -0500 X-MC-Unique: JReApqYgOjavD14OGWY4tg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1D9391019C82; Wed, 20 Dec 2023 22:46:17 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5947140C6EB9; Wed, 20 Dec 2023 22:46:15 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 27/40] mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:51 +0100 Message-ID: <20231220224504.646757-28-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B5A31A000E X-Stat-Signature: 7q3fhs14qte6r6txzrp1akfywnxj1ibs X-HE-Tag: 1703112382-91608 X-HE-Meta: U2FsdGVkX19Px2JcmwOZYE5W7S69mU7He5zErujDA4gejwbPPgx+ExZkOVw+7PXJ58vXCD0Cj+VbUqchoe1OLAISexI+8PHopw8AF7NqoWDKBu/8agJagKE0PWQ1wdDADgHUB5MxpgELnf8ab7ux10QcCduT+ZBc3gxFe8Ph4ErCVaXQWoEPB/4KMEqgQKXoi8YPDWLV43Kv8XTkGC5+6rZOWls3XVHWkKdrSc/mvT2ShcZAbOb0xxubROqW+3KvKEdFz2Pebdb4aBRoENMfGVbrVojyB/Bkvqp3H2VNKXPvveWGHUapFM9FIc44Ck90rWdI3ksjn7fBC/cjHWLzHT1grht653/IVmiB7K2QgAuKaIUGGLYYFSWJMYr1efyC9dKq/8OYBMy2wwcZDR+GZZ9ZJYDbqIvUcfpfl3gLI6eO3/80+soGsRqxjHoIVZ1rsiAzi1viV8P9njVph5v/JIBChhYXqso19/aNeUnIO6xQltFXL9BdXQv896HjMtUHjrNOUTV36drMpy7T5IWgFqJLOqaKoF5fs8jcRqHTGZCJQGkKaVmgsRbkud+rSd+o4l2xZ8LFVtpg3Yc3K5+Zkl+tkmZhuN00LhceVQGl+xdW9FqTejyLmG/MomaxSONh7sY0873jNmG1qkGKORBDmdkeyoeBi5yWJspGkJ4cP1myBwxqOHl2pTJBqkiahB1zINo5mSUjux7185uTEmeDo7wb7btbD7yz8U4vAzioynWZsa3QjyMviNbXYHV6svh9CR/WMex0ZK/LG93psyePnT8b62lPYwefUrtvjHAzW4MnT5E1Ur0WWaw+1q7liizLoCOt/jbNn76/oxPkhZjLmoU5YiGLzkZpk0ozCCYhB/h5ucibflkKtk80Fmofwtz7GxwwceD/Ux1Eb+2Rr19rF+hqx77GNvimEZ1QFsgwuryAegxEGsvyuKw3Yip+km6Wz5yaXEtsVSNG363Sgfg D43MR8eB zWi7M7j7OtFrYzeuVk7MiZQ1zDbczMVSeGSoKxP3PzVKUcrnURsSvxUK7DQpFMdiXhLnwwTGcG7Yp7HbLfarjsDaIF8Ev1xLUgfKe2Jxy0LI6pybDgbbLdLHHmixEXoCukgS2lr/iq2WWMyuoIdIfNcfkGEC/4hMtAJ/boSW9T+/g+2cOCXK3ecYDWGsC9sk/b+AirjDEV8r9TuWAQu/nH87XALg1iBgQfDY+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert replace_page(). Signed-off-by: David Hildenbrand --- mm/ksm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/ksm.c b/mm/ksm.c index 146aa75fa6ff7..716e2f87dd795 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1449,7 +1449,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, set_pte_at_notify(mm, addr, ptep, newpte); folio = page_folio(page); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (!folio_mapped(folio)) folio_free_swap(folio); folio_put(folio); From patchwork Wed Dec 20 22:44:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F885C3DA6E for ; Wed, 20 Dec 2023 22:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C0FA8D001E; Wed, 20 Dec 2023 17:46:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66C658D000D; Wed, 20 Dec 2023 17:46:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E4DF8D001E; Wed, 20 Dec 2023 17:46:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3AB328D000D for ; Wed, 20 Dec 2023 17:46:26 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 13340C09B3 for ; Wed, 20 Dec 2023 22:46:26 +0000 (UTC) X-FDA: 81588681972.13.46F1085 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 5E85F180018 for ; Wed, 20 Dec 2023 22:46:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IoXzXfwE; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Io0fNOjJFUJN+cQn2T3HaexKpR7MvH4XyxQbyfgzgvU=; b=v6jaW2sCHyNMYqg4s+FU4zBtzFpONDv09/fGfp7v5VRsrZmEy6Lh2CwRT8mhmqLf0a3ywE aLi8gCKkaDUQYGFZR7EtK9j5C0wCfRxVzKcS41IS9JAi8HTZyetC3HJsV4ORk5Ifq5ICsw CMw16l/YPdvFr+2YN0qcAQXavmkm2rw= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IoXzXfwE; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112384; a=rsa-sha256; cv=none; b=RkdSeADAgX69JyUxA+ezQ3RVR3puH7dLBoEalxFsnrsdl2uIS3vgqcJnJKAjopQEzxSFM4 kDlHl4V7yY1q9K1Seb4y0A4sWNFzTFEfAAU9xvGb4OS49kR5Vf7pOkqpp2GUPK0o4buEJQ VTq7NrB1QuvE6kfZ6ehwKW9uAJ5s9gY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112383; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Io0fNOjJFUJN+cQn2T3HaexKpR7MvH4XyxQbyfgzgvU=; b=IoXzXfwESpWvRsmikGjPEn4LSrJ7qA8J2vaX2708lzRBlMF8Mmgo6eDsu9OR5w264wXPVQ Jr/Kg7kqfg/ITHqyEAzGyVBkubfqMCmL4NWl3j/o2SeSpjLHl1Rwyi9AQ0l9BzAUDWmjyF tZY2ksZy8CMAYZizO98kzTiFlBjRovM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-99-EEpn4mmfOKyp69_cUJHDvw-1; Wed, 20 Dec 2023 17:46:20 -0500 X-MC-Unique: EEpn4mmfOKyp69_cUJHDvw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9284985A588; Wed, 20 Dec 2023 22:46:19 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6B17A40C6EB9; Wed, 20 Dec 2023 22:46:17 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 28/40] mm/memory: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:52 +0100 Message-ID: <20231220224504.646757-29-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 5E85F180018 X-Rspam-User: X-Stat-Signature: eqnmuez1hmx7brtr8nurp6cd6n97j9rd X-Rspamd-Server: rspam01 X-HE-Tag: 1703112384-630537 X-HE-Meta: U2FsdGVkX19TnWRVqUs3vYKV+mOX41i9NJEu1V832wkOrWZDTpigx6YOufk7IQKrsuZ0VOHQTnBW+h12oMwTjYSR+WV7zW/ih1Y8aalW5J3iloVrbyn76C+9ZaPl83YDf9dvupGvMhinXSUU4wroDBkC2OyFQ0eGApCnj3mgxDaa26V5c78OkMsjO0iasQMGWSVqoOawd/gfzBPZvdoWVXMlzDi8bc3q6a64uW2tqnKbLiksTVHVzMgL1FWxAuntFD0dBHVp5xbxeJwgUYjeq78v4NRqD/cswODMnzwpFXhzu26WuOO9b51IbIpug7I1Ezr2G3n1x/2gVzLlgGbNiahcK8t1ZmKfnF8W+6HEsmpCNMQfHx5FBc3vhRuPdMMxPhi7cs6b7Pqxag61haozJwkHWb2Uz/Tb41aXX5aHSMZFcxzyujFmDxZxktCafxQ77nJh6+PThInwaxj21xsmx2hj3JoEdvVoqG9AZXz30G9N5bWBORSMdgjXKxFjq11bmd79U9nkNPRjYfeQY67fsdDgavRJC8LZbZkWgUrFMUNRLkZjL7QN3NCHlG71TvKUajM0t8t9iJHpeJ5jjP11/ax7zu2EJLTpW75sIt/hpsSFYRAzpRKKVUiJbGv/+8nJNNuTBlHZ4kAVaA6bD4SMngS32ON4Z8SjhzcjEoWuU2BSoqOgPeQq+mliFTuWCHx90GMvbKNPANiITVN0Gji8QRdch3PHOC8EacJKuTdbkuBcuUbrg6pUfknhktW/ujFBoqjL5pWa2zFEu9o8spEuG5/Sl1APOECGVvECvkqThnff0iQt56EgCpsMpGdmoDm4apPcUGqcKt7li6buSxW9subBe983Nbs8xxuaH7eFIIRou6BYNY1G4iwvTQd1LriTH+o2OeNRltNz+4vJwfMplUFYwnGxvRmizsOi5s2SyNvMg0WDneW3B/w2FYmI0jWplCDswCryqzBh+L4QncD JJzIBjur 3R6HicnYs4lCbQxFwHnZkVKZMnEMq+1RGNhSskaV6JqZZyRqT8T8G7qVNyUG7iK7PtmhSJOyzuzWCicdSHCt013mg8OM0FfZ3Qvvdos9pD1XZtcJH9cb5/4mzusJjXPk2xp9TrFtQD0/0B4Vu7K8Kb7H4c8n7JE9BYm/jbV2FBBP4yIOcm8FhClby7RQJXKwVp0VW5gn0/HWvUbb5+kceBK2YTQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert zap_pte_range() and closely-related tlb_flush_rmap_batch(). While at it, perform some more folio conversion in zap_pte_range(). Signed-off-by: David Hildenbrand --- mm/memory.c | 23 +++++++++++++---------- mm/mmu_gather.c | 2 +- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6552ea27b0bfa..eda2181275d9b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1434,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1459,21 +1460,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + folio = page_folio(page); delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } rss[mm_counter(page)]--; if (!delay_rmap) { - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } @@ -1489,6 +1491,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); if (unlikely(!should_zap_page(details, page))) continue; /* @@ -1500,8 +1503,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(page)]--; if (is_device_private_entry(entry)) - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -3220,10 +3223,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * threads. * * The critical issue is to order this - * page_remove_rmap with the ptp_clear_flush above. - * Those stores are ordered by (if nothing else,) + * folio_remove_rmap_pte() with the ptp_clear_flush + * above. Those stores are ordered by (if nothing else,) * the barrier present in the atomic_add_negative - * in page_remove_rmap. + * in folio_remove_rmap_pte(); * * Then the TLB flush in ptep_clear_flush ensures that * no process can access the old page before the @@ -3232,7 +3235,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(vmf->page, vma, false); + folio_remove_rmap_pte(old_folio, vmf->page, vma); } /* Free the old page.. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd217..604ddf08affed 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ if (encoded_page_flags(enc)) { struct page *page = encoded_page_ptr(enc); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(page_folio(page), page, vma); } } } From patchwork Wed Dec 20 22:44:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3EB2C3DA6E for ; Wed, 20 Dec 2023 22:46:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 428638D001F; Wed, 20 Dec 2023 17:46:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AB3A8D000D; Wed, 20 Dec 2023 17:46:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24D738D001F; Wed, 20 Dec 2023 17:46:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0B0968D000D for ; Wed, 20 Dec 2023 17:46:31 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E20A41A0512 for ; Wed, 20 Dec 2023 22:46:30 +0000 (UTC) X-FDA: 81588682140.15.DCCF695 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 37B3C4001F for ; Wed, 20 Dec 2023 22:46:28 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Cf9xRjjY; spf=pass (imf07.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112388; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aKdh92e0zNAps0SI5GoWH5U4FWHuHogZgqgLcAEKNJQ=; b=UqT/EH3KeUErAAHsRVaRwdK9ZxDxF4vOSqZO6DatpDdRXz9Ka7fyCyGjJfuElfYsC3b3mk 8MpqY0t1oCqpfHduC40bH/2nbdI9uJwEjyq/sRKxSJs7L82FbHDTenDnAAKlxYbcnZxfXV TTsvzRf59cKRGIjSeNs880TMeItYHFc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112388; a=rsa-sha256; cv=none; b=7JDTbiZVD9TUhVTmnPyegtsP6GNerol3tdrkupwR3lg0/fjwxMny3ASFdX4ZmJamfNQC9G MwUzfpxeTS+wKICjdjg4wqcfGWdX/hpEFwqa/tt3kZ6EvT0CP1BzBRd6Wbc+IDCu2XbFvt CYqGj/DaRm8aSvu8+OXyOJUZMNWVpRM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Cf9xRjjY; spf=pass (imf07.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112387; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aKdh92e0zNAps0SI5GoWH5U4FWHuHogZgqgLcAEKNJQ=; b=Cf9xRjjYEMNjI0sFRnIotGl4J76yyzvKIqi9aHX/iFIylhKH9rjRsB7QBwkjZx7oPB1EUE s0W0Kh681HxYCRN7qCBiUDhtYQTOzWp6mrwiHReLw9HBHdPvyWy9AmNo356uO5SAMs+/PJ xR5tb5lDZV+RpRGDIf5iQn9krjk2t6g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-513-nZxpfJamMleZQ6ECjJXe1Q-1; Wed, 20 Dec 2023 17:46:23 -0500 X-MC-Unique: nZxpfJamMleZQ6ECjJXe1Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 87B8A86EB22; Wed, 20 Dec 2023 22:46:22 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0D2E40C6EB9; Wed, 20 Dec 2023 22:46:19 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 29/40] mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:53 +0100 Message-ID: <20231220224504.646757-30-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Stat-Signature: p4md87n4uoj6n9jknuexcobf7fqqoy58 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 37B3C4001F X-Rspam-User: X-HE-Tag: 1703112388-614220 X-HE-Meta: U2FsdGVkX1/BZCyFNhdSO4+ANg9vW7wDrLD8Vm4jwFsqNSX7WO9VLhisPlD14y9azi1ava9VQvxyzHFy0O3qcgqygkZ5XrTWDBcqbkCRl9v5IaObrloYtp/z+MFUniXcnFWEqAP3muZryFaOMgLi7NAs3jyXfrg8C8qwYe7fmC/cBxWKXYuRmaUn8goY5M/8/smcmxnKgj/WhQEbLOquLjX9Zf9Gpbncl3fe9igW92KKS9Tvgwwtr2XVbRybC4qjRtmguyA7zJneDtTuQquGHSIsQ5UyCeHDSec+99UmuvEoJlID6Zc8cmFRjELAzr2WournnVCmDZ4nZ/+T0ha8WVDBqVAP4TTmKRtQq+cV0Tq1NhPI4359Wu7y8juc+ORy9OZq3axFEXhD+nr+A7jwFBAU6ZwcnX15lRztM2eA3J8ML/9YUWbnJdDQani43NMrQCqjVNtn4y8PWMSFzTeWzDkRBzP+fdct9FQjErp+0MPEm+PpfoeEM9Q59+rpgCxjoNsxQjemq3hWc4HPz/s6JFldeOduipp7Ed5CRSWIgx+YxPQsrvOaBU0JTPqlwPgop7HRDQHYrri/j2jvJXQJpHM9YtcIarGj/mFyRbZZaQ5Ix9YfuuY6Rm1jTPvZI7nzYCLaN8hvB1X/M6J6d7lFBWhgsVw0xqA/VHgHMJdY9TvQv7J8+1eKxfhSyGwIj9AOqsMirGl6hIktCjyvTtiZN6I+9KTbY3cdIm8OBs/xFDQlV78DsikOJGDpSafo7G+657dNCBbh1/gIwDzmziMbMV6TUEvs1EhTqLgLurwG6mMUcfSYv4AhkmcAvMoBm8z3utjrH6x1U58WeBr7wHEQX/2oShBecyygE1GIfUFkSM9QrQxSFAUUE9WIQ9DHPwdPmd1/NNlb5/wHqTCAMQvdD+YJY/KdK7y+TwSjaW7q8m3w7s5Kd274v0sefQBG1WMIr5DOdL5KTG7prq4hQ61 91buIqTe nOJIxk14RWfxKKlyGwnUeCR+lCMJVBg7X9UZexO3IMuc4AhsLgQlzokuldgzxn97MqqgMNqezHl5yJ9hiLViR8cJFj4zeoyl/qox9lM8S0rqvZl++uclzB8R1W6eghPp+5IGEiO1ybfYZG7jcnnNJoyyrPrF4ybJwSm2aBoNYh2qeWlXZr2RZ3qyV+WMes96ipncJiBEkVXYvWvOG/ahfWhPY7Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert migrate_vma_collect_pmd(). While at it, perform more folio conversion. Signed-off-by: David Hildenbrand --- mm/migrate_device.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 81193363f8cd5..39b7754480c67 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -107,6 +107,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, for (; addr < end; addr += PAGE_SIZE, ptep++) { unsigned long mpfn = 0, pfn; + struct folio *folio; struct page *page; swp_entry_t entry; pte_t pte; @@ -168,41 +169,43 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, } /* - * By getting a reference on the page we pin it and that blocks + * By getting a reference on the folio we pin it and that blocks * any kind of migration. Side effect is that it "freezes" the * pte. * - * We drop this reference after isolating the page from the lru - * for non device page (device page are not on the lru and thus + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus * can't be dropped from it). */ - get_page(page); + folio = page_folio(page); + folio_get(folio); /* - * We rely on trylock_page() to avoid deadlock between + * We rely on folio_trylock() to avoid deadlock between * concurrent migrations where each is waiting on the others - * page lock. If we can't immediately lock the page we fail this + * folio lock. If we can't immediately lock the folio we fail this * migration as it is only best effort anyway. * - * If we can lock the page it's safe to set up a migration entry - * now. In the common case where the page is mapped once in a + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a * single process setting up the migration entry now is an * optimisation to avoid walking the rmap later with * try_to_migrate(). */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { bool anon_exclusive; pte_t swp_pte; flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && + PageAnonExclusive(page); if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) { set_pte_at(mm, addr, ptep, pte); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); mpfn = 0; goto next; } @@ -214,7 +217,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty(pte)) - folio_mark_dirty(page_folio(page)); + folio_mark_dirty(folio); /* Setup special migration page table entry */ if (mpfn & MIGRATE_PFN_WRITE) @@ -248,16 +251,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* * This is like regular unmap: we remove the rmap and - * drop page refcount. Page won't be freed, as we took - * a reference just above. + * drop the folio refcount. The folio won't be freed, as + * we took a reference just above. */ - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); if (pte_present(pte)) unmapped++; } else { - put_page(page); + folio_put(folio); mpfn = 0; } From patchwork Wed Dec 20 22:44:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7A11C3DA6E for ; Wed, 20 Dec 2023 22:46:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 276F98D0020; Wed, 20 Dec 2023 17:46:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FE7A8D000D; Wed, 20 Dec 2023 17:46:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 006A18D0020; Wed, 20 Dec 2023 17:46:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DB5F78D000D for ; Wed, 20 Dec 2023 17:46:33 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B80901C09CA for ; Wed, 20 Dec 2023 22:46:33 +0000 (UTC) X-FDA: 81588682266.21.334DF57 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 147D014001D for ; Wed, 20 Dec 2023 22:46:31 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fpddgJce; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112392; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HrqmxJFgmC1fiGIelPJz980NZ2Q1oauxy+LU1liPb0o=; b=uNsUbXJPvEPu3qgTsi0n9SvO2JdwIuIx51U9JZ6lfdtmXZNvFteLbDuC1AQ2XTw9c58F3m SYgV3uJclu9wmAEAhL5bJynCV+uTM1zq0uaGeUXP4fIBvuYGgmxHbUOu+mvZQ+MpDfVz4M ewfYKK3AvU+cWMplJOeaQBa6awM0iAE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fpddgJce; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112392; a=rsa-sha256; cv=none; b=38xW+0u9QITy1iWpB0YX5RPbkVtdGc25FCucS7HUWCe45Sj7Xs/6iu83+7pwUHh5t8oI7+ VMLxKRwrCqh0Z5HEzM4XCHsoOiA6YzeZVJJnjWvxcwCfSBY3h4M2iZtt822arslKgvR+Cl S6JFGNH/FW9w+L3ZrxoBy3B8liKEgUg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112391; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HrqmxJFgmC1fiGIelPJz980NZ2Q1oauxy+LU1liPb0o=; b=fpddgJce8hSScfgzx/WUOKqW1RrCaLovKV2kXl5SiOCvxWrsFjmzEAPHv0VMydXqO4NV6G l6RfemE24fWgRtQn8OTSidhtA4UVV+2B0O+MU1AVEABDIwdJb0RRx71JxnXK6lEhPFivUw xWiPMk6DKcQTIJOHUZJtRPsakvc8JEw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-388-IzS3I1BaNGycfEoIX0J4Hw-1; Wed, 20 Dec 2023 17:46:26 -0500 X-MC-Unique: IzS3I1BaNGycfEoIX0J4Hw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 99362863B83; Wed, 20 Dec 2023 22:46:25 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id CDC2940C6EB9; Wed, 20 Dec 2023 22:46:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 30/40] mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:54 +0100 Message-ID: <20231220224504.646757-31-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 147D014001D X-Stat-Signature: 661yfdbbggqku5oujbbpxp3sw647nnqc X-HE-Tag: 1703112391-582046 X-HE-Meta: U2FsdGVkX1/s5xq3HzQdHblw8CAVdZo5xE+r6Xh2o2+3MP7Wx6APGVMcnesEqGKGi7iNkndrll2HXY0iuFnflzn5pxxY1HrhIgRdyCV3JnmCukftc27NQiND+lIVzXhYJXH47jOseL0V6EXBMW9I8J+Cq7R9Vv6XQEBQufUV5UaLc2PPVEIweNLTKXT9HC+Gux3pVpZqKHr/ilQeWctfVTggYJ6wTAgq3olDmycB9jsnIsBBsFiSaCyg4XFk+eOX7+uJCis0msiN8OP1WmP3IttFntETv/GEoWQvbIvX8aABt8SmHC2j1Aon+Ojy9R08wt4lQ9lFNFPuz6K6n+uAD14N6MuOEyn2IVINZM57GUk4psjj2QsLo2FkEfITSz82EDDWH0FPsIEF2n90NIsxDYDnlcOzQWggPBAmKZW+rACNl8LsCxcwALLEutL8n7io//MQAC53Fs3dD1FcG4hcueGvsANfFs/8tLopScVnuYPKvge1ywmxFvlE+nPqWPdBRH6nszQRWUS/kJ1O7V/AOy3oWkafS1CdM0iIkVvqRIl7qoFai6tPcTTfVdSAa6eSrErWGvxSkCVntpTdFfxqEk0ivLFF4gxBbafc+DmPdZR6rz46F3ciyIjN50hIAO5N5wXXpBJZwCkg41jTxEz1Roff9KlQMUGNiZoovXQUoe2QSCk0e65N5QPuuYzEMYRriSHyqTHU2PHlNebQn2FqcqsHn66pEE3ApuEwo4H6vVwEHfKIeAzBmB0StuYi2O25bg+wTsGrHZzw1kXhbq8K/2uvgeR/LqpkvdgbuWaW2Dnd4+uJDc2rlRL8XJ5eG3udVgcbooS1dLMTqf92kp4wuIOfWXc2WPWOOXPG4RYQB37jXYZloTWmoQYfSCsMSWplutUwuV0uuhxgP5QOhgTMmN0O9Xr7You3jCrOiraYrb2vXK2epFsDPpM3U7umpe9LX2Eu/3TQ5dqx8NjCxo1 90s/M5Wd LmglrLIgQE+OrypPO+3g1xNarXXw3UkSeDpzkoRQoakKN3BF3s45Ec6zcONL0vszXKR+wQFlG+wtmpcwn2gkcA5/w8PpjjJN2D8CJKJfXP9/cN+9NQOgjbJwE0ouHNzBcwcKUciAmqJXb8vhGIZIgEffF5UBb/W7THqh3JYHQSpcx24UdVlMhq3Re7rKnEPBRpKUtsWFdLEWr0abCWc6GxXgenw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert try_to_unmap_one() and try_to_migrate_one(). Signed-off-by: David Hildenbrand --- mm/rmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1273180753953..a3ec2be484cfc 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1647,7 +1647,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_unmap() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -1928,7 +1928,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -1996,7 +1996,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_migrate() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -2289,7 +2289,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2428,7 +2428,7 @@ static bool page_make_device_exclusive_one(struct folio *folio, * There is a reference on the page for the swap entry which has * been removed, so shouldn't take another. */ - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); } mmu_notifier_invalidate_range_end(&range); From patchwork Wed Dec 20 22:44:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83DC6C3DA6E for ; Wed, 20 Dec 2023 22:46:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02C828D000D; Wed, 20 Dec 2023 17:46:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EB1A28D0023; Wed, 20 Dec 2023 17:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C119C8D0022; Wed, 20 Dec 2023 17:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A59B28D000D for ; Wed, 20 Dec 2023 17:46:37 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5E7C81608B0 for ; Wed, 20 Dec 2023 22:46:37 +0000 (UTC) X-FDA: 81588682434.15.B78E434 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id B0D07C001E for ; Wed, 20 Dec 2023 22:46:35 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fRSj2ykF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dFfPUjt3XA3pNIs6QrEG4XUFOfbPY73VNaCA30/uDMs=; b=1q1ZDniQ18UcZxqEV0EWcgRFbzx23+A44+rFexeSyP+XmQV1qnFn2MAvEXReC1bprvKHC6 rua58Sw1HfWop1PQVM74PaRjfcTohLnbbQ6mosUmNsAHHN73nnug1w6w+rtYEU1/l9hrUQ Eu1Dd0WZg1J562d5O3LV3txu7hvxFn8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fRSj2ykF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112395; a=rsa-sha256; cv=none; b=kcw34G4gPVa9d9lBLWBx8uFK2/S0j93EMzwtwr8dmErdMHvHB3+f0pzMMBKU11zPXvroFp zJgcFFtOJslJD/1AeJ9zZF0vXnIwSZY/c+Br1ti0G3pCqiT/1wJzx7ZTYJ1WE8pCzQ3m9P TPb1HScx+AD4lRA0HMBjtZV/VBzE1iQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112395; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dFfPUjt3XA3pNIs6QrEG4XUFOfbPY73VNaCA30/uDMs=; b=fRSj2ykFjU5bLXjIxLa5J+WdDyhCwgnNtLn50a/Vb4YVLbjWr47aLnd8MLpyoPehofFCIc XeWBrLQg2btnNH2QxA8rNtSIen0b8nW9l3Z4tV3mV/xJB9dYBjiKP9vuJKLxy2p0sHs0w/ gtSM5R0voLhBzgvjLkhQt+Khc8sW/6o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-441-eYmFnTyfNBSFscisqBgv4Q-1; Wed, 20 Dec 2023 17:46:29 -0500 X-MC-Unique: eYmFnTyfNBSFscisqBgv4Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AC44F848C06; Wed, 20 Dec 2023 22:46:28 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0429440C6EB9; Wed, 20 Dec 2023 22:46:25 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 31/40] Documentation: stop referring to page_remove_rmap() Date: Wed, 20 Dec 2023 23:44:55 +0100 Message-ID: <20231220224504.646757-32-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B0D07C001E X-Stat-Signature: ga7t9qimn3onxg1nd1dna1pmg1ho9dnf X-HE-Tag: 1703112395-294528 X-HE-Meta: U2FsdGVkX18x1bWBg+mtQ2m82fpqr2w6D1ljkyNmrVeeKRb/tCKj0bdcVPlkRS9FQFx5e4YLCrlQRgMxSKk9Jax2EpyMWOfZoi1lOiANBkARCrNEs91b2JhM174h4LafNlQ6fDBzuAyDHyUm9TxLnD7o5axjSUMLzoNo3l8gaC9XdYhfn2MkUf5vrhHzSn74qWqhQSKtTXvOLtjTJAUW08aYtMIK2fKHjpx3jJTgQwI1KbPn9d0eK+qqngk5Ax4ca2KyFwuGl02C+mOknM3l6lWEZl6yS7hMuk2wAoByCEldl22lgAYIZ6qhqxH5lBD06cbaBfhYsQxpb+MCI2Lra9ZZ/q+mBZCllVMD0h8REXUzOIDPDzhXoE4PfA99B7TdkMSpYFzYs1Yi49TnrDE0xyuWgcPugu0F2k+BFctisHZT22tlx92kKKguXMGJf1j/vFt2Mkvd3biH9UaMIM0fUrHdAOFstyC9O7DepEZHNpugzMQz4AybvyBl68PgsUIsT2S0Y+wJGUuaFnKX3sAGsf+lIGasT34ub/16l98tHyr/PWWALg/PKNSC6xx/SDzCwnyn8Ava2ef/Urzg1PX2mvZS45z+l5Z2isDGd1PSenJGLVPAiXDijqgo+joOhCDw1uaGEPybw8nbyEJvIvNO84irH76QuuRSNTQqO31UtPKLCB/9EFZGEWcw/f5Rk3pCjJC5GHY1NvVeuLWuQ/beWQ9Uu43rzPCijDMcZPecl3OIYR2FSOuF/sK1Rls30+iQ9Y67nZnkQeHdil1TUBvWFYj0Ssc1DeI5mf9YKI9nBNxyS+F10YvkfcImRHHPR3bFS11+3H7zTAx9QxxtuBt64gOEwkPN6PcaJLCm8HUkT9WJcOjv8ORUV/ojz3VrpyZUZgXJJYFXhlDn0p5i9/QfPNLdg/VdZJzSSQ1spb2LLCrj8ioyyXOMg5KzoGTiPCJm5qexjttNx+gxCTIuWm3 YanBjJFG 241TW119E8GHHjEhB1tzWiT8RdRpqHAZj9GH2GXlgAb5/3wfxxxcBXZYN3Vk/zKsh4w5ceyrK/WEKUSKPevqIKKhavOFnOWfk5/7NJX/jIaJbAU0YFsbdGmXEcvRKN39vQo2/yt3jhNPQPNluYIHAA0TVcvk5drHQULjmbqbGJy2BE6kn2fnEpMcP/aRPvU6KxLf4k/9EGz8AMBoLCO61NgubkQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refer to folio_remove_rmap_*() instaed. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- Documentation/mm/unevictable-lru.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index 9a607059ea11c..cf81272a6b8b6 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -156,7 +156,7 @@ Partial unmap and deferred_split_folio() Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use -in page_remove_rmap() and queue the THP for splitting if memory pressure +in folio_remove_rmap_*() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages. Splitting the page right away is not an option due to locking context in diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevictable-lru.rst index 67f1338440a50..b6a07a26b10d5 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -486,7 +486,7 @@ munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. Before the unevictable/mlock changes, mlocking did not mark the pages in any way, so unmapping them required no processing. -For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). @@ -511,7 +511,7 @@ userspace; truncation even unmaps and deletes any private anonymous pages which had been Copied-On-Write from the file pages now being truncated. Mlocked pages can be munlocked and deleted in this way: like with munmap(), -for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). From patchwork Wed Dec 20 22:44:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1947C3DA6E for ; Wed, 20 Dec 2023 22:46:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB8BC8D0021; Wed, 20 Dec 2023 17:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C61358D000D; Wed, 20 Dec 2023 17:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A0E58D0021; Wed, 20 Dec 2023 17:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 873468D000D for ; Wed, 20 Dec 2023 17:46:37 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5EA06C0A93 for ; Wed, 20 Dec 2023 22:46:37 +0000 (UTC) X-FDA: 81588682434.16.FC40C02 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id BC644C000A for ; Wed, 20 Dec 2023 22:46:35 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Iwkyu+BP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vX4HFQixyHeUqudW1I+7GoOAzQbfOI98S0mk8R6Q4EQ=; b=JssnusVqfh/dMHd7+UfqG7iPyWM2Ouh4IORVJvvN0CWA5o/RAqAVFUmH/SXV6kLEY5SvPF XXhZcFlO/CDUUC/Z/bxNrnftLJiZYLc6JO4iCk+N+oVWV2LlWgNE1ij7srj37kSBQUAjzj 5xp/CQ55zT20T+G9/UQ0x21CoARhp7s= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Iwkyu+BP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112395; a=rsa-sha256; cv=none; b=v9mRHaa+EnW4qOh9M2Rj0IIb7g8g1eYE138trvyws7LotkoHfI4zNBBnqqu6tV6pAYskf6 0YGMCtOSCvfWehw/C1PofOSOEbzBth2h7p9kLSsL8uT4H5Wvlnc6S0VgO16v2wQ1o633GM qMMzdJPMB8U4SxzzQUkjA01o5lAMrGw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112395; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vX4HFQixyHeUqudW1I+7GoOAzQbfOI98S0mk8R6Q4EQ=; b=Iwkyu+BPE5jnsbkbdG+s+crY+49jyinQTbt46QkHZZz6Fg0Q9ud4lMzJ4BrKCqpNzwpXv0 6qwjdcSaeCf6kIEB7ScjPpJ4leftLN4PEi1mIuT2HD8LRvvGGiknwWgfCpJjGOVPotr03t baOo+hggxJVwH2ao7eVw2OJJ3SB+FL0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-548-1zduYvExPhKGzCC_CaEsSA-1; Wed, 20 Dec 2023 17:46:31 -0500 X-MC-Unique: 1zduYvExPhKGzCC_CaEsSA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CDE6C185A783; Wed, 20 Dec 2023 22:46:30 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1364A40C6EB9; Wed, 20 Dec 2023 22:46:28 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 32/40] mm/rmap: remove page_remove_rmap() Date: Wed, 20 Dec 2023 23:44:56 +0100 Message-ID: <20231220224504.646757-33-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BC644C000A X-Stat-Signature: a8kawi9uqt5769iiw4i7nscc9kdynk5b X-Rspam-User: X-HE-Tag: 1703112395-294214 X-HE-Meta: U2FsdGVkX18QsWv8m7TjfLDpt7vVsg8ftDj3l9zSDn70L1dvh1gDae/shMEwd1kDZfNsi62042GsjuCSqGFeEP6GLo+CwUJNUQaMJ5IkGgjkWZBkUHkMe9+wBKGEX0ygpzY8OoBmqppI5DBTXY0c1jRCr4o7yzhH2ECSRTnNcCyeQFEE7JhUDKZBDyUodACzPi+cM8BgFWl7ieUbB6P2leiAIONXsI6gnMSYsfKZ8cDBhmwru85EGRqMqQbcxikRVgUbNUek+EAbhVZtBalin7RERyCWlga6Ixf79DIEu1dhcz/g417CNOUJFkA2Gl622OToFbbMMSElNLSXJtwUv9aTF1mZLGYDhHngJOkNOJpUffrjgLaJILdfEOZ9t4nwLrD8tDzBl/8eaDBKfgqUnc7ljESZ+PNdTCDzc6ZhKtbHnsXRKuWQywZN/KRh480uy5cYFTsbcloc6OImNJLE997JSanANg+DDvwpBQsuWUmH/PPqPJVidX71TJVrsWeemB9GM1qHq6N4Jy/w5pvUmP2c55rDVJsQdOC8t0O+/w8gBPVPe5flrcuTE61QLZyvmRdamgQGsHNAa4GyDtWXZUgBozdp1LweDL2YK33PuIeNGnxIHYmcs0OwkEj02XV+AQuOWmp9FZQ4cHbcSY1ZGDhYTPhYPCGrLtXPX7ZnUzzrZo13H6036v1lS2WmEyN6Go/gz5IHOAgt54MURLII7OI6fX6Gn4LE1hUMzuh6PMu59snWvxXRhK/yW00qqm2LPm3BNU5b2TfvvKSe30+kr8N7yRocQoDJcpNOuHml+O6i7WIjty0S6n48jtP7xadK6yXNK3gUCPNJg7+IG+dVd+/qwjAjCLe1GKpPxGGfFADV9KDTlEFvO7Tdf6LkAgveLQxBusApObXXKQGSZuKxj5D4MseORmzA0JbKx3l8P1UaohhehyPIMWOIsFaCOuLx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers are gone, let's remove it and some leftover traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 +--- mm/filemap.c | 10 +++++----- mm/internal.h | 2 +- mm/memory-failure.c | 4 ++-- mm/rmap.c | 23 ++--------------------- 5 files changed, 11 insertions(+), 32 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 57e045093f047..fef369e37039a 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -241,8 +241,6 @@ void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, folio_add_file_rmap_ptes(folio, page, 1, vma) void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); -void page_remove_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_remove_rmap_pte(folio, page, vma) \ @@ -389,7 +387,7 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * * This is similar to page_try_dup_anon_rmap(), however, not used during fork() * to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via page_remove_rmap(). + * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device * private pages cannot get pinned and consequently this function cannot fail. diff --git a/mm/filemap.c b/mm/filemap.c index 67ba56ecdd32a..c8dafe70d4cce 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -113,11 +113,11 @@ * ->i_pages lock (try_to_unmap_one) * ->lruvec->lru_lock (follow_page->mark_page_accessed) * ->lruvec->lru_lock (check_pte_range->isolate_lru_page) - * ->private_lock (page_remove_rmap->set_page_dirty) - * ->i_pages lock (page_remove_rmap->set_page_dirty) - * bdi.wb->list_lock (page_remove_rmap->set_page_dirty) - * ->inode->i_lock (page_remove_rmap->set_page_dirty) - * ->memcg->move_lock (page_remove_rmap->folio_memcg_lock) + * ->private_lock (folio_remove_rmap_pte->set_page_dirty) + * ->i_pages lock (folio_remove_rmap_pte->set_page_dirty) + * bdi.wb->list_lock (folio_remove_rmap_pte->set_page_dirty) + * ->inode->i_lock (folio_remove_rmap_pte->set_page_dirty) + * ->memcg->move_lock (folio_remove_rmap_pte->folio_memcg_lock) * bdi.wb->list_lock (zap_pte_range->set_page_dirty) * ->inode->i_lock (zap_pte_range->set_page_dirty) * ->private_lock (zap_pte_range->block_dirty_folio) diff --git a/mm/internal.h b/mm/internal.h index 222e63b2dea48..a94355e70bd78 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -651,7 +651,7 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma) * under page table lock for the pte/pmd being added or removed. * * mlock is usually called at the end of page_add_*_rmap(), munlock at - * the end of page_remove_rmap(); but new anon folios are managed by + * the end of folio_remove_rmap_*(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). */ void mlock_folio(struct folio *folio); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 5a23da5eb8c1e..a0d9b4ac7d545 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2315,8 +2315,8 @@ int memory_failure(unsigned long pfn, int flags) * We use page flags to determine what action should be taken, but * the flags can be modified by the error containment action. One * example is an mlocked page, where PG_mlocked is cleared by - * page_remove_rmap() in try_to_unmap_one(). So to determine page status - * correctly, we save a copy of the page flags at this time. + * folio_remove_rmap_*() in try_to_unmap_one(). So to determine page + * status correctly, we save a copy of the page flags at this time. */ page_flags = p->flags; diff --git a/mm/rmap.c b/mm/rmap.c index a3ec2be484cfc..3ee254a996221 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -470,7 +470,7 @@ void __init anon_vma_init(void) /* * Getting a lock on a stable anon_vma from a page off the LRU is tricky! * - * Since there is no serialization what so ever against page_remove_rmap() + * Since there is no serialization what so ever against folio_remove_rmap_*() * the best this function can do is return a refcount increased anon_vma * that might have been relevant to this page. * @@ -487,7 +487,7 @@ void __init anon_vma_init(void) * [ something equivalent to page_mapped_in_vma() ]. * * Since anon_vma's slab is SLAB_TYPESAFE_BY_RCU and we know from - * page_remove_rmap() that the anon_vma pointer from page->mapping is valid + * folio_remove_rmap_*() that the anon_vma pointer from page->mapping is valid * if there is a mapcount, we can dereference the anon_vma after observing * those. * @@ -1498,25 +1498,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from - * @vma: the vm area from which the mapping is removed - * @compound: uncharge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - folio_remove_rmap_pte(folio, page, vma); - else - folio_remove_rmap_pmd(folio, page, vma); -} - static __always_inline void __folio_remove_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum rmap_level level) From patchwork Wed Dec 20 22:44:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 250A6C3DA6E for ; Wed, 20 Dec 2023 22:46:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B7E98D0023; Wed, 20 Dec 2023 17:46:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 86EC18D0022; Wed, 20 Dec 2023 17:46:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61D598D0023; Wed, 20 Dec 2023 17:46:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4C4FB8D0022 for ; Wed, 20 Dec 2023 17:46:41 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2643080B92 for ; Wed, 20 Dec 2023 22:46:41 +0000 (UTC) X-FDA: 81588682602.17.62BF9DF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 6CCBF160019 for ; Wed, 20 Dec 2023 22:46:39 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QcRJmHv1; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112399; a=rsa-sha256; cv=none; b=AtEZhswoLuDVu9Yf4Ed1KL6VJRf62AYpzzlzakqWTe/kaU+OkKb6B61IlTgy0nPzNYCbZs jxbGxQj3+UTpCgi7a5M/b2L3TB2gHFcbEjeyCeMEV5cRSio4jRuvnP5Jjvj0RN4FdWWnZV G27JiT3gHJXPVTwfWjJqXSU/4dQ4FgM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QcRJmHv1; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112399; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FS6TinGTKiO3H/lfNCqQPkYyWdSgfpvkK+CFwNOsPIw=; b=ySDE5007ZE7D/AOt2AS7Xp7czrChmgERkVaVNZ+XO5P7LmaVvJRDwrWg4AomR0tXNBI/K0 ZFEiUuuflgFrhbmMDJ/82Ta0yl7iadQf7mK0jeRuH3XG949LVOBwhJ9zDoznC2RHLbEPpi ri+pb8D1X4E/ZYhzDGYN+66Z5Hyq8KQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112398; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FS6TinGTKiO3H/lfNCqQPkYyWdSgfpvkK+CFwNOsPIw=; b=QcRJmHv1aSOZwhkVzwt8zD8vA/pEEPQ5NRK/TjaqxSdr/aLru6PvOjOwuD99MGXNTmG+Xn 3AbMeo8J4GOX+seiuNcKTZGNF8/HuhUU95rM+XGdMP6G8AyMPhiSzEI48UZtpw2CP19Ruy ZQ2AZTsu5Kk4U0YGFh1Kw9L0cgvvJDc= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-314-hNtzcJbYOtScYX6nbBDUnA-1; Wed, 20 Dec 2023 17:46:34 -0500 X-MC-Unique: hNtzcJbYOtScYX6nbBDUnA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C6BF838425AF; Wed, 20 Dec 2023 22:46:33 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BC8F40C6EB9; Wed, 20 Dec 2023 22:46:31 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 33/40] mm/rmap: convert page_dup_file_rmap() to folio_dup_file_rmap_[pte|ptes|pmd]() Date: Wed, 20 Dec 2023 23:44:57 +0100 Message-ID: <20231220224504.646757-34-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6CCBF160019 X-Stat-Signature: 9wmwhynhw5m185hyo3a567g5d65uczzc X-Rspam-User: X-HE-Tag: 1703112399-525971 X-HE-Meta: U2FsdGVkX1+zk1i/0/7qsQ9TMQPn+0dE2VK2TNUKOEK3sw924IQKvOliBUVew1Vc7z6Xkg/RQEOAhUOJPKg91OQm+Rt0YVy/ToE/y6OsMiUguo4ARdSyBCryrz3WkwgTfM3CchjTpwaNUCcMBNN9/o+ofL2l6Ucd489OwQbumjBSUr0IOyzLLCgmtpC3AjcDxKoeoPm79KfG1AtOS7AT1+e+8vtvZ0pU40Q7fXdgKAiSoi7Fi0Z9/lZZq6IqbbPsCmpB9K7zSDIZW3aUaeyQqm4vtVtLHPQoUMO85i9v9dLiw8m0N8NebtOdbq7Y/PCg8kN4TdeS7fqAKXBkYODqMLxgH4WEkoYc9CzHXILbuyQ8QjZgIYWCr90B8IzddtY4vXWxOMQBc9C4znVI9t/cnq8zRwiobilCSZUBhcoDU5ztQInx4Z9tNHPzSpM2oRykISXkp+GL8W6GoeB+JUUgKnaL+uMxREx0EmlXBcuTjFo5x79014dqBHJc3ptliF0I1EkdCj+Kd/ADqTtJB0dhX28MLHBjrP+kAUTvYu+jdYUP0dljguClVQt1PY5kX0S71Arya+pg9v7ETwqvRifCE0VTV2qSU2ISONuTJuF9cRSs3iSIyr1vU25R+oDKUaAwNxBrfXtRUfSKIYFLvphVmy7uqsYLiur3N8i+b8kpeTFvcPPKgy4QgTfyqcZgYXazkeJXDfcMp5iyHytNfNDjRiLjpaxVEGDlsjnUXmjGajBnWPGtyQjWgMo31f/bFkzO0ubp2HJXOve5dq6MrxQqIdOFS0xXfH4PPqzaSoRc1V3ksc7uheabxSpAtjDhL1PhmOroEWZ4BPRhGaCAZMx6VhWVRrLrH7kdKCrt56mHdYJEuDT0nDUybRT4bbpfONfBU/Gzb65EQe9WGaGDI6VrnBc1QWJk9EZEFKisg+C9LNB47sGaT35CMt0/TycIxSy+HtLbYRIt7WsdEZrVmA9 udCcrgwl 3o38Cn43onqNDdkKEfUpwTtPbFAPhobzqL1zRkgROckB/ezA9Ne9tYm7ZLSExUYof+YIEpj/brs74t8wMi7QH1pG4Q41sGtFKh4HzbUj+5ndfwaFrSwsdMW06cwoG1CfFGI098NAkaoyLkyA1hs4DkngBqMylIoQY4kBiL6hZA4H8741XHWjk7XfUxvSzFqbiQQug9E+X+Ft0xzXYcylH4l/ojq3x4YUCCZ5A X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert page_dup_file_rmap() like the other rmap functions. As there is only a single caller, convert that single caller right away and remove page_dup_file_rmap(). Add folio_dup_file_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 59 ++++++++++++++++++++++++++++++++++++++++---- mm/memory.c | 2 +- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index fef369e37039a..7607f862e795d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -308,6 +308,60 @@ static inline void hugetlb_remove_rmap(struct folio *folio) atomic_dec(&folio->_entire_mapcount); } +static __always_inline void __folio_dup_file_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_level level) +{ + __folio_rmap_sanity_checks(folio, page, nr_pages, level); + + switch (level) { + case RMAP_LEVEL_PTE: + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + break; + case RMAP_LEVEL_PMD: + atomic_inc(&folio->_entire_mapcount); + break; + } +} + +/** + * folio_dup_file_rmap_ptes - duplicate PTE mappings of a page range of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_ptes(struct folio *folio, + struct page *page, int nr_pages) +{ + __folio_dup_file_rmap(folio, page, nr_pages, RMAP_LEVEL_PTE); +} +#define folio_dup_file_rmap_pte(folio, page) \ + folio_dup_file_rmap_ptes(folio, page, 1) + +/** + * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, RMAP_LEVEL_PTE); +#else + WARN_ON_ONCE(true); +#endif +} + static inline void __page_dup_rmap(struct page *page, bool compound) { VM_WARN_ON(folio_test_hugetlb(page_folio(page))); @@ -322,11 +376,6 @@ static inline void __page_dup_rmap(struct page *page, bool compound) } } -static inline void page_dup_file_rmap(struct page *page, bool compound) -{ - __page_dup_rmap(page, compound); -} - /** * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped * anonymous page diff --git a/mm/memory.c b/mm/memory.c index eda2181275d9b..dc2a8e6858179 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -965,7 +965,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, rss[MM_ANONPAGES]++; } else if (page) { folio_get(folio); - page_dup_file_rmap(page, false); + folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(page)]++; } From patchwork Wed Dec 20 22:44:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DCC7C3DA6E for ; Wed, 20 Dec 2023 22:46:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97BF58D0025; Wed, 20 Dec 2023 17:46:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DD588D0022; Wed, 20 Dec 2023 17:46:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 735238D0025; Wed, 20 Dec 2023 17:46:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 545E48D0022 for ; Wed, 20 Dec 2023 17:46:46 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2690F1203E6 for ; Wed, 20 Dec 2023 22:46:46 +0000 (UTC) X-FDA: 81588682812.18.165CB62 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 5C306C0012 for ; Wed, 20 Dec 2023 22:46:43 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PJLWuO22; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qNA8pWEc7GnGKj0IRh6+wWeJkGUBzLslpaDMoEDhhnE=; b=fTxea/FXBOle+igBP90iPB+FkVFTO5dCHMudf7xaFHoE6m83mzwrToXvqwwsa3931sRF0r RJN++jlChohZ3H/KXAODb1HF+pEMCg79SthiLDT2v4b3QfGLFMnzqtvRJtxRD36GAePu5b /000VmyOEo+4GoLAjiaHNe2OnR3hAqc= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PJLWuO22; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112403; a=rsa-sha256; cv=none; b=TvNV/0sImEsrNf/H0spk5+fFwvQo6wRQRe3SpODHxHJuiwZ0FyKnb5m+HhvCJUGrIY1701 8QqY5Pl2DjJR2cOShYyJ+5gpYf7JRiKXQapE89wd3Z6TmBP93qsGQ7fIztgiR84qWpjXHZ iqFPtpPrKDKWBEO4N9+PENI1Nc1U010= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112402; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qNA8pWEc7GnGKj0IRh6+wWeJkGUBzLslpaDMoEDhhnE=; b=PJLWuO22VvPtfULpCk7t1XzGZml0/TAtWbveSKTwfnhX36wFQt3w5zwuE3yYOiHOXqlAoq GkgBE6OGe4y3hZxmZmWbKOIm9AGteZFK2/UT/vp2uHt+qnz0Y7UAekjKnZpfbYjCbEV2FS jqgbsNJkH/788zigWzJLbzw9VAKQgLc= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-464-FgP96C90Ok2CEIy_NFv94Q-1; Wed, 20 Dec 2023 17:46:37 -0500 X-MC-Unique: FgP96C90Ok2CEIy_NFv94Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A735D38425B2; Wed, 20 Dec 2023 22:46:36 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F9F740C6EB9; Wed, 20 Dec 2023 22:46:34 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 34/40] mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]() Date: Wed, 20 Dec 2023 23:44:58 +0100 Message-ID: <20231220224504.646757-35-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 5C306C0012 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: xf8qw5fnxwrc8amtc3wi83eain11k98r X-HE-Tag: 1703112403-30294 X-HE-Meta: U2FsdGVkX1+Ene5x+/JIANChPtW5HLxhnViw+660UKJn+goPiaGpV1uv9zdMj6ieUj361N/SrZwJ80LimmICFUQc5jFukO4SP+Z1NA3jp7UAaS5IaDppGkv7OjFIKjbNXLhhKSiZMY0g3HeT9Iu3ayTQDZGAk0XXXTa9hz4q57q0iYIa97mlSXiEDTpkyyrB1po71SbCL4kQTKf8vvABSKXAF6UBikJnnS4xTyF2yc8AvXz2nCElIUpULqyPqFRP20tTnu5UeWS/hJI94Fe1CgTjl1RLFFAzxCcIINGRmsHw88GxRXm2I3QJQUx2IJjWN6hd0huSt+O+kE86Wf8A0bUlLzZlRUndaU3VEG3N21RZKwT0KP4YnI3KACJlEr8yFGCYBEQfRBElp84thIhYtpGYywZNarCVkiRlGt19AByeb5iXBgLmuZ3b0AMt87ft7PptSAkS4/uYSrS5DBsA90RQZ+v+prQbgoGhGh2vby0JSZGTfNLcjS+qX0KSTquODUAKxwDQGq/JqCXXcVkLMGDwhincw1+xUAjJ/Mx5kV9nWrdaQf3lA4AZJLNCokf16Muc+St9bKkINxK9lKYAjPLi7Y/06GkprownmZ5cjZQ7Ls/3b6hOFfa9BBHsR0UQ6nF8JB6YzKe3387DiQEwaj/XrlQY2fyTvTVXf83Nbj8iCX2ShxELW0k5zgDrcDClOLmCfmizGDKiYTuAkLQkC7WVSSoj5EWSgJiS1CHID/moR30868tVwHUWgGP7cYPhjFp6Yrr3by6DWlqNp/n4VJm2N2Wd+RkTGqJZFpKTFzOs1lYm68PfSag6qcceRCl7+282TonIGwbz4RX9JuuT7zTmxSkkhKZB2CmlbNb6L0xHu4VYX6iYL/0sZziLRbyaraI91qZibr+sbfr7aMPFe6W3Hh/51SL4QKaxzPR+EKscxKoz3ycwA5iUEqNmWJ39uUqRQ+lRXwRsx0YQQeR wc16hXwp w767jDf7sD+ta/G6+JVZnNKGm4+x2YmyErU4jfulIcAYrmVgtK+Dpjh/yThzMF9jmorQ2RGFv0q1YIBdjHmD35fElCsI1JEZPemwpg+fo8ykZiMQBl1WA37JOszQvZtQzE6+gtaU/QYMcxojCp4YQN43jjBKiilowCCYRZ0FmQWyZPvHylzbn38prZCZajijSE/jZx2ixdwCyXFEcFMZ+5kmNJN/FL2UCV/kC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The last user of page_needs_cow_for_dma() and __page_dup_rmap() are gone, remove them. Add folio_try_dup_anon_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 6 -- include/linux/rmap.h | 150 ++++++++++++++++++++++++++++++------------- 2 files changed, 106 insertions(+), 50 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ae547b62f3252..30edf3f7d1f38 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1975,12 +1975,6 @@ static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, return folio_maybe_dma_pinned(folio); } -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) -{ - return folio_needs_cow_for_dma(vma, page_folio(page)); -} - /** * is_zero_page - Query if a page is a zero page * @page: The page to query diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 7607f862e795d..850aa74b6724c 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -362,68 +362,130 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio, #endif } -static inline void __page_dup_rmap(struct page *page, bool compound) +static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *src_vma, + enum rmap_level level) { - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + bool maybe_pinned; + int i; + + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, level); - if (compound) { - struct folio *folio = (struct folio *)page; + /* + * If this folio may have been pinned by the parent process, + * don't allow to duplicate the mappings but instead require to e.g., + * copy the subpage immediately for the child so that we'll always + * guarantee the pinned folio won't be randomly replaced in the + * future on write faults. + */ + maybe_pinned = likely(!folio_is_device_private(folio)) && + unlikely(folio_needs_cow_for_dma(src_vma, folio)); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + /* + * No need to check+clear for already shared PTEs/PMDs of the + * folio. But if any page is PageAnonExclusive, we must fallback to + * copying if the folio maybe pinned. + */ + switch (level) { + case RMAP_LEVEL_PTE: + if (unlikely(maybe_pinned)) { + for (i = 0; i < nr_pages; i++) + if (PageAnonExclusive(page + i)) + return -EBUSY; + } + do { + if (PageAnonExclusive(page)) + ClearPageAnonExclusive(page); + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + break; + case RMAP_LEVEL_PMD: + if (PageAnonExclusive(page)) { + if (unlikely(maybe_pinned)) + return -EBUSY; + ClearPageAnonExclusive(page); + } atomic_inc(&folio->_entire_mapcount); - } else { - atomic_inc(&page->_mapcount); + break; } + return 0; } /** - * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped - * anonymous page - * @page: the page to duplicate the mapping for - * @compound: the page is mapped as compound or as a small page - * @vma: the source vma + * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range + * of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * @src_vma: The vm area from which the mappings are duplicated * - * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq. + * The page range of the folio is defined by [page, page + nr_pages) * - * Duplicating the mapping can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. + * + * Duplicating the mappings can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail + * for them. + * + * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in + * the parent and the child. They must *not* be writable after this call + * succeeded. + * + * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. + */ +static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *src_vma) +{ + return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, + RMAP_LEVEL_PTE); +} +#define folio_try_dup_anon_rmap_pte(folio, page, vma) \ + folio_try_dup_anon_rmap_ptes(folio, page, 1, vma) + +/** + * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range + * of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * @src_vma: The vm area from which the mapping is duplicated + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) * - * If duplicating the mapping succeeds, the page has to be mapped R/O into - * the parent and the child. It must *not* get mapped writable after this call. + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. + * + * Duplicating the mapping can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail + * for them. + * + * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in + * the parent and the child. They must *not* be writable after this call + * succeeded. * * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ +static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, + struct page *page, struct vm_area_struct *src_vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, + RMAP_LEVEL_PMD); +#else + WARN_ON_ONCE(true); + return -EBUSY; +#endif +} + static inline int page_try_dup_anon_rmap(struct page *page, bool compound, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(!PageAnon(page), page); - - /* - * No need to check+clear for already shared pages, including KSM - * pages. - */ - if (!PageAnonExclusive(page)) - goto dup; - - /* - * If this page may have been pinned by the parent process, - * don't allow to duplicate the mapping but instead require to e.g., - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future on write faults. - */ - if (likely(!is_device_private_page(page)) && - unlikely(page_needs_cow_for_dma(vma, page))) - return -EBUSY; + struct folio *folio = page_folio(page); - ClearPageAnonExclusive(page); - /* - * It's okay to share the anon page between both processes, mapping - * the page R/O into both processes. - */ -dup: - __page_dup_rmap(page, compound); - return 0; + if (likely(!compound)) + return folio_try_dup_anon_rmap_pte(folio, page, vma); + return folio_try_dup_anon_rmap_pmd(folio, page, vma); } /** From patchwork Wed Dec 20 22:44:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 784BEC3DA6E for ; Wed, 20 Dec 2023 22:46:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C49A8D0024; Wed, 20 Dec 2023 17:46:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FC5C8D0022; Wed, 20 Dec 2023 17:46:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 273F08D0024; Wed, 20 Dec 2023 17:46:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0ECA18D0022 for ; Wed, 20 Dec 2023 17:46:44 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D53A01203E6 for ; Wed, 20 Dec 2023 22:46:43 +0000 (UTC) X-FDA: 81588682686.24.4033812 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 1B7C340005 for ; Wed, 20 Dec 2023 22:46:41 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J+NmOtc0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112402; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eh/NeqNby3APEgZIeJ5Gbh/kC7oqyH0NpFSKRwTjdSA=; b=6nqqOusTb1YEghqZ5xQhItveom+J4gQ3gtTlIju42up3RvUPFZCRc13EjxqSG4FVhSFiEu Y3yTV5VfP2kge2V3d17Fr+UkBbsPZxJgsw1TAwMsg3MrMa3+fHZynJtOzOuk/VTZWZam8S fdf98KY5oiMpiPe3U2TjaDZxeRploJE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J+NmOtc0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112402; a=rsa-sha256; cv=none; b=u/U1R1kuneMATZy68yNVjlLSAklBRqKAcifGZ4ee9F2V5fVXgzJESoUynvNZRMIKQkrbIx dTq+FdrZIQOOBNdE2K3ypjXnSnz2H5wPF6jILlQlAhbUzPWXFH52rcmF8iXWKkXCvtofNX HM8a5cYQTRGKMDCcq+zV8T1aDbDjjO4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112401; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eh/NeqNby3APEgZIeJ5Gbh/kC7oqyH0NpFSKRwTjdSA=; b=J+NmOtc0go66rDawAPBVajmuJYaoHFHbn0iuM15iVcEClZFhazgnHmRy8x20ozKH92wWIu VT4H5iQHFkipNLTL5wXAKQW4O6gcJ4AJu+fdnpwaXhYya2Yfx8H94zdfwtBXULViAHOdbM PsHe/kQGushF7lhX0FaaAyjkcortwc0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-313-ZfLMWBpIMNiX3NN0QdVaEA-1; Wed, 20 Dec 2023 17:46:40 -0500 X-MC-Unique: ZfLMWBpIMNiX3NN0QdVaEA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 538C13C23FCD; Wed, 20 Dec 2023 22:46:39 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1209840C6EBC; Wed, 20 Dec 2023 22:46:36 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 35/40] mm/huge_memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pmd() Date: Wed, 20 Dec 2023 23:44:59 +0100 Message-ID: <20231220224504.646757-36-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Stat-Signature: wqbbanfh7mubbn8zuwsyih3yx3rjxqn4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1B7C340005 X-HE-Tag: 1703112401-703393 X-HE-Meta: U2FsdGVkX18GbF8r2/Xo76LELMkp3sUsT0z22RsBmIrKHvegqzCD5jmREv60eS5ZbCaMMfMFrZeEoEEX/n8YxPG8zySSjyGDr7h4eo0V5m5PhxuD7j7tVLTftxz/v8M4m7okVhG/HDGS2iF+hDEGC+FDo2F8kPObB3IN6SCRuw6mXAvsZ7qrT+TK9rjGhVm2vBoiijnNHezMc3R+kQgBrJNcDaI2mven3fQzSiiG4qDdYr2qVy50Y7S41q/2o5ASzQ6tDglC4P++IsvxfGNyWu0DJnMtAUaCQjTqVP+1+hRuhOPjNs9Wdl1p1oKILWxspBFk8dJYCZr2KH6klPsON52s2mSWz1FrccUwxkEHsX2BA/OGB9o+kD10UtYfRhMTjVkzYKgVZua4Wc7ipT3AM845+6w2LcvuSim4cHAgAt2Do4WdtPNG1tGROVXZ2ZTOVvLDEEVRJpu78HaJzAt/EfjI3duLyRBxAGiebdxUFn8hm1bitNihWPYAmgYy/ycVBgWW+qHcg5kMWgP/juUBAjnIvnwUglu/R/zOeEX6aGr976Fur9hRB0qVNsv9d4cBluTtHUANu9CGiNNhwa+DhumSktChL1pdAAb0mOUK76AMdhov9Asar+mTGCEktkvrbLnOse2GUxlCKuCQUscBWL7ciudpeUNLmUVaFlTbgUYHnk6oC3lAYCdrzDdlnPFWLLJcEh5YVA69URDfOQrOXYmXaQjJPUK5G26fFYmqSM2RxOno8NcMAjZ2M0GcomplpIJkKjHvmaGw4Bk7EthBSkushRfrq0BNh7wVV59dtkooY9dOVIeynKwp+Ry4JX4KWTdMMgOiduMsSHbCUIBYWU5i60e8eKkW0HheNQ9m6c4Rap7ptLVHyfhddUY+6XgmRcU/1Ld89RHxHhbGL0/JvYfIYvO19nT4n9YHfi0IxZsQHypzMDJZWc2CWaHPLbnX68v1hP7jgxN/xNvwiip BSkp422O 8+0TlHd0aQDOV9yE8m2T1r9tMr6QNyZQ8OjVUXgjm01/LoBfHQMh4yxIA6tQIgbBiFqI5y5wZyWR7ufaedaFObwfpFsjFuwCVuGl5n2wEiOgutA4Wn2Jo0qzvPCoAsqz2ArVrX9d29thz4rkD/aGwujHwk2kGvRtfiwDsENQ56QGqggXRkMnF22cVzp7b0X1iy1hrC1lHJNO6VjdlZaR80lIvFA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert copy_huge_pmd() and fixup the comment in copy_huge_pud(). While at it, perform more folio conversion in copy_huge_pmd(). Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e7bc0f38ddc53..edbca08449357 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1275,6 +1275,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, { spinlock_t *dst_ptl, *src_ptl; struct page *src_page; + struct folio *src_folio; pmd_t pmd; pgtable_t pgtable = NULL; int ret = -ENOMEM; @@ -1341,11 +1342,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_page = pmd_page(pmd); VM_BUG_ON_PAGE(!PageHead(src_page), src_page); + src_folio = page_folio(src_page); - get_page(src_page); - if (unlikely(page_try_dup_anon_rmap(src_page, true, src_vma))) { + folio_get(src_folio); + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ - put_page(src_page); + folio_put(src_folio); pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); @@ -1454,8 +1456,8 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * TODO: once we support anonymous pages, use page_try_dup_anon_rmap() - * and split if duplicating fails. + * TODO: once we support anonymous pages, use + * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_mkold(pud_wrprotect(pud)); From patchwork Wed Dec 20 22:45:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52E15C3DA6E for ; Wed, 20 Dec 2023 22:46:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2AE68D0026; Wed, 20 Dec 2023 17:46:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADB058D0022; Wed, 20 Dec 2023 17:46:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 841E08D0026; Wed, 20 Dec 2023 17:46:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 69CBA8D0022 for ; Wed, 20 Dec 2023 17:46:50 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 44AFC4041D for ; Wed, 20 Dec 2023 22:46:50 +0000 (UTC) X-FDA: 81588682980.12.35FAB92 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 8D8B84000E for ; Wed, 20 Dec 2023 22:46:48 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Zpm5g6Q0; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fTcpbW98US35msxaiX+BCXxL5QAKVhrGk2lkGfrSlfw=; b=26mZEhHOIoGrfe31swK1jubcp+0XIGToviQNtzLD3V3iqlMKFYJUALZqE3HmojVForwAQI YaDXrHhOd1qc8aK2s/G3Xyxniby/RIUgsPsSKv5XoLlnDFKvNKl3fY5f10FWHrGeTp9ry7 25fOzSKK40/1EZRplZ0bLMTZHhGNJyI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112408; a=rsa-sha256; cv=none; b=hcbPzvC5oy4/RU3oYrkXjpRn+pNuW0vTF2PvmGUuYjYlRDlzAIYtSsJjKriE9kBM6/Yd/P A/giLMKdGd4fmUtOVuCqcVy7EywoRfmUbLbdmkeVQoNFyzdHcD6sx+2f4iG8U3lfZW1JH6 S0WT1UKaI8SFJrJWC7kijHBGhACLQco= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Zpm5g6Q0; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fTcpbW98US35msxaiX+BCXxL5QAKVhrGk2lkGfrSlfw=; b=Zpm5g6Q09NBAYfk/y7xEWtjLQ13Z7H/pZ/npKNe5aL54NWJPfTdJa5DtQ0wDzcoICT1yI8 /uZ5XVJAnlLOKIhmgruZo3oLWAaZgihTldbMVBPCmF84fOVX6FPOciSzISQABQr1JqvgaH J8I+turVCO0bhI3blCwko9nzUZ6HIR4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-7CmBAUaHOZWiWfdpU_xHgA-1; Wed, 20 Dec 2023 17:46:42 -0500 X-MC-Unique: 7CmBAUaHOZWiWfdpU_xHgA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 034AF85A58B; Wed, 20 Dec 2023 22:46:42 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F08C40C6EB9; Wed, 20 Dec 2023 22:46:39 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 36/40] mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte() Date: Wed, 20 Dec 2023 23:45:00 +0100 Message-ID: <20231220224504.646757-37-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 8D8B84000E X-Rspam-User: X-Stat-Signature: trwup7rr95gznjigfcj9pzk5j5f15z6t X-Rspamd-Server: rspam03 X-HE-Tag: 1703112408-630035 X-HE-Meta: U2FsdGVkX1/Jhw8liwNhmeW/Qyihi4bn7NVUtTlgbbOvyQOhJjZ6EF3atX6Nh2AewW23nonvce9RIgVLgXPn5e0r/KWmWdvIZsYCkSSTamHaod775YoYX4hbkgCfMoQOPtTWj81ukhzUlg6hgIM88Q9VWVxsJ4NtXdBXNxB3cywXS203E/hBfDSfETIedi5tLZcKTcLMFuHBRsxnmBwJ8fG3RGb9jUxmfQpysMl7HyXJSpvMF1bABuDTSzAIeS6h0odZHEZwSNsnAd08DAxItS6cvaovL2uMz6QqQe+gzZqGis2piRIdIhpPjsMLPHWr7waYn8351Mk5/kVAX+69QGIMKVS7eZJFYSMaFxIHSfI1PNz5oZhiSTQc2En3ATw3px9FuBzKHpVdAMrHzFtTtR4j37FNqVBYtBeWYG6J21ERMM4qUakL8vv+Zv7BAm5RGcVKBtBPvRxuI/++K7fwLJUyjdhsiBrhmube1IaJsQQw67ybyjakuSPEEVpzRGyDxT7KZ/Na3DAwoEZ10mR+gDEoI1XgaprDGlJ6b7za8jqw/oXWrcHdonPhYAdkx92reT/07MMbjN8D108dQSzIEVDPjcW1ty4REAptZDRAaiEnG/s+Q4YRa1OtdW9KIBe+C7N0dEZU1l2j9zKmBcrFWQuOLM94OiK9PlLNyvb+pmS0ZfNLFY8ZgT9jCWbf+hRwDZcy4y26dy/bVa7Scd5o8vfoILA2b13Lfs2gwhGj8ouvpkaG8YItQokfYrFyfVtzxcPcCzDvutdYRKQ5M33lwaihGJIGTtpMCZOD/AQPv5sdKYXlzk7aYCUbNXJJ2HSDzjqCKPHWfvTGqcr7yDU7MfXmKjNYw3ZMdJMKuLA4cnC48rd+uk5IxSZBc0it4apMxViweeR+QGm+TQnSFodPvckMvOAmIzw79oBSLnksnJpCI8N0KgaJNzpkiv8qnnRsAF7nJCoWE9un55E+6kK CpOPOIno 98oBvi5bO4VqkJK6KAQwqju8m1KTtoEqvQK+wOTlfe6kSBbkPak5oUYbvwu3aIpNXEyocaeQsPiPir3UyTK+w3VaodNDTNuoeUQf7SG3xHLn/1o7pKoSd74WWDGN0fVnvqVDFANEmP1kafNe4D0riouAR94x1GFn4W4vjwEeteFOlomk9wKHw+IEJF7dLX2qmixgQNllqVr9WGr+1GKvQq4RcOA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert copy_nonpresent_pte(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dc2a8e6858179..d995ead7a3933 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -785,6 +785,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); pte_t pte = orig_pte; + struct folio *folio; struct page *page; swp_entry_t entry = pte_to_swp_entry(orig_pte); @@ -829,6 +830,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } } else if (is_device_private_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); /* * Update rss count even for unaddressable pages, as @@ -839,10 +841,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * for unaddressable pages, at some point. But for now * keep things as they are. */ - get_page(page); + folio_get(folio); rss[mm_counter(page)]++; /* Cannot fail as these pages cannot get pinned. */ - BUG_ON(page_try_dup_anon_rmap(page, false, src_vma)); + folio_try_dup_anon_rmap_pte(folio, page, src_vma); /* * We do not preserve soft-dirty information, because so @@ -956,7 +958,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * future. */ folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, From patchwork Wed Dec 20 22:45:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2727AC46CD4 for ; Wed, 20 Dec 2023 22:47:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 164718D0027; Wed, 20 Dec 2023 17:46:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 114B88D0022; Wed, 20 Dec 2023 17:46:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E83B28D0027; Wed, 20 Dec 2023 17:46:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CBFEA8D0022 for ; Wed, 20 Dec 2023 17:46:51 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AAB69A0250 for ; Wed, 20 Dec 2023 22:46:51 +0000 (UTC) X-FDA: 81588683022.04.A88C282 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id C136F14000B for ; Wed, 20 Dec 2023 22:46:49 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="FLuJJv/N"; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112409; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9XkiTziRJNAJJWRX4gshftH/Gk4w9uSE1IPr6jMrvv0=; b=NF2cZbdrwTxVZ+QF71msir70/ivIK7s+u5DGgNx/xlCw0ENa69BEExBAYnozgwwqYgBjRs BqTtooz44S6vhHGxc06chU8nAcmmiGCaFxufZ/flmwHQLTSCpw+VKeMPO6rKHnTmmOZ8jy u6sIzPfD3E212T4CbkFgFDEFY05c8Lg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112409; a=rsa-sha256; cv=none; b=q4pLoHDrJrl/iXRFxNDIMVRK9VmB1Na0jnRziXa2n0ybE1GV1Gb4xEQTk5IQM/wSlEOoZo XmeB3j69edPeVmzYDV0ShPN0ZtgpwHTcrgeKdaN3VA6IILDQowfMgiGfTuUYXHRN0ETqUd 8ZGHhnFGfyVO0811leJSPQJpUbUqGNU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="FLuJJv/N"; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112409; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9XkiTziRJNAJJWRX4gshftH/Gk4w9uSE1IPr6jMrvv0=; b=FLuJJv/NGSs52X47AWDQeItoiey9nMqObOEkxpYx2C2HPw3m0KaTdCHY32HBLIPFR16fSL u4BMFVNsycp809VGV2pFdFqtH2MQ6o8+rohgF3UEy79s16tBxNCAck2wBXmxUK+eInjXIo EzDD9mWpeDI8YQluPGOuitc3fHBD8GA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-451-NmSSg2uJMuaP3fhgO4P08A-1; Wed, 20 Dec 2023 17:46:44 -0500 X-MC-Unique: NmSSg2uJMuaP3fhgO4P08A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 62C1E38425AD; Wed, 20 Dec 2023 22:46:44 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 61C2740C6EB9; Wed, 20 Dec 2023 22:46:42 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 37/40] mm/rmap: remove page_try_dup_anon_rmap() Date: Wed, 20 Dec 2023 23:45:01 +0100 Message-ID: <20231220224504.646757-38-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Stat-Signature: xktjyf3hg7gebdr57y5x8ufumowgrkzi X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C136F14000B X-Rspam-User: X-HE-Tag: 1703112409-732244 X-HE-Meta: U2FsdGVkX190UOGKlzNYtoCl+QUw8qvozjZbIWr1KfhHIKXp6P1Me3s2vpTKqtD++KjmZDR0WOEX13WAD5pbGCgVr5laYE+Qmk+2HJ7oeAapte1FczDLJAc0NTFAPDAJegDdaazhNZ5oZcNDh9rf2GuL84r2ehiI4kGUkgRnzjwDxqUdi0oTWBnsGybnmFW8QQQKkMiu+eRQpR4nIcc0h6FtdKdhUow0c0tmTAX73a6pxYFW+7xTO5KTDbQcA/wdf5+Y84/LbpOsD8+W6olB3nC57c6wom6+JL29vFsu6viPFC1BDCtMInNiIvdiPFC5djkQomVwivtA+4ssODiN+ZZjp11TXygyqRozMd7JO1MVp76mZZS0MbneF0lR/0gRos39zxH/fTTrj3bcjPjWrJ420heg1uqo6SH4QUc8+o//xdrQq3MQ7fMg663UbWamrPKOCWRFcGDuRLyRZ21h1J9sLq9krqS6zXBs82pXt1DSY8yOk9OgTMn1bqbUcrImEeRZIwGmel0Aab41xYs1knS71xZ69MEd0KPJ8yLkjRPekbuyvWm77uu4W9XJws5jYZ0CxLUwQzgPmPMsrqmilDobuz4fhGk9j1Il5XhQJ4cLQFFHKqa5X0VaP+7Okpz3X5RNkxfUjCNbanbCh1eqAleGjj+lV2/9eBoWxgDPLhnWXg83h5pLGY/XIepoj6ikR3oqhUBlinoweobV2vLxXmGw1YnKVre0ewsPjxzgVaWegvu67wbIEriLYkSywo7aT4cFslxBs+EwZObrIp2NXjBH8ey9N7sG4/DI32JcGgjFD2S3Df3JK3weaBwDn9DpNgkV/gk3HQbPGeJAV6RJga2EcGetRSzD0x8UjHMFAQPpko9heo4q213ics6bgP3GYJSALzq8/wpqbocWcpYUAFN8FYxHs/p7zS3XWinVpU0OlGmPUROR2HZFYoo3lUhSWizWOY6rJYx7g9C4kx4 Oe6ouayg 6EVRcADfVUwzNRXFx0d+l78ryw3MLINHd83pB2b1+CoPKgVcGAm3DZdfImByp30f8rdDFyKYi6SQZ1WyDfxeXtMuaEjZL0m3vvLP+o2qLITFfstjbVq6PZRTztjYHzBJvsO0fzi3qaba91YceepaFAb0Kg4qfMaahXwdgD0gjruuIsoJOG33/w2QBTi9GcVhvynDbkQsaEndB4TtLYE3vFd444A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, remove page_try_dup_anon_rmap() and any remaining traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 850aa74b6724c..0ad2ea2734e4a 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -253,7 +253,7 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -/* See page_try_dup_anon_rmap() */ +/* See folio_try_dup_anon_rmap_*() */ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { @@ -478,16 +478,6 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -static inline int page_try_dup_anon_rmap(struct page *page, bool compound, - struct vm_area_struct *vma) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - return folio_try_dup_anon_rmap_pte(folio, page, vma); - return folio_try_dup_anon_rmap_pmd(folio, page, vma); -} - /** * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly * shared to prepare for KSM or temporary unmapping @@ -496,8 +486,8 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * The caller needs to hold the PT lock and has to have the page table entry * cleared/invalidated. * - * This is similar to page_try_dup_anon_rmap(), however, not used during fork() - * to duplicate a mapping, but instead to prepare for KSM or temporarily + * This is similar to folio_try_dup_anon_rmap_*(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device From patchwork Wed Dec 20 22:45:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8A5AC3DA6E for ; Wed, 20 Dec 2023 22:47:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A1EB8D0029; Wed, 20 Dec 2023 17:46:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 007BD8D0022; Wed, 20 Dec 2023 17:46:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4ACB8D0029; Wed, 20 Dec 2023 17:46:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BF7178D0022 for ; Wed, 20 Dec 2023 17:46:55 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 829C2A05C3 for ; Wed, 20 Dec 2023 22:46:55 +0000 (UTC) X-FDA: 81588683190.06.E136253 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id B8A7E180015 for ; Wed, 20 Dec 2023 22:46:53 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P6dPApAY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112413; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SSs7kxR08TFUh2upMd1uVQYPxntUe8H10GgxXN6/rpw=; b=ByycnZwOg+wJZfIDMSKOsbJWSpTnpLa4reCBn9FreY7GAk1CSqgFtrzWQ1CRoD/TRrOyt9 l5YPo5nH1oPkfqUeoeYyR1MdKjrNM9Qmwftw3SIqlC/vvrhrMG7bnqkcT+R/+FPJHS7A4O Cl5NGzfo9EosuWMR5I9dgFnxTdZibDE= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P6dPApAY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112413; a=rsa-sha256; cv=none; b=j12XNanbc1QZTf/xE3uXnGC2Ajn6jjrewEMwE1Llz8x8pL97HyJ8x9ssPZg/Q1n9mgwhWw Aq4qJxllRtPELTAUsorV/yktgajI/pEf9M/3Ns687lnUv6WWpLn5IUSevRknnJr8o6DeLV AcRs2dWu0z8yBh6POwQtuj31OkwhBn8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SSs7kxR08TFUh2upMd1uVQYPxntUe8H10GgxXN6/rpw=; b=P6dPApAYIiwV6DXEYJrjb3o6YaAqauQzAtr80DB5F6keSstxDsh4CJlMZTo4efHCfoeJUn dyfOujr84Is49ghmqlgFjSJAAIAbK3ZL3v4wHTZ/mMNSkjYIWdCpmVSNg/9Quk1HkEJzs6 ISZYuBGL5zpA+z5DyLlL2rj5fxqOQEk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-x_NUJPmCMxC9Uh7JS4QKgw-1; Wed, 20 Dec 2023 17:46:47 -0500 X-MC-Unique: x_NUJPmCMxC9Uh7JS4QKgw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E2C76101A555; Wed, 20 Dec 2023 22:46:46 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9DD5A40C6EB9; Wed, 20 Dec 2023 22:46:44 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 38/40] mm: convert page_try_share_anon_rmap() to folio_try_share_anon_rmap_[pte|pmd]() Date: Wed, 20 Dec 2023 23:45:02 +0100 Message-ID: <20231220224504.646757-39-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B8A7E180015 X-Stat-Signature: 3u1z8r1qmt9g7h13f5m9tu6qmudgbpux X-Rspam-User: X-HE-Tag: 1703112413-715086 X-HE-Meta: U2FsdGVkX19CHjaCO+ysxoM4F2aVp10FHj43uL5UT9hJfClVTK78zAJDOWluDogSyKlEOsThD2JxedAc/BQmT/oRF6T6kIS7gvduZttdFmTr1Lmsi2UQ+FZVFKsTqVZB6S0e78cbGcJZu6NlETeUiIp7DkR1Hd0QbMyBMpY30/oKMly3jj97SZkm9JdqCfg8SLs5UonL3tgIdi8pqd/LrR6sP921mioWgrjjjjf5412MiMe6kvxKG3QO/MPNESiJaQrMfvBvnFGzTitGzM0VYd6l2mHY9rOpdBBSZYGyz+zRa3q31PFrC00FBvslH6oEePKV+wZ5pbe9yQYyPm93byIVCbCs8R4RrXh0AjTWxKDyEMWP1UOLIZB341CFL56lYGFLhNgSXIjIkdX89/5YyzKBf0Y1j1vLdoj5LaKCvpZkU16itKk3dNeE2g7H5KEgRalMb6M9ZH9Oq7Q9S5Dvjf9tBrI91CpVuYM2qRQ7DmL6E7LBlPlJ9RmYG8BtK/dAtzoN6jvmkhyp2jPW2sLzNVhaH9TJMFuBZar1Fex/dSVxshmFYl/ho9ZGTp19KEMXWxSW8Kt495NMlukKo9oS+1FcsWh7TJ1/cUhXGg5+mefmnOOepHOXEZHbNEYauqNVr2m6e1OTx037pHQ84jyL4+6mNbT2wCDeuuXsURbDrrCPaOGd7hSlJ7nZEBPWCFSAodHkMM7Mm+Me3iqTAS0T/GTyKmC/p1RgqWLs/nNwYSraFQEWZ0Bv6AOHO6IhmdMWB3mq8abcBmHl8Rq3fql1cZtQX57KLCINakCk9eA41NPkIyPsFv6ZuXWtD0MwD7gsDFblYSGvANeVOPVL4AT7mq194CvHmoscth7BYIu80vs2kleuUvrdRaq6wb4IH0KBKt3r5/q1ZOg1phmD7GCZwH9PzyCOim8nTHwjssBv5TsKM5E7AoXBtSao+bEIKFS52M84tFwlHDJMbcbY9UF H/V5UXUZ NgLIqzwys160/q2WEkj7C8qGLo0ew3Lg2IA5zwkHjvRhWa7mPuhPLWCe/suhtR0S97/QSRs+ru3iPUZ/i5cr6PokjDcOPiOqHOvBd/IPaUatawtuZdu+QEo35p669biihzJDyatpZOHBrCgjx+QABq7uJPjNXWUSp3ssVD4NQGOJlRDSs/20X9APZTS9bpQo8XOv6PJgXCpDWswpXiK0m5kTdFb15fRo19MNJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert it like we converted all the other rmap functions. Don't introduce folio_try_share_anon_rmap_ptes() for now, as we don't have a user that wants rmap batching in sight. Pretty easy to add later. All users are easy to convert -- only ksm.c doesn't use folios yet but that is left for future work -- so let's just do it in a single shot. While at it, turn the BUG_ON into a WARN_ON_ONCE. Note that page_try_share_anon_rmap() so far didn't care about pte/pmd mappings (no compound parameter). We're changing that so we can perform better sanity checks and make the code actually more readable/consistent. For example, __folio_rmap_sanity_checks() will make sure that a PMD range actually falls completely into the folio. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 96 ++++++++++++++++++++++++++++++++------------ mm/gup.c | 2 +- mm/huge_memory.c | 9 +++-- mm/internal.h | 4 +- mm/ksm.c | 5 ++- mm/migrate_device.c | 2 +- mm/rmap.c | 11 ++--- 7 files changed, 89 insertions(+), 40 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0ad2ea2734e4a..fd6fe16fa3583 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -269,7 +269,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } -/* See page_try_share_anon_rmap() */ +/* See folio_try_share_anon_rmap_*() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); @@ -478,31 +478,15 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -/** - * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly - * shared to prepare for KSM or temporary unmapping - * @page: the exclusive anonymous page to try marking possibly shared - * - * The caller needs to hold the PT lock and has to have the page table entry - * cleared/invalidated. - * - * This is similar to folio_try_dup_anon_rmap_*(), however, not used during - * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via folio_remove_rmap_*(). - * - * Marking the page shared can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. - * - * Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY - * otherwise. - */ -static inline int page_try_share_anon_rmap(struct page *page) +static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_level level) { - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); - VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, level); - /* device private pages cannot get pinned via GUP. */ - if (unlikely(is_device_private_page(page))) { + /* device private folios cannot get pinned via GUP. */ + if (unlikely(folio_is_device_private(folio))) { ClearPageAnonExclusive(page); return 0; } @@ -553,7 +537,7 @@ static inline int page_try_share_anon_rmap(struct page *page) if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); - if (unlikely(page_maybe_dma_pinned(page))) + if (unlikely(folio_maybe_dma_pinned(folio))) return -EBUSY; ClearPageAnonExclusive(page); @@ -566,6 +550,68 @@ static inline int page_try_share_anon_rmap(struct page *page) return 0; } +/** + * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page + * mapped by a PTE possibly shared to prepare + * for KSM or temporary unmapping + * @folio: The folio to share a mapping of + * @page: The mapped exclusive page + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pte(), however, not used during + * fork() to duplicate mappings, but instead to prepare for KSM or temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pte(). + * + * Marking the mapped page shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped page possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pte(struct folio *folio, + struct page *page) +{ + return __folio_try_share_anon_rmap(folio, page, 1, RMAP_LEVEL_PTE); +} + +/** + * folio_try_share_anon_rmap_pmd - try marking an exclusive anonymous page + * range mapped by a PMD possibly shared to + * prepare for temporary unmapping + * @folio: The folio to share the mapping of + * @page: The first page to share the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pmd(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pmd(). + * + * Marking the mapped pages shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped pages possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR, + RMAP_LEVEL_PMD); +#else + WARN_ON_ONCE(true); + return -EBUSY; +#endif +} + /* * Called from mm/vmscan.c to handle paging out */ diff --git a/mm/gup.c b/mm/gup.c index 0a5f0e91bfec5..df83182ec72d5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -177,7 +177,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) /* * Adjust the pincount before re-checking the PTE for changes. * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). + * barrier in folio_try_share_anon_rmap_*(). */ smp_mb__after_atomic(); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index edbca08449357..ed0f66545e9fb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2523,10 +2523,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * In case we cannot clear PageAnonExclusive(), split the PMD * only and let try_to_migrate_one() fail later. * - * See page_try_share_anon_rmap(): invalidate PMD first. + * See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = PageAnonExclusive(page); - if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) + if (freeze && anon_exclusive && + folio_try_share_anon_rmap_pmd(folio, page)) freeze = false; if (!freeze) { rmap_t rmap_flags = RMAP_NONE; @@ -3554,9 +3555,9 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); pmdval = pmdp_invalidate(vma, address, pvmw->pmd); - /* See page_try_share_anon_rmap(): invalidate PMD first. */ + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); - if (anon_exclusive && page_try_share_anon_rmap(page)) { + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } diff --git a/mm/internal.h b/mm/internal.h index a94355e70bd78..29589bc3f046d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1047,7 +1047,7 @@ enum { * * Ordinary GUP: Using the PT lock * * GUP-fast and fork(): mm->write_protect_seq * * GUP-fast and KSM or temporary unmapping (swap, migration): see - * page_try_share_anon_rmap() + * folio_try_share_anon_rmap_*() * * Must be called with the (sub)page that's actually referenced via the * page table entry, which might not necessarily be the head page for a @@ -1090,7 +1090,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, return is_cow_mapping(vma->vm_flags); } - /* Paired with a memory barrier in page_try_share_anon_rmap(). */ + /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_rmb(); diff --git a/mm/ksm.c b/mm/ksm.c index 716e2f87dd795..8c001819cf10f 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1331,8 +1331,9 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; } - /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && page_try_share_anon_rmap(page)) { + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && + folio_try_share_anon_rmap_pte(page_folio(page), page)) { set_pte_at(mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 39b7754480c67..b6c27c76e1a0b 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -202,7 +202,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); - if (page_try_share_anon_rmap(page)) { + if (folio_try_share_anon_rmap_pte(folio, page)) { set_pte_at(mm, addr, ptep, pte); folio_unlock(folio); folio_put(folio); diff --git a/mm/rmap.c b/mm/rmap.c index 3ee254a996221..6209e65985a26 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1866,9 +1866,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); ret = false; @@ -2142,7 +2142,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, pte_t swp_pte; if (anon_exclusive) - BUG_ON(page_try_share_anon_rmap(subpage)); + WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, + subpage)); /* * Store the pfn of the page in a special migration @@ -2213,7 +2214,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, VM_BUG_ON_PAGE(pte_write(pteval) && folio_test_anon(folio) && !anon_exclusive, subpage); - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ if (folio_test_hugetlb(folio)) { if (anon_exclusive && hugetlb_try_share_anon_rmap(folio)) { @@ -2224,7 +2225,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } } else if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); From patchwork Wed Dec 20 22:45:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CE4AC46CD4 for ; Wed, 20 Dec 2023 22:47:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F55E8D0028; Wed, 20 Dec 2023 17:46:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A36B8D0022; Wed, 20 Dec 2023 17:46:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 539448D0028; Wed, 20 Dec 2023 17:46:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3C9E68D0022 for ; Wed, 20 Dec 2023 17:46:54 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1AD08A1A87 for ; Wed, 20 Dec 2023 22:46:54 +0000 (UTC) X-FDA: 81588683148.15.8BE1370 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 68A5C4000A for ; Wed, 20 Dec 2023 22:46:52 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VVhx9Ni1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=97Mkj9KBAth/M3QAsUCGduTq4AF8YjGPbQx442c84RA=; b=IanMod8EMY7zhzm6fgVh/DcYWXMSmpej/QSETmDQnCv9zJ01PInGUTZ/EEzPRwQeR7OTEe SIPcXjPOkFSv8IadcYrONAFrW4n6ZrCXevl/51Wqm6lOlS3vce8JwPckR8EwZ7ox2vFD5z hhO7HEHwrcxVT2uoQmo4ob5hhe/eCzc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VVhx9Ni1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112412; a=rsa-sha256; cv=none; b=GrxKLbsVS5VXLeszehginZ+f2Dt02gXSv9KVJgajuEqiGb29AoGrzlgGJMzOymSvelzuK/ 7h4UwNfkKjvH832a+p5WNMPZxZryGoQFhS3GlnNAYhnn89ee1qlHarv19G15nEJWJxq2vY rZyRlz/fSIs6s5I1usswWNo+yNyqGjQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=97Mkj9KBAth/M3QAsUCGduTq4AF8YjGPbQx442c84RA=; b=VVhx9Ni1fnpb2bw/L/boWveZejNoR+XT7uEZzWBjf0tDYFJrV106uHXFCV6YOJ2wb/6sCm F9yLMuDTiy885K0loR5dFN73MtjppWzsejYy1N3xPbGn0JonPvIwQoRS58WD/aEp6dHfLb kGyiRp400tyO7yEmRTrZKjjZhpC+gno= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-636-w_aS-3J-PiOb71UIJo4vXQ-1; Wed, 20 Dec 2023 17:46:50 -0500 X-MC-Unique: w_aS-3J-PiOb71UIJo4vXQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 93C361019C89; Wed, 20 Dec 2023 22:46:49 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 28EFF40C6EB9; Wed, 20 Dec 2023 22:46:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 39/40] mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED Date: Wed, 20 Dec 2023 23:45:03 +0100 Message-ID: <20231220224504.646757-40-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 68A5C4000A X-Stat-Signature: t8djc8aq63wf4ob6m5p1w799d7whp4xm X-Rspam-User: X-HE-Tag: 1703112412-917971 X-HE-Meta: U2FsdGVkX19mYJd6RU1HtzhLHPmyfCUd3Mg8ukxzxfdfKYdXf2jtI0DxBba4ZK8Kq8o4oWGB/sIbf428A6yNj/ZZvsXb2gT0266s20Nz6uBcWrGC1QuzaKASt4nOYhl+nSTktHVX4Zj/oyo9qfSO2s92nvNbaovtchSpSkZPuMGmH4qUC1T2Why8Zd2TPesbFyRS19rcjhXWvb+o4dsOK717LtElJaGq16PPSEYhDLXFRbmvMVrQntxAdh8J1JR1TtWAbZJxknyQ+TED1DUg96B4pqB3sKjGUSBwr/OcIGcMV5GdF8stMHzMZXrn/1HwNGGjfzASXT9rekqzkRDfrLSD93N45NBEjoS/mD52PzRotTCPuDXX0c/ToHuhFEAPwZA7nJJNNnM1dFZacpuN5l3qQV9M7FkHMPtjMMKHCNi/0g8U8oQRqvHKB4LvXvoGxcDbX4D+tXG2UEVkVMNZlvETAFR2Vb1UNckDV5XZZeML3Ciw0ru1KuP3Z8FHWNUM3WW/zqW6xtPY4+NbRSi3D1lXcx3t4cHD1PGNpQBjzELTqlXokEWNI5QESm9th4m/57769fzwg1JJpeuoAnLac4L36BhkYJAbcLaMpzR1H+oGmYrgVE13/OJmBuc2xDm0qnOhaHCYqZ6BGiYhULG++nN4RORVZgbwFRZT8HYjuDYxurj48/DZOqDp8ddqRDEA2i2CoPDcSJiA9Iz3wLD1u1JGe3W0MXUefkNoUs1H9BvubV2ch+o9QusqDsQSbhsduWVjHsSBEoUN81Ef/6R1R3Bh2DmL7G7p0OdCYgtRb0+ev8pv3ejEel4YTmxNlwS6FUnJIfkcwBn3ZRLThoe5wyvsWAfkxfGBXHj5T0enbQO7X/qiUC59clPxLhI1h51U7//OZiU8Nvmwf1whkkod6PmGgpgA3mKiRiS2LOWbTmlhemC5WkO62AUmrAo3Pk2BcxI188a9zrbynePe2mu YMdA0+pQ FPMOeQbMXxtW05Q7Z+ni8dFPEOFCJORGCo5RK5Rx5hSa7d4ucEYlN6s9GMUzC8/SImy+M8evAbjG5UBc5vngui4vXdQ8z2nU078JMzpFy6tLD0XKCgTBiU2y5OBsaHGNYGCX/zfGFWNbiBl7WFIlc7vPKF7KlfppPeyQY1V9z9A0M8FeiZTCp94dyLNvKpShYne83waQsg0m91CzyuW1MsgdmM5GgTiZVNfIX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We removed all "bool compound" and RMAP_COMPOUND parameters. Let's remove the remaining "compound" terminology by making COMPOUND_MAPPED match the "folio->_entire_mapcount" terminology, renaming it to ENTIRELY_MAPPED. ENTIRELY_MAPPED is only used when the whole folio is mapped using a single page table entry (e.g., a single PMD mapping a PMD-sized THP). For now, we don't support mapping any THP bigger than that, so ENTIRELY_MAPPED only applies to PMD-mapped PMD-sized THP only. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- mm/internal.h | 6 +++--- mm/rmap.c | 18 +++++++++--------- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index cf81272a6b8b6..93c9239b9ebe2 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -117,7 +117,7 @@ pages: - map/unmap of a PMD entry for the whole THP increment/decrement folio->_entire_mapcount and also increment/decrement - folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. - map/unmap of individual pages with PTE entry increment/decrement diff --git a/mm/internal.h b/mm/internal.h index 29589bc3f046d..188807d2aebc5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -54,12 +54,12 @@ void page_writeback_init(void); /* * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, - * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * its nr_pages_mapped would be 0x400000: choose the ENTIRELY_MAPPED bit * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. */ -#define COMPOUND_MAPPED 0x800000 -#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) +#define ENTIRELY_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (ENTIRELY_MAPPED - 1) /* * Flags passed to __show_mem() and show_free_areas() to suppress output in diff --git a/mm/rmap.c b/mm/rmap.c index 6209e65985a26..f5d43edad529a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1172,7 +1172,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); + first = (first < ENTIRELY_MAPPED); } if (first) @@ -1182,15 +1182,15 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, case RMAP_LEVEL_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { *nr_pmdmapped = folio_nr_pages(folio); nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ + /* Raced ahead of a remove of ENTIRELY_MAPPED */ nr = 0; } } @@ -1433,7 +1433,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); SetPageAnonExclusive(&folio->page); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } @@ -1514,7 +1514,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); - last = (last < COMPOUND_MAPPED); + last = (last < ENTIRELY_MAPPED); } if (last) @@ -1524,15 +1524,15 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, case RMAP_LEVEL_PMD: last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED)) { + nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED)) { nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* An add of COMPOUND_MAPPED raced ahead */ + /* An add of ENTIRELY_MAPPED raced ahead */ nr = 0; } } From patchwork Wed Dec 20 22:45:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13500653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38B96C3DA6E for ; Wed, 20 Dec 2023 22:47:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E365E8D002A; Wed, 20 Dec 2023 17:46:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE6188D0022; Wed, 20 Dec 2023 17:46:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C11B98D002A; Wed, 20 Dec 2023 17:46:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id ABFF08D0022 for ; Wed, 20 Dec 2023 17:46:58 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8B77E1C0BEE for ; Wed, 20 Dec 2023 22:46:58 +0000 (UTC) X-FDA: 81588683316.17.FA400A9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id D9F8F8001B for ; Wed, 20 Dec 2023 22:46:56 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HkZltjKZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703112417; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zwHHhlXrWPXO/4jePm24jC5xrVtR1eDI9aBc8Gwcrbg=; b=TAVPvTMe8G6T61EY/ElOjETRwpNa9GdSdKHZ51w0iDLozXxcN+btWZUW111nPORFxM2Tdc hGyZpOonu4ASRgScZMoWGd1oLP+1yk2KJUbxX6ndHaI6++iopdTGEhr1xFDHWBrXmG9io7 qQRBDScgs7Hgb+fqn6PaJgWkS5naGGA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HkZltjKZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703112417; a=rsa-sha256; cv=none; b=kmNtRpUj2X7aHRNDqfEc8unlDpqu86GCcnO/CKhlwmT0lSKQxvAcqCd6jPvollfrGTeHNc G3gRuzRlXRgMjdW5RDEi7hlcuHQtNSrZ9ML0vdBs6V9Hz2npUw0hfPhSpTiaaxp5HhsmuM JBR8XYtpmjmyxIQcxRGLNbM6EkIrjBg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zwHHhlXrWPXO/4jePm24jC5xrVtR1eDI9aBc8Gwcrbg=; b=HkZltjKZC3ul53coi6baNmb8Xc+F8tG2XV18mKdXiKgeiYbbn9OPBeUWdh5ztAimjbnpFB 2GilE/3J5f0GR7BFekMsnA105T6Tj1V3o4vskWNbm00HErq2KJUn8n+qngWdk5GDZkKe0E o5RyMOJJs+PYvY0E2HPFUJ4lXC+XsoM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-176-ZzFw3tvFOO2FjpEXoDI-bg-1; Wed, 20 Dec 2023 17:46:53 -0500 X-MC-Unique: ZzFw3tvFOO2FjpEXoDI-bg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 405B3863B83; Wed, 20 Dec 2023 22:46:52 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id E2D0D40C6EB9; Wed, 20 Dec 2023 22:46:49 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v2 40/40] mm: remove one last reference to page_add_*_rmap() Date: Wed, 20 Dec 2023 23:45:04 +0100 Message-ID: <20231220224504.646757-41-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D9F8F8001B X-Stat-Signature: twnxg811rqa9fg9f3a65xdc1rhtupa8i X-HE-Tag: 1703112416-672763 X-HE-Meta: U2FsdGVkX18MRJSQtl1ikyqak7WIU5huYQ8xMNo3xNw2RoRBJjdRPbiLl5xKrh+F1/Nbfx83xufKJOQYlmQmerhKr27NIGlLghwsRT3mWcZFEzSUXWL5rnLHr8cV+AWHlsix5SrcKwLhg7+BPlUbuNtTvuWQxwGKtsp6eMe0opp5txnhwHcNa4KqYFSf7vHpVTBnwq577wswXnsYLfCPHarsYwN3dOA8BU+I2opHrHpBV0nnf6kyZD4DoU7/E9eUtylUr3ry7AUaw3k9OqA9HMGTGj63aA8ydCTx71KPzesfK4nCMdu42Lh4b4mLLWg2z0QaoS40nQCGNdig4Q/1yfaqqyb0aMUUG1uKjcTN8bPTXulkzWDr2yXB/sUFtw8mDTRmctv8z3cS64XlxiPi9dxyq/tv/cg/4ZQXEB1qrFo5pEd/1crG0Nm9oDZufcCmFyhIiMwfelnLDjyOPyxxOr0mpLGq2uwJ1rx1/TUHojnd/2zWo3DcR9/N3T5NpqANmrDSlt0gPDWiuWtXlD2ogP5ONxXgTRY08uCRWXbKPs5jSyeSEakKaOrhoshHOvuPppeMwX0HjVUGFcGZSCEqvfap6IPN4iVj3is0k6tfJbM0FRodVwFOdAvPxwbCplR+XOgYEAXWGWxU8fRako4FWsbg3GeGKZ5mqbyzTCZ5TiiNHiwiQERNibA5Aj6HdjsHk2JU9trVXpe+l2SR5rFRwvNwBpYKjsEIyUv8t8i4ByAHObA6DAUkR03CE7JnADdNTjT5KQccU3VMVtEeQW5QE4z4+yMGhmjc+8GBQ/LVx9gLML+qAR/DBb70K4zw0XKFpY4BKRYNNy/7C/bPKkh7WxL51xhvSaFgsCCZ6k8CO3/EzGL38O8upjVcxppl2HEWDucKvPMoL+IVWOqWSUVZ9WrOrnIrpA0AirWlEoQyw0qhE6oqcSzKbw07QvlXiJ8o8vxyZAZgV0ZoXPHQVR9 16GCM0+l pInAz8ixrgE2dbbqXmyKXGa1lKe6AG7oirkjKh8yX8Tej4B+rKN13dFLBNXjW+jIP90dNiaARVtyUbsgygZ1rw6NRiUBJcp33/wOhuAW4NBASFQXIlSW+0Bo2GV14d3oHEYaPwCUf4O5P9Qf/j08q0LkGaDFJxaFWm8ORAu12KDK/rwpez4tRScFFgZC/qQd8W3KL7c2klO3XCqugjkGDscWTPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's fixup one remaining comment. Note that the only trace remaining of the old rmap interface is in an example in Documentation/trace/ftrace.rst, that we'll just leave alone. Signed-off-by: David Hildenbrand --- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index 188807d2aebc5..ac40c3d003368 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -650,7 +650,7 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma) * should be called with vma's mmap_lock held for read or write, * under page table lock for the pte/pmd being added or removed. * - * mlock is usually called at the end of page_add_*_rmap(), munlock at + * mlock is usually called at the end of folio_add_*_rmap_*(), munlock at * the end of folio_remove_rmap_*(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). */