From patchwork Tue Feb 11 11:13:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13969521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24AA3C021A1 for ; Tue, 11 Feb 2025 11:14:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F9D26B008C; Tue, 11 Feb 2025 06:14:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47CC56B0092; Tue, 11 Feb 2025 06:14:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 259D76B0093; Tue, 11 Feb 2025 06:14:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0B1146B008C for ; Tue, 11 Feb 2025 06:14:51 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 764FA81516 for ; Tue, 11 Feb 2025 11:14:37 +0000 (UTC) X-FDA: 83107405794.10.115702B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id ED890100009 for ; Tue, 11 Feb 2025 11:14:34 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739272475; a=rsa-sha256; cv=none; b=QDXRW7E77yKa3SrfUI/Ey5mEOgOC1UrH/XOvha6WKgIvLE2T865W8eXxNV1MyyoxlM5ZNI AUF33uVNqkl0jHwPjs0ZMJNHqNJnHijsEUe8LtFGO9J3juu14uz5aQrAw0vpkGWSEBoiPU TeCM0Ah/3z1ScvoDbZZABBKYQKvS4Fk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739272475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nSf+OiB3/bLwawjGnCMjW6lhWp2HI9Dj7B3czOtKfxw=; b=TEDby2Ih2oAMPAg7rSnsVZRp3HQVhLVeLvXCWa76R9uMgD7OYY6ls/+2feqwXo5odfdHgj VjgtWOyF9iy4DqUxjiUFPpyP3NlSApbyaFtyaJP1D99hhBZhKIc2xx3YS9xumy9OuETULm lsO2thB7B/Dofdq3SHkEE9VwTCJGK2w= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8F15213D5; Tue, 11 Feb 2025 03:14:55 -0800 (PST) Received: from K4MQJ0H1H2.emea.arm.com (K4MQJ0H1H2.blr.arm.com [10.162.40.80]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E91073F5A1; Tue, 11 Feb 2025 03:14:24 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: npache@redhat.com, ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [PATCH v2 05/17] khugepaged: Generalize __collapse_huge_page_copy() Date: Tue, 11 Feb 2025 16:43:14 +0530 Message-Id: <20250211111326.14295-6-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250211111326.14295-1-dev.jain@arm.com> References: <20250211111326.14295-1-dev.jain@arm.com> MIME-Version: 1.0 X-Stat-Signature: 31gnzsrqwnsg6hfyo336ui61sgxrcksj X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: ED890100009 X-Rspam-User: X-HE-Tag: 1739272474-388043 X-HE-Meta: U2FsdGVkX1/nXigN3glqgTCO9bs/YPSJWbsioXO+rn0ewKUiwV1Q46N1aWdDNxECqLS60LiR7WaNLrBNIXSq7YaJnf8yF+x09A16a3e6JB0mToY7YxfKHY5yKCj44mFNXK45PYspKcrn/x6vMHPoN52Fswd25dab/Yvd0zFs54/Swx09e8jKlnIdOq3s8eOrc6E9hTLryi+W9UxpOkC1JWRvL7lcOLTFeJWrLmjVBRJCUyXsE4Kj6kuBIifyx+YeudXo98u6QepwPf7385bebSCekEFsNcoUcg0tY6l3tXJAuQGj2/cTKJyVXWWNzqvT3k4sldOc9iPhoKebhuCOaCfFdTsnnW90Rw5g0olxIbO1SDu9NAoyMSiyv5m5xufzPWoA1ciKH0FwzJYweyNTZHR4Ghdc7EA9eFIyWLDT5Nb3ivyOhzxsKof/O6OXtL3r2iYESgvVZYWDwrjJrDLp06UH2LYFOzP0cVg0B3uvhUMhLZ5eqXwpzC9B2GVubiAOdbS1Edna6iXD0KBn6pdaw0ZD3NZBpc8HrYvcZTqgSWnDugmoMP06UBROMQP6FHMA8rOrRrTYLUXrnjeGyehsb506haVYZaulUoqd+EXQkvt7qkDNeSBVRH3zmDo9fD8jkAmfL7MAY/kYJw2eKBGRpfr8nS942okrsnefgJG1LZFoYVCLeEgeASB+gdQccBgJwkxM9Z1fJQKav4sKOApyoL93Ml+/nhIOkAQuSL3mC8b9AFL3ybuTYjCwcMG4g7klVwlwmCjtE6AX/7sOb8SNtF4lqV/D19lxc/6sKtvzLmKBmyB8rmDA4mvOtQZa+DdlFhrdBgcAXUBtqRc1lS+hRVyD8MyccedXG9oGfoLjuWa70Uy8qCMIZlzjx9ZVjuTj0M5IS5N84gKTbZIsyQX5KD5mvEOigKTkOFin2aDMxhI3+c26bdP3nps3D0/0lTeJJrI+y6RIAwcltWMPRbJ cG4Qtbs1 2m+Lt9vPLEdA5go33iUE2t6xAoPLiI0pVbZ30WSzeHOEo6zckBjAstBj8e9panQ/BY1LwvaVL2Cesr6k3LRkBrIpILrsGyYz7AUqyv3Ym4swntCwoKQrhJN0NKDunuGsT4u8voTjQbDTEXHYD0L+tj0HZDK0FbQuQjZfOepcORkcX9Lioan0vINwElfLv+NWWJOHQqxedzP6y2irw5+FbldFB4cgL4/KkusgAelJTST9//YXyapx7Md9H7CzBum8LMxqrjLhlkHS4EV34Pl5y9FrVKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Generalize folio copying, PTE clearing and the failure path. Signed-off-by: Dev Jain --- mm/khugepaged.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 0ea99df115cb..99eb1f72a508 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -712,13 +712,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + int order) { struct folio *src, *tmp; pte_t *_pte; pte_t pteval; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_pte = pte; _pte < pte + (1UL << order); _pte++, address += PAGE_SIZE) { pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { @@ -765,7 +766,8 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + int order) { spinlock_t *pmd_ptl; @@ -782,7 +784,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, * Release both raw and compound pages isolated * in __collapse_huge_page_isolate. */ - release_pte_pages(pte, pte + HPAGE_PMD_NR, compound_pagelist); + release_pte_pages(pte, pte + (1UL << order), compound_pagelist); } /* @@ -803,7 +805,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, int order) { unsigned int i; int result = SCAN_SUCCEED; @@ -811,7 +813,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, /* * Copying pages' contents is subject to memory poison at any iteration. */ - for (i = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0; i < (1 << order); i++) { pte_t pteval = ptep_get(pte + i); struct page *page = folio_page(folio, i); unsigned long src_addr = address + i * PAGE_SIZE; @@ -830,10 +832,10 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, if (likely(result == SCAN_SUCCEED)) __collapse_huge_page_copy_succeeded(pte, vma, address, ptl, - compound_pagelist); + compound_pagelist, order); else __collapse_huge_page_copy_failed(pte, pmd, orig_pmd, vma, - compound_pagelist); + compound_pagelist, order); return result; } @@ -1232,7 +1234,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, vma, address, pte_ptl, - &compound_pagelist); + &compound_pagelist, HPAGE_PMD_ORDER); pte_unmap(pte); if (unlikely(result != SCAN_SUCCEED)) goto out_up_write;