From patchwork Mon Oct 16 20:05:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13424036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C3B4CDB482 for ; Mon, 16 Oct 2023 20:07:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 750B58D00BF; Mon, 16 Oct 2023 16:07:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7003B8D00BA; Mon, 16 Oct 2023 16:07:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 553B98D00BF; Mon, 16 Oct 2023 16:07:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 32CC08D00BA for ; Mon, 16 Oct 2023 16:07:00 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EE392B5C05 for ; Mon, 16 Oct 2023 20:06:59 +0000 (UTC) X-FDA: 81352408158.10.E3000FC Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf17.hostedemail.com (Postfix) with ESMTP id C9CA540016 for ; Mon, 16 Oct 2023 20:06:57 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RGF0AruC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697486817; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dowlsYISY/BEh4ltWtb6fy0PpeiHEk3DajdxWZY/uS8=; b=p2XuNhyrLvGAmdagKgH6R3QdQBHgdkbCjW912ZjtwvQZYVTEFwZnKOYGXWTiU2SYXM/846 zK8iPRvrmHaG9J2ykyEORyjFLAEPP9Mny8G0g/hnec7Lsb9dte8Z++M/+spfZSSbUMgLPH TzOJl5lVSNwWj82lt2UOgdW1sZ9lqMQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RGF0AruC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697486817; a=rsa-sha256; cv=none; b=iUaATz55kk7MCvjY0KhLMnbFKL1U0E1crdQOeHZyflQ133OSUZXsfk7xf1E9fyoTnyU4/P eCwhntyqg/HuUqLsK0X12Lu9PZiTO6659AF08McSI911VRacvFpSHni252VIwZKO/QlHcD J1Yx2/qv52WE9p/L0rDskrIIlRKQQ4s= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6b6f4c118b7so2242064b3a.0 for ; Mon, 16 Oct 2023 13:06:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486816; x=1698091616; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dowlsYISY/BEh4ltWtb6fy0PpeiHEk3DajdxWZY/uS8=; b=RGF0AruCsUwxk+vbZ+noUHndq5WuiHgWiP+tn2bVDgcMVu9lz3rAjNWc0H1PxQtiB6 MkCqdCSqHoqURqNEEy6XgQw4yzA4E34EW+cO2Tne9KB6zaUg9YCsE/LpDnJ9M1hdxZrF 3cQ89H3CT5W8r1+MmuteSPLOJ9fHD7NXEzzLdOsnMo/f3CrHPYoZXZK8OLKVZ+c0B5ZG xpd7f880J5ieRvh1S+bL+yWyoE84InTytPX5JO5gAX1HuunXl6U56ZvJ0Rx33BTwBKzM 0RqAp3pAK9kwRtaYVKORw2Tjrul9UfLpjAf3l8Dm2lmwFl4wIP/KsM7FN7+EtgogdAyc mZNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486816; x=1698091616; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dowlsYISY/BEh4ltWtb6fy0PpeiHEk3DajdxWZY/uS8=; b=kXX219u2w5t4QGIDILAJp8umyP9IBKMHCZmoNLO72s7cBRzvmgr2SvbSGcPAYgGEaj um1aP/macWxztWKdtEgqutw+NhB5Px1j2bDFElKkwAF0Xgq6ADj4ObKOfPVnLQzL4olY RALbAYo4qJVeWTBcquhg2amRHXWlk5ED+dRwI87F2dfOujwH1qJRkqYQCtbZH+hjnMhx aArmlnNQ7lfLzBPnjBCSIXgtPEQPpXoC4sYr+cVtqVMLy5Fp+lEKFRNEaWSsoLgBwODt xhlCA4v7dCvU63c1O4vb9WmhkXYulELhucieEjYVdWn580OKxGWiI5EKwVyZLOypLF1g RyHQ== X-Gm-Message-State: AOJu0YwhqOb/qaBieG+4FJ8f3j/0DfftEtM1Y5xdIOgaR4Sr9MzmUYbg Jmg2ohWOOd17nRSGIB4UhtcE6AeT0Xs= X-Google-Smtp-Source: AGHT+IHNAbR5P9OXapRD4T0zOXCwzPhLQ3rt7wyj9Cgg9D9ndiZau71+lTiikiZfhKBGtwidV+lYHQ== X-Received: by 2002:a05:6300:8003:b0:12c:b10d:693d with SMTP id an3-20020a056300800300b0012cb10d693dmr97230pzc.6.1697486816530; Mon, 16 Oct 2023 13:06:56 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:56 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" , Matthew Wilcox Subject: [PATCH 1/5] mm/khugepaged: Convert __collapse_huge_page_isolate() to use folios Date: Mon, 16 Oct 2023 13:05:06 -0700 Message-Id: <20231016200510.7387-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C9CA540016 X-Stat-Signature: 3wkyc6wb5axryszfuibjfbqgf7zjpdbp X-Rspam-User: X-HE-Tag: 1697486817-502444 X-HE-Meta: U2FsdGVkX1/H89/dlgYazQUjtCTDJ8tPiZFZtOWj0fKUYQULhx6rEulvYKNVIfZYLPxtv7s7DhWjl/cLf88S4tqB5eTd2QdW1cH2l1pS7GoJkA/HCpOdLTPqMcnwB3dJrqMPWDo0U/V0IMm7qoDQhvmliQbm6k9uSHQ/A+ucxtlhOrdbeDuue8Li02Cp1ZB3sc3HL/2QNmL8f7nvc5/gdvqmEOMUZKo0O8u4kd9K6xdHiv/pQp9Rrdv9uYdmDPzvNVEOusoH+o84ZAD9jB9hXn+jijoyUUT9Kg+QJ3s0CtGAEznMzLlXzfHzH07G0d2MLiT/dpPMZwH3P4vsuVIagBksmRjGl57AQ7Hbbf7DtGT4kfgmifGjidVhGhPopTqjLy/odG9vRZVQcvDZPp+zydFmVeReixRx5zFbMC0NDxD6BC39m9gAL/bodDh72xooSb1/+aCT+h94WaFEFnCvxcOD1/zKIn5t3i9/KkBTyMIJ2m8ufLAtNE3OTmK38uZCbvWHhyrFqQvko7eb9qECfphaO12djawcehX5DDPGEjRxF7RteaJuIDt1atV+UqRhkA8FWePp5WLo5ujDqUyz3t+UFRXzTv4oihhOY7b8ZL70AIgpGR/GZ4juhVostXhCLkzknaZt52VS5exUNVIZbehro6YLxHEgJn8IDcHgPCBMihTiP6Gx+2NuGgae01NezoaoZ6iL3UkXo/DJa6jrAwHjTdHr6oNlw++BmQzAAHEO3jwSzMWd/FDPLyF7/O7zYfoFdKzrf5Jo6vyFfEl4lNfdQqXE2ICaq/k4Yb+ojlrTWzEvgYB+hogI1ds5k5CHbLNTFFFUIQdoSvSZn+Sq/DAYURviyInfuceLGmd9Sf4OkCUt1DDIkvnUeGQnHMimhk9fwzzjauAPaGcPmUEMVkiLo5HrlB4sP5fLJX7YBbzJsK4GpxnLfpqAreOGPmpDbitRzPZnrkykhuBVy5d EZMaGdA+ /s8/ZOtoCx7g8XdvIRE6TEllVYwqSECzXCp9M5PmQixXQkBcGAu7yeP/lKwORNxgZ36NlGYtYkARMivvTJkoPefWqLT5R8fnyWOPXnqW1qnASb+rAolmRCrfQuXQQA/3udje4NRByT2RBC/uq+GuhAkR58UDX/wC36k9xu9DAHa6TApZJA7BPwHzLKp3egwWCjrfPpQuwu6jHHl15Qovn2z1ZCZGQTnkd29RJaoiM6hsLMU6IMWm25hGoXbmvYNZC7n7VxWBopybNDHK/QTFS7UfXr1aVIpxJVEKBfdCVgzpVqwT83x+Vsc8JLum6VWAqrVKRzhk7v4yC3Cn3lp7Rpr5Vc5ocJKIQl8jTdkM1MGViT/0o91+RAWe4qPuVxopf6IPyQr1XXX1NcYkA/oyFNe6OZplkDDdQkw7Q6/WOV7VCHNIWoT3WEZUFIF+N+bt/VGav/S/VPbgQCd5r8icqjjCcfU6cuTxIjYtY5Jx1xvGlrzvAs0DbrJhXYfLyUl0nT56fPHSNT7aviKd0dsUF4k2nBH0y8OBBu06VhXEyckZJxwmlgHLlLidsXsZBI+OqGEOXRcrpPOrzeHggXQo74vcJkNipajNdUI8zEZq+A90k3A0QA2+eLMW7W2Bzdqokgdcg7pedFLTu4x0SlQy0ssqI5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replaces 11 calls to compound_head() with 1, and removes 1375 bytes of kernel text. Previously, to determine if any pte was shared, the page mapcount corresponding exactly to the pte was checked. This gave us a precise number of shared ptes. Using folio_estimated_sharers() instead uses the mapcount of the head page, giving us an estimate for tail page ptes. This means if a tail page's mapcount is greater than its head page's mapcount, folio_estimated_sharers() would be underestimating the number of shared ptes, and vice versa. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 51 ++++++++++++++++++++++++------------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88433cc25d8a..7a552fe16c92 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -541,7 +541,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, struct collapse_control *cc, struct list_head *compound_pagelist) { - struct page *page = NULL; + struct folio *folio = NULL; pte_t *_pte; int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; bool writable = false; @@ -570,15 +570,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_PTE_UFFD_WP; goto out; } - page = vm_normal_page(vma, address, pteval); - if (unlikely(!page) || unlikely(is_zone_device_page(page))) { + folio = vm_normal_folio(vma, address, pteval); + if (unlikely(!folio) || unlikely(folio_is_zone_device(folio))) { result = SCAN_PAGE_NULL; goto out; } - VM_BUG_ON_PAGE(!PageAnon(page), page); + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); - if (page_mapcount(page) > 1) { + if (folio_estimated_sharers(folio) > 1) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { @@ -588,16 +588,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } } - if (PageCompound(page)) { - struct page *p; - page = compound_head(page); + if (folio_test_large(folio)) { + struct folio *f; /* * Check if we have dealt with the compound page * already */ - list_for_each_entry(p, compound_pagelist, lru) { - if (page == p) + list_for_each_entry(f, compound_pagelist, lru) { + if (folio == f) goto next; } } @@ -608,7 +607,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * is needed to serialize against split_huge_page * when invoked from the VM. */ - if (!trylock_page(page)) { + if (!folio_trylock(folio)) { result = SCAN_PAGE_LOCK; goto out; } @@ -624,8 +623,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(page)) { - unlock_page(page); + if (!is_refcount_suitable(&folio->page)) { + folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; } @@ -634,27 +633,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * Isolate the page to avoid collapsing an hugepage * currently in use by the VM. */ - if (!isolate_lru_page(page)) { - unlock_page(page); + if (!folio_isolate_lru(folio)) { + folio_unlock(folio); result = SCAN_DEL_PAGE_LRU; goto out; } - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - compound_nr(page)); - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageLRU(page), page); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - if (PageCompound(page)) - list_add_tail(&page->lru, compound_pagelist); + if (folio_test_large(folio)) + list_add_tail(&folio->lru, compound_pagelist); next: /* * If collapse was initiated by khugepaged, check that there is * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; @@ -668,13 +667,13 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } out: release_pte_pages(pte, _pte, compound_pagelist); - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } From patchwork Mon Oct 16 20:05:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13424037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D929CDB465 for ; Mon, 16 Oct 2023 20:07:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15CC58D00C0; Mon, 16 Oct 2023 16:07:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E6908D00BA; Mon, 16 Oct 2023 16:07:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E06148D00C0; Mon, 16 Oct 2023 16:07:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CDF538D00BA for ; Mon, 16 Oct 2023 16:07:00 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A316B1A0C02 for ; Mon, 16 Oct 2023 20:07:00 +0000 (UTC) X-FDA: 81352408200.29.3E70AB0 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf09.hostedemail.com (Postfix) with ESMTP id C9121140002 for ; Mon, 16 Oct 2023 20:06:58 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Kxjpjboa; spf=pass (imf09.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697486818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mMefHNBDnWRPz8cjT6IRO34+spfB9/Lnaff9PAvwfj8=; b=WBYc1LQtpsI0+RrcqwkdxqgtA6BxgbwGjh3VmYLxKFkCW6Gegcvbchf5KD4tVtSOgPapla 5t2UXn5ebxZgi8teug/UsJth5QyY9usnz0z3akGCbK4To6c2p+fSDhtRhZaktOscrZHy90 UBg2UC1YDncIzaNZaZqFhqn7CeR90eU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697486818; a=rsa-sha256; cv=none; b=jf+Hzl/7NXDb5ICygHBPyPN4gIIS7yai+nlB3rii/VH/38phKXgSz+sK+tYZx9VfGQ+oV+ Mp6KixYqqDIW52XyGOgNpzRhC7WKUXrxxF7VU0PEOk3249rDCCJSyK6Fn3EK7F8arVitDX zsVCBEB8fUbCUejJCi4pn7VhgyS+Kl4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Kxjpjboa; spf=pass (imf09.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1c9b1e3a809so31331015ad.2 for ; Mon, 16 Oct 2023 13:06:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486817; x=1698091617; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mMefHNBDnWRPz8cjT6IRO34+spfB9/Lnaff9PAvwfj8=; b=KxjpjboaVkiBJCUdGahmP7thGCdUoYKa2/lA0e+1dz4pn9EAwMAyuAm5w4kNTxmj4s 2T4fPqQ/1cA5frW5eITqvxiPpKDkMcPgIw9z3s41IZu9/88vMTIQihZhuausgw89t1Sv 9/H9TBZAJ0uMO5tQ06Sc2l/tFSvcciU1DhoPNYPBJhe68jPPkyqWrYJ+FMvdiDdR8fZI SZst/odM39hoL6XsnalqrIQjhuqzZ2FvdKbRFrmGQdsaVJoxC+zJH3DaDU8ozND/v17N D65X4xwVTeMEriEYuhxk7OCpkyZehLR4L/0JGdbWkAwu+kPZBvmJSgIAk2JjY+tcECBh puLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486817; x=1698091617; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mMefHNBDnWRPz8cjT6IRO34+spfB9/Lnaff9PAvwfj8=; b=bNA6Htbg8ECaCFJwnNvrOgPrRpJ6NUdyR4qsPuktZHrXg8woVNPddCW/Qw5m+jTHGI j0Wy2ELblmGiHqKTcuN5QpvL5wR43bUTJrcPbNfmyyOkVQglIFkZZFBJhRlqi4xcol53 WhxLGuYpoJRVfmYoG4YbVu4HzTNjiv2o9Y23g0aw2VKLxROo8NPbOCA3gGqUGpowRn13 fpu4Srwv5hjKfHSWqnG34nVNLO84X8dnBnSfmk5RDqog7mUJBzKZbWDPkkhjUULpxFcV g854OlL0xkevfKIRmJpWUpq6aXImOMRrY+/KYeqREgzAy37hwJMHtwLEthlbeIMfkSdv jn/Q== X-Gm-Message-State: AOJu0YxW8H5w6YCFIYVzvS0vwsDW8Cr6cvY/lQkx8UqjxiY6JaNAcUNf RTPYX+wTSJ/niEGh1xdtnmXnJBNV/M4= X-Google-Smtp-Source: AGHT+IHA265DKTJz97l5pSxxF6Sh4mCEGazOefOWWexfxpvvfFDKrwesNxGieo7cqzTIX0SPaOQ/3w== X-Received: by 2002:a17:90a:1959:b0:27d:10ab:2325 with SMTP id 25-20020a17090a195900b0027d10ab2325mr161116pjh.27.1697486817436; Mon, 16 Oct 2023 13:06:57 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:57 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 2/5] mm/khugepaged: Convert hpage_collapse_scan_pmd() to use folios Date: Mon, 16 Oct 2023 13:05:07 -0700 Message-Id: <20231016200510.7387-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 9wwcc4phiu6rfhzys5ki7h1zm4j4jopr X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C9121140002 X-Rspam-User: X-HE-Tag: 1697486818-72078 X-HE-Meta: U2FsdGVkX19k0vv/CqfrSC6lbS87Q36a/qXkYI8+A0S5E1G5KQwv9MByaVieGV/LIk+OMSWQj8qe3llzEB32MIY384NsYJKbiGkGPNWbaEc6gkgyOxCtGOP2+Iz5beSfLh3beVLq10RLd9b4CZ0cqcM/1SSKDruq0m2JtfzkX/S84oR+hlwQsnoXwp2YvUjU3hI2rcnMzJS/WAOaM/qOBtF6Ks/XozhejbxL0REoSUzzqcDLV463QDI02qN1QnsauRnwtZReIdLpJS+L10dOWUSSeUmItRgsJ4WZEizRsY3ENOgDzzvdBLccNbYse4+7cl1Hpv1Q6Ku4tNxeT4KBzBCEiDGERLrHd89LNUehrpc3cnsh2xSt7/H5NmOLzL4K35eIF8HiAPnoDpRnKAmcnV2GBxeC5uEn/MJ7WJk+Nuxwa7Aa78/3ulabh2UMeX/qqiZQF04Z0GPISCuimU9xd1wDim5nceao/ikWzjVwEvwkbAN8ERrMUHsUjdyfqw85OdHCHh92Q5RpB508g/Ri9gjurtX8ew2F1CVVkev+AgjsBjm/UIfiUPnOJti/UiIXiESyj7eyMwcOuBLd/xNPck/aZcppHbnC9WWRs6IVGIZJqFZ1/lkOqFvtgJD/W1mlz+sU38R1uPly9XfjF0fufxwjR+XMGHfkN2l2HFnvkvE5R6uvCjAMJiN7wjrpWXX4BuKBRN9eklBKa7ugM3jw08uuxlXbJ87WGC+w+P60EPDcF0MMOdvjQsKZ8wIxkw5bZZwwKLmuAHhz8UnopfinyLuUc8wbPqd+1LgIPY9abttS/3s2u7pZ+o/a0MdnQcQRC4QPueRZ9imJH5+7+v8gbVvjg3SvJrfqbYuHQvWa06g4y3TPaikkyxuPnvjHHAXCkV+bltm16tbPntR2qJUGKXDOWGAXhv3nPbfyVarS1vkBlEuLRy8matUeuZphyPVzHmZTBWUyq97bbJdP+6R BENcc3J7 5QS07zza9cp5xbCqmpC4XResLpX7HsW7lgz4mDOHPJZ1oNu8FYHshH6Koph1dXzbM/SyIg8AdQKoiZ1JqUd+7gzHNZuXYreb+SMrRoiNGMG9a0AHiJ9HJJPdyFcUx+pZMvdNwyL0m9hX30Iktxdnrp82I9QhOtjP3HFOB6KIbAOFIOUPLEfJ2n6Vmsm6Z8h6xbBYo2fZ7e05IO0ZkG9o0navl7r6Xd4d8kti2fbN/3RyXI+vFSanv50HXNa7hOjZUoQNs4ECOTg7GAcuoMhOcYY28vwbIvwOoaJ5rNiQaf23iL77Mo7f8qDutnajG+A6FsCbYnK+Kgc6Nfvqm8ezSIJtaSnvAGQ6yT27Mjq0ZyiwvxTa5h5kWrK5JwwCH4JiLVQ1d85ABkLjzatiq09lBxC4KmOGTxCs8pWphmZBIk/Xul0eRhx6iuExiHn0KBwbXp1krB6L+slNVt6/qrs3ca3t8I1g7mQOtNDfzL+63ZaXOpWmqeadK+NHZrUK2pFuaE6xAxGQt8pIrCRdRr29e2GKqaQpRK/PQsH8TWOEtA7AujOuPr7jCSp+EHIJdONH1GYPdqK49nUwWJiC53YVatqu+Og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replaces 5 calls to compound_head(), and removes 1466 bytes of kernel text. Previously, to determine if any pte was shared, the page mapcount corresponding exactly to the pte was checked. This gave us a precise number of shared ptes. Using folio_estimated_sharers() instead uses the mapcount of the head page, giving us an estimate for tail page ptes. This means if a tail page's mapcount is greater than its head page's mapcount, folio_estimated_sharers() would be underestimating the number of shared ptes, and vice versa. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7a552fe16c92..67aac53b31c8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1245,7 +1245,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, pte_t *pte, *_pte; int result = SCAN_FAIL, referenced = 0; int none_or_zero = 0, shared = 0; - struct page *page = NULL; + struct folio *folio = NULL; unsigned long _address; spinlock_t *ptl; int node = NUMA_NO_NODE, unmapped = 0; @@ -1316,13 +1316,13 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, if (pte_write(pteval)) writable = true; - page = vm_normal_page(vma, _address, pteval); - if (unlikely(!page) || unlikely(is_zone_device_page(page))) { + folio = vm_normal_folio(vma, _address, pteval); + if (unlikely(!folio) || unlikely(folio_is_zone_device(folio))) { result = SCAN_PAGE_NULL; goto out_unmap; } - if (page_mapcount(page) > 1) { + if (folio_estimated_sharers(folio) > 1) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { @@ -1332,29 +1332,27 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, } } - page = compound_head(page); - /* * Record which node the original page is from and save this * information to cc->node_load[]. * Khugepaged will allocate hugepage from the node has the max * hit record. */ - node = page_to_nid(page); + node = folio_nid(folio); if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; goto out_unmap; } cc->node_load[node]++; - if (!PageLRU(page)) { + if (!folio_test_lru(folio)) { result = SCAN_PAGE_LRU; goto out_unmap; } - if (PageLocked(page)) { + if (folio_test_locked(folio)) { result = SCAN_PAGE_LOCK; goto out_unmap; } - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { result = SCAN_PAGE_ANON; goto out_unmap; } @@ -1369,7 +1367,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(page)) { + if (!is_refcount_suitable(&folio->page)) { result = SCAN_PAGE_COUNT; goto out_unmap; } @@ -1379,8 +1377,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; } @@ -1402,7 +1400,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, *mmap_locked = false; } out: - trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced, + trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, none_or_zero, result, unmapped); return result; } From patchwork Mon Oct 16 20:05:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13424038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52476CDB474 for ; Mon, 16 Oct 2023 20:07:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E61218D00C1; Mon, 16 Oct 2023 16:07:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC3128D00BA; Mon, 16 Oct 2023 16:07:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0A118D00C1; Mon, 16 Oct 2023 16:07:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AD5EE8D00BA for ; Mon, 16 Oct 2023 16:07:01 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8ADD0120ACD for ; Mon, 16 Oct 2023 20:07:01 +0000 (UTC) X-FDA: 81352408242.22.A9A3EA0 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf02.hostedemail.com (Postfix) with ESMTP id A6F1A80003 for ; Mon, 16 Oct 2023 20:06:59 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="Q/XXFvhL"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697486819; a=rsa-sha256; cv=none; b=6HSIeJqC7N4QDPwswlRm61727JrtQRKYWwUmgz7pNEXL2dmHXg/dhZ79ebafsK/dpPNTF+ iZoPiRQOuvlhANAtwGXPWrBjapAFK3fqHjqx50xyj2X43XXTBZMXB94TgechZrHgpFvzWx NiWViKd72z71ojTytLcKHDBP+jVkEKM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="Q/XXFvhL"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697486819; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vjFYTTptetyjp2cWLw2h5u7A70TW4Bkl9MqEU9zvv1g=; b=yleNn+3aQ4K9JxIsz4WhfehOYIgaI4aW9P08U2aQdJKFt4PCtymk6sUha7/wJnu5AlrLPg A7Ws5Bj+wpkfMVRkObdb30X7TYDEil5EnKBo14w4qtnlrKPxDqtUjuLZiHuHzhb0hIwi2J C23kf2JWzxDDm8h198zRZb7tL4mifJU= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-27d18475ed4so3662613a91.0 for ; Mon, 16 Oct 2023 13:06:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486818; x=1698091618; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vjFYTTptetyjp2cWLw2h5u7A70TW4Bkl9MqEU9zvv1g=; b=Q/XXFvhL4uKp9PaNHioE6gmShZLAI9CymahXsE97BGn4JlWCyYPytvTOaU+7K/TrXE WyzHdbOOWvfluqxQhCZZmp6UpAw7oWMlDWqrqlfm/u/Y8J7k8oX6ZxKaqfF0A0VHKa4J Rwj0sW+miHl/s0q3rR6/vggfLpoPCmQhQPsD3hR0cmuwCibNH5lzl/1jF9K1/J5IJh2V Os7Ms/WSRB6FgZTitZmOKbNAQuohazFZiLNGuFGffxHV0weFnTza5iQ8AktQHvdyg05Q r0PDKSuY+pDLuMD0Ey5UkoZ5OZYoc1DwmkwJkiegxc0zYOmcY2B/RQx0rXv9yppzZJds LSSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486818; x=1698091618; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vjFYTTptetyjp2cWLw2h5u7A70TW4Bkl9MqEU9zvv1g=; b=vOFZX1eufDoqlzoV/iuy3G4P+hbIX6AtTkdzaPbCQadlMxIWFE7l1xMEFZhLt+tQnX zxLju/u5Fks1+C4Kl+k45j2wAE5dm2EJezjawV1qImibElwdwOrer0qAu+hGb5ht4EbT GGYM8OUkik3fYfmIfBhdybao0Pv+ypvoo/HpZMNwTWbGbuBPAubMz2XB+b19BtxYNZ94 AJeyzdx063L0atUvdZrviL+7JNSiIK+56LH13O5Qm1aXbCy7M816WJhLLQMfd91ndfSN 6NwbC1T8YgZkmZMCRkcTSa+hqn1pJWOdW7kyKBThbuiO7IF7eSxfMIbNRsssVIuKsGaI Tu6g== X-Gm-Message-State: AOJu0YzhH9NQA6U1fP868ueah1nunJh+asj9IPEY9jKKiNpH4KabC0A2 ERHPM1xfVDH4Joe4rsi5Ho6B3B8d55I= X-Google-Smtp-Source: AGHT+IGtbGzgcQz/00iryQWZ82rvlGPg9aar7vEHyd9Xs5MULzqeh3bW4EbQYfV2V5h2DPzj6QHTog== X-Received: by 2002:a17:90a:eb09:b0:27c:fc09:401d with SMTP id j9-20020a17090aeb0900b0027cfc09401dmr179801pjz.38.1697486818298; Mon, 16 Oct 2023 13:06:58 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:58 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 3/5] mm/khugepaged: Convert is_refcount_suitable() to use folios Date: Mon, 16 Oct 2023 13:05:08 -0700 Message-Id: <20231016200510.7387-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A6F1A80003 X-Stat-Signature: 3sdg44th8jts48qqku59gex4px3npq7a X-HE-Tag: 1697486819-331524 X-HE-Meta: U2FsdGVkX1/FnsnRN2doeK1KSZNtcYr8K1Fw0YaL0F3EkhQ4nS/I1UjKJA+IPjdXpG9gyf4PXbgsOAjbWl6FgqOoHSU9jBTCkfVtDEGqz45dqh8QGySiI7DA3/IzBpmXC4HKIEHKyVe4cmFtTqE98tVdQS1R8RIp6X6X7Qel9UQCmkTG9cQfwI2FqZiLwpeG+LItEXq9UPMc6hl6ORWZ7UA7qcS3fcn7slCIEOExb/UMjxS67vUFA6BoGhFENYvSMSxYmXGW4GdyYISI8lY/prBWZkRmcBMb9++rPh6f5MQLx4EuX9DWrdqJZIcKFvvAmKgGeYqyyqL3ldIDEj4p91TyYYLadmfgfSp5fn3CCvFjv70D5OVDjxCH300EmHG8ibWGHCwV9gALz29wHJxcJa16sC7wa93G0r2QfHUVByOgsyu/klSq4c9ShQ7XqLW/db4cORGAbsOAX+njprSmxbl1PYqedcgnYEA5ixGQ51pL0u9664PiExkZQhMH6qQFzFQ8+54dR0t8oZvTTHzkWeQxhGojrYuBIvF2NkwP1x5Zo5rIBXZBZiWCXeoahKGca7ecHvc46sP27LCHeLzcKYIrJIsaN2rOCveK6/rp/ydU9fLghU8uprfBVw+K0xyyh8lbOSl1F1k4IuBCOZh5XKeWe+3m+8nXmw0IWLQRZ1pFXs7SOP4tB9EV2UxLK9jO1KWNefUrVg+x1vq8zIeVP27cwttv2MrSMBi5r3ZoJuJi0MK23M+8lROGV9OiCaODhBloAIHWu/q9CyudXjqglvyGPlBfHHLIltKbmmEpRZD4xf7y29lXjZhB14w00cjS56td4PKtY83zNeqnXJxdbU2mw9vergOhuilqiEYKubT/EXcLwIneGUaJMoxJzpKM6BTcXN+kbGnQMSDk02CHW7eenCylqbRrGKylzpJfFe2rSl9re8qmPVRDvvHJHqIOuRzVvpbz+Mdlh5TrBxk XYJWWDO0 ekJh/cicj45BmylIF13mZ3r1WEfNkuqDsJl8TpB6PYWi0lSMQPDudc7djla03htWbuVX78IKkh9Da1FWymUXL6FbkuNhxM69mGjBb+5YVmUSrna226CPAwnI6vZE0KzUzzJdy2dBfCmOQuhKkxgxDlEZ+9ak4HhEtokYcR/nLaR2O+yjjxQOQObm3sGZK303274F2yGe9ipVNR94wrxHkxbR8MlbjDh8ZKCiav/qTCqXa3ip28oIUPAZ7MWWuEp+8+qSUcfiDBxCylw9iNKwmhmp/lM2FUVOjajQNU1y96GzIUlbnKOIL+2Vp1HMOROls4CSIChHylEHHKeQftYYV9GjBO+8b2KpQoiYGYX3JII2pqw1eY4EuTQLoWUVJE04SFoPNT30CE3q8JD4rbWZoQJTFfuG/8oA3ME58GtHK9KjPxxaZeokiUA6N6RRf3q2M9WKjOGHm4EKVCm2YALYLZqyscD+WZjjS0d3M3e8rbILe50w9P53CUlKcVO/4c1GPjbGa4cxMwma0PuTCrFaOWMwSCV2TiQwH9ez7h2OBFOjrTgAPzUYHyB1JC8PDGzhYwiM4kVPg9fN2cccb9Xq2LcGFrQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both callers of is_refcount_suitable() have been converted to use folios, so convert it to take in a folio. Both callers only operate on head pages of folios so mapcount/refcount conversions here are trivial. Removes 3 calls to compound head, and removes 315 bytes of kernel text. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 67aac53b31c8..fa21a53ce0c0 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } -static bool is_refcount_suitable(struct page *page) +static bool is_refcount_suitable(struct folio *folio) { int expected_refcount; - expected_refcount = total_mapcount(page); - if (PageSwapCache(page)) - expected_refcount += compound_nr(page); + expected_refcount = folio_mapcount(folio); + if (folio_test_swapcache(folio)) + expected_refcount += folio_nr_pages(folio); - return page_count(page) == expected_refcount; + return folio_ref_count(folio) == expected_refcount; } static int __collapse_huge_page_isolate(struct vm_area_struct *vma, @@ -623,7 +623,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; @@ -1367,7 +1367,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { result = SCAN_PAGE_COUNT; goto out_unmap; } From patchwork Mon Oct 16 20:05:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13424039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9AB3CDB482 for ; Mon, 16 Oct 2023 20:07:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C28F8000A; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F405480008; Mon, 16 Oct 2023 16:07:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF5648000A; Mon, 16 Oct 2023 16:07:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B1A0080008 for ; Mon, 16 Oct 2023 16:07:02 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7EB47A0B91 for ; Mon, 16 Oct 2023 20:07:02 +0000 (UTC) X-FDA: 81352408284.18.DE4194F Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf08.hostedemail.com (Postfix) with ESMTP id 84CA616001B for ; Mon, 16 Oct 2023 20:07:00 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fcv3WcUc; spf=pass (imf08.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697486820; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mwAjlmpq6KKWWorM+tiqR60ij8VY2bcU8cxZ+UqJifo=; b=5MGEaqLpirZjisDYn4UFowIUI9QUSUKO/vPphC0ezDPW4jDdCLeEiGDF6Yfzk4cpYX0XPF S96YoJ7Uhye1yUlWBqsXf+UcLeAwNlmjW85+hrbqLWgV2giwrCGLNuo6vVU8ZDkUhBRFFS 7grLx+Pk7A9NdWwELo/0aeWva/JsMEM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697486820; a=rsa-sha256; cv=none; b=kc94+PW8yASqsAeQ5PeqHvShEVgLssdLZQ7kQk+TeYABTbFNJQAk9mNl9Ax5QCK53abaqA Hv7VS4qxX8P1lsDgeXNiEIHMsQu5xrBbzUmJ46mksNTIAKdMr8/J6L4J5L88BeDbBLkYq5 zC4qwgvqFEPwayFWfHCeZ8kuaaHwd0g= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fcv3WcUc; spf=pass (imf08.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-27cfb8bc7eeso3940650a91.0 for ; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486819; x=1698091619; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mwAjlmpq6KKWWorM+tiqR60ij8VY2bcU8cxZ+UqJifo=; b=fcv3WcUchj6wKCkns0jYmJJhwp4t0oNNCWQB1pSbPsDpmgPxKSNdgiG0IfQLSuYWgG SX7r6cSgaZx/Ibog0hPasX/e7LE+jxCxia1E4S6Ivq1YpnoYYDfYaIsnoYroLukHL8+q oKV3OpPcHIEn2TqVCVVg9rISJjrNg2v9IHM2u6YnB+inJzKySOx0XIx7Z/L9sznFUaZ+ Q3cVY8dRmbNEn087RzavLMWiJxDRs7m14nV7nsz4dpMaEiqcQNLOChP9fklzFaFdjIvP SkYSByBNIx5NetXAQ9mByerYKW0Mql0/qnBBHyjIoxp0G2Ep2WsjsOl6vfFdHMV1kwYS IbVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486819; x=1698091619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mwAjlmpq6KKWWorM+tiqR60ij8VY2bcU8cxZ+UqJifo=; b=vXOSpniWSdV69AsrUBUizXE0jMk3UOQpfYOWyOiPiwP8qhfgNDq9j4eGg5spPxipSC xMa2enbkCwA18vqAY+p1g9sb77kX9AaoBBLciKNL++mYMTpyW8m3fUKShQDvZY0NXsm0 dwHyBv/2UI/dgpeAwhS1b00l/rkZX9NnA8uWAlEX4rWWt3Vf5XQ3idYPscSzVn8CZ1zn xjfbF2vVPAD75FN26pGWK435ABQJt1NrwTvNmz5c/KZpJWKtZgRXQ0Ca0A2suS6ELQ6x 9xHxTRODjmAXFUKHqxhqSIcaFwLFNvn0acHwC4Zqt7nkv+HaD2ONe/nbscJQa/luhWX5 uofg== X-Gm-Message-State: AOJu0YwXElz9/lReWTZBtTUvlpnnm4pVrmF5gy08PZLY7wOwB/ag9vU4 4wxehPhqrHTb7LaBMDDz5anheEzumFQ= X-Google-Smtp-Source: AGHT+IG2u33OIftxV+vG+uCRE0PArlnIRhFf/JTq4llkJ/7L4qH2959ZYqq3PoHzeqWuyBIqSdin2A== X-Received: by 2002:a17:90b:245:b0:27d:4259:b7ef with SMTP id fz5-20020a17090b024500b0027d4259b7efmr190516pjb.23.1697486819203; Mon, 16 Oct 2023 13:06:59 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:58 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 4/5] mm/khugepaged: Convert alloc_charge_hpage() to use folios Date: Mon, 16 Oct 2023 13:05:09 -0700 Message-Id: <20231016200510.7387-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 84CA616001B X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: f16u7udy6d7etm7iu8n4pqz619fijipd X-HE-Tag: 1697486820-825180 X-HE-Meta: U2FsdGVkX1+BxnKRsRUnVvJhjqGC3+WQDBmUZ+Zfg7etVVVOtUQMVdEo/uRu+NHbrwuWqay0Uco+24NxU2fOXePd6WUvpkQzSAZYq2mixxIrly9FbC6nGGdOkWVV4RjpR+JXdbxb5M7vHlu0nHhK4Jm/OSxTUYxJCezzZNkZs3vZiBpVSOOVIxVo71xzOVehPvHGZeok//pfyjiS0CfLMrn/OpWvdZ4upUAWRVz4/7/n4eh7LAAPZOcc/7c+1Fd4FF/+h+rM4SGn63bEoz+QP+Ncsr48wSzDUxeBo2++m3WZW6HFfRSJSy1smhVLqhXAb3qEYLPD6f5sATA8BknCSlQIF9Gqcc1E/ysW7K2Rvi/Y9Sffdi+oOaAJf0MJCzyB2+JtrbWYyc5Ra769eTJeAQhN7GYMJ7OeEv2GSV/b6xiMUyXUo3tgLT5w/FecL88Zc9pNLKbB5GGut/nVsXDGe2CgUTjY3qhN9QQ0y96Q0so4gVyYL1dmULJFx5QVmATF1fDaGZ3nwyEQ63wexIMMtO2m3YF7rmusHQWcK12khwtqReY1XE+q6ai7o3ITYhJ2N8CfQ13pZ/QN93dbYRvpbTF1eeF47cX0pCFz0MqqK4k5psihYkFobPCA7Q+2ttXctwpjczamJMTNUtfXnSeEtLMHF9kN0EQkEugPEWtWODAk4s9YA8vT2j+KlPWA1iTcYhduHqAjVnKgu/DeocGW/ruVMaMBOKtVjeG5q9meyAfPCeRyDh+bpp6DT9QGPqbCrKBstmz+1gDlyGZEZlfjNdYG1U6vd5ZSERdDZ226YVPle2MT0DU+tDJJCI3++gQBx/4dPsTCIv9cVKHIwz73Jgj3no8W0TgVxbDvE7WjwrD7ChgXzNECqvbNvd+UKhuleYeLlI0lwi/gW5fuFmeIBsS97rL5GHdwlX1OItU6IEk3wHo8q/EZFYDOY12KAh8JMfEY2oJ243PzfvcctOE 3ZCvDljU dxCAGKSDR7IlPUg9t8iU/lq5fCY+CTvVYNblKu2RyLL4GSYuGtI+n38ZmXZkkSJE+yeLmlrbrIbhc/XxnYaTi08EdiBgFq75NCGIOqYUPkqj9ovYU9fpbDmtZnUC7RLysahhO3WUa3shmdBndP3NLdjNVeVsFzLjrbHakY83FvmEwwOzLuzGSSNpbWBxRh26ZW3nMuWzktlN/CwN5wsh5ziqwSsAVGoEezknGDzlhUajvL/YlsMRpUYPEiYI0cRnJLhrIWHD50vsH5FAyP6W4+Hf2R/Iy7FshhpC2bxWFVhDBlxXHTKsIPWTS3UtRIWKAOfO+AtmabUQB4rTNEAY5k5o9tPmUnH4xjBWlV6BIeHJqAV35RD7w5o7YeyaZ8TKjbhf3Z1vPusc/njd7zOKhuOvDvkmeI+f+ov3FM2cbUbhkV+A3MwO1vE3Q9PTGhJCxVTI7Vdo+YXUVfPgGi38Y6B+bj3+2A6lcfzCjDGK4HdbcW3mF/ojzqCb3VOIijXll9fHFwSQAO3EmE7fagGukc6hYk+R8j3vzZg8pUk+iYiMANOIhxOeQwaqYoj+6AfEx8wSNM+nCmnsWA3/Sm+hI+8wcPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000021, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Also convert hpage_collapse_alloc_page() to hpage_collapse_alloc_folio(). This removes 1 call to compound_head() and helps convert khugepaged to use folios throughout. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index fa21a53ce0c0..70bba8ddea13 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -886,16 +886,16 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) } #endif -static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node, +static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, int node, nodemask_t *nmask) { - *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask); - if (unlikely(!*hpage)) { + *folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask); + + if (unlikely(!*folio)) { count_vm_event(THP_COLLAPSE_ALLOC_FAILED); return false; } - folio_prep_large_rmappable((struct folio *)*hpage); count_vm_event(THP_COLLAPSE_ALLOC); return true; } @@ -1062,15 +1062,16 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, int node = hpage_collapse_find_target_node(cc); struct folio *folio; - if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) + if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) return SCAN_ALLOC_HUGE_PAGE_FAIL; - folio = page_folio(*hpage); if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *hpage = NULL; return SCAN_CGROUP_CHARGE_FAIL; } + + *hpage = folio_page(folio, 0); count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC); return SCAN_SUCCEED; From patchwork Mon Oct 16 20:05:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13424040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75504CDB474 for ; Mon, 16 Oct 2023 20:07:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D45DD80027; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD11C80008; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B478880027; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9D79080008 for ; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6CE6F1CB3A6 for ; Mon, 16 Oct 2023 20:07:03 +0000 (UTC) X-FDA: 81352408326.07.317DC36 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 9708540004 for ; Mon, 16 Oct 2023 20:07:01 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B3a2rpc9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697486821; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=vCGQZOaijy6IwvG6JWEIu/pagfoyMSKpk26+Yv+3chcrBYIRthrIEK2zmW3hTnOLaMsJgc HxPacgzQ+Mexbjz4TXJKM2fExNtxTQ7JPb2WV7o48zpQCc3GAGYTK7GzDnjugg10NmSrSX 5WbR0gpPIKngOqtkTWgArL3S0jt/tis= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B3a2rpc9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697486821; a=rsa-sha256; cv=none; b=DpnRlAmfqMLPKrYiN0kZkZL+UfUvCC717I+aPNyIN1oI7FSGpbCplJAi30GH/mIeRP3e0r cVLKcuxti29QY3wALDNYAMVGyG1woWnERbvlnvvlt4mJK7id3egHsm3l4Ae3txbm6ShKP0 dd4CtoqKC+dxLUnj/ZGfgr4JiL6Km68= Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-53fbf2c42bfso3665655a12.3 for ; Mon, 16 Oct 2023 13:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486820; x=1698091620; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=B3a2rpc9f2yt8X4FzcDPWsSY6/+mFaJTKki6vukwAkUq0z0gNksP1SAMmEuF4cQXCK JDyKfEQhajlR+tvrd5sHmx+5fCUTlJTTi2EQha/kt7VIxaDycUNbBVbpmctS7csxYXr3 wsbJy4z0aF1uC62XRN0usW+g1UAm3/3FdJzEMJ5EpOdJ3d3+1VMdKmWOf0mZf0Z3dGxN m2uizjRPCOQXhi94dN9ekJNpYcvsPqGlFBCmePtgdFjfnv2TvRWYV/eZonA+0SJgRKXZ Yg8nIWiy3sw1msiSC1909bNNr4nTCF9rUUVNo58ZnxKRhw+Vl55yd0jdZ92gBYqSXPKe 6ZPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486820; x=1698091620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=QRxw4l8WQd4/AK1YQABGoM5tSNievREIkf+j+TBHC8GbCmpUqYAQlIRsaEyA/Y3jRW b99N1uEeM4rcllgeZkEuR1EcKjkcWoUlyFkDHP/weswBzRBxn4eaiwJMjIZaTXwvL9e+ 0cUOe6y8Xoe4sa97PJ/lZZtyoYUMNEBsIn5fhE+4DAYx/v5lF65HkYBxwH/8T50dJe4t XSypEau8eRK0e2ZTbeY58sW/xq1fBEPbpNuC8V++LP4JUGSpA9dwLqneOsd8rxCz8t/7 CeK+ucaNT5NAONK9yiQx2VKwobZY0vO2JwfYY0Nkduz4YbcYc150Nu4EvEPjDLay8syK CNsQ== X-Gm-Message-State: AOJu0Yyz4FVMoThn+FRsgeZUSULjDf2Tz3GGizBrIDSzcqcUS0fftd10 r6BHnGV7db7uP8fsYvbVwgVGOhO0Px4= X-Google-Smtp-Source: AGHT+IFPrGp2LxyePxJ2SGP9Uv5jwNSK/jlVnwKjorWrK64yF1KjrffZTKbz4NdtbcNvcYG72+5GCQ== X-Received: by 2002:a17:90b:368d:b0:27d:f85:9505 with SMTP id mj13-20020a17090b368d00b0027d0f859505mr159453pjb.24.1697486820185; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:59 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Mon, 16 Oct 2023 13:05:10 -0700 Message-Id: <20231016200510.7387-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9708540004 X-Stat-Signature: wehsqmdt13mu3eszgfpo6oig74rcyo5w X-Rspam-User: X-HE-Tag: 1697486821-377195 X-HE-Meta: U2FsdGVkX18RtrQCCDE0izASUEq9GhRdEF31rle25oxE9sJtmgHNtdde0MHgXVeRMmbE/+MSq4ASKQalk5lWTDgGW98BLUeuiNLEXppoYPksk7oQGhmNDzggVrNvPTDgID5KkqQzIfhNT6SZe7xet7Mo5P9sfkJ/iSC0n2k6MDLdMYfop89YalcShTUukFZdRTCnn2AyzlRsyv0BwBtmJRP6C/YnBcYno8EXIpg6ZTkj2MqrCG10ONrVh+7h7v9Pb36LwLYcnh5AZUZaJufIMlahpBPlAUNOh2Om6YLkUhzNF9PapNWyOzFgLJcb8uGeJWGqARWWdwBVkzCu5PLt4ctynvhf8pZZRRqHvXXcjU5qbFyds0BSQ4THlidc26Jhj4u7gZF6QvduRjuox/LhxRO49HeNmIYKIZHehin44EISyl62g1UClI7+Hl2CB/PLmo18cedBQbu3vOyI5f12V4jiLwmtalySGOD+fUW6pG+OxYF52mH47VJ6DdMjyXCj0fLZB8M4t41Wjlvnt6OrQ7zbYFmAJa4tt0JUmnSN/+tHzAaLfzmrYtHFi4YWMMnSuw5Iq398B5q9goqWuu8yGQfEPxfE6pPyMkzhvWad01h05Cql5SncsECXPCDivR3g3kTpUocAc12OIPkDcmIPvEzUmq8j5p3eZYaFCkIJGGlHW52EPcJzpiMiQ+kCgpzryP4M0krZXXna5Qpq1fPcaV3bBBJH0TW/WEma83yNo+E5w2p7Vd6F6Sy9YNoF27R1xkKf0RuNANkpJ2JOzKWmxZx0PsvF2NRUsmfGMZ6dI3wCuZsZRfHtJ/1QD3b8bfolPzCzZl4hVSyzmnR4F5ABzAv2U0qi0hsR30A+srnWM+zbeo5N2tBKLehASJ9U57t+Uxg5+adOPRp9ensr91NJUHWH84yP6qGAxrdFNeaXOfnSZ7471SkqQPAf94GKqMIG7hoTE9QjMrMXPTolc2u JJIh7hjm Ktg2pEoCaGnYJkNBfyEVx/rUoQJOpi50R+GUbcmJXo8KlS8enwxmUElVI5aBD0N3aB0L40vupIyCwhvZ859WuBbD2VVPH3hBoc9n5uGi1dlwqO2KN9/T1d7MnXJwkHcVdyAx4zsXKnAOfeUNK40jhZ2uoUy1HEHfDjLSSvZcWL+6xUcf7De9xp8KpkgPgIgUPqaaCXboHqpWLxgwWmUpkLZStTNt/QzYbdEGStRMQhAfkhJYS5iXwG32RC+cpA1XSWmFxjN+JnAXkXf4oUDQE/QPV8tnHy6/FFsSYbLM1ZCvIbHoJO3gYqaLwq87RyI9F6tjxA8Ty9Y711gAKrbEf4waJIfDVtZoRsDU7oi1rHZawiMTo4t0/rVfJg5bxiQdrr8Y41gh8QoY7SoBJSZveQirKw0oO3aDjGYxgcfd/WMD/jILMLTNIyS8dbQEgpPMU/whb/d5aCUy9pD6+HYBVHi7XPzpDFxKQbwIJi0JGtHLhsKlbNgN3MCSsfm0R8tzd58LxbufPkwhttNEoIo4M3Dlp4g7b1a3+n6MWW15LCnUf1z3ZQ39z+m+aRhG0j+UJ7nuV6v4cJNSBtqAhowVQfxHl+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we can get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 70bba8ddea13..99c437979848 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1471,7 +1471,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool notified = false; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml = NULL, *ptl; @@ -1504,19 +1504,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; - hpage = find_lock_page(vma->vm_file->f_mapping, + folio = filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (!folio) return SCAN_PAGE_NULL; - if (!PageHead(hpage)) { - result = SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) != HPAGE_PMD_ORDER) { + if (folio_order(folio) != HPAGE_PMD_ORDER) { result = SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1530,13 +1525,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1561,7 +1556,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; } @@ -1576,7 +1571,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1600,7 +1595,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1609,7 +1604,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page = vm_normal_page(vma, addr, ptent); - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; /* @@ -1628,8 +1623,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } /* step 4: remove empty page table */ @@ -1653,14 +1648,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1668,9 +1663,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; }