From patchwork Fri Oct 20 18:33:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13431017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96F20C001DF for ; Fri, 20 Oct 2023 18:33:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E04E8D0003; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9908780044; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76E658D01D5; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6A5DA8D0003 for ; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3D11D120891 for ; Fri, 20 Oct 2023 18:33:47 +0000 (UTC) X-FDA: 81366688494.14.7ABA9F9 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 5E1DFA0016 for ; Fri, 20 Oct 2023 18:33:45 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VWDnFcpM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=lPDhgJYKGty2jMkkb+ajWc8zWl3FcTfQUOWxQ0qdQPfL7hW4iIjp1ptORNZs9kKIpuUHTA VLWmZuLTWiLuj8DEatwYuKvaiPOTvinPrZT0vPo6b8XM645xjhgFpjrhl+43EL2k8rJWO8 Er+5YYGWeGMIOWPaJx2XNmm5KXDBE7c= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VWDnFcpM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826825; a=rsa-sha256; cv=none; b=nsBvWPBPFAq4G0e5HQPjxq4UozmxV8SQP+0QrmfYkgUjc+M8xAhE39WKtR1UWpmTrx0FdO chRt4xFAhby8L1kV2SOBP114190apMI6NGQceZVeJCFvMKo1r7GIpfATL/NJvVJuUEx7b4 bWCr7SWgfPrwXmTMDPuvyVz3OnyKkcc= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6b5af4662b7so1057133b3a.3 for ; Fri, 20 Oct 2023 11:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826824; x=1698431624; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=VWDnFcpM1SUsjPJSsaaxM8IpoLObUAr5eJlSNgQbKY/3harjkn5PfWox8kN1+4zm3w zaiCQhlbjqjC05sIKPFixdl3BYw0H9G1aSdg9ir8G6dw3UGJCSen2HluvGXvAyx3fWOK +chsgEkStjWH1uDZKJCHLUi/xw9F4K59Z9eq8m5hqaMStLJFnAFuqvr/+TinM15qfVD5 JiY9oIXYSG+ButSrNFiYOmJ4YM3Rv8YSWRNQEaMgSJDzZ9VFsu+RGnoz5DQKWB/LD6Qz UmAo3NdLA48tsYQSsZ7KmA1tpOOXPpunNf7kTcw0zfhpBazaFlbnZSgTWSvvmT4bJdzg EJDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826824; x=1698431624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=VQK7aOqTn6h/lq0eKmvBFrQQsKvibLn0/rXrFdgGOJoKIsXbLVpX1Nr34SgvYcRt7q /Co1BRpsYKtl+yJqYOuzLKdVClZK+LvShdTtVCE1f0/xKLbUf8/wXJY4bd9ZujV20tf3 xg6A3ezg3nK5z3LjKM5rZZp+IbCiAS4YmFAcGHZZ36WGNeSnxQ4wolch1dYzxQxEiKBx BYnFd6q0tnjlHCqE4L4J9uHr4s+v3IoyyDIMMuJR1LkpteM7+aUC6h6nTuC1FROHPIdU gZTgd9qkrr9pD4jegMC5ffqPPLdtu4ExoehSGxdLlwtJcbw0PA+siOUDRnozinf2Vj8B iwKQ== X-Gm-Message-State: AOJu0Yz+Ab75rW9Aw0hF9lqanAf0clMJD8wLx2XQrzuovrMty5x6Ms0t ythadthhAcV3sk779mqWC7ZI9obsN4I= X-Google-Smtp-Source: AGHT+IGHwdGrDYHJU3ar4omw5xYzuwotpwlRbNDAoj/IKDWChBLhpJBOvVjPjaCZMcDR1LHeQ2sY9A== X-Received: by 2002:a05:6a20:158c:b0:17c:cd46:73d with SMTP id h12-20020a056a20158c00b0017ccd46073dmr386382pzj.20.1697826823892; Fri, 20 Oct 2023 11:33:43 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" , Matthew Wilcox Subject: [PATCH v3 1/5] mm/khugepaged: Convert __collapse_huge_page_isolate() to use folios Date: Fri, 20 Oct 2023 11:33:27 -0700 Message-Id: <20231020183331.10770-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5E1DFA0016 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: tu6omiutfgaop4rzijrc9rz7cqkcwj3j X-HE-Tag: 1697826825-45399 X-HE-Meta: U2FsdGVkX19uxjvEp+yMOJmfCeHC4xBoNKWO879Aeyh/+2N9PWjvbnZtFR2JBJsyPMv4AA1j+7blyX4S35/9pjaiIbuM5R5J+tOq8qIf24P0RMBGjOys19SOTTAkKZ8bLaXlj+0M4Oe5soRhfwlG7Eh9Trxb8vXF1PVWOfX/hvpQHtlEcAteccyi3YzXgti8kvysmTKqis44PSSNmYWBba7aVNNEjpIdMuljYk9D0MpylHpT+5MirOtmG5jOSMGe61CJFLhmDVtl7mBLCcVUe8h6hyAWAWKH0FaGEWp512pueu2oc3D5jqriGll9XTphsfX1HkJ7vbpdz8y9P7vAovJA02u6zv8rchPs5gb3jr2Ff3OJTfI9U+EjJWTtmIAOvvDzarMsC+kgU1PiUnZTE5dBLNuBu4CdU+BbENARBf4PhbNv+ZBzNQ8HOwCoTuE3QZlLdYa8tIk0w6KNJlMsAPSLhzHFn5YaaAcGmyrasNf5c/VbpruYvIMv8MQvMr//2ixARmsg524x+CDckiYR3SZ2Af1ZncV8FSgZkp8/12n9M/whkqsF5doGkd6myp/TittrHpl3D2P7qVxmHW7mZK/pytte3jBsvzAzsir5jeX1nkSs1fZwIWQnB3uhru83M/qBKWZFsv6Hc/GmwCanRQafQ5QowKtuker6RVJnG83odte7Qa072DTFY+PocM/FqBQJIQJqIZFuEXmQPYraqbsRGWz6iHZuSoFD6/DruzEadMwX8iaiAJdJF0PT9JpcUVQrqMZe0NOiXmWtOtNk8IBnmgc+pd2xsS9RsL9Xo4wXOc/n68AyfFQzJAmISJaJHlkmBZOTx3noh6OGPM4KsoUPBa2QzGAqzSLXiBmA+4SkQAwaxsc9ELDhFUKjVfROdvwjh6tsfeRBgByK4YeDDCq7YXg5c47W8POf3rE1+nI7JWyNrZMbftuL707ghr7JAVfkONDkF6P30wH687v MQCoQXGJ ydxvOqiCIn4xA6s7wq/EnXaQfFDE3LOGt0oAOmHNFk3fXgO2Iucwc/A2SHLg36VtGyxYtxgtdzN+rpqU1BaxsNBDW/7WPT8VPma0SUJTecDofz0zNlqltPs/+QxpcPtPnsoaCuNTJSEecO1Fy1Nc+nV4HU3KM9XTMs7wNR4g1buQTUQbgYVEhQd79n/3miTV58wBNrkENUwM2U8qjyp0YG2DLRIp2xCwnoF8nPlreVpV5SCzlxHQcphwyMDqOGUze7PwA6s1caZ1xDC+qvzh6ycjIdvwFjZuZtJACftjVZw1HEOUQBZfxsMzz6GxpvJTGmAz2D5Af5e1UKY5qZ5dSDxgKS2c1U46foBQh2jP5e8gmxErdDWiHpTJeZqoYYcy+d3JLkHFX43qNvd0mrYhFgoYtIBFQGgHRJqYDqT/94oZNkWJb2s+NW3yvCMyt/szy044pNMLx+70v4Qe3L7/UR07CfgB/YHmUvLVsrI369CwNsKNO953BEYdZX7NVxGDt1ZhnlhHkB4RPcQhLw6f6nUyZXU0DkFPODcuxY3JM6h7mSpDQqDrMNSBUvpQRtHCMExETXOnCudEqlcT8ASwiePbcV+d6IcjxmEaqaABZq1W5PUXS7u/lHQxLf8DhP+z3dPKilyzhGFJ6w3carzucFsBsAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replaces 11 calls to compound_head() with 1, and removes 1348 bytes of kernel text. Reviewed-by: Matthew Wilcox (Oracle) Signed-off-by: Vishal Moola (Oracle) Reviewed-by: David Hildenbrand Reviewed-by: Yang Shi --- mm/khugepaged.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88433cc25d8a..500756604488 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -542,6 +542,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, struct list_head *compound_pagelist) { struct page *page = NULL; + struct folio *folio = NULL; pte_t *_pte; int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; bool writable = false; @@ -576,7 +577,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, goto out; } - VM_BUG_ON_PAGE(!PageAnon(page), page); + folio = page_folio(page); + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); if (page_mapcount(page) > 1) { ++shared; @@ -588,16 +590,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } } - if (PageCompound(page)) { - struct page *p; - page = compound_head(page); + if (folio_test_large(folio)) { + struct folio *f; /* * Check if we have dealt with the compound page * already */ - list_for_each_entry(p, compound_pagelist, lru) { - if (page == p) + list_for_each_entry(f, compound_pagelist, lru) { + if (folio == f) goto next; } } @@ -608,7 +609,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * is needed to serialize against split_huge_page * when invoked from the VM. */ - if (!trylock_page(page)) { + if (!folio_trylock(folio)) { result = SCAN_PAGE_LOCK; goto out; } @@ -624,8 +625,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(page)) { - unlock_page(page); + if (!is_refcount_suitable(&folio->page)) { + folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; } @@ -634,27 +635,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * Isolate the page to avoid collapsing an hugepage * currently in use by the VM. */ - if (!isolate_lru_page(page)) { - unlock_page(page); + if (!folio_isolate_lru(folio)) { + folio_unlock(folio); result = SCAN_DEL_PAGE_LRU; goto out; } - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - compound_nr(page)); - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageLRU(page), page); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - if (PageCompound(page)) - list_add_tail(&page->lru, compound_pagelist); + if (folio_test_large(folio)) + list_add_tail(&folio->lru, compound_pagelist); next: /* * If collapse was initiated by khugepaged, check that there is * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; @@ -668,13 +669,13 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } out: release_pte_pages(pte, _pte, compound_pagelist); - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } From patchwork Fri Oct 20 18:33:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13431018 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB1B6C0032E for ; Fri, 20 Oct 2023 18:33:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74954800BB; Fri, 20 Oct 2023 14:33:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F90280044; Fri, 20 Oct 2023 14:33:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54A96800BB; Fri, 20 Oct 2023 14:33:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4914D80044 for ; Fri, 20 Oct 2023 14:33:48 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1AB5C1401B6 for ; Fri, 20 Oct 2023 18:33:48 +0000 (UTC) X-FDA: 81366688536.05.492F0EE Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf29.hostedemail.com (Postfix) with ESMTP id 4BD11120002 for ; Fri, 20 Oct 2023 18:33:46 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YNQnSr8C; spf=pass (imf29.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kOxvYaksrDFINgvTHvg7i3BLH6nikNi8tBwFFhT4+xI=; b=qEAoaivB8GGXxVdfu1ftS0P3zBcR3kYjhNpg42SDraRtr1TlqASfzWC6jQuGUgpllc1yaJ PGUM3GxmzDxt+5Wj/9ZYxXk/fpcslswtpK2jMk1Wz03QAWK455xhR1wGLKWASjiHXoagdr j6shtsVVu6BpGY58PdouPgPl9oKapMU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YNQnSr8C; spf=pass (imf29.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826826; a=rsa-sha256; cv=none; b=lZSdOtoi4LjRMIVjikwiqKUguUEIUno85FAit5N9L3SlcrLnrbP2YZ5tquqDV72r/omeFs CGMFlv1DSKoexcqy3r4VYgqG72h6PNwcdxG+TV3Y5mxaOQgKNLkCUb2G4CJB2nXfJYxJo+ PjBBH4nWT5ebBqCwSAqYFmvtyvHiFIk= Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6b3c2607d9bso1064067b3a.1 for ; Fri, 20 Oct 2023 11:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826825; x=1698431625; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kOxvYaksrDFINgvTHvg7i3BLH6nikNi8tBwFFhT4+xI=; b=YNQnSr8CL3EdAaBELXCWS+6jaR2jZ36riGWOLpY6pFKbDYrZAU+20pMxKWD82EK9aD +/gXWI3feFZlHxUibQCbiK9Sr+Z2nDHnAHAVZicF1rJbZ59pKKJV1JjllGV9LZCH8AuS mZ4fC7DUYbELduqbbufEEy2R5qFHTnvmF6+lDoiIT1IR0Y9DpDTJUVYt7h0muNFUHa/B OWybA+MwMDyLIhkiHEe8D0qffVfXww7bJpF+nTvazLsQJJcxqqY7asjpHW0cujuWz0XV Nd1cfFxkeHN/QBuysp0NjdBIRzK0zSNnMNoFVvFY7XnFjccBS5TBU6QIm3XhDxaSq0tk 1JKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826825; x=1698431625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kOxvYaksrDFINgvTHvg7i3BLH6nikNi8tBwFFhT4+xI=; b=pXeb4doTvhP+a8o7H4bdR0vBbz+uDGrVVB0b3I4vr2SGi2cCNK0VmUQDFKIiLp4+0J ADaQX80qSmptGoKPuXVo/AwZBYapxZ8TqustRqXfXca4TFuuHTHNgFupjHYPAe6108WC TVl6Y/tjfqcZMr+KCbdPKExkxsHH6AnOOrrSEFnpKoGNb0KqOcUhqy03XdAbLPmTiNTY EHeVWYf0AA9fu7u+tvQFcW2obSTW3qLib9ZQE0AAVjFFjdgaU7cnNY1MDhunOIveQ074 3r7WrlRAZMKX9XT6ch83Qiol7EfuCAQdaq2HBrhdvMDXYMTL8EhIUkyQYJwENlBPVEv9 nwSw== X-Gm-Message-State: AOJu0YwJbxXTyr8zJecvm8LbXsHU2ujWPJwMuf0c17tVeYb5yDBseAej i5Dn3PJzG5bCRouCQWPh+s4M+QGiRIo= X-Google-Smtp-Source: AGHT+IGjvWBVIPvS8d/Tp9SPsRWhqao+5hgFvjwUvgPczwPUlTiLiRzRdGbqQM/sCIzdkV5OmPNgVQ== X-Received: by 2002:a05:6a00:1393:b0:6bc:62d0:fffd with SMTP id t19-20020a056a00139300b006bc62d0fffdmr2943268pfg.15.1697826824857; Fri, 20 Oct 2023 11:33:44 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:44 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 2/5] mm/khugepaged: Convert hpage_collapse_scan_pmd() to use folios Date: Fri, 20 Oct 2023 11:33:28 -0700 Message-Id: <20231020183331.10770-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4BD11120002 X-Rspam-User: X-Stat-Signature: dhbym7hgkzcwrd37zgemthwceb6exk6t X-Rspamd-Server: rspam01 X-HE-Tag: 1697826826-382685 X-HE-Meta: U2FsdGVkX1+dSeFXQzr6HHgPbbW0KEKEUcsiOvTCDlyR500KJ4ZOlImr5k7weQes68dKzmLsMzfS9vXy6Gm7wBtqMWUlWYxJTS6TLBeAjp5FhU02UtP0VY5DBiyN86saGkrAG2Jvci/Vb+JWd9h2FNu+eB8tfiiX/F6Bs3QjKBFr1j0Xjr1NpvV7zXgy0fT9/SPn/XOKnsKoz8rn2ojIpBN+91GXzAFmL2XoloulweV1ffAUx3XqrxoQRNeMudb6zwjSuXC4Kl4CE0t6zr4/+YU495EWWTpG6EhB5W1F2GnvZscT8GdkNl6DNd/Xfw4H8Hjx7ZiZXRbjPHTMbSJ/L244NJLJCYKts8voZQ0kIuSZ2Du8xIAar1mBa0LlO3q/ltsIpl6UeOvGDb0G6q7gFfPvlomktPTFaGX+khgwLiSZ3IQb3P7VJpf5IZnbuTgZed6XU9BAh1Ko2c8rvcKoEuFHgsHJwDrzl+VvS3MmwR5if1JinwsoZk5yuwZL/F7/5KFr2m5rFDqq2YGwZ6GqDTTUSF3OTffi1STPVDSMeOrhgv+ZUK0uF/bm29c0YbhKW7PxYILiHPmFBGFpW+HnxVPfNDFk4BMhMfjP23T/mjAG3fJrACoqdVdY8U4zPcPcL1HNmSh5mJkDnpAzFLA0MYGpq3R1bhFkbyMwuHA3HToOv3ZV6ffNom5V30idoLYF3OoJihInf9SLdTwqOEl0796s8+25R5sJSMC+ugDdYpg5mpcKZgqLRytSQTHawJoAujCar6xEWJsbJ+Jwpnpk4/dSnp25COhe2G5lFKp+qeWPCDQECMmXIoYggR3GvvktO+ieqfOSGCsGMZmeQDcZ+t4g+/kmtEkolaks31PKq8sxB4FKM7zugqotii6bGt/ggIBzR3I7o6OkP/V2z4PhHzY14zGbfiG1j5lCJ1sQjbNTuO6Oi5NgpXfsXShNTE6BdyMSdgNN1ztD8yAGjcF zTZhl5vW 2DSdaZus2zwffOkzfLuWtOgwagj8KJthZGIGiZ3g4J/zD3hGqa4DnV1ym+jAhGBHsE3J0AJG/TpJN+8K/lBjeChbvOyGNkMofdcd8RLTDp55r15sDP1TA73eJUmOvkMPVtSm0FThQPWGPNxzkbcUU1em+EoU8PjUZ7ch4Kalo9OzRGqLNHAz9i57Zpzz99lUsdd2IOl8MOM0q2T8560x/LCfJZbNXcoVMmcOFQsrm9WQ3qiz4XZvWqFQQGRxhp5fOPT14L2lOj948BilF290t7m/NCzDBF7AaiK6PsbsUP5fYCi7gQ1xnP8YIT6yRKDiqvX2gHFgixq0sFtkQrsnnIkkk3neIbe2KU0eQimKjmkUyh20IkPBhYDikwBizZHEDPR4vBer8TlfeZgvntFt55+SGKYjQwv4oGQzIcbCApgzKEORCnbZjbamap5vkUmsS7WscqWX/Fq+d2JDj46WqQTvMxGp6jzSj1l9Iwkv9ez0CyeoXeOlTLtzi+rzmWmtirAm73RfJoXW7J+vI1dhloLRuX0uD/ck/NSZlsCaU0uQdCfYNM003Ls+Esw8sUkmu9GfY4ZJRQ1oaSTAKzOQLTRnLQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replaces 5 calls to compound_head(), and removes 1385 bytes of kernel text. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- mm/khugepaged.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 500756604488..6c4b5af43371 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1248,6 +1248,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, int result = SCAN_FAIL, referenced = 0; int none_or_zero = 0, shared = 0; struct page *page = NULL; + struct folio *folio = NULL; unsigned long _address; spinlock_t *ptl; int node = NUMA_NO_NODE, unmapped = 0; @@ -1334,29 +1335,28 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, } } - page = compound_head(page); - + folio = page_folio(page); /* * Record which node the original page is from and save this * information to cc->node_load[]. * Khugepaged will allocate hugepage from the node has the max * hit record. */ - node = page_to_nid(page); + node = folio_nid(folio); if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; goto out_unmap; } cc->node_load[node]++; - if (!PageLRU(page)) { + if (!folio_test_lru(folio)) { result = SCAN_PAGE_LRU; goto out_unmap; } - if (PageLocked(page)) { + if (folio_test_locked(folio)) { result = SCAN_PAGE_LOCK; goto out_unmap; } - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { result = SCAN_PAGE_ANON; goto out_unmap; } @@ -1371,7 +1371,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(page)) { + if (!is_refcount_suitable(&folio->page)) { result = SCAN_PAGE_COUNT; goto out_unmap; } @@ -1381,8 +1381,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; } @@ -1404,7 +1404,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, *mmap_locked = false; } out: - trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced, + trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, none_or_zero, result, unmapped); return result; } From patchwork Fri Oct 20 18:33:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13431019 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29D50C001DF for ; Fri, 20 Oct 2023 18:33:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80BCE800BC; Fri, 20 Oct 2023 14:33:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BCA980044; Fri, 20 Oct 2023 14:33:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68502800BC; Fri, 20 Oct 2023 14:33:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 56EBC80044 for ; Fri, 20 Oct 2023 14:33:49 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1DFB51A0673 for ; Fri, 20 Oct 2023 18:33:49 +0000 (UTC) X-FDA: 81366688578.24.52C35A6 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf17.hostedemail.com (Postfix) with ESMTP id 4329240003 for ; Fri, 20 Oct 2023 18:33:47 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WBYSAKp5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/rAUjlYbudj5wnw8lHkufWutUwYDRgsClPJWjawwgo=; b=SAT6LzmdehaZiR4tKk4RqPO16D+GmKhlHwXjktt6p2wh//Y4j1CbUKCBIToS+AVDC983jK o9BaYDiniZXTbHIqsfur+rO/OfN6XcOKyVFILb/lGv7tvw3t4Y14i2LvZTeR0WgxsClNzv VDaaJn1jQVmow10oG2s8NiLkfF+xvMA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WBYSAKp5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826827; a=rsa-sha256; cv=none; b=zCTMkwPko0ge9p5I8Gkq+Pa38P5dd7gbBO04ID33Jmb+VxwZLVZzg3o8sM3OT5EFAj66iK tKMaSQG3BHwVXMem0T9gHtySr8/haWPOTG+r4pF76pEhyiUIt967s56dftCm0eEbzEn22g 0XuR9oLnKyIqcNe7wqBi0ftWp984++0= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6b3c2607d9bso1064084b3a.1 for ; Fri, 20 Oct 2023 11:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826826; x=1698431626; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q/rAUjlYbudj5wnw8lHkufWutUwYDRgsClPJWjawwgo=; b=WBYSAKp5e0k5+sJ82wqzPxPiUv9i622raqy7L/ddlGSrE7TWAeC+Dfz90dXzCohzNr ImWe+bBUpNrSna198oVSlAcpqwm/wQ8AmmwVe94jA3fiDeyZWGY2kzD2uCm3PKUO2va/ /R38fl0NwMIDfZHFBJZO6Y/n1P5bTrYMbEOatlUt9MsFF41/YAHzI0M7hJ8P66v1GIwd yWnknbnUwhK6lRaRspDEIitC7ixru5itquExk758kcZiC9saQxvrQ9doEAmXe/QcD8b4 a5k1ivoR+GZPXJIBfsGk7aR9WM042KObfVRPuSLcFL9emAZgpdnIyRKD9OW0uBUEU8VH 56zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826826; x=1698431626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q/rAUjlYbudj5wnw8lHkufWutUwYDRgsClPJWjawwgo=; b=bbCIdiwITkDwtA7/G1npIio7aKGXxId/qiWuYvTJe+yQvKoENS3sbOAQXbA7pkwOTQ Ursj1YxNRnsgRgmIJut5v0VQunvVLcXp7IOC0WHI2dRqt4NwAnYV5gqmK6LlPTHKJPi9 c+KUCh2lGKWfg6ih8MjYBI1e3KiFQwRhXR7Vn51I3dZDDteucfFSO/La8LuiRevQGfnA jmIeWyBMPMtByxCL/LA2pUNqZyDIH6stg1spYpBuZZSkBmkPzK8Df0q6vo8A6L8FZLkj UZecGFjXGKygD8YxR4RAX7b9ST+q4CN26AFlHSJdkJpMfInbTvs7LrakqL+5aeygT0Z/ E9Og== X-Gm-Message-State: AOJu0Yzz/sgWMiNaaT7QCPv6Rtsh8dL98gwK9ENXx/I6qgE7W6B8kONz jTUe3+XFfdSTuBe2FhzigwByYb+JO/Y= X-Google-Smtp-Source: AGHT+IHyGeFVYpw0wd/VaSvhNfJGFkjxPCXQo0amteeNn2H2xRJuWdIEmOzTNv8bZATyXXNSf6XFFg== X-Received: by 2002:a05:6a00:2341:b0:6b3:2106:94b0 with SMTP id j1-20020a056a00234100b006b3210694b0mr2956504pfj.23.1697826825800; Fri, 20 Oct 2023 11:33:45 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:45 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 3/5] mm/khugepaged: Convert is_refcount_suitable() to use folios Date: Fri, 20 Oct 2023 11:33:29 -0700 Message-Id: <20231020183331.10770-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 4329240003 X-Stat-Signature: 393g1pcawy8kww3onfz1t3umbsi96fxb X-HE-Tag: 1697826827-942734 X-HE-Meta: U2FsdGVkX1+Rb1UM+N0WNipJa3am+RumF0bI18xNqq4jlJyyZs8xaM7LoXPR1ffJoG9qsb9Eonoyceld3kSJD/wNWd3aSmgYIw3v0nR/8uBaqpAy9qz2/FAl6Nii8kvHWBApvb/QIOvFduG4T5lmnNSoWoOP007vcbSztGPEjHfnd18tfVrmO2w4X/lUXfjti0+SBlyh3gjDv1F2zh60fjx4AW8/NX0+Od5Nur618iPkhLeDWq0cecBtI8+DHsbIDTXYpBMm5IZsR6qUbqsr58LWtMl97BhxtdcRogDI/pVWcT8AOUQceaS/2t9hVjRZh1452agu6CpBR+vg1sdsu4zWdwmQkIbwYFxCOw1eMPD+TuWcZCV1iLvR1e2bPB0bWh20QN6Y2jb/YSwPwd54yCbAFctumgT+PwJ68j1X7CsXYZzKUtxKsSn4JUlFHjDK8HR9UH6BPKZxdfOhrSgp2jss+qk/y4aLhpMo30/y4We4eVzKNlbBRI4KCrBdCKrjhOlsNmxgqH0GyssPbYZCCzi9bcAItjBS1QV1lKSJ3IOkvDKSQu6uSnR/bvNsWGaqrJUlTBZjfPgxUTfGEC1twUUXNhb/RK1jaIWs/QVOzbbQ9EySCLEI3J5ii1s4MCDFEiaAZo+XVqkQbTgW28jpxfXqxIXOqPSHZno6QO9PIhQ1PFD50jFT3s8OiTKYyWL1pClDepLAlaIqcVu/PAcDeF7ch8odPM8Gnq0ZPjVklj1dD08Npv5WG6P+kKIBZoozh6JS7ITpvwkeBRL7RCijYjbMVuEhmkkZ+MhZsHGFuIf8pOMwvMG17qPX/GU0/Eiw2xRkCxyG2N8LdCqdp/dM8XDBsknUyyzt0ytqNjLzJyDX2W7hKguasxMRi9+b1TOXqNhrulTVOrBr5TkY4p+slpj04FcraD71Nz9RqxEAg9GPW5BsgNn61+UzV/+uomBPXbFzu++QjXNufdEwYZ4 GADNAl7v O8CQHe6vBSTbXrS11g+YYyUCcy3ikTdbPeGf2fqiwQS3+0h9S8CTXZ8yPoHRClADGtLtvIPBbJxiUF94saPHfOz6gvq1z3138FIkHoKkj1Xit7gFtozjVdM04D7MEgRgRklxKElKIe6sRVRqFfSlI/Ez+aF9HFKHtAbay0q6VAhrUTRFtSnXtP65+WwVPete775/oqrOWDDYypyU5N0hBaKdyXdzKd4hrIbNQBfcHqaOYVwc4JYoXFDgIRGUOa13nqhpMvs49YtYhFTXhk2IiXmq9EfcRB8oIol5M+WPrAK+qdBMw0Z7VBcanpEjScxUOfQ0UydPfn1d84oWqcZ5UpI6X+27Vo50BJQjmUjDLB/rwevZNwz8DhTykL2a3uBh18c1Gx0jNHYQOI9zHTxjkvHyEUWSEFSUW55XyCHQ594U4hGkweFjhOuoGv5+sgAlYNVGCoSUjDUjunsCVLSEGGc0WIT17VE/95KfrQJo5DTEnKejrzhV8sJEbM2uuwLcVejSWzlW/mCSq3llkCwPZ9b4j58gf7JT79POMGLuUcXpGhQHOV8YwagIcJkguzaqmr1ybpYXiNseuOXzoboZ1IJJcvg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both callers of is_refcount_suitable() have been converted to use folios, so convert it to take in a folio. Both callers only operate on head pages of folios so mapcount/refcount conversions here are trivial. Removes 3 calls to compound head, and removes 315 bytes of kernel text. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: David Hildenbrand Reviewed-by: Yang Shi --- mm/khugepaged.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6c4b5af43371..9efd8ff68f06 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } -static bool is_refcount_suitable(struct page *page) +static bool is_refcount_suitable(struct folio *folio) { int expected_refcount; - expected_refcount = total_mapcount(page); - if (PageSwapCache(page)) - expected_refcount += compound_nr(page); + expected_refcount = folio_mapcount(folio); + if (folio_test_swapcache(folio)) + expected_refcount += folio_nr_pages(folio); - return page_count(page) == expected_refcount; + return folio_ref_count(folio) == expected_refcount; } static int __collapse_huge_page_isolate(struct vm_area_struct *vma, @@ -625,7 +625,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; @@ -1371,7 +1371,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { result = SCAN_PAGE_COUNT; goto out_unmap; } From patchwork Fri Oct 20 18:33:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13431020 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66BC4C004C0 for ; Fri, 20 Oct 2023 18:33:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68648800C5; Fri, 20 Oct 2023 14:33:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6356780044; Fri, 20 Oct 2023 14:33:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41329800C5; Fri, 20 Oct 2023 14:33:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 31C4280044 for ; Fri, 20 Oct 2023 14:33:50 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F32C91CB5D3 for ; Fri, 20 Oct 2023 18:33:49 +0000 (UTC) X-FDA: 81366688578.11.A6C11AF Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf30.hostedemail.com (Postfix) with ESMTP id 2F90A8001E for ; Fri, 20 Oct 2023 18:33:47 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IyLnUqtu; spf=pass (imf30.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826828; a=rsa-sha256; cv=none; b=3XTHdnLn4EqEYvHpaBpT/WrIOA+dBd3dMiAAuHovtmYM37GCZbDa1z+lB9jPLpXaM0W/0Z fx6c4Pshr13X+OvyhRyxZsei5bBHjec7w8JpDitV0xe77My2/Qbj4q+cSoI24j5kAdJGfK /VEweQqF0fFWFyibj0zifKMdBZ7NP/g= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IyLnUqtu; spf=pass (imf30.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xk5ybvp+SNZIaZ3pEA3Be7vi8BfZn75lCz2iohEvZyM=; b=Xz3MU9VS5Bj1HnmvMAs4Na2qlXVyfgTtMkVJWSF/uWot0Qhdmw2hFK8QdZW+8va8Dp4hd5 4u8546M5FWWX6tc5I7ypbeR9ggXZMXwGPvgf5pa+/55ksSzpLNVNIo3Zxd2XpuVuCBuYy+ 9DfVDO/HVnkgle80eGXUJ+KXGNhhZH4= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6b497c8575aso1107725b3a.1 for ; Fri, 20 Oct 2023 11:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826827; x=1698431627; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xk5ybvp+SNZIaZ3pEA3Be7vi8BfZn75lCz2iohEvZyM=; b=IyLnUqtuD5l/+L9a9vP08QOgkSFVThoiQNZeEYZkmiadt+aBOUoXd+NHR6whFnm7ve YFyIS5MAUe5hre3B4yDpBswDbfYCOyFtjC4Bon6MfsjWaIxZqKkDwO/UD8zZziUORNOU dm/OrpEyIMmPbapGDRGy6Z5zPPnQNunjDDKw01w+FpBx+fPsbFNMm5XxsgWtsOiUKglb 2tA+BqA4JkVcG5O0R7G7CiBrFm6/r3s67+/ODt+71p/pvvOmjxIRqS98ayvudhFi2EMm IyJpTumVirVPHJ6I5CTrF04pYQxXZOWlly6VWem8rS02uxOGPN/5qL9zIbbaMcraT/xE kHXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826827; x=1698431627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xk5ybvp+SNZIaZ3pEA3Be7vi8BfZn75lCz2iohEvZyM=; b=k75mc0VQBcBKm460JvRDHRdZTLHCBgA1gjOME+nB/zSaGYIEL+LJmyM90pHxuKIJRv +09z6kMLpToTLu/rMU55I3rjjvKGYr1u5CA9p1mPMP67GOjSLnx5fHCNE/jDSWaNzjZE /F3SuiIMWLsgzH64+ZGbjgMyUpQbwv3J1wuCgM/fzGCZXvh0uOpAKayP+gF9aoC0D4qm n+Q28kgJ4W75+VKrWECZVaR0lxA84CLoLPkLLi8BDhrjkakC0CcfzzGdtNyPVNentCjo j72c6x7agbJnE7dR5L0Vh0oZ/DZ2ih/+D+VmWuphLSnCwX1s9twjNxxwvAh3H9x2RTr/ OGOw== X-Gm-Message-State: AOJu0YwBVU7NRwrjjrSpnWi/ftakrdXNiqjuqrg7jYQBeslcfHpmvZTr dYFEkx9IcPArL3Pbpczag/GOv2kxDC8= X-Google-Smtp-Source: AGHT+IHGxs5mKwV0SIfJTy/1PmcUSW12rF0lShgSZsg2OVE3kvZusMC/81y+OW2KnKIPoi+leXoxBg== X-Received: by 2002:a05:6a21:78aa:b0:141:69d:8041 with SMTP id bf42-20020a056a2178aa00b00141069d8041mr3176731pzc.48.1697826826732; Fri, 20 Oct 2023 11:33:46 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:46 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 4/5] mm/khugepaged: Convert alloc_charge_hpage() to use folios Date: Fri, 20 Oct 2023 11:33:30 -0700 Message-Id: <20231020183331.10770-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2F90A8001E X-Stat-Signature: x349ypst3yxnn3z5o8ituce5ftziimo5 X-Rspam-User: X-HE-Tag: 1697826827-306497 X-HE-Meta: U2FsdGVkX1+eLYkH+qGCcsRglHCfmjgFAFf73ajAPiu65iTQO3Fn2Zj+IUeHvqTrumX4sBwhPs5fDQuaJZ/gqAyhygdanRnKDM4+TtfZnd8Zri6yBvfWEaSZuzSCdTYQwUupS/3Utkz/sc+gZiowRSu+dmGpj0RYMmEa9/7vK6XtRE8ZJ//ncQOHhJDtEdSYYWZKdsJ9jBOtO1luyMpmLDhAoduh+Za0yFlk16QHvkWWrlZQs0R7b4Rm+2fB9o9TLKXYh0eFimk+3RUrlAz0216tpUEsa/Wz2HrZxhX07BsIxDxP32zmeaNf9fdHHpcBzNF48mYs0/8YO6RdO9vFqRJhitFRhGLNYfLwjXc9bh2uxiJ1LweX37NdEeHCzQ+TfZ/OzL7JtrBLwrcFdcaQ7lIEjQ8yt4ir4XnGLMnUopZvXOt55nd2GUV0hzmv8p63ZgKyIptD0OLwzEY+EPihxxmjhxJuhvIKtPFs9lChH1ZoqKD0FqVVKZZZvha0xEwUhIFYyZs75SPw60W3Q1X+UURqwVJJlczkY+JSUSVKY1iRbPM/q1a7qBFRiE3OPthpdqmQ3ou1ajC5mn5vVSgaRPOhlQDe0Zb88Qu0Vyz56W2quLFGUnjUg9GUHZTq4QEw9m7zIc4g1XZAS0SI1k4Oxb3TJ0Jtkg1q02+v5mmKGDgklmV/Uq0V6eJG0ThdOaOqSXJe+s5xZTwB/SGJDRxLapgYQ0wwVryLFelqEqWKpph9xX3+KcahpuqLU3ijXHAAkwzty13t9z5xbsUJBJuvvMtndsk0T0rHmzKv3I0jyTzdY9JntGgXWZTcqf9gGahWi78SO3fRgdjBCBYRHzhBD13sj5PQF3KWCZMe2WYrZurLtcU7QN4qg+95HlD46bDzUPtXLJyUOIucS1E5RkuCC0/+ZdRetfbyhlbhl0cVvZbKcN081BmAxBM4p3w7gLHr3bi97IHE+kzdOnGlu4X 9TmAdrUg KD9JhgBQ9fHwgl11/GnCXWCcCMJN5M47E6x/Kl3BhCYF8pSCipOyjomSZ1GGFO12M8dXWB8rv26xmd/Sg4/NytjgYyQB8YrN+iF/cjdOqI6myK/o7HO65HrC1JKY9e+EpHX7kT11L7kjfvr+dXxbzMhckH9Dh3UkiYi5KYRXs1WAXQQsxNEX9Dzx7Xe0yPfmDMPap6TBUBCDftrmqDkgGYtEdX4ctLmrkEtOrbzI2LhMXAz5baCrW4nC6379wT9toGBh0OU4hgpBQzU0dDchdxPpoksBsERF6O0Mekf97HERbtwAswB/u9QItjQa+YkvjclYq+8tDAGke36BWH3H7VMctSf75VFh51X+1b6G+FAiuIsdS3GuLJHPuk4twopjAsGi85eo/RvzGO1sMM3hFNerXIyLH6ZsTbU8MXq+F2OzE8eXSQIQ0wIh935Yeg9GIHi71WAOqqKDri8784/Fc5gmYLoPkoS5DY44EaMYDIYrGmQR/sL4NJDam1C5EPSxgft62CJ/zqk5iE+RLF8NUOwH37xPmz8HTabogrYPsttL0fwSEytt/eJgcZv2trbWf1XwKz2EEhkxcZrg/C8zn0FmwGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Also remove count_memcg_page_event now that its last caller no longer uses it and reword hpage_collapse_alloc_page() to hpage_collapse_alloc_folio(). This removes 1 call to compound_head() and helps convert khugepaged to use folios throughout. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- include/linux/memcontrol.h | 14 -------------- mm/khugepaged.c | 17 ++++++++++------- 2 files changed, 10 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..3126bde982e8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1080,15 +1080,6 @@ static inline void count_memcg_events(struct mem_cgroup *memcg, local_irq_restore(flags); } -static inline void count_memcg_page_event(struct page *page, - enum vm_event_item idx) -{ - struct mem_cgroup *memcg = page_memcg(page); - - if (memcg) - count_memcg_events(memcg, idx, 1); -} - static inline void count_memcg_folio_events(struct folio *folio, enum vm_event_item idx, unsigned long nr) { @@ -1565,11 +1556,6 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, { } -static inline void count_memcg_page_event(struct page *page, - int idx) -{ -} - static inline void count_memcg_folio_events(struct folio *folio, enum vm_event_item idx, unsigned long nr) { diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9efd8ff68f06..6a7184cd291b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -888,16 +888,16 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) } #endif -static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node, +static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, int node, nodemask_t *nmask) { - *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask); - if (unlikely(!*hpage)) { + *folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask); + + if (unlikely(!*folio)) { count_vm_event(THP_COLLAPSE_ALLOC_FAILED); return false; } - folio_prep_large_rmappable((struct folio *)*hpage); count_vm_event(THP_COLLAPSE_ALLOC); return true; } @@ -1064,17 +1064,20 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, int node = hpage_collapse_find_target_node(cc); struct folio *folio; - if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) + if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) { + *hpage = NULL; return SCAN_ALLOC_HUGE_PAGE_FAIL; + } - folio = page_folio(*hpage); if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *hpage = NULL; return SCAN_CGROUP_CHARGE_FAIL; } - count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC); + count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1); + + *hpage = folio_page(folio, 0); return SCAN_SUCCEED; } From patchwork Fri Oct 20 18:33:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13431021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F331BC0032E for ; Fri, 20 Oct 2023 18:33:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39FF1800CA; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3518280044; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F097800CA; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0959D80044 for ; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D679CA0882 for ; Fri, 20 Oct 2023 18:33:50 +0000 (UTC) X-FDA: 81366688620.06.296988C Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf02.hostedemail.com (Postfix) with ESMTP id 162E080022 for ; Fri, 20 Oct 2023 18:33:48 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hHmOFlfD; spf=pass (imf02.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=Re7idhDgGRNh9/tOJGq89jRRg9yjSnl7lkEGKkT4Bo0dUVC0HMNEXUH5xmO56mGldKj0Bs 0Rt4xTg2WNjzocrxzRX8p8CfpCUwVnJR8UlW/ovrTl0pBut/MxpTQfdjJWHIXruX9SSeXb v7oheC75gFn+cmjD0iUxEo2eLHU5xSw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826829; a=rsa-sha256; cv=none; b=GeKf7LDKi/hirdj+pJKqlH4NgTCXeWMJOWKCKri9zl3Ua2K4igXzpD+Ptx46JZZcEyK9Nr SXXa9IXknYh7yQ724C0YipAa7Fe8RzUM6JdNBtYI8k+DCTuJ6PkeyvjU24OPfov9Re3ONU YxG2hYOAlAcxAOXYRFc0HJ5/Pd7JjGI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hHmOFlfD; spf=pass (imf02.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6b1d1099a84so1066413b3a.1 for ; Fri, 20 Oct 2023 11:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826828; x=1698431628; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=hHmOFlfDlbqklrfab9qHnY0CQNI5g5ayjRBBP8gQfvnJHfetoKpp43pfd2Qm9zfQAp CEOT5G8GgzPXhCxqJixNl3rMfFwYV/Cz1uG5upE2bhlWzqjZBshxV9SWdKVmLPm3kWCp CN9nfPyIwfzqN7ztMd8ShSMTbXiN9xDX+HUNFSEGqeFBkGMjjGvBjB9a+xcdKU7gnAs7 P/vDRxwZCmIkm+2XVm+RXkmBItuP9itCRtIeDa3UOe8aDWDwI5gqgMA+H3nsXXe9zsKq CZaGu0gwNLEpiZXBDbmY80iLGg5CHXR88sx3ld4uXiMzwoT+fkvVHeDOFqfHP5DdvN+i rlVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826828; x=1698431628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=BjuS8+t382Tb7z2URqH6/Ne0uDxmckge+ZxpKr3D5z2akClMNZUtf3qijD1HBjPBsu ij1fRzWRx18am+lcoterofMr+DmFd/cFa3e1GMbxx3SWZs0YL29D8V62nxw7IbetQOjL wHpGh3f2fxCywDJzN02ob6Wo+8FPxCWBa5TlqrVgtL5dGkhbaAr4NiF1akMXryxWotD/ YE7pa6ofA3JRPudPxmAucV3p3LER+HIt67Qi6ZaH+znHPcHkwx0fkl6W4eOvcAlQHdRA BTzBvTV+zbNd0xNrIuVF+u0mi37Jr0YcLZVkj1J8puk9Q9Wi9+s9LtkOh15RuLWBb07m dCbQ== X-Gm-Message-State: AOJu0YzcCFtpWCYJb7EghSXj11Um6lBRfqwXXANtVTIsgsYndZUemlWt PJHkRuK1Af1nYcAZEPu4N7FJ96m9fTw= X-Google-Smtp-Source: AGHT+IFoA9q8+nleOsVaWYGKJMR5PUXVnKaxoWBcmjk+PdUFvUSMwibx6e8j5iJCbuEQcfjDcMuY/A== X-Received: by 2002:a05:6a00:1ad1:b0:6bd:a7ea:5c7e with SMTP id f17-20020a056a001ad100b006bda7ea5c7emr2751158pfv.11.1697826827696; Fri, 20 Oct 2023 11:33:47 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:47 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Fri, 20 Oct 2023 11:33:31 -0700 Message-Id: <20231020183331.10770-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 162E080022 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 3jor3tr4w5nm4wrz4nx86iac8j6qgraq X-HE-Tag: 1697826828-604334 X-HE-Meta: U2FsdGVkX1+DVhrfUp+KioBTN7/8xFUi5n/DmQSbwMoMpW+lflMMeJgs22Ti8ILU/nVO4oRj68G36QJgd5fetkiR+D56Ft3sYqgalUi9600u7w3BOAbXU/fSR7r8ERhLX5q0yVFnn5yvPnGNwzym1OS6h0TIQKkVK1zQ97OEpQSc/svegY/S5Dgwt6oKP1WkrinoqrCgH6NjmYo57kc3cHzH1660AXMA/a31othmWHBo4bD7VdqBUCyIl6Pf0j+pHQRHHXsVO7skaejy+S68T1M0b+nGGO2uNbafdjBoSWrTTthrVfUr90edq688WEzw0xvSm1VOM7AxuhutN9ihO5ZpUccCzAhtxZ87DVlmYqAWXBB91MO0AjGLKsjU/n1KSBhC5KKQ1ptEQUTJigGyuTH+2iD2+5wFCmajMLVHfsZ8YN5OKLta+Jxrt1qVOVpF4jIYOR/YWA400pI72Wpc9Q0DfJME3RzzeNUo+uGNi+6Gk6N9vcHrm7N9PGL35zYIoB9aa5VzEk8dP36m/n4Z7qPLXsc1aZ8CQON9N7grMKlbTBuWWt7LFkYeb4EI6sRbjzHKFjJfRyyq+t3tx4zZxb78X7zQbMM1UJDIy/ypJcgbxn1Hz+7F5texU+Zy3PUuDXNhloEC7rmDkmcJuKQBfrcUWAuHvMt9KrhmMiEvJLOsGGjoizflb5052lOhn2DNhH6CbU9wCsHIxqR4otUewKotX/5fSYqABn664hQLKZq3+UWZIvuoyCMqdBM8H+xhXJEhLu/M5X7zBlWuNDJfkKq/ME3YlS49Qv0zwhbJ2JfeKZQ5CPKWLJUOeEE+HJweJp3JNBXk1OKzv5d7VaKHY+t2RwB/zzU+MSfuQODGk2w1mAWG91r5GcLrCpZKrA9dMTOY0TiATm5jGWkb8sugicf+70YxjUBIe4gxxrCyx7dh+q9IRCeLUhalxKxaht+c7gDfSF4+hB1itq0sa30 +dqoT1E/ BQvefgK/XWbDzvbleTH/gpb/e+bEfr3KWlEJ0gObgTJo9s8KhzzilL4JX7kl3f0jFRv+/NCZM4UYu2pT4ikz+qfTSyUio5mrN58QkfiUGaFUy5Ju3tgngWlpIi/2RP98tTDvH4hdnyeGDRpec9jM5v+VtyjcK44ddm889MHOfcjJ0wdABvWOPdmKzcWlXvEl0mT8I9thTkaEfroFD9qBNtdUdlrNR+IPCzGu/JbB+UNJMSjTW0PnBKV3/R4exVBodNF1aLA87lep1ps0QaT72mGvmialJuKPn8y5zYnft+lyKu3xw7m/c+VbLPMX+1ASJ3vgG7FKkFf1QEfctlfv8fbHU4bBGlUGGIuqlEbNw3yVI09zoE6AWwa6Sr3FPybSURRAcx3lYv9lj9sc4iQ2XOsFWvoTeEPlphmHSOuMlMu2oz1B+zIWg69CI0bjDvlaPLzwWHIxVJEsy1ebxnqQ2ORB8c9UT/WY+xA/yw+GCmK1QxlTWSlPCseayaveR2Sr90EZ5zgGM2TH2y+tpwTMp5RimqyxZoUZ7eBtY9gYwGjGb89zXUmvmJq7G38WoawykTSIIWSt7Gsxc5JRmK5PlJJxvhQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6a7184cd291b..bc2d8ff269c7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1477,7 +1477,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool notified = false; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml = NULL, *ptl; @@ -1510,19 +1510,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; - hpage = find_lock_page(vma->vm_file->f_mapping, + folio = filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (IS_ERR(folio)) return SCAN_PAGE_NULL; - if (!PageHead(hpage)) { - result = SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) != HPAGE_PMD_ORDER) { + if (folio_order(folio) != HPAGE_PMD_ORDER) { result = SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1536,13 +1531,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1567,7 +1562,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; } @@ -1582,7 +1577,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1606,7 +1601,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1615,7 +1610,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page = vm_normal_page(vma, addr, ptent); - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; /* @@ -1634,8 +1629,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } /* step 4: remove empty page table */ @@ -1659,14 +1654,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1674,9 +1669,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; }