From patchwork Fri Oct 20 18:33:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13431017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96F20C001DF for ; Fri, 20 Oct 2023 18:33:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E04E8D0003; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9908780044; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76E658D01D5; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6A5DA8D0003 for ; Fri, 20 Oct 2023 14:33:47 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3D11D120891 for ; Fri, 20 Oct 2023 18:33:47 +0000 (UTC) X-FDA: 81366688494.14.7ABA9F9 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 5E1DFA0016 for ; Fri, 20 Oct 2023 18:33:45 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VWDnFcpM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=lPDhgJYKGty2jMkkb+ajWc8zWl3FcTfQUOWxQ0qdQPfL7hW4iIjp1ptORNZs9kKIpuUHTA VLWmZuLTWiLuj8DEatwYuKvaiPOTvinPrZT0vPo6b8XM645xjhgFpjrhl+43EL2k8rJWO8 Er+5YYGWeGMIOWPaJx2XNmm5KXDBE7c= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VWDnFcpM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826825; a=rsa-sha256; cv=none; b=nsBvWPBPFAq4G0e5HQPjxq4UozmxV8SQP+0QrmfYkgUjc+M8xAhE39WKtR1UWpmTrx0FdO chRt4xFAhby8L1kV2SOBP114190apMI6NGQceZVeJCFvMKo1r7GIpfATL/NJvVJuUEx7b4 bWCr7SWgfPrwXmTMDPuvyVz3OnyKkcc= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6b5af4662b7so1057133b3a.3 for ; Fri, 20 Oct 2023 11:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826824; x=1698431624; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=VWDnFcpM1SUsjPJSsaaxM8IpoLObUAr5eJlSNgQbKY/3harjkn5PfWox8kN1+4zm3w zaiCQhlbjqjC05sIKPFixdl3BYw0H9G1aSdg9ir8G6dw3UGJCSen2HluvGXvAyx3fWOK +chsgEkStjWH1uDZKJCHLUi/xw9F4K59Z9eq8m5hqaMStLJFnAFuqvr/+TinM15qfVD5 JiY9oIXYSG+ButSrNFiYOmJ4YM3Rv8YSWRNQEaMgSJDzZ9VFsu+RGnoz5DQKWB/LD6Qz UmAo3NdLA48tsYQSsZ7KmA1tpOOXPpunNf7kTcw0zfhpBazaFlbnZSgTWSvvmT4bJdzg EJDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826824; x=1698431624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=VQK7aOqTn6h/lq0eKmvBFrQQsKvibLn0/rXrFdgGOJoKIsXbLVpX1Nr34SgvYcRt7q /Co1BRpsYKtl+yJqYOuzLKdVClZK+LvShdTtVCE1f0/xKLbUf8/wXJY4bd9ZujV20tf3 xg6A3ezg3nK5z3LjKM5rZZp+IbCiAS4YmFAcGHZZ36WGNeSnxQ4wolch1dYzxQxEiKBx BYnFd6q0tnjlHCqE4L4J9uHr4s+v3IoyyDIMMuJR1LkpteM7+aUC6h6nTuC1FROHPIdU gZTgd9qkrr9pD4jegMC5ffqPPLdtu4ExoehSGxdLlwtJcbw0PA+siOUDRnozinf2Vj8B iwKQ== X-Gm-Message-State: AOJu0Yz+Ab75rW9Aw0hF9lqanAf0clMJD8wLx2XQrzuovrMty5x6Ms0t ythadthhAcV3sk779mqWC7ZI9obsN4I= X-Google-Smtp-Source: AGHT+IGHwdGrDYHJU3ar4omw5xYzuwotpwlRbNDAoj/IKDWChBLhpJBOvVjPjaCZMcDR1LHeQ2sY9A== X-Received: by 2002:a05:6a20:158c:b0:17c:cd46:73d with SMTP id h12-20020a056a20158c00b0017ccd46073dmr386382pzj.20.1697826823892; Fri, 20 Oct 2023 11:33:43 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" , Matthew Wilcox Subject: [PATCH v3 1/5] mm/khugepaged: Convert __collapse_huge_page_isolate() to use folios Date: Fri, 20 Oct 2023 11:33:27 -0700 Message-Id: <20231020183331.10770-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5E1DFA0016 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: tu6omiutfgaop4rzijrc9rz7cqkcwj3j X-HE-Tag: 1697826825-45399 X-HE-Meta: U2FsdGVkX19uxjvEp+yMOJmfCeHC4xBoNKWO879Aeyh/+2N9PWjvbnZtFR2JBJsyPMv4AA1j+7blyX4S35/9pjaiIbuM5R5J+tOq8qIf24P0RMBGjOys19SOTTAkKZ8bLaXlj+0M4Oe5soRhfwlG7Eh9Trxb8vXF1PVWOfX/hvpQHtlEcAteccyi3YzXgti8kvysmTKqis44PSSNmYWBba7aVNNEjpIdMuljYk9D0MpylHpT+5MirOtmG5jOSMGe61CJFLhmDVtl7mBLCcVUe8h6hyAWAWKH0FaGEWp512pueu2oc3D5jqriGll9XTphsfX1HkJ7vbpdz8y9P7vAovJA02u6zv8rchPs5gb3jr2Ff3OJTfI9U+EjJWTtmIAOvvDzarMsC+kgU1PiUnZTE5dBLNuBu4CdU+BbENARBf4PhbNv+ZBzNQ8HOwCoTuE3QZlLdYa8tIk0w6KNJlMsAPSLhzHFn5YaaAcGmyrasNf5c/VbpruYvIMv8MQvMr//2ixARmsg524x+CDckiYR3SZ2Af1ZncV8FSgZkp8/12n9M/whkqsF5doGkd6myp/TittrHpl3D2P7qVxmHW7mZK/pytte3jBsvzAzsir5jeX1nkSs1fZwIWQnB3uhru83M/qBKWZFsv6Hc/GmwCanRQafQ5QowKtuker6RVJnG83odte7Qa072DTFY+PocM/FqBQJIQJqIZFuEXmQPYraqbsRGWz6iHZuSoFD6/DruzEadMwX8iaiAJdJF0PT9JpcUVQrqMZe0NOiXmWtOtNk8IBnmgc+pd2xsS9RsL9Xo4wXOc/n68AyfFQzJAmISJaJHlkmBZOTx3noh6OGPM4KsoUPBa2QzGAqzSLXiBmA+4SkQAwaxsc9ELDhFUKjVfROdvwjh6tsfeRBgByK4YeDDCq7YXg5c47W8POf3rE1+nI7JWyNrZMbftuL707ghr7JAVfkONDkF6P30wH687v MQCoQXGJ ydxvOqiCIn4xA6s7wq/EnXaQfFDE3LOGt0oAOmHNFk3fXgO2Iucwc/A2SHLg36VtGyxYtxgtdzN+rpqU1BaxsNBDW/7WPT8VPma0SUJTecDofz0zNlqltPs/+QxpcPtPnsoaCuNTJSEecO1Fy1Nc+nV4HU3KM9XTMs7wNR4g1buQTUQbgYVEhQd79n/3miTV58wBNrkENUwM2U8qjyp0YG2DLRIp2xCwnoF8nPlreVpV5SCzlxHQcphwyMDqOGUze7PwA6s1caZ1xDC+qvzh6ycjIdvwFjZuZtJACftjVZw1HEOUQBZfxsMzz6GxpvJTGmAz2D5Af5e1UKY5qZ5dSDxgKS2c1U46foBQh2jP5e8gmxErdDWiHpTJeZqoYYcy+d3JLkHFX43qNvd0mrYhFgoYtIBFQGgHRJqYDqT/94oZNkWJb2s+NW3yvCMyt/szy044pNMLx+70v4Qe3L7/UR07CfgB/YHmUvLVsrI369CwNsKNO953BEYdZX7NVxGDt1ZhnlhHkB4RPcQhLw6f6nUyZXU0DkFPODcuxY3JM6h7mSpDQqDrMNSBUvpQRtHCMExETXOnCudEqlcT8ASwiePbcV+d6IcjxmEaqaABZq1W5PUXS7u/lHQxLf8DhP+z3dPKilyzhGFJ6w3carzucFsBsAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replaces 11 calls to compound_head() with 1, and removes 1348 bytes of kernel text. Reviewed-by: Matthew Wilcox (Oracle) Signed-off-by: Vishal Moola (Oracle) Reviewed-by: David Hildenbrand Reviewed-by: Yang Shi --- mm/khugepaged.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88433cc25d8a..500756604488 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -542,6 +542,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, struct list_head *compound_pagelist) { struct page *page = NULL; + struct folio *folio = NULL; pte_t *_pte; int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; bool writable = false; @@ -576,7 +577,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, goto out; } - VM_BUG_ON_PAGE(!PageAnon(page), page); + folio = page_folio(page); + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); if (page_mapcount(page) > 1) { ++shared; @@ -588,16 +590,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } } - if (PageCompound(page)) { - struct page *p; - page = compound_head(page); + if (folio_test_large(folio)) { + struct folio *f; /* * Check if we have dealt with the compound page * already */ - list_for_each_entry(p, compound_pagelist, lru) { - if (page == p) + list_for_each_entry(f, compound_pagelist, lru) { + if (folio == f) goto next; } } @@ -608,7 +609,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * is needed to serialize against split_huge_page * when invoked from the VM. */ - if (!trylock_page(page)) { + if (!folio_trylock(folio)) { result = SCAN_PAGE_LOCK; goto out; } @@ -624,8 +625,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(page)) { - unlock_page(page); + if (!is_refcount_suitable(&folio->page)) { + folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; } @@ -634,27 +635,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * Isolate the page to avoid collapsing an hugepage * currently in use by the VM. */ - if (!isolate_lru_page(page)) { - unlock_page(page); + if (!folio_isolate_lru(folio)) { + folio_unlock(folio); result = SCAN_DEL_PAGE_LRU; goto out; } - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - compound_nr(page)); - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageLRU(page), page); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - if (PageCompound(page)) - list_add_tail(&page->lru, compound_pagelist); + if (folio_test_large(folio)) + list_add_tail(&folio->lru, compound_pagelist); next: /* * If collapse was initiated by khugepaged, check that there is * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; @@ -668,13 +669,13 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } out: release_pte_pages(pte, _pte, compound_pagelist); - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; }