From patchwork Mon Apr 3 20:18:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21005C761AF for ; Mon, 3 Apr 2023 20:19:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 733AB6B0078; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BCD16B007B; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E78D6B007D; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 39AA16B0078 for ; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EBCB2140D60 for ; Mon, 3 Apr 2023 20:19:06 +0000 (UTC) X-FDA: 80641193892.12.A8F9575 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf29.hostedemail.com (Postfix) with ESMTP id DA49F120009 for ; Mon, 3 Apr 2023 20:19:04 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=T1hetx0z; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=sXEJDM7K; spf=pass (imf29.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553145; a=rsa-sha256; cv=none; b=g7a7MROYBSZ6+OUuaE3PjRs5z1C+J60Wn/p3qzPD5qE7ZmdUM3JBxzTy9zREypO9Igxk2t Q3bUaQCJrdMgkTmeAHGumYnzRwgwD1W+Hu6OCCk+zYfmIckoLgrt/g1NUD6faiCxGl/aFH fIW1ZmfWIbAO963VuvtEj9NW/gaBf3Q= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=T1hetx0z; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=sXEJDM7K; spf=pass (imf29.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553145; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Yr+uFjBJL/0qCDkJMJuMYA8+4U49oyS8kQRIZ4QnQ48=; b=kbLNV3DQZ6vbgy9I3jmEgfgY3tmogKd8S0B98E/mZafZx3isWUF8g1QizUEUNBvkrCE0dK hxLE2jmpXyhDxnBdUdrzf+nZ6r4S3M63qh4Z8i/86+3rURqGkHtIIO16IMzMUuIh7B/JLr ShVrWs2HH2PpbqIYN5bkRNJvYx1VY3c= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 3D5C55C0077; Mon, 3 Apr 2023 16:19:04 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 03 Apr 2023 16:19:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553144; x=1680639544; bh=Yr+uFjBJL/0qCDkJMJuMYA8+4U49oyS8kQR IZ4QnQ48=; b=T1hetx0z1aJK2O8iBf1XvbqRSf2LI5NbMRJIZbD7uStCjfYTwWu LJGnCoWYcbaqpoiUrntElnYceDUE5dsSh5Ku+hBbgvndzIdg1RoPOlpMQblhIZEj Gj9N8O5K106uCqM170WRD7HzCaPjrz+LrVUSe85Y2mDdavpdEAQwrn4H17Ft3gZV 4i1+Y9GQdBSJAMy41kpcfLa/idIEkFFLl3C4BwKjk/9zNnr+jhhFMl8lURKOJePr WUFqi+HV8uSXTAynXN9pclE+IFtgQQ5tFcHeF3Hdw+WTPtgbvTJf/fpYERORW3zX w9l3SsHoMN9zpXheXCSyiSDa8U3ezvl9WFQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553144; x=1680639544; bh=Yr+uFjBJL/0qCDkJMJuMYA8+4U49oyS8kQR IZ4QnQ48=; b=sXEJDM7KET2Hz2FHW38uAUF1gFqDFEOhVRtiWCqszLoqwd1QDsa XpDhu5g5zULZEpHyMzC0XVamd3bmgfGCMbzr8ic5lVeNWN+70QI+kqdzTDEJVJlQ lF5zFAKYBWZxkHlKZ/5jObCMfNlOGCacD3+oDf582G15K8Un92Y2c5oKigRFMxqa ZXeJYHTlEteuxWOv4laEB07bJgRtG5KeX65vfCX09baLQsAuneHGOMJxAm6qe1z0 +eOvk/9J9zol6wjU0NxawPfRaYWPNDFUvbyjqYXN0nwNts9yMrGDgrS/wuklV3dq WcJctYgIhA+94PoWt+7EwS1GfgHOq/BOyvA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:03 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 1/7] mm/memcg: use order instead of nr in split_page_memcg() Date: Mon, 3 Apr 2023 16:18:33 -0400 Message-Id: <20230403201839.4097845-2-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: DA49F120009 X-Rspamd-Server: rspam01 X-Stat-Signature: imqab9j4j1pum77y3dhxgmcm5nekraba X-HE-Tag: 1680553144-69691 X-HE-Meta: U2FsdGVkX18/o6jjvejz8mYXoDiqXpFtwaVf/38RNi0T39W+htHwjTYMp7PGqJNUFjnNWOkNSvZrg62nrjFcZKqrTqkaLG6Q9HiRbsOI64uzkRI92lmZOOUkj3AK6iH38zZX/Xye6BPj5/bxukX7Laei6f7msb8uJGB1SijVKKUAxZr0Z/9X5tYkaE5QwspQkWB6LqOFObVUmXJPEgYOllZsLUNntNdV8HXGD3ZkdK7tpOMctvUOqF6pNPOB7Arfk0iczOmlzpDlJzLdy4yWY8ytCEg2Mdw3mVQfkIBdaGcix+mE608quGbZS8a02jvqHx2usb3xZ8kzMtwqptd7VCCAhJ94M5QMD2Grx45vAFWKJTkC0FhZWcysWrUSuf+Tt0vkBHhsFv+tZqP6RKbobnW4QZ2/1OUzAT02ekg+s0C5neCBuwyXzDQCBEGWDEV8+E3IpfYaVIRV0bj7zoCeztL1ufQbl9mAy7azI8IQHLeYTgynTetoV0/Dfili39edmDbwqV4Kqxab4Wp2+th8gH/Ay4SU2evzKzMvFOtxy0Mzba2Qt27R8Yn9sowVznUILXrGR/m9vV98yOKL1VFgtnOx4ntzaFD6ef5oq/rmJCK+Kngn8pIcjIJlTCHItmTkq8jQ6SQFYHNF28eavZ/r8axQk5QQT4WTEWvrS+jC300RHeeTHsvXrQjm8AOkmTJAU10jxwxS+xHy+cFHKSAkw09UvXt1Ut0RXM5yDkBR2bG3Bw+WYauyEO958WKFZX1BuCtc2usHJIxeexBIjSxfjkEfispBlM+wx6oxL35wLwCqPCcIUqkyKEDug01oH3KnIYF8IRt2+tharUcsncRAwKpYuCSUBpJzJECGgCd07gm9qeCu/q57wTti3TLQKVfspU+qc317hyNUzTZykLaRzWx7WVwvd+OjK2nsUXJFVSO5VPL9t2kui9pdXwXaJpua0P/wGjjuBX1JpCai4pP QOCqPJnN tajx/4cQ04FiSebrQHdJd7uXou6zcluRQOOIr6qMxWvC47l65Vb+4V20hm9meCgpbjiFWmbQRfuiED57pytAuSRaOgKAf8SIPMcur+rkZype+hy4jIbSJdt25dNAeaC+zEmHnbx6xaCln10gKMMakFGLG1VzxoxlW78r7e/5bBpozyKsXZxbzrxAcyRSStnDdSQeV2+biQP5ra/G4Z5e7XvUkiTbUtHEV8f+Ot+3ecgM8e0GQHhHxkIus0j1ki3xs0L3l2KlfhSPLO4RP+HeVytx9qt1/bFX4qr2vsBa67GQiCI3bSNcfclkmwIIQUyyin9o23niEvzKxizCuygIwiuFaBWRwtaUV3o62wsKOeu02DJRy/ES7Jz+Y38DymqZ+q/8VJ1OKP7zqfknH0m4mHcAjcPoWr1a/FXPTqTUB06YHYCorr8zH99X0UyT7QAR+Oid8Tp6x72ydkrAFijfCkdP1tA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan We do not have non power of two pages, using nr is error prone if nr is not power-of-two. Use page order instead. Signed-off-by: Zi Yan --- include/linux/memcontrol.h | 4 ++-- mm/huge_memory.c | 3 ++- mm/memcontrol.c | 3 ++- mm/page_alloc.c | 4 ++-- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index aa69ea98e2d8..e06a61ea4fc1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1151,7 +1151,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, rcu_read_unlock(); } -void split_page_memcg(struct page *head, unsigned int nr); +void split_page_memcg(struct page *head, int order); unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, gfp_t gfp_mask, @@ -1588,7 +1588,7 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) { } -static inline void split_page_memcg(struct page *head, unsigned int nr) +static inline void split_page_memcg(struct page *head, int order) { } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 81a5689806af..3bb003eb80a3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2512,10 +2512,11 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); + int order = folio_order(folio); int i; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, nr); + split_page_memcg(head, order); if (PageAnon(head) && PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 681e7528a714..cab2828e188d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3414,11 +3414,12 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) /* * Because page_memcg(head) is not set on tails, set it now. */ -void split_page_memcg(struct page *head, unsigned int nr) +void split_page_memcg(struct page *head, int order) { struct folio *folio = page_folio(head); struct mem_cgroup *memcg = folio_memcg(folio); int i; + unsigned int nr = 1 << order; if (mem_cgroup_disabled() || !memcg) return; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0767dd6bc5ba..d84b121d1e03 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2781,7 +2781,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); - split_page_memcg(page, 1 << order); + split_page_memcg(page, order); } EXPORT_SYMBOL_GPL(split_page); @@ -4997,7 +4997,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); - split_page_memcg(page, 1 << order); + split_page_memcg(page, order); while (page < --last) set_page_refcounted(last); From patchwork Mon Apr 3 20:18:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8EA1C77B6C for ; Mon, 3 Apr 2023 20:19:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3A0B6B007B; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0FAE6B007D; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3A756B007E; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AECCC6B007B for ; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 81CBC1C614E for ; Mon, 3 Apr 2023 20:19:07 +0000 (UTC) X-FDA: 80641193934.04.66413CA Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf08.hostedemail.com (Postfix) with ESMTP id 7D195160008 for ; Mon, 3 Apr 2023 20:19:05 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=KUwl7e1a; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="H1/eBrev"; spf=pass (imf08.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553145; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0+1ccQx8AuLgSqfkQ9qAEQMi/lCcYIby1X8SrOj7vms=; b=o4jQi68n77Fi+KXE468hDlIx4Q1YSBKMI16LitWEvL+HPwnuaikXqPr/UXp21KWCvJQnMU x1iDQ5lh43XLfItJU7RbzsrlyGnOTL9DDbWeU2DoYTIYCZlF7Phn6lyrxN1S8kQT2r9zDU Hdk+73Cd0myPBtkkYC7LfsLa0e7OSlU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=KUwl7e1a; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="H1/eBrev"; spf=pass (imf08.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553145; a=rsa-sha256; cv=none; b=bf2f8g8j2H+xap+iVxwOMuxQ/3dj+CksmqR5JC3XguR6tFvmcZIZUfH3NPkH1WC8Y6hWA5 2Tizu+wIXfQMX4r6HUg5tn1nUnv00bsYWLgksqChDRWDW2r57CMHEqpd/zofA8wU7jMNM4 iSk2TZ+NX4sAr90/hQd9xI3aUVrk5Rk= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id DB5045C00B4; Mon, 3 Apr 2023 16:19:04 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Apr 2023 16:19:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553144; x=1680639544; bh=0+1ccQx8AuLgSqfkQ9qAEQMi/lCcYIby1X8 SrOj7vms=; b=KUwl7e1aBsGFQzG5D0syDzxeWbJ3GF3fvksO8O/36dURW8FqYMv SZmNQeib2JGKzildDgbeERIuwZu+26p3d5QGEMmQLqnbTgdbRtGECgXXZ4R1y4f7 G5LavpoRwEp5Me3Ov0l6eM6zMpcRj5OTFUUuG1A5NHmVuQTS5Vh3PMSSsYLusCMa 9xLD7t4TNL+IZBKoI7BB+cqubyRv/opGyrQyeYTx2n5SDve5SCbWTHgi1ATf5h5E LFRzjk+tkSmVdnujGmCNvrqKW/XYpCWCWWMTFXZe9Py4pgqZ97R8zFuxL0GF5AxG gQVRC/CejfgmQW8A1yjsYXroTbXvrNSnHqQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553144; x=1680639544; bh=0+1ccQx8AuLgSqfkQ9qAEQMi/lCcYIby1X8 SrOj7vms=; b=H1/eBrevt0g1ytaLRjDooVqSD9DjKANjiNur8QMCYBINWoRKxAM wav5DMHaX3kn6pl8Xh3A9XcfMNeATpKNRSU4Llq+Fm1pC+f0IT2AMLvBUO3/jkYD KcILNwTsTu0YBLszjyQhHNAb8XNM0C0vWoAdo6PBjz+PT1qwwJlzRAkN9LeiRtaC fKd5PlLQI14MgXjnnRGBUiRWTKUgaGGd7k/3QwPxo5fZcDb/fBEy7Cuz6lcSrRg2 Ac3xfSiB5kDm6eQrFul6iQyFkISdftmQfbhidUFwBtoLvV0pdTvnq3BaDJdMhhyk dTqdvWK7XYpUy7rnUlvCLDgjwlyVC/tT3CQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:04 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 2/7] mm/page_owner: use order instead of nr in split_page_owner() Date: Mon, 3 Apr 2023 16:18:34 -0400 Message-Id: <20230403201839.4097845-3-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7D195160008 X-Stat-Signature: dtjh3d6ca6i9pu3a5yme3kh746wduwwo X-Rspam-User: X-HE-Tag: 1680553145-510926 X-HE-Meta: U2FsdGVkX1/t99R6TkN3KXk5XBP3bNaKyS1M2ON4lMxZojJHA4ebzq5kVb1odmEjj3NVxbx3OunmYZvlwcI5piU4aGGKM6gyL1/RcUUdw53YFQJkIVVr6kLlTa1iNdqg+MPpfbOSpLQsc0p7YpDHn4mGZRYSMGv0EPt+veBH0bb9c37lLUG+75T0wMAasz6NqYCU+CoQ8NYtF0eFGt2XVAMM8SL91aLsN1Rs6ylIGpU5ON06IfSY7NixNdO+dICsge+W6G6DLGRov2Uz0yp4B1N30dFh77UMWyOdEDe41LaGPx/xL4+X6EDVl8mDlRef3TniMVEOaD3aOhlEgA03bt1yYAWBE+9mni28JEbq/nKmn43MUOvCa0b4whK5UgbKam/SVQagbjDxb1Qejlg8nrMCrHz5tMQeMxvbvfvliJTupsVF6Qkjwi5b5aKag7da0+oClYqDozNNp1qSEdxu+VwFvJEVs7ivnJCX20QHklqVxwdRVEWtORKZNnojd1BLjGTMO0w8Th2Nt2u2d4+eAfhuP9lNIYRUwUeTYd0aMTKkKFWg1jU6xEUtI33GyR3WzRP69/kGpa5W6NbeqpTDpYKQ160X8JpHpGEzdHyIzdrWDiqNGPKhVRA/ShZdhDca8xaQRKnnIoINbae5wHQVmZ599xgTcPnLYAn+L9O5SXXx/cGxh9JUTNrTOGHwKoiai0v6966FvK4NYDLS7pLWl/IC7ypI5YZRrrApPO62X25VWEtQ2J06/aTbkpYYnblcroqb47l3X6yBhFDwC3IM6nFH6O2sSFjPqbldwTYOU/LJQhSbFKzFCh1UpEh/00E82+FYm2FNlUYprp5zUAICYQysX9YJNOWgmadrY2nHrWoEtg7TkmyMkr6/DTq7DFtVbiekFU5e59p4XcKCW7V4nmUlvxCQ/ggey5k38Zz3jd08rmDUJzuHLKDAwLvOpyPxyLfBKxVYEIZmnoSy3PB lmltG4/S myVZDRjftmYV7u5IHuZj+s4LIbJ19+WhaGjv2ZOgaU/WGWPNxPeVZqKDjBUKF4Viy4nhf23rSrA7x2N8I2mp9R5c2M1Lc2LmnCcwq5DFIP6k3Pabfx3jy4HryfhqL2jmlX9XgUDUJtAw+kQZ+XODxaChEpKOEBm+uodhlRrVKcGj202hoH3Mrp6a8/cly3P1CcR4MZVe0ecuNbpTZajIqsfbFll0MDgwqnssaWdtEAFASDxeq5Lv7DFPtZCZ6wEG2xeYcjXnqEdss4w8NyQQgKXBO3kWQ5eGyz9OmLKvwvqlVL3gB72RlxFrjkzAET6S0nmJOMoE5IFRLIUPBHcgu/i34j7AZ/wTCRPs5b2kREPH/LyPH3ULEEgp3pREJMUmKTyUTvcZC1DbV0WhaSong1zehjii6Cs0Rl8e2Caorn4b211dWibqpcbzy+G1wT8NCr1dWNY76bS4NPjByvCN3Nnw9tA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan We do not have non power of two pages, using nr is error prone if nr is not power-of-two. Use page order instead. Signed-off-by: Zi Yan --- include/linux/page_owner.h | 8 ++++---- mm/huge_memory.c | 2 +- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 3 ++- 4 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h index 119a0c9d2a8b..d7878523adfc 100644 --- a/include/linux/page_owner.h +++ b/include/linux/page_owner.h @@ -11,7 +11,7 @@ extern struct page_ext_operations page_owner_ops; extern void __reset_page_owner(struct page *page, unsigned short order); extern void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask); -extern void __split_page_owner(struct page *page, unsigned int nr); +extern void __split_page_owner(struct page *page, int order); extern void __folio_copy_owner(struct folio *newfolio, struct folio *old); extern void __set_page_owner_migrate_reason(struct page *page, int reason); extern void __dump_page_owner(const struct page *page); @@ -31,10 +31,10 @@ static inline void set_page_owner(struct page *page, __set_page_owner(page, order, gfp_mask); } -static inline void split_page_owner(struct page *page, unsigned int nr) +static inline void split_page_owner(struct page *page, int order) { if (static_branch_unlikely(&page_owner_inited)) - __split_page_owner(page, nr); + __split_page_owner(page, order); } static inline void folio_copy_owner(struct folio *newfolio, struct folio *old) { @@ -60,7 +60,7 @@ static inline void set_page_owner(struct page *page, { } static inline void split_page_owner(struct page *page, - unsigned short order) + int order) { } static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3bb003eb80a3..a21921c90b21 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2557,7 +2557,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, nr); + split_page_owner(head, order); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d84b121d1e03..d537828bc4be 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2780,7 +2780,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); - split_page_owner(page, 1 << order); + split_page_owner(page, order); split_page_memcg(page, order); } EXPORT_SYMBOL_GPL(split_page); @@ -4996,7 +4996,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *page = virt_to_page((void *)addr); struct page *last = page + nr; - split_page_owner(page, 1 << order); + split_page_owner(page, order); split_page_memcg(page, order); while (page < --last) set_page_refcounted(last); diff --git a/mm/page_owner.c b/mm/page_owner.c index 31169b3e7f06..64233b5b09d5 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -211,11 +211,12 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) page_ext_put(page_ext); } -void __split_page_owner(struct page *page, unsigned int nr) +void __split_page_owner(struct page *page, int order) { int i; struct page_ext *page_ext = page_ext_get(page); struct page_owner *page_owner; + unsigned int nr = 1 << order; if (unlikely(!page_ext)) return; From patchwork Mon Apr 3 20:18:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC52AC77B6E for ; Mon, 3 Apr 2023 20:19:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F23E86B007D; Mon, 3 Apr 2023 16:19:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAB706B007E; Mon, 3 Apr 2023 16:19:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D25616B0080; Mon, 3 Apr 2023 16:19:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C03316B007D for ; Mon, 3 Apr 2023 16:19:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 96285140D55 for ; Mon, 3 Apr 2023 20:19:08 +0000 (UTC) X-FDA: 80641193976.29.67A145F Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf04.hostedemail.com (Postfix) with ESMTP id 5D6774000C for ; Mon, 3 Apr 2023 20:19:06 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=wpT7oFFV; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=LYskHU92; spf=pass (imf04.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553146; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tqMM6rDGpE/k/HR8iQPMcAVX1ewArHJYKHHgDopL/rw=; b=CeMwXIFVcD+gARs4QZwIv+Xde9p2IPtaaknzfa7hCHxJaFqMgCbquov/C7yT6x5KsrMS8R iN0J3rBHldAASxqg3ZEw9a6I4EDBQPlJjNHCdYzhUMeGrXULT7Ga+w9sqTD2zJnhVQRQRH r5HteSU62Ss+cJS+3kC6eqpTaqUYET4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=wpT7oFFV; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=LYskHU92; spf=pass (imf04.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553146; a=rsa-sha256; cv=none; b=TqCFyMJjkFwhlQqYpbG7O65grP1d0+pRKwJV4n8CHWgijMilAJR0KvgXomZmbAqhbC40Dv Ns9rbdHapa8q3LS/r/F7UtDKfEDVM4hbiT4L0VCznDKwxD3V/8qrutohIdhdsf8DITeJTN J+mYHNt5dJkB7wuDLJpDSgbdSMS70bk= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id A5D195C00EF; Mon, 3 Apr 2023 16:19:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Apr 2023 16:19:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553145; x=1680639545; bh=tqMM6rDGpE/k/HR8iQPMcAVX1ewArHJYKHH gDopL/rw=; b=wpT7oFFVwFLTvzK53KS7nLSNOrYngSEpjzH/GvK/K+BtVaTnpwt 5WptsXlmg5k1kHLlbg5xu14eV0qlNbBd8gKVeJQHTdF8nAi5J7qTHAnRPJ8WkyJ5 IM9PFa3jP7ubn+t+BAIhdOwqHPOlwttbXaVr1p7t6lOj15mPajIvUXCIL2Sio5aJ hGikiNIKLcCAViAhW1uJT/yV9qiUES8t+qtBuHOLdB3wGuQXtfJIeNELBUosj/dT FndH5PU1wXGL0OImBgdZf7xV+22idrUM35tByYvVKQ+fBvTUbOu4AZSUiwxVN8Fz ExcR6OmDo7o0h3U0wdY/tjzRDw/49GTN4wg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553145; x=1680639545; bh=tqMM6rDGpE/k/HR8iQPMcAVX1ewArHJYKHH gDopL/rw=; b=LYskHU926t8XHzszhhYJpGuo0avkqMeVWPCHXgMQ1zrW18Crwes /MlQVH3/ujVcFW3iiLUeBHWiio8Bj2I/5qY3hhTyAidX47B+PqaIcC25o+ael2Ka HKLuQ1jmke6snUE9DJ+wg7j7nyjeRwyYf8VBURZBHV5eCtEhaEBQhO5Qn+9QCXLB +W7XYiey+UO5E7r077araEX7XMF+Y8s82cA06dSNDCrGjO1Mlqlu4plM/xpTW1h9 jfVnd0SC6mmC1iNWLGfo13aa3SE8Cmd5JjwCCz62OCwWj3dJC/eUqz6y2L1poqdm bVgaEEcpq4+70fJp5F5yX6gMtoMRpAVpstA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:04 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 3/7] mm: memcg: make memcg huge page split support any order split. Date: Mon, 3 Apr 2023 16:18:35 -0400 Message-Id: <20230403201839.4097845-4-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5D6774000C X-Stat-Signature: oec6prhpwmck3xtjuds6oex7ywa49hnu X-Rspam-User: X-HE-Tag: 1680553146-823185 X-HE-Meta: U2FsdGVkX197aYiRctxHOlOAQaebQemiykL/FJ+HQBM9D1U7aSjvB4pKFVoLzOsDfAxVPp54lm0RtYVBshEoUYdF0BETe5OUCeahrwQePbm/WXWHsy4Z5W7jNMGZ5UUMiAGF+5GpQNnoyd6q/ukPjg7+AqiXIPwYbiyR9QeOXAH3tHuqAfG9PsgrGSgKjEC3K2vmK4d1aSUQDz9nbnhuocnNaRScdtQJibYblOCEJQr/6CKOYynO5IOwW/GzOeoDrrB9HviM+8y0jW4LjDz5gRWOwN5YUA2EjCbF0AEgO5MKYBGaGVprsiRga8RgHttCo17XgKV3L5RfOi6v9AZbNUyl7dA+l01xNdDP7zr58SUb2st5iwMPbowk6WNy7S2/bHh2uWRmsDlJB6XXbSjCQLS20n+YB4mPAEGUFMy9Y+xBGvCXdwzH1abIzTjYB/O68BmY4aMCOVkP/ON86FPgHo3m8cQjRNGpsGv6B/w6r3Z3iLFyq9J3EC6Bmo4Mzi+FIIUvAP1lRPOvu7p6Seilb+yhozeSnzMk3cK9CN6TvlAA4o/iZc+f6ULLwgDqyQcbizOv2onFhn5wWMufvxKUh6v1qenDKMMC3T4dREneTPjE8CJh32RTS+CZBoEmpBEfAmqJ8MLNSTnDU8jsl2WodFUWzrbjxc4KKgOX6+vQRuf69cLxOOUWKwoW28sNkkcptFh+Nei0K5ZE4+EH7AyaxzafPPTZqsg8fyhKW2MBvxEJHxWCfZNSf7Q6QTlVgj1VPrFvpvL9ZU/KY+u0QgnU6nrnQNcJ0UrU4pJ/GeJaogcFyfa0JicpV39pMrzZNfw2Kq8dGpTs3TJGmdgdQz51yFb8HjhDxFoNeJSzTCTdkvaJoDoUN85OrLNKIqYDYaeeVfDP9DiBfl9RwMRut+Gd5jIXWXRtGETc0IPBn75Gu/iDP0lJliQZ2FQGodBOhWachuOIXXVUZlvIZzoZPpe DRBIQLhA 0ftPD9XHORYXhmpQr74bSe0PTjMvOYb+XzX67mLc9wDgnEoT16Z5CvL4FBj/CLpOmjEgizUo9+PP1c/OdA42LKnnTl3AeSp+8x6EDBpygxn5pA0p1rMnjmGAIvTqFP3iChw3h5832IVrVOWJwHQVTqlir64PH/T2bIlHXPd2NdQb7+TsWHWZgZYLXruzjiWZUM3jqemOCvSdmYj06j6FTdbmYM78OyUrYcJTVuJkcalhDeNWwNpvvxK5xwZ/Ajd3CKxzLfdhdyDxLNzgDOT6bhyAKLYePC3ZBD3mskWuf0h6U+CjHSo8NOSUcz+QAPaX4Ow2JntLDXsD5jzz/cb1js/vpnmTv9FlxjokN3kqA3LhMC1qYvPr9nKkOvg1+VLe+F3Zit1VHQpygn1NLr4HA0g3ntZa8y7ippoQg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan It sets memcg information for the pages after the split. A new parameter new_order is added to tell the order of subpages in the new page, always 0 for now. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan --- include/linux/memcontrol.h | 4 ++-- mm/huge_memory.c | 2 +- mm/memcontrol.c | 11 ++++++----- mm/page_alloc.c | 4 ++-- 4 files changed, 11 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e06a61ea4fc1..1633c00fe393 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1151,7 +1151,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, rcu_read_unlock(); } -void split_page_memcg(struct page *head, int order); +void split_page_memcg(struct page *head, int old_order, int new_order); unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, gfp_t gfp_mask, @@ -1588,7 +1588,7 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) { } -static inline void split_page_memcg(struct page *head, int order) +static inline void split_page_memcg(struct page *head, int old_order, int new_order) { } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a21921c90b21..106cde74d933 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2516,7 +2516,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, int i; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order); + split_page_memcg(head, order, 0); if (PageAnon(head) && PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cab2828e188d..93ae37f90c84 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3414,23 +3414,24 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) /* * Because page_memcg(head) is not set on tails, set it now. */ -void split_page_memcg(struct page *head, int order) +void split_page_memcg(struct page *head, int old_order, int new_order) { struct folio *folio = page_folio(head); struct mem_cgroup *memcg = folio_memcg(folio); int i; - unsigned int nr = 1 << order; + unsigned int old_nr = 1 << old_order; + unsigned int new_nr = 1 << new_order; if (mem_cgroup_disabled() || !memcg) return; - for (i = 1; i < nr; i++) + for (i = new_nr; i < old_nr; i += new_nr) folio_page(folio, i)->memcg_data = folio->memcg_data; if (folio_memcg_kmem(folio)) - obj_cgroup_get_many(__folio_objcg(folio), nr - 1); + obj_cgroup_get_many(__folio_objcg(folio), old_nr / new_nr - 1); else - css_get_many(&memcg->css, nr - 1); + css_get_many(&memcg->css, old_nr / new_nr - 1); } #ifdef CONFIG_SWAP diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d537828bc4be..ef559795525b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2781,7 +2781,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, order); - split_page_memcg(page, order); + split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4997,7 +4997,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, order); - split_page_memcg(page, order); + split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last); From patchwork Mon Apr 3 20:18:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85A9CC761AF for ; Mon, 3 Apr 2023 20:19:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B519C6B007E; Mon, 3 Apr 2023 16:19:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B01D36B0081; Mon, 3 Apr 2023 16:19:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DFC3900002; Mon, 3 Apr 2023 16:19:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 74EB56B007E for ; Mon, 3 Apr 2023 16:19:09 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4765EABDA6 for ; Mon, 3 Apr 2023 20:19:09 +0000 (UTC) X-FDA: 80641194018.22.67C776C Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf16.hostedemail.com (Postfix) with ESMTP id 255F8180004 for ; Mon, 3 Apr 2023 20:19:06 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b="nz/6OkYj"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=N+H9wuit; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf16.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553147; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Fyda6DdrS5eQAxPutJLWU7rPSEY+1UbDEAwHHIWfJU=; b=lPkB7Vw4YEezYRokPRhnGUh4oGdkB7FLLB5PuVYQjSfmdz6juxHZEPiIrf81pR3Yk0frJ/ RlmolN7GocGj5iGGCvfMHBjUp0dz2KQzP5ZN/OhJPZhcwDcTsQw8kY53w+A02CjrCuIBYV hsKyH7u+8ZRfmWsza/zIdcNt6qfhDVo= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b="nz/6OkYj"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=N+H9wuit; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf16.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553147; a=rsa-sha256; cv=none; b=bkNcPuZwyScil41+JG2GDbc36AAdlca8/c4jJMjwK1XCvTC6lM3XwM79jRvH8jmMdrhm5D O3YolR5QKq0ETZ558f/UB9qlVpf3jQpub5S06s1sLuP5NWeI7BmVJn8NgK4762mfUtKuuu joXxFiOL+JP3fmkz4Tank+ifPsEEfXo= Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 7DE075C01D4; Mon, 3 Apr 2023 16:19:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 03 Apr 2023 16:19:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553146; x=1680639546; bh=/Fyda6DdrS5eQAxPutJLWU7rPSEY+1UbDEA wHHIWfJU=; b=nz/6OkYjLbCZX+ixfPsX979MsgC+XBBrD4ldyEfa/BZo60sfHsp z/ityPXAu8IL/HKTy3HBCG3CJn6kCJWKjTcsf8y3lAiZlzSuMmKnUEhuRZajXGeo tm2Kslm94MRfstlsxAeWjXTfyZYyVK9zt9N1AocGdn5hWS/lNGatQfldBWmMfkZS dOsE2fMrcgFu2KJ5PFqRylGCWWeHqjxUVlcKN2T2tKEagRTtNo0GpgDrgipRLGEE +uADnX0J4NstF3xR7llS00aEZXSLFOK9VLm52ytHn+6BbMXN8Oq61caDfYiAZUSp b58E1ySQUSi59D7M3+ZB4fBnCBL1mqAQBgQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553146; x=1680639546; bh=/Fyda6DdrS5eQAxPutJLWU7rPSEY+1UbDEA wHHIWfJU=; b=N+H9wuitSDaOyzs7mde+61NwgHhTfzBzdBVkWra5jyk0ReWUrrX rpK0QabHxDx8blNWX6nbr262B1krQ92qPL7mwEguLhuNhmPrm/bGm1wjNQXDFWF5 5+X8ZeX1Y5++dP1TmUj5rMaUdNfk3xLEcfeJo4fK80qP7qBe0RHLPdJw8eWK3Urr srsfAYspeZ7kyUHsa48gvNQWMyr8JbcqY474rlV3zpWAxJ078HENXuMZS8oA7+D4 a7tOAGxMgd5F1W5jZmpCDFo3sqqIIZk+Y+ftLUZ/esjhYpq1Pb18aoXSKFee6O7I 3U0L7NaJHqMvYxnRnaJoEOMMiF9uRp7atDw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudeglecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:05 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 4/7] mm: page_owner: add support for splitting to any order in split page_owner. Date: Mon, 3 Apr 2023 16:18:36 -0400 Message-Id: <20230403201839.4097845-5-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 255F8180004 X-Stat-Signature: me1z3ns9bk69m18m7j19my5jreqimxwd X-HE-Tag: 1680553146-620027 X-HE-Meta: U2FsdGVkX19h0jMAJw5xqE2hTT9zY0RXMmvxDcZMrowgYeO50My1IH+yBa189i577XYqKtSZNz5nJOfCr0wyfLKhOg12LFtgYeHXSTB9iA8I1AjA989W4h2vDnWMYnl0At8as+3ydatU0huTIzqcj+iOyXt/i6qR5euyHCibAkCygqt9HBpcGCBTmtjCI3OrPu+nsT7SgNVM6r8f7SO0dv3JGEzQ/uGj6x9U/5lC4gxlghY5xezQabQ1ZVZgKAVqEG2T1h85+Eh0LWMac/qqcg3XYbcXRvhkAFXEZUMI5VkcDJ7VRxueiSGLSRBhJL0fbQpBQnJ4iWo7h4Prgz18jMIdiDX3KlBOlWphyGx0QIBzzkwJG6V/mICK+59vB5sj3WCvrBry08csHNAeFqD6iJJwIYO50Zx54SCgRXXIO1g0qBdRlGKVang7lclztOs58yq2NDcUBxeeafmLA+fUWCotU2gZ2QZdPfCeUJ6xBfTe3UNMqXboPzBItBajwIPyeAU8/P+cKVM13a/80q3MPQ9MahfUsAurpODc7lIeEwpSfoTONhWrbFQM93Ezj3h4zLgI5wzQ7sMuvW+XDukkHKy7DlUIIEA9ScWqctm2mqpQRhRwcTEVnlVinvj7G9IdTuHCYHEpmUbecbBZj8KNgwsq+OlANtp393pkmdAGwn6Kb/Zy1hvNgKWlOmNFauQYNa4Oem/7IQWeMnPcyGswYohJDpQAh5D+3gb25SQvpnTmJpeY3YE2TVxm5X/JFPHpZqTRXUDrIACzYexR1KvoCdguvEyI1cee6BXE6dJgY3j3u5QfqtG1ZsnMncj6UzM7jS7iTC0yr8B0N7bQ4VvXLd/nVoqq7VyUP2uFRf67ORFCTSOODXTkPnV5PhwDcAviPUICVF4MK7s1r7iysqF6MOepCE/kUPXNHQk5iRh0xOhU81C4F9ztWm0YKJzJfdr59CNDsSMGi1O8XdbxuPR 5yo5digs 3RzgB7TRvFyY6sTdLkm0oxLqDDXyrTadiOChSaX9+B8b/m65OkO8ukw1zpgOfuv2VqRVEw2lfsqBF3l+eeKAjDuo7wPD1my2phEYA3JwDsptVwuk5tYdBUYRvnNr9LkUVioq70OQIjlmLi6v1eYbbCrsoa2BtRXq9OHfhl0qWm21jCCdxcqCz4nU8lQ62tnMpFAO5445MEx8GmwniD+G1uYUaP/lCJG8UN55FOySCrywglK4cIdbZ2bDLrFbTPZ/253RQHHRPb79QILfFVE+HOE979fnRdr5T5RSH37WdOHkL0w2dAyfkatlTAIAoDpQEDQfQMnSQxRHoBXraE/qiR6QtWlUbF4B8MzqMVy9Be7Zcx5HpoD41veKWLFQ5RRcQ/7ZNTVU8Tm4AtfyIftHFEIU062cbbUyydYGy6M9SVBtFXscFwZVAZjBYj/8HV3x6/by7P6kNCvpqfv/JD9dX7CjSGAID2zm+wtCSkBW7iTWC3RNab9lSAd5CFUMmLGO+/XdeLyQJwuwsSpwVEEhv+1kGVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan It adds a new_order parameter to set new page order in page owner. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan --- include/linux/page_owner.h | 10 +++++----- mm/huge_memory.c | 2 +- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 9 +++++---- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h index d7878523adfc..a784ba69f67f 100644 --- a/include/linux/page_owner.h +++ b/include/linux/page_owner.h @@ -11,7 +11,7 @@ extern struct page_ext_operations page_owner_ops; extern void __reset_page_owner(struct page *page, unsigned short order); extern void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask); -extern void __split_page_owner(struct page *page, int order); +extern void __split_page_owner(struct page *page, int old_order, int new_order); extern void __folio_copy_owner(struct folio *newfolio, struct folio *old); extern void __set_page_owner_migrate_reason(struct page *page, int reason); extern void __dump_page_owner(const struct page *page); @@ -31,10 +31,10 @@ static inline void set_page_owner(struct page *page, __set_page_owner(page, order, gfp_mask); } -static inline void split_page_owner(struct page *page, int order) +static inline void split_page_owner(struct page *page, int old_order, int new_order) { if (static_branch_unlikely(&page_owner_inited)) - __split_page_owner(page, order); + __split_page_owner(page, old_order, new_order); } static inline void folio_copy_owner(struct folio *newfolio, struct folio *old) { @@ -56,11 +56,11 @@ static inline void reset_page_owner(struct page *page, unsigned short order) { } static inline void set_page_owner(struct page *page, - unsigned int order, gfp_t gfp_mask) + unsigned short order, gfp_t gfp_mask) { } static inline void split_page_owner(struct page *page, - int order) + int old_order, int new_order) { } static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 106cde74d933..f8a8a72b207d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2557,7 +2557,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, order); + split_page_owner(head, order, 0); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ef559795525b..4845ff6c4223 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2780,7 +2780,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); - split_page_owner(page, order); + split_page_owner(page, order, 0); split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4996,7 +4996,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *page = virt_to_page((void *)addr); struct page *last = page + nr; - split_page_owner(page, order); + split_page_owner(page, order, 0); split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last); diff --git a/mm/page_owner.c b/mm/page_owner.c index 64233b5b09d5..72244a4f1a31 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -211,19 +211,20 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) page_ext_put(page_ext); } -void __split_page_owner(struct page *page, int order) +void __split_page_owner(struct page *page, int old_order, int new_order) { int i; struct page_ext *page_ext = page_ext_get(page); struct page_owner *page_owner; - unsigned int nr = 1 << order; + unsigned int old_nr = 1 << old_order; + unsigned int new_nr = 1 << new_order; if (unlikely(!page_ext)) return; - for (i = 0; i < nr; i++) { + for (i = 0; i < old_nr; i += new_nr) { page_owner = get_page_owner(page_ext); - page_owner->order = 0; + page_owner->order = new_order; page_ext = page_ext_next(page_ext); } page_ext_put(page_ext); From patchwork Mon Apr 3 20:18:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74D31C77B6E for ; Mon, 3 Apr 2023 20:19:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 909696B0082; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8435F6B0083; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BCC5900002; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 57EA56B0082 for ; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1D49012073A for ; Mon, 3 Apr 2023 20:19:10 +0000 (UTC) X-FDA: 80641194060.15.8549960 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf20.hostedemail.com (Postfix) with ESMTP id 074CF1C000D for ; Mon, 3 Apr 2023 20:19:07 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=bYdnJ5QB; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Z9UKPLXQ; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf20.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553148; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aksTP0wsXOFiVzEDaAS1LD1g3uMK0049FZHtL5zgUuw=; b=aoub4Wxc4tj+rh6Ee8Hjas2SkG75L/Jfy6oaHlnKexrYCeqrq4897LMS9zSdSGmh34ouf2 HomhDERgfl0pi3lmWkFkInsxkH5qN8XL+r/kQx4YUnTYkpVPYCpWAjhN4a45HVA3aaTBeh LgnwBT8UuhPhWkamWe52t96v38j8Qe8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=bYdnJ5QB; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Z9UKPLXQ; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf20.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553148; a=rsa-sha256; cv=none; b=DhyT4IP1Pgj0NaHDRLTgERDpBvla6+pFJguRvFWkXqkm1N51UGhFBVqxViYAFZt7jHoWBm w+Kve04+JpHR3BeooEGSmMOCgXOG5oIhR60vR09ZuWFKqsbsvCSuLmeYdYw60Ft7GpjO1o UzKqvnzHuXAgbwuOptbDUITvaPbgJeU= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 5773E5C01DB; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 03 Apr 2023 16:19:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553147; x=1680639547; bh=aksTP0wsXOFiVzEDaAS1LD1g3uMK0049FZH tL5zgUuw=; b=bYdnJ5QBNBJu3ZJXstuE2YWzj77VMJt3PPI5qvUwBTn5Mfvy6UP OvN6rJlshP4EJE/ppLHgOV+6woXIBSQn48IXJBGdXnJBpsEQHQJGoByyWYll0HHY IoT23jhtTOFUuPptabv2bP1LT88IASZKqFaa3iDH8/QpPiz7yYRMWoNrrTnkQnc/ izzYHzEcu1F887QvGyepRAojOkwFJA060KYDciqk34TXsBVs38lV/RBCMrSh+paD h28yFdOzDRiLcuwb3EGHDTULuUf6ngjiv3J1O1bhKeGEDX8cwCh1+p0HBDJYnSGd 2Lefjnz0M4pVUhTwN+EkqA2U1q8tcVXqx8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553147; x=1680639547; bh=aksTP0wsXOFiVzEDaAS1LD1g3uMK0049FZH tL5zgUuw=; b=Z9UKPLXQGewR1AAUglbCW2XnniXMqwNYdZYYljH4El75rSI1PZS UisJFudWXq9mKF5YWxQGeDbsTI3kcylaQu8cD9jIMvrjR1JTZ4uYIdddaQoCOBT4 cevzys70blRonh504OZXmmz7vVCsXJohYm7my1N69wJOt9scgSZpx/TUlh6FygrX EyOzUs9NNS7PdWLiUTkeKNJYzCqQNIaKVca0dxWYTW7YSGbBFsaHV9RlLFtX1NF5 Cr87vOYzTa08mYpes6IbdLJilwUdy/lcke2pugO1hCalzuuLbtIi+9OfWf7mGZ0s erknzwyW48OM86MsafK1s2+8rzpSiNmYX5A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:06 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 5/7] mm: thp: split huge page to any lower order pages. Date: Mon, 3 Apr 2023 16:18:37 -0400 Message-Id: <20230403201839.4097845-6-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 074CF1C000D X-Stat-Signature: 6exw6osywpre7jw4afouh67wafus483e X-HE-Tag: 1680553147-895859 X-HE-Meta: U2FsdGVkX1+dOUaJnQWRHmK2S9GQLqn+0hAefzsu5kaXqwRO78iGBJs5ApBWfK2Om6EpMLka2Mw9m9/eEePoo7m3768+y4GxQf3+nZifqUhSvzlrmTdAorsIJ0sKucE6l8ToucSIlDUsZ9IlUlEx0OFkrsQeiUrLepxKbvdJ4Xo3WSJBqQnNrb4ery+Qk7tWGkQTgPFBxQlJcCmj9rmpPfFDBayAfIR22zA3ALZ4zwh2DTqpyoegcS6KCf1OG3XVqVT9M8gjnl1MRATHmBWWMnGYgjo6sg83muB+QpvrMFQ/tHoOAw6f36gHi2ol6n2KLYVQJIMPvjwMsw6IScJNN8BTH7qxH57IwCzplNMpd1NbJpFl82z2jWD5YtDKm2dZBFavElkjWwxmhTOWx9AFbE0yF7uI2tudw2n/9Hzh0hxtp7QAZGS91ys3LAM1tNuFKrPFPDIBsw+qPZRRVa4hIPGT9LmgvoaAijXFxqXq9omzS516s4r6L/G6hM0RiuNlRa1oHhvx30srX7tTdNNMiQpd82A0nRNraMJpvhc+aMGXgTF6/gcAWqr5quZIo7+2TzvFjiTB1ToGugxei3aXFEa6kfICFTRPHkWm30ESiqnOjO9pDjpx+nBKxFvOZK+h5Dcb+JpL3ZLtqo9ngZKLZOhMQESpgbbOyTBP1U2b5fZujOusoaE3v71fOQMayqzmW4uVmLZ3bh68W/5BqbzvjFXylBm5fqtxOPiNY7QiMPDRchxR9ehGIdVN2wQBxdzi8LKdCf5o36A9lMetS78gH1mOc0o2g1/B/lIpmm8yzd5ic31/MKJ3Mr0fzKqhJtXe6rgPpkGYYjGRQqq5bLg1io10MX/0TubCoak/WcDs/kwQlfj4Y9jDnjHfK4tAQXOEURebRtjZPh1IMKuAag70ew9nOpUxji/suwxAWqlF6uL7g0IqWw1Ha4/mamW+MeUVp/sxulEJ47waTZq4UXs gkK80kCS AcTHIhv2DXQnMcUXGtslwnmWmyJORQH1uLTT0/D4hCy7/RHsXuhXc/6IRAp/7rSNoZBy+VfuW0iTAD5LHOPNP2ma/n+99VH4sNYRU20Xvhi9vlyQsKb5PCl+mYF984d/dcreL2XZJoqjvZkkFoFixSjYUDtrrce1YP9P1pUHXDkVN1I8Eq8uIiaAPzU3T8rcSH68rrMGAXg9qlRaVaVBrDVrB4iz7TmYnlxPvL2yUbt0OGoX8olhYVQibGsFgWlEYiio4Bzw6PtH/OmJTRSIHExuLPaQyn/P8Wk6pYwqS32ODT0QME89OvmVqCA78ntGJOxw+/PjLn2VW1C8T3SvAeje8jHRuMOONCZrO8VBFqcP9zOgyjQWwvbeXbZuhxIn/5oaVw+sKpPMa8oyqQt/GKOSlGvL0EQiLXI1x X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan To split a THP to any lower order pages, we need to reform THPs on subpages at given order and add page refcount based on the new page order. Also we need to reinitialize page_deferred_list after removing the page from the split_queue, otherwise a subsequent split will see list corruption when checking the page_deferred_list again. It has many uses, like minimizing the number of pages after truncating a huge pagecache page. For anonymous THPs, we can only split them to order-0 like before until we add support for any size anonymous THPs. Signed-off-by: Zi Yan --- include/linux/huge_mm.h | 10 ++-- mm/huge_memory.c | 102 +++++++++++++++++++++++++++++----------- 2 files changed, 81 insertions(+), 31 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 20284387b841..32c91e1b59cd 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -147,10 +147,11 @@ void prep_transhuge_page(struct page *page); void free_transhuge_page(struct page *page); bool can_split_folio(struct folio *folio, int *pextra_pins); -int split_huge_page_to_list(struct page *page, struct list_head *list); +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order); static inline int split_huge_page(struct page *page) { - return split_huge_page_to_list(page, NULL); + return split_huge_page_to_list_to_order(page, NULL, 0); } void deferred_split_folio(struct folio *folio); @@ -297,7 +298,8 @@ can_split_folio(struct folio *folio, int *pextra_pins) return false; } static inline int -split_huge_page_to_list(struct page *page, struct list_head *list) +split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { return 0; } @@ -397,7 +399,7 @@ static inline bool thp_migration_supported(void) static inline int split_folio_to_list(struct folio *folio, struct list_head *list) { - return split_huge_page_to_list(&folio->page, list); + return split_huge_page_to_list_to_order(&folio->page, list, 0); } static inline int split_folio(struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f8a8a72b207d..619d25278340 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2359,11 +2359,13 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, static void unmap_folio(struct folio *folio) { - enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC; + enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SYNC; VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (folio_test_pmd_mappable(folio)) + ttu_flags |= TTU_SPLIT_HUGE_PMD; + /* * Anon pages need migration entries to preserve them, but file * pages can simply be left unmapped, then faulted back on demand. @@ -2395,7 +2397,6 @@ static void lru_add_page_tail(struct page *head, struct page *tail, struct lruvec *lruvec, struct list_head *list) { VM_BUG_ON_PAGE(!PageHead(head), head); - VM_BUG_ON_PAGE(PageCompound(tail), head); VM_BUG_ON_PAGE(PageLRU(tail), head); lockdep_assert_held(&lruvec->lru_lock); @@ -2416,7 +2417,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail, } static void __split_huge_page_tail(struct page *head, int tail, - struct lruvec *lruvec, struct list_head *list) + struct lruvec *lruvec, struct list_head *list, unsigned int new_order) { struct page *page_tail = head + tail; @@ -2483,10 +2484,15 @@ static void __split_huge_page_tail(struct page *head, int tail, * which needs correct compound_head(). */ clear_compound_head(page_tail); + if (new_order) { + prep_compound_page(page_tail, new_order); + prep_transhuge_page(page_tail); + } /* Finally unfreeze refcount. Additional reference from page cache. */ - page_ref_unfreeze(page_tail, 1 + (!PageAnon(head) || - PageSwapCache(head))); + page_ref_unfreeze(page_tail, 1 + ((!PageAnon(head) || + PageSwapCache(head)) ? + thp_nr_pages(page_tail) : 0)); if (page_is_young(head)) set_page_young(page_tail); @@ -2504,7 +2510,7 @@ static void __split_huge_page_tail(struct page *head, int tail, } static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end) + pgoff_t end, unsigned int new_order) { struct folio *folio = page_folio(page); struct page *head = &folio->page; @@ -2512,11 +2518,12 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); + unsigned int new_nr = 1 << new_order; int order = folio_order(folio); int i; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order, 0); + split_page_memcg(head, order, new_order); if (PageAnon(head) && PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2531,14 +2538,14 @@ static void __split_huge_page(struct page *page, struct list_head *list, ClearPageHasHWPoisoned(head); - for (i = nr - 1; i >= 1; i--) { - __split_huge_page_tail(head, i, lruvec, list); + for (i = nr - new_nr; i >= new_nr; i -= new_nr) { + __split_huge_page_tail(head, i, lruvec, list, new_order); /* Some pages can be beyond EOF: drop them from page cache */ if (head[i].index >= end) { struct folio *tail = page_folio(head + i); if (shmem_mapping(head->mapping)) - shmem_uncharge(head->mapping->host, 1); + shmem_uncharge(head->mapping->host, new_nr); else if (folio_test_clear_dirty(tail)) folio_account_cleaned(tail, inode_to_wb(folio->mapping->host)); @@ -2548,29 +2555,38 @@ static void __split_huge_page(struct page *page, struct list_head *list, __xa_store(&head->mapping->i_pages, head[i].index, head + i, 0); } else if (swap_cache) { + /* + * split anonymous THPs (including swapped out ones) to + * non-zero order not supported + */ + VM_WARN_ONCE(new_order, + "Split swap-cached anon folio to non-0 order not supported"); __xa_store(&swap_cache->i_pages, offset + i, head + i, 0); } } - ClearPageCompound(head); + if (!new_order) + ClearPageCompound(head); + else + set_compound_order(head, new_order); unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, order, 0); + split_page_owner(head, order, new_order); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { /* Additional pin to swap cache */ if (PageSwapCache(head)) { - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&swap_cache->i_pages); } else { page_ref_inc(head); } } else { /* Additional pin to page cache */ - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&head->mapping->i_pages); } local_irq_enable(); @@ -2583,7 +2599,15 @@ static void __split_huge_page(struct page *page, struct list_head *list, split_swap_cluster(entry); } - for (i = 0; i < nr; i++) { + /* + * set page to its compound_head when split to non order-0 pages, so + * we can skip unlocking it below, since PG_locked is transferred to + * the compound_head of the page and the caller will unlock it. + */ + if (new_order) + page = compound_head(page); + + for (i = 0; i < nr; i += new_nr) { struct page *subpage = head + i; if (subpage == page) continue; @@ -2617,29 +2641,31 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) } /* - * This function splits huge page into normal pages. @page can point to any - * subpage of huge page to split. Split doesn't change the position of @page. + * This function splits huge page into pages in @new_order. @page can point to + * any subpage of huge page to split. Split doesn't change the position of + * @page. * * Only caller must hold pin on the @page, otherwise split fails with -EBUSY. * The huge page must be locked. * * If @list is null, tail pages will be added to LRU list, otherwise, to @list. * - * Both head page and tail pages will inherit mapping, flags, and so on from - * the hugepage. + * Pages in new_order will inherit mapping, flags, and so on from the hugepage. * - * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if - * they are not mapped. + * GUP pin and PG_locked transferred to @page or the compound page @page belongs + * to. Rest subpages can be freed if they are not mapped. * * Returns 0 if the hugepage is split successfully. * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under * us. */ -int split_huge_page_to_list(struct page *page, struct list_head *list) +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { struct folio *folio = page_folio(page); struct deferred_split *ds_queue = get_deferred_split_queue(folio); - XA_STATE(xas, &folio->mapping->i_pages, folio->index); + /* reset xarray order to new order after split */ + XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int extra_pins, ret; @@ -2649,6 +2675,18 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + /* Cannot split THP to order-1 (no order-1 THPs) */ + if (new_order == 1) { + VM_WARN_ONCE(1, "Cannot split to order-1 folio"); + return -EINVAL; + } + + /* Split anonymous folio to non-zero order not support */ + if (folio_test_anon(folio) && new_order) { + VM_WARN_ONCE(1, "Split anon folio to non-0 order not support"); + return -EINVAL; + } + is_hzp = is_huge_zero_page(&folio->page); VM_WARN_ON_ONCE_FOLIO(is_hzp, folio); if (is_hzp) @@ -2744,7 +2782,13 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (folio_ref_freeze(folio, 1 + extra_pins)) { if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; - list_del(&folio->_deferred_list); + /* + * Reinitialize page_deferred_list after removing the + * page from the split_queue, otherwise a subsequent + * split will see list corruption when checking the + * page_deferred_list. + */ + list_del_init(&folio->_deferred_list); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { @@ -2754,14 +2798,18 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (folio_test_swapbacked(folio)) { __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr); - } else { + } else if (!new_order) { + /* + * Decrease THP stats only if split to normal + * pages + */ __lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); } } - __split_huge_page(page, list, end); + __split_huge_page(page, list, end, new_order); ret = 0; } else { spin_unlock(&ds_queue->split_queue_lock); From patchwork Mon Apr 3 20:18:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24E4FC76188 for ; Mon, 3 Apr 2023 20:19:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C72E6B0083; Mon, 3 Apr 2023 16:19:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2546D900002; Mon, 3 Apr 2023 16:19:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F233A6B0087; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E38346B0083 for ; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C01BF40C0A for ; Mon, 3 Apr 2023 20:19:10 +0000 (UTC) X-FDA: 80641194060.11.3078782 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf06.hostedemail.com (Postfix) with ESMTP id A7297180012 for ; Mon, 3 Apr 2023 20:19:08 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=CknXjAfE; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=hNICcSGN; spf=pass (imf06.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553148; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n0I3rKbVmW11Ko5MdyrWnLEXy5CTcBBnWUTtQZPt4TU=; b=cCPJPrTngypI0Bx0TrU8IH2js75edLu72R2Kewu6MQkrc1/BKSiGyr+2ZkyOt0juhCSJQ6 ZdYs5GbbE8ksQrjaxoa9alQUm76ievRegrrGkfJt0QLbT/jYjKjUyvcUpd8rgo6jOItHBA o3E3tUjVb7e051HB1AAxIikohHDEKYo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=CknXjAfE; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=hNICcSGN; spf=pass (imf06.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553148; a=rsa-sha256; cv=none; b=g5IsDNUcmtNrfYTlSBnw8u35ZX7ItrIRP+F1MrJDtRs2xvkrYzf32UN1mT7+nc9MpKeQ91 Od4hucf6VkhZlBlILgbMHoCzT39zYpVQPpCCUUjjyj996Xp7NAIwreRq7RFgdSx2Hx3NeS kQdo/mTH9QdQdIzthXOn/956PduR3xA= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id F34585C012D; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Apr 2023 16:19:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553147; x=1680639547; bh=n0I3rKbVmW11Ko5MdyrWnLEXy5CTcBBnWUT tQZPt4TU=; b=CknXjAfEg+BlpRI36eBMzyBQgGufPONGhPvWy7s+gy/bhNUeDoy XrZZS36kGlcl7YvPRJZxh2QYUhLMrXCglMzsbdi6caUVkB32GPU136oGWOw9D7FN a8wMhxHbFNmlt1qrJ4P/OxqrHkHDhHurwSRTuukUPsm7F/OfAzu7PwJdwNe7hY1d dOGCKNRadN2+ZLi0jXCvtfukxAs0chGy99+JG1ewv3jNM2OLkjwCNlw0I19tiNiQ FnGOZayjMtLBHucHfEbFGOi/9g1Wg+PAq0cU0Ne/rfk5Pt2zZn8rEnuff6x7Lwtq wL12JuQLFZHNFUdi0ohPgNaCwuQhZRPtcow== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553147; x=1680639547; bh=n0I3rKbVmW11Ko5MdyrWnLEXy5CTcBBnWUT tQZPt4TU=; b=hNICcSGN1EXXDV3uYu5QpP5/qwn5lAAgv19FTzFTRbX5N0FlNAD +5Ly6h8+eRZyyrSEBPSN01hlDZia4jeE8G0YMvkRUcpZK9YRyKP43ep+HjxUeE20 azKRsRgrYvZ0GHFOvtBFsAnC1eAK617AZaxsoAL4N0/2EGRvbUPQr0Hgug9brVGx wq1vCYXFWpRBjdnVF9SmQ0OIxtvSSxi76UNt/wFGEF57YmmJAUg0kBVUaVJRBARw Peey3G6reNzAV18EFN4QhsoRg4Ow1+bNDTN5r17diEi2jc8HfCGGMbSUEL0f0Ikv GUwL+HnFRkobihhK9fbtiKJ902159F8YvgA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 6/7] mm: truncate: split huge page cache page to a non-zero order if possible. Date: Mon, 3 Apr 2023 16:18:38 -0400 Message-Id: <20230403201839.4097845-7-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A7297180012 X-Rspam-User: X-Stat-Signature: tngyg1c3j7qu4txesq6anyg1b9q6fnqp X-HE-Tag: 1680553148-20425 X-HE-Meta: U2FsdGVkX1+Ykmp6f3M4NrIQOG9Oilbu6pFuOtjDf3GCWOCFYT03UasjmDgfU5RmtpUs6D29hjZD43IoasIl8wIou8i9jPzylG2LoEl+BjhPTv0y1wL4DExQiiTlL2XktxnAt/52g4Y5ZzEh/Ko4tfBqc23en+TADEqo/whU2ZmPY+jKokMDtUCm1lEjMXacrQea6dR6kNoTGB52pj+/EAuuphypI95vwO8Y+srupBIXdTwhH65g4UGzzxE20GirUP9KFP4nDnkECDevenhg+9KRTWPN51dwAcYKoJ3glwUgCmriB7l6oe1RVs33a986yQx/iWwgBMtJYiPsixQzvpgtTggvm83gpyWXO4GyN3so2/a6dMkFl76ru6UdORSBqF9iKOPNkROMqLwRJgGPwQqIUBgtOj6UMeFCA6zCMFNh50uENkKoDCFZTjO1rTvrPTKTBrGhWVcns5R2KnN/a6rSMk3kuZ+t1HRDOoLWUMMNMZBNdXsME8dXqvXF1E0s2aUK25NiUUg+bXCe41gxNnh7I3xG0nJI+RDqNx4ZMWJhkyRfEj/5LfFCQN+veHQW4j8Rbp6O6t0/kr5sS+bbbKuyRHCG9Pxr5GIxoMxzlskjRh+54NGwRecTm+dAoHkPo7qF/hP5Pn/lTkOrDUYumzZfiW41Goeev4B+t8ImSnV/C5IXF5Wdm3faBmqERrAoXJR7R6H0CqVLpwA6hOQM73f4WJmAaR5KGRo8dLImDQVtT3LMU8l0yz4/0SynLvdhtQPXjf74xz0Mgek4hUZRAZPsAaqEE7w+DZZMJTTEPTe+jeqB5SLtvg5hUd3XtoyX1U1vTpDCKS1Zww4W0wPAZjufK4Hm/yoDlJhnXS2nhObqgl4bVTcY/qLepcxzvttDl9EH5B1wGf92p9+M6SduP59etJf10o3RuJ432WxMFU2rKqmQgwdmJuTdDsHF+jhmSu/M2lvHJBoK16Uqyn+ 7LOYHv3q RAQ8BYJlAZSygSEb7j3RfsNfdr+aLn8Des2AKSloxZFjcG355HLmVIhk+iroTvVzjyPNrzd8rBSzEWho20xZN0SnqJ5F+hBH5Mtx0FGF2m5R/iPvVZm9EOhvAEkivDMExMWwJo/Urzh0qHUu7WmGJytZ7F8Zx2YbOC4h4ZDnBa+cZu3TXiAffNPJUaEh0M/n6RAIRx2oOmB/KNxUgY7f2Z7Eizgsj8apmUMms/VEJyuZFPGn2mY71bn84z1Q+NtqvgodGIt/QX8ZY5P1j3Iw9LOWXlYJZBxLJURcIwqN18y36u821onCSI1orIUFp/tgZQbdX39kS8P02G12b1kvXIW5dvOzzOX33WonYxOTejHB7j1ZLjccmT9nykSqoRE+sEAAjD+ONwk4Nl24CpNw9hBlz8bw8DacB3ppP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan To minimize the number of pages after a huge page truncation, we do not need to split it all the way down to order-0. The huge page has at most three parts, the part before offset, the part to be truncated, the part remaining at the end. Find the greatest common divisor of them to calculate the new page order from it, so we can split the huge page to this order and keep the remaining pages as large and as few as possible. Signed-off-by: Zi Yan --- mm/truncate.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 86de31ed4d32..817efd5e94b4 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -22,6 +22,7 @@ #include /* grr. try_to_release_page */ #include #include +#include #include "internal.h" /* @@ -211,7 +212,8 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio) bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) { loff_t pos = folio_pos(folio); - unsigned int offset, length; + unsigned int offset, length, remaining; + unsigned int new_order = folio_order(folio); if (pos < start) offset = start - pos; @@ -222,6 +224,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) length = length - offset; else length = end + 1 - pos - offset; + remaining = folio_size(folio) - offset - length; folio_wait_writeback(folio); if (length == folio_size(folio)) { @@ -236,11 +239,25 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) */ folio_zero_range(folio, offset, length); + /* + * Use the greatest common divisor of offset, length, and remaining + * as the smallest page size and compute the new order from it. So we + * can truncate a subpage as large as possible. Round up gcd to + * PAGE_SIZE, otherwise ilog2 can give -1 when gcd/PAGE_SIZE is 0. + */ + new_order = ilog2(round_up(gcd(gcd(offset, length), remaining), + PAGE_SIZE) / PAGE_SIZE); + + /* order-1 THP not supported, downgrade to order-0 */ + if (new_order == 1) + new_order = 0; + + if (folio_has_private(folio)) folio_invalidate(folio, offset, length); if (!folio_test_large(folio)) return true; - if (split_folio(folio) == 0) + if (split_huge_page_to_list_to_order(&folio->page, NULL, new_order) == 0) return true; if (folio_test_dirty(folio)) return false; From patchwork Mon Apr 3 20:18:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C47A1C77B72 for ; Mon, 3 Apr 2023 20:19:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 141806B0087; Mon, 3 Apr 2023 16:19:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A3336B0088; Mon, 3 Apr 2023 16:19:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC4226B0089; Mon, 3 Apr 2023 16:19:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C52526B0087 for ; Mon, 3 Apr 2023 16:19:12 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F12AABA7E for ; Mon, 3 Apr 2023 20:19:12 +0000 (UTC) X-FDA: 80641194144.02.0FBFC34 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf10.hostedemail.com (Postfix) with ESMTP id 57D7AC001F for ; Mon, 3 Apr 2023 20:19:09 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=ksLbbax5; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=gE5fJx9S; spf=pass (imf10.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553149; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BxqGc0evZec652+iCiid1CZrCHeD4aKdhQKP2Da/cdE=; b=AoGtVmoYkifNkQqGXpY139XMVSkdmQVRNwXf8Z9M1DBN7ud5IkWFgzV/GOAht6IeLFhpDH piR6ZUSN5Hfxwuq7LxGa+eoXR6lk5Fh5KfalhUelYRfAAYulEX58VsGmFglI3RWulC86KM m1ymWDHRHdb6UWyX2Pp+qahXvpHGl9U= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=ksLbbax5; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=gE5fJx9S; spf=pass (imf10.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553149; a=rsa-sha256; cv=none; b=Ev1YlAVRHUbh1X7LuJmi8BuaHOoc8o5rbavm/4EMg0cSqH8thpRRIc9cLRVXzK/b8qRkSe OfALxGBhInkc70ntGi5E/LYCuFLhesRVXf4BOVPAUbcMkrwCfu+6PmJ7oMH2/SzLJdp2oB 0khpiWxBehnI8t6FluZR9EkqnT0sFSo= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id AA3EA5C01C5; Mon, 3 Apr 2023 16:19:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Apr 2023 16:19:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553148; x=1680639548; bh=BxqGc0evZec652+iCiid1CZrCHeD4aKdhQK P2Da/cdE=; b=ksLbbax5Uv7LmBwo9gDNM14bjc1Lft9lnuZya29aXZtsrl/HFsK WyyR/1KNaTAaxUX8BIiqr+RMitpjRu0W0pjMYihpDC4zMHKX4YXaxfmQkX2QccTQ qD1fWDOrtSVYl2tmjeAaLhdfA8RiQPN5IqT27aw2xCsb+2kiaiW6GWQRDg4grlrs MnXEQbKzo215lLNwQgH4IxkVdOeWJ+grbZH8xX0JScSAZzajmH/Qfh/xDm6G/6fa XXG6NqjOTghReZ9C/iHBbWOhkJAWhoPiRTbz7tGnnPhlyTDLqeCrNj9SS5L1kdNZ PB0yaNC06/d2tmwMyX3WUKrxqCTjuecieVw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553148; x=1680639548; bh=BxqGc0evZec652+iCiid1CZrCHeD4aKdhQK P2Da/cdE=; b=gE5fJx9StrHBvykREsXrYV2Yh01h/9SXNya+QwGlgSEk3JzXEGc 4R8d8GudaLb4foyGX+8RhaWK/Z/A6X09D7UCB0wYTO4zCgc66UZyEu6tqSH9/aw7 OJFwDjFAl7PUvvowlk4u4MmXZ5SgSdMUr1C6AhwBG5wfC09kxSy/WaLI/lb08OZH a3Vo75uBMzhZbqYGT3/idPWT5QqfqvWZ3eAgq3kjopBenNPVPY9cgd+IiyAZukzY f3AEe1/kpwA97UwSEl7FlJxxlnuD8SgXXQiPP72GcNMgWmwlAkIIxe6bd0e5U5w0 SUWy/EzObDuhBZ1caAxdLbuIekuJkSt196w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:08 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 7/7] mm: huge_memory: enable debugfs to split huge pages to any order. Date: Mon, 3 Apr 2023 16:18:39 -0400 Message-Id: <20230403201839.4097845-8-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Queue-Id: 57D7AC001F X-Stat-Signature: g893n4jieo1qir19ah1iu6ug4xhn9dbr X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680553149-250870 X-HE-Meta: U2FsdGVkX19/rdoJMPiFMIBBUqDF12SU6KXQ+7RFlst7HwIKybi2yyDMzeszE9EgQ45AefaPrGeVhJhFVVZjG58+1GjR83JIy/24kK2Tn3KAEKvXg7htPztLkZzQHwWUJiHl4oLtzKrGj9Bs5nCghYoNbMtJ/c1djkdZHJ2l+pg/+8oOgLY5hDcEFIr9m0VsdEofuMd+hz+NrSXY/bHTXTqMwDIchv1yS+NuGjCfK4hvdJPNxPCZ+/0qE/9m3D+9S8xA7sgrx25HKrrzQ++2lQ85afJfUpvGNSniWq2ePr/JML7fAmKVYJVjCeTxH3naeKEgN9tXVCPMVgFBFi/Cleh2ae0QSVoCOUcf9z0a0NIk7zYlpTIgVM2XTg8CMul+Kx1w3UqpR7v2sM5nsnx491BAGsxnGW3EXWsbX2wWsepuKDDAQ9dTQymZ/g5BtqNIJ3W5qBqjmbTY7zwL8xwgEKPFtV8+ODL0NO1IxePqYfFbclPaYbsWPwW6gBIftHd/9AjcfAq/UQCT8RrvDa71KMoPEOGbmbu/0kiGxyOnCiRtSBRKt4fIQHMFw6DeEHAZ5QSJyETC4rw8e9cITlDN2XA+0SjDeGz/k3TNrJr2Agd8yJxwm4yOvjIuRmlwdFO9AmKEv3MO19MbXlT2BL2B6PGLB1G3eXb9Isrw/ItOvu35YeK3V8oxR0rX2Q3S1TuanMLaJCl605lgtrFSdj2p7qH6KRlx/u/Z/Hr8TNCuTlC9t8KE2LFpjCx5W4EuOhBc5QKymLmncD69tHNxdbp6RBLWomt146vx+CEUma456gia4uklsItufHLu2YCDYqNsGnK26enF0Tj10PTbxn7HyuVNS9ltETw4w0dZNgquqr81cqVvWDWlpaYyVjxb4u/Isq9RSQXjnoOJFLv+aSpjEvQ9xscnLrjkegf+qBAPlZ9PUxaNYoOh4pRXKz4QrsnWkqIfmnIH3JlFcfTdcCp wHUEeJAL KgFPcJb8z+Z1K2E++pdk7qnRBXR1UsDfnZcd7xH3nsVwNhesileu67ZdkaNbMf5mf5MwX+IqSim42l8xSeM3EDzttd5bA7TVEtebE+1+OMTstzjt6xjDgGCOQw5ZuTFZe5AMvEszT+WqpPV9AF3F3Lnhz8it/V8lKUbcmUHwIidzITu93kMl6OxyXT44anD/6uK0oRL4onH3+LbWT0qEl1shtl9fCraZ+eIFfAU8y6CzTn/jVc5y0FqLxDqA+3J+GhSdxs3NfcvM+ON4zcY2fNg2Btw7YpLQji93Sutk2vviWTdQJ18JvDWDmtlZCnmXTZ1tCWesRCJtee9lN4en6V6ZVtXRuisEt16WaJaz3BFwWz2tMKbJh5hk+e9gn/J59TdbFjheDyzCEqlfIaCmStl4qQ4goAaytK/xFPMPldqIyU/Cex0RjGufXwTwVIEPwp10/o4fbKotOnJVM8h3rxuOacg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan It is used to test split_huge_page_to_list_to_order for pagecache THPs. Also add test cases for split_huge_page_to_list_to_order via both debugfs, truncating a file, and punching holes in a file. Signed-off-by: Zi Yan --- mm/huge_memory.c | 34 ++- .../selftests/mm/split_huge_page_test.c | 225 +++++++++++++++++- 2 files changed, 242 insertions(+), 17 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 619d25278340..ad5b29558a51 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3023,7 +3023,7 @@ static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *vma) } static int split_huge_pages_pid(int pid, unsigned long vaddr_start, - unsigned long vaddr_end) + unsigned long vaddr_end, unsigned int new_order) { int ret = 0; struct task_struct *task; @@ -3085,13 +3085,19 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, goto next; total++; - if (!can_split_folio(page_folio(page), NULL)) + /* + * For folios with private, split_huge_page_to_list_to_order() + * will try to drop it before split and then check if the folio + * can be split or not. So skip the check here. + */ + if (!folio_test_private(page_folio(page)) && + !can_split_folio(page_folio(page), NULL)) goto next; if (!trylock_page(page)) goto next; - if (!split_huge_page(page)) + if (!split_huge_page_to_list_to_order(page, NULL, new_order)) split++; unlock_page(page); @@ -3109,7 +3115,7 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, } static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, - pgoff_t off_end) + pgoff_t off_end, unsigned int new_order) { struct filename *file; struct file *candidate; @@ -3148,7 +3154,7 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, if (!folio_trylock(folio)) goto next; - if (!split_folio(folio)) + if (!split_huge_page_to_list_to_order(&folio->page, NULL, new_order)) split++; folio_unlock(folio); @@ -3173,10 +3179,14 @@ static ssize_t split_huge_pages_write(struct file *file, const char __user *buf, { static DEFINE_MUTEX(split_debug_mutex); ssize_t ret; - /* hold pid, start_vaddr, end_vaddr or file_path, off_start, off_end */ + /* + * hold pid, start_vaddr, end_vaddr, new_order or + * file_path, off_start, off_end, new_order + */ char input_buf[MAX_INPUT_BUF_SZ]; int pid; unsigned long vaddr_start, vaddr_end; + unsigned int new_order = 0; ret = mutex_lock_interruptible(&split_debug_mutex); if (ret) @@ -3205,29 +3215,29 @@ static ssize_t split_huge_pages_write(struct file *file, const char __user *buf, goto out; } - ret = sscanf(buf, "0x%lx,0x%lx", &off_start, &off_end); - if (ret != 2) { + ret = sscanf(buf, "0x%lx,0x%lx,%d", &off_start, &off_end, &new_order); + if (ret != 2 && ret != 3) { ret = -EINVAL; goto out; } - ret = split_huge_pages_in_file(file_path, off_start, off_end); + ret = split_huge_pages_in_file(file_path, off_start, off_end, new_order); if (!ret) ret = input_len; goto out; } - ret = sscanf(input_buf, "%d,0x%lx,0x%lx", &pid, &vaddr_start, &vaddr_end); + ret = sscanf(input_buf, "%d,0x%lx,0x%lx,%d", &pid, &vaddr_start, &vaddr_end, &new_order); if (ret == 1 && pid == 1) { split_huge_pages_all(); ret = strlen(input_buf); goto out; - } else if (ret != 3) { + } else if (ret != 3 && ret != 4) { ret = -EINVAL; goto out; } - ret = split_huge_pages_pid(pid, vaddr_start, vaddr_end); + ret = split_huge_pages_pid(pid, vaddr_start, vaddr_end, new_order); if (!ret) ret = strlen(input_buf); out: diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index b8558c7f1a39..cbb5e6893cbf 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "vm_util.h" uint64_t pagesize; @@ -23,10 +24,12 @@ unsigned int pageshift; uint64_t pmd_pagesize; #define SPLIT_DEBUGFS "/sys/kernel/debug/split_huge_pages" +#define SMAP_PATH "/proc/self/smaps" +#define THP_FS_PATH "/mnt/thp_fs" #define INPUT_MAX 80 -#define PID_FMT "%d,0x%lx,0x%lx" -#define PATH_FMT "%s,0x%lx,0x%lx" +#define PID_FMT "%d,0x%lx,0x%lx,%d" +#define PATH_FMT "%s,0x%lx,0x%lx,%d" #define PFN_MASK ((1UL<<55)-1) #define KPF_THP (1UL<<22) @@ -113,7 +116,7 @@ void split_pmd_thp(void) /* split all THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, - (uint64_t)one_page + len); + (uint64_t)one_page + len, 0); for (i = 0; i < len; i++) if (one_page[i] != (char)i) { @@ -203,7 +206,7 @@ void split_pte_mapped_thp(void) /* split all remapped THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)pte_mapped, - (uint64_t)pte_mapped + pagesize * 4); + (uint64_t)pte_mapped + pagesize * 4, 0); /* smap does not show THPs after mremap, use kpageflags instead */ thp_size = 0; @@ -269,7 +272,7 @@ void split_file_backed_thp(void) } /* split the file-backed THP */ - write_debugfs(PATH_FMT, testfile, pgoff_start, pgoff_end); + write_debugfs(PATH_FMT, testfile, pgoff_start, pgoff_end, 0); status = unlink(testfile); if (status) @@ -290,20 +293,232 @@ void split_file_backed_thp(void) printf("file-backed THP split test done, please check dmesg for more information\n"); } +void create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, int *fd, char **addr) +{ + size_t i; + int dummy; + + srand(time(NULL)); + + *fd = open(testfile, O_CREAT | O_RDWR, 0664); + if (*fd == -1) { + perror("Failed to create a file at "THP_FS_PATH); + exit(EXIT_FAILURE); + } + + for (i = 0; i < fd_size; i++) { + unsigned char byte = (unsigned char)i; + + write(*fd, &byte, sizeof(byte)); + } + close(*fd); + sync(); + *fd = open("/proc/sys/vm/drop_caches", O_WRONLY); + if (*fd == -1) { + perror("open drop_caches"); + goto err_out_unlink; + } + if (write(*fd, "3", 1) != 1) { + perror("write to drop_caches"); + goto err_out_unlink; + } + close(*fd); + + *fd = open(testfile, O_RDWR); + if (*fd == -1) { + perror("Failed to open a file at "THP_FS_PATH); + goto err_out_unlink; + } + + *addr = mmap(NULL, fd_size, PROT_READ|PROT_WRITE, MAP_SHARED, *fd, 0); + if (*addr == (char *)-1) { + perror("cannot mmap"); + goto err_out_close; + } + madvise(*addr, fd_size, MADV_HUGEPAGE); + + for (size_t i = 0; i < fd_size; i++) + dummy += *(*addr + i); + + if (!check_huge_file(*addr, fd_size / pmd_pagesize, pmd_pagesize)) { + printf("No pagecache THP generated, please mount a filesystem supporting pagecache THP at "THP_FS_PATH"\n"); + goto err_out_close; + } + return; +err_out_close: + close(*fd); +err_out_unlink: + unlink(testfile); + exit(EXIT_FAILURE); +} + +void split_thp_in_pagecache_to_order(size_t fd_size, int order) +{ + int fd; + char *addr; + size_t i; + const char testfile[] = THP_FS_PATH "/test"; + int err = 0; + + create_pagecache_thp_and_fd(testfile, fd_size, &fd, &addr); + + printf("split %ld kB PMD-mapped pagecache page to order %d ... ", fd_size >> 10, order); + write_debugfs(PID_FMT, getpid(), (uint64_t)addr, (uint64_t)addr + fd_size, order); + + for (i = 0; i < fd_size; i++) + if (*(addr + i) != (char)i) { + printf("%lu byte corrupted in the file\n", i); + err = EXIT_FAILURE; + goto out; + } + + if (!check_huge_file(addr, 0, pmd_pagesize)) { + printf("Still FilePmdMapped not split\n"); + err = EXIT_FAILURE; + goto out; + } + + printf("done\n"); +out: + close(fd); + unlink(testfile); + if (err) + exit(err); +} + +void truncate_thp_in_pagecache_to_order(size_t fd_size, int order) +{ + int fd; + char *addr; + size_t i; + const char testfile[] = THP_FS_PATH "/test"; + int err = 0; + + create_pagecache_thp_and_fd(testfile, fd_size, &fd, &addr); + + printf("truncate %ld kB PMD-mapped pagecache page to size %lu kB ... ", + fd_size >> 10, 4UL << order); + ftruncate(fd, pagesize << order); + + for (i = 0; i < (pagesize << order); i++) + if (*(addr + i) != (char)i) { + printf("%lu byte corrupted in the file\n", i); + err = EXIT_FAILURE; + goto out; + } + + if (!check_huge_file(addr, 0, pmd_pagesize)) { + printf("Still FilePmdMapped not split after truncate\n"); + err = EXIT_FAILURE; + goto out; + } + + printf("done\n"); +out: + close(fd); + unlink(testfile); + if (err) + exit(err); +} + +void punch_hole_in_pagecache_thp(size_t fd_size, off_t offset[], off_t len[], + int n, int num_left_thps) +{ + int fd, j; + char *addr; + size_t i; + const char testfile[] = THP_FS_PATH "/test"; + int err = 0; + + create_pagecache_thp_and_fd(testfile, fd_size, &fd, &addr); + + for (j = 0; j < n; j++) { + printf("punch a hole to %ld kB PMD-mapped pagecache page at addr: %lx, offset %ld, and len %ld ...\n", + fd_size >> 10, (unsigned long)addr, offset[j], len[j]); + fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, offset[j], len[j]); + } + + for (i = 0; i < fd_size; i++) { + int in_hole = 0; + + for (j = 0; j < n; j++) + if (i >= offset[j] && i < (offset[j] + len[j])) { + in_hole = 1; + break; + } + + if (in_hole) { + if (*(addr + i)) { + printf("%lu byte non-zero after punch\n", i); + err = EXIT_FAILURE; + goto out; + } + continue; + } + if (*(addr + i) != (char)i) { + printf("%lu byte corrupted in the file\n", i); + err = EXIT_FAILURE; + goto out; + } + } + + if (!check_huge_file(addr, num_left_thps, pmd_pagesize)) { + printf("Still FilePmdMapped not split after punch\n"); + goto out; + } + printf("done\n"); +out: + close(fd); + unlink(testfile); + if (err) + exit(err); +} + int main(int argc, char **argv) { + int i; + size_t fd_size; + off_t offset[2], len[2]; + if (geteuid() != 0) { printf("Please run the benchmark as root\n"); exit(EXIT_FAILURE); } + setbuf(stdout, NULL); + pagesize = getpagesize(); pageshift = ffs(pagesize) - 1; pmd_pagesize = read_pmd_pagesize(); + fd_size = 2 * pmd_pagesize; split_pmd_thp(); split_pte_mapped_thp(); split_file_backed_thp(); + for (i = 8; i >= 0; i--) + if (i != 1) + split_thp_in_pagecache_to_order(fd_size, i); + + /* + * for i is 1, truncate code in the kernel should create order-0 pages + * instead of order-1 THPs, since order-1 THP is not supported. No error + * is expected. + */ + for (i = 8; i >= 0; i--) + truncate_thp_in_pagecache_to_order(fd_size, i); + + offset[0] = 123; + offset[1] = 4 * pagesize; + len[0] = 200 * pagesize; + len[1] = 16 * pagesize; + punch_hole_in_pagecache_thp(fd_size, offset, len, 2, 1); + + offset[0] = 259 * pagesize + pagesize / 2; + offset[1] = 33 * pagesize; + len[0] = 129 * pagesize; + len[1] = 16 * pagesize; + punch_hole_in_pagecache_thp(fd_size, offset, len, 2, 1); + return 0; }