From patchwork Mon Jan 13 03:39:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13936681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EDEFE77188 for ; Mon, 13 Jan 2025 03:39:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AB556B0093; Sun, 12 Jan 2025 22:39:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 334336B0095; Sun, 12 Jan 2025 22:39:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D5096B0096; Sun, 12 Jan 2025 22:39:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EA8436B0093 for ; Sun, 12 Jan 2025 22:39:55 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7D6631C9343 for ; Mon, 13 Jan 2025 03:39:55 +0000 (UTC) X-FDA: 83001024750.19.E9A9E0A Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf12.hostedemail.com (Postfix) with ESMTP id 7B53640003 for ; Mon, 13 Jan 2025 03:39:53 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=My2FmB7A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736739593; a=rsa-sha256; cv=none; b=I4gO/zDA6QRcRuI43XuLTA8EYgzhvqdXH4oqm2gCEg6d2wDQfhep70djfbw2jq1vc6Qdtb xSSAj4ifOkBuiIVAICw//TeqB4u96sgdaZ9GiVqDf7IgawP8VF3U/AK9mIPaQ9kT0aQjGc tFfjVQ9hs72Co+1+yM1nP/HKyfhJZIs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=My2FmB7A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736739593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t1XMRg54nQSp8/EH5otPlOs5Y//8NM069hYIc60Rxo8=; b=ttlcxW9jAgsaiaIN2PxuA0roZRKX8U07Deea5xkqCw1tDmh8qIwmID4L+whKWd3MgaEnBb u67vnC1A4OVUvHZOx+8fz4S6ClHJpZZ8tNmB0syGwQN4s6IghusXAsaSwhcmj2fK0SEGIu YgIa9edwgNvCcOdVaTCdPniHtvvblrY= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21669fd5c7cso66199655ad.3 for ; Sun, 12 Jan 2025 19:39:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736739592; x=1737344392; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t1XMRg54nQSp8/EH5otPlOs5Y//8NM069hYIc60Rxo8=; b=My2FmB7AKo/mQwEJKMxlMhcGx43lUAaA8eN+8TV5HTMPxDpNgAyaI6mI+Ad/TMXmfD TXQZPhAZHbOhpzoeuLvcanaBaTPgkV02R3Lhu5TP0MXvdONrHpDdacoIM+Wf3LSvFsyC Nxyf6T/8kFYFGbehyw1rB7UMlJedpTCSYpIBKSYPftzi+vcY2Ri5bWUGLwMNEsDKs3pb K3UCtpE5+vdeHgMwWf60VYE/XT3y2Roy2IhVAPcLFH3K9s2coK75s8MOp3z4rlDJXwqy 7uWunJjJXE+MOK8fpOdURbxDcEInttUfCGfssPj8+tNdTDXmgpzn43BOw2T6gUN2ubh9 BR9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736739592; x=1737344392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1XMRg54nQSp8/EH5otPlOs5Y//8NM069hYIc60Rxo8=; b=RFQ0sjWSnEDUUDMwuNk4dxZb42fGz2ufSwVhQ9cz6gs/7U5BcO9IPnV6XN2b64SEzr 5aUGQCO5Tvc/SpcJo0+FvRhPK93FHkb/01wcpL72n1Ah8org2ov4ATOzR3za5wSpTIeq bmsdzNiRq9yjiv5z1H0sq/REnye2MQLdTb9JQyYCA1nrppvNQRw6G+ynFwUPlJPuWuCh nrKF1gJ4fsWFWXt8BpT5dfsUtmmEv3dFde0qsoZ92gkkzaOjSszHeEqi0y/pAIaDeQyY aUmVSNMI7VuIPYnqXFAKZ98My2/P/C5IhNl0dyloE3UQvpztndinXyuWHDdsYZd4ZVC2 duUw== X-Forwarded-Encrypted: i=1; AJvYcCWpaaVhe2RAHNzBxgdY2XK7baQBCCLrOKyF88Abs7uNPd+9r0nxWc77TiTBgUib6yLmDNkIkzYktQ==@kvack.org X-Gm-Message-State: AOJu0YxSsdzFFGCkANY2yT2MXoLO6+O577bDu26E/Ta8lM8u1m6trQY6 ThSWugKzW8Cy6cfoBXmdJ9otg6IE8lxPUdo1Niv7sPNXy9KWNkhF X-Gm-Gg: ASbGncse8QXnSjd75WRumoGqns2lCEqulIz8UXmFN1332onQ/Bf4lhbUuOMQML6PqSG ycxpvJUziQnobhObjD4Wt7wTf0l3SsYxdr0hIeM+r8BKXbTv775FOQfa82Av0DhPuxgnArMWtA/ xY38SwjdBDR7hhQW0ts/asi25lcdtRZPqywkp6SmX0I0nmRhXTiUQrBLwx/r5PXxj5pPSAsUUpi p2HPlRZ7o7IGqdT6ngMxJUF0W6q0zgHT7kzEu8lChIW2DftahXJ7FFzhKM+V8TI783Wjqn9jBbV J2NfoOOt X-Google-Smtp-Source: AGHT+IFvTUatRCa49/9Qoa3okmsFGYsqNW658p8oE5OURLMqB1h00MlvT5OAmq5hWQRPMmGvVdLulw== X-Received: by 2002:a17:902:cec3:b0:216:6fb5:fd83 with SMTP id d9443c01a7336-21a83f69d84mr299035325ad.29.1736739592302; Sun, 12 Jan 2025 19:39:52 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:39b5:3f0b:acf3:9158]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f25aabfsm44368405ad.246.2025.01.12.19.39.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 12 Jan 2025 19:39:51 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, linux-riscv@lists.infradead.org, ying.huang@intel.com, zhengtangquan@oppo.com, lorenzo.stoakes@oracle.com Subject: [PATCH v2 4/4] mm: Avoid splitting pmd for lazyfree pmd-mapped THP in try_to_unmap Date: Mon, 13 Jan 2025 16:39:01 +1300 Message-Id: <20250113033901.68951-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250113033901.68951-1-21cnbao@gmail.com> References: <20250113033901.68951-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7B53640003 X-Stat-Signature: 6o776d841n8gdrtb7f6ixmdomi4syan7 X-Rspam-User: X-HE-Tag: 1736739593-659929 X-HE-Meta: U2FsdGVkX1+f2z9urunt53LzrxERudo7MNL1fz/k83LPyOC3fvEC4BktNw1yYh73mt+cj5NdZi0DnQ2VPsdpHLZe9yYQTYJVClNwNELXDlsTfh2AoqLncfRt3Gi5TnPnLJgfTbD+JsAb7Hnj+s3W7WKwoM9JCWt75DNoO9fNnkDix2lw8Fr8k52+iSmD4GV9IseDWkByc8P9CYpL4BhjHuo1MtFjqGl+jjRuxYNuX68Cb5KpeNvvpB0LxLa8Wj3/6oUhM7Q1jUPzmvqim/90Ag0gL4zsu8Eb8H12MX4fc1xxHwQ+6SwLUXHK2Gvh4yAaJoS2LwTFlJY69NWQ67qPsgndzDfh7VpCgQL0CWnkTme7dnPsfIS1Iynlf7cVt8m2xobsp4bzpNHHenyN0kD+aOgplWSxr67b/Ipde2k/ocjBKs/sofVeu/mLNO4e233HYDa8nH82cDnDrQ7J19iSz+tBdA4GFwyUPT3BqDtHFP/EBCxiHtwADohEGM6uMN2poVvu3si+qlGBY3DfwnFF9szkrjgXvkRCaN7GvEVpNDlYEzZJo1Xn/gm8rOSTUR0u4eXXmIrPiacN55GwUYg4/qtYoQSwXK0xySu5VmT/wLtTew10mo3ImSOCLEnpyLv9I3tPLJBY/gX1Gn3FpaDVl+NWwPsOEb1f3WfmoXUpKrOpq9uzIYlvXNgEYqVligeYrIsp56wZ8MeMi7e59OoHHE+iKJ5FLqLUA5SGb3WmAyv8FfSEi8uFDjEgEiirtagb3B1W3SvzUoTMBgMuMr3nyDr4qoNF0xTFRgwSUOehUKoSAoR74TQQ1VAA/9TCyegZImzbxckdnwdQSjutHQOHTPGtZV6Ud7JIU/+C4oT5YDeJGCcTSn7oOKL36DPRW2eyrtsOwSC0RKpiKmSB7qCjYACNjrq4nGBuphrgw91kc5U+OSwFojn4uHlpkP3sMv/BC4pgpanLVNKtXF4CXQC CHsTtLhG XI9f8HvglmVpMq1MlVBjGT20ZjxJHU1dk2GhuYbuki2hQEPl9bVzO5qdPJXGdlJP2MHZh/FaxCP/TrGOgPa1KfOsbRzs5eDv+DzZJE24b40gYY2I6xfVAU/T3tlbFnXFvwZlyCqDtK5tZTdNyRhfYcIZS2jaqppo0V9LymTQXEkf5T8pG5WNvDZ7PIQY6tREqPYz6XQLcsO1Jl24LujiORP7kWA4NNzwiLKB9Y0lrIb4QUZRQsDgowW1eijxw657+D9sDfeVlpXwsdDk/u4P6zZZJRjioiznIEElKYOHYgjAVMkgTQM9R1X1igsxsX0sxLYJzlsfd47VZxjca6ZGjdTPlUk9Y1z4htk278v2pRB6Pm6g9GsAjOeASFPcoOx23g1esV55VRV6TjHZjtpHXNhP5oI3SkS6NB9K8OjFBCKtwB05ls0vcEN6W4MLPVUgQ+GqlEpZ5HCeub82JrXP1rAImPrfsoJrwkVSacssUWOUwgZxfF9m4Kklycg/t6mMEkS7EdjLmJPSz7k4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song The try_to_unmap_one() function currently handles PMD-mapped THPs inefficiently. It first splits the PMD into PTEs, copies the dirty state from the PMD to the PTEs, iterates over the PTEs to locate the dirty state, and then marks the THP as swap-backed. This process involves unnecessary PMD splitting and redundant iteration. Instead, this functionality can be efficiently managed in __discard_anon_folio_pmd_locked(), avoiding the extra steps and improving performance. The following microbenchmark redirties folios after invoking MADV_FREE, then measures the time taken to perform memory reclamation (actually set those folios swapbacked again) on the redirtied folios. #include #include #include #include #define SIZE 128*1024*1024 // 128 MB int main(int argc, char *argv[]) { while(1) { volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset((void *)p, 1, SIZE); madvise((void *)p, SIZE, MADV_FREE); /* redirty after MADV_FREE */ memset((void *)p, 1, SIZE); clock_t start_time = clock(); madvise((void *)p, SIZE, MADV_PAGEOUT); clock_t end_time = clock(); double elapsed_time = (double)(end_time - start_time) / CLOCKS_PER_SEC; printf("Time taken by reclamation: %f seconds\n", elapsed_time); munmap((void *)p, SIZE); } return 0; } Testing results are as below, w/o patch: ~ # ./a.out Time taken by reclamation: 0.007300 seconds Time taken by reclamation: 0.007226 seconds Time taken by reclamation: 0.007295 seconds Time taken by reclamation: 0.007731 seconds Time taken by reclamation: 0.007134 seconds Time taken by reclamation: 0.007285 seconds Time taken by reclamation: 0.007720 seconds Time taken by reclamation: 0.007128 seconds Time taken by reclamation: 0.007710 seconds Time taken by reclamation: 0.007712 seconds Time taken by reclamation: 0.007236 seconds Time taken by reclamation: 0.007690 seconds Time taken by reclamation: 0.007174 seconds Time taken by reclamation: 0.007670 seconds Time taken by reclamation: 0.007169 seconds Time taken by reclamation: 0.007305 seconds Time taken by reclamation: 0.007432 seconds Time taken by reclamation: 0.007158 seconds Time taken by reclamation: 0.007133 seconds … w/ patch ~ # ./a.out Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002116 seconds Time taken by reclamation: 0.002150 seconds Time taken by reclamation: 0.002261 seconds Time taken by reclamation: 0.002137 seconds Time taken by reclamation: 0.002173 seconds Time taken by reclamation: 0.002063 seconds Time taken by reclamation: 0.002088 seconds Time taken by reclamation: 0.002169 seconds Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002111 seconds Time taken by reclamation: 0.002224 seconds Time taken by reclamation: 0.002297 seconds Time taken by reclamation: 0.002260 seconds Time taken by reclamation: 0.002246 seconds Time taken by reclamation: 0.002272 seconds Time taken by reclamation: 0.002277 seconds Time taken by reclamation: 0.002462 seconds … This patch significantly speeds up try_to_unmap_one() by allowing it to skip redirtied THPs without splitting the PMD. Suggested-by: Baolin Wang Suggested-by: Lance Yang Signed-off-by: Barry Song --- mm/huge_memory.c | 17 ++++++++++++++--- mm/rmap.c | 11 ++++++++++- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3d3ebdc002d5..aea49f7125f1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3070,8 +3070,12 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, int ref_count, map_count; pmd_t orig_pmd = *pmdp; - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd)) + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); return false; + } orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp); @@ -3098,8 +3102,15 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, * * The only folio refs must be one from isolation plus the rmap(s). */ - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd) || - ref_count != map_count + 1) { + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); + set_pmd_at(mm, addr, pmdp, orig_pmd); + return false; + } + + if (ref_count != map_count + 1) { set_pmd_at(mm, addr, pmdp, orig_pmd); return false; } diff --git a/mm/rmap.c b/mm/rmap.c index 3ef659310797..02c4e4b2cd7b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1671,7 +1671,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); pte_t pteval; struct page *subpage; - bool anon_exclusive, ret = true; + bool anon_exclusive, lazyfree, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; int nr_pages = 1; @@ -1724,9 +1724,18 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { + lazyfree = folio_test_anon(folio) && !folio_test_swapbacked(folio); + if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) goto walk_done; + /* + * unmap_huge_pmd_locked has either already marked + * the folio as swap-backed or decided to retain it + * due to GUP or speculative references. + */ + if (lazyfree) + goto walk_abort; if (flags & TTU_SPLIT_HUGE_PMD) { /*