From patchwork Mon Jun 10 12:02:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13691943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8326EC27C55 for ; Mon, 10 Jun 2024 12:02:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18E006B008C; Mon, 10 Jun 2024 08:02:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1166A6B0092; Mon, 10 Jun 2024 08:02:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED1D86B0093; Mon, 10 Jun 2024 08:02:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CBC4C6B008C for ; Mon, 10 Jun 2024 08:02:24 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4CA3780E8B for ; Mon, 10 Jun 2024 12:02:24 +0000 (UTC) X-FDA: 82214841408.30.3415FBD Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf12.hostedemail.com (Postfix) with ESMTP id 46A364001C for ; Mon, 10 Jun 2024 12:02:22 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="GPno7t/E"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718020942; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=26tuUrqnEQfFAp3h3pgs/MAqgYUu2Sg0W6tKE5DaxLQ=; b=UosagkFiBwT+4DXEu49oZYjHe5DZlP/tJqgcP7oK4AfC/7n7otFLBuuxhIbI7qdLOIF45n jIEUsrY4rOQjUaBV78zAyB3i2P0kokPc5Y0pbzzrdl+3Ye0M3FPHg2rBSGRyNRL/sey9oV iMtGGIZvP3ZicG6TGn4UVhVdZTyAsGk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="GPno7t/E"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718020942; a=rsa-sha256; cv=none; b=ULoiBaKGv3LkrRHVSVJXizaKNRDtTHj7WBT3FXQO4xUF4CPwrfgmf4tr+FGXWCoUtNOOIa YkgAUjFO3nqWjbynZuyfGBKpTJ4KqzwwQILSOTwt32v3xLLQnF66/Fq+QJgHHL5Fih69e2 w3VBgbk4NhM7AVggjY1LlxG6Fgp/a2I= Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2c19e6dc3dcso3629454a91.3 for ; Mon, 10 Jun 2024 05:02:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718020941; x=1718625741; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=26tuUrqnEQfFAp3h3pgs/MAqgYUu2Sg0W6tKE5DaxLQ=; b=GPno7t/EXdSWmyXlMJI5HACTkeskMV3UnKICQp131z9ufSKBboIzROTAiMuiMjwOxo iNyrkXOefyd+xvTxIeo/B54ivHey+J9CUlnidSicmERnkXqhkh+DDLGqZ3s6rDhGQOH9 mT2WMxSqFzIBLonfd1ifOjqMTqli7mab5sEUO4N7Y0FkE9u79RAUicbr/KgN9dtBs2ne hKYDBeKsrnG+NE6bkieR4im+RjoxieK3qdfZtiGhjFLHDp8tAP4Yc6uosGLKK89R6Es6 xU5EJ3boc90MZBUGRTefMoVYjkI1jaTTIJLGKhLrj9QPCgn1isSGlLZHsf1yHLrrWKjU 1skg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718020941; x=1718625741; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=26tuUrqnEQfFAp3h3pgs/MAqgYUu2Sg0W6tKE5DaxLQ=; b=e23X2sAjHtufbl5K/t67FFdhBR7gxcbd2tEax2gpvSwdDIdvfcK5gqh3AIvvJ5EAzr vi7jyQpj07uyrGpdFCyhd6+BTOfRq/oGoYyYyRBp1zUZp6eu0yGPjhO8kjE8wH1Vv9UX acIcJMfh/fwGbrCLe/BbsHxRvKQMXO8My77iL1dMn7FX5lKBTTmOWPWWsPWRetqfn8CC 3KC06bHh/NiNHglBDS0k0UmQy/g23h/xiJjbna0jBliwUkU4+4mTU2l7FPfqILNFMVQ9 Z/taax9laAourOPIcVi9Xy5j3NKPCghh2RU5KQ3RsDXQI6l5bAa6neP1NpCwIUa3ekGb O2nw== X-Forwarded-Encrypted: i=1; AJvYcCWOM3ugFtcQqjL1Wse366zF6etJTnQ7YktuPPrktQIqXYJUI5biqgQcDesul35Ag4wlceYkiuQCqdaX1GgZWGQRu1U= X-Gm-Message-State: AOJu0Yz8hjs9zDm8mnplolebLZE6z6ThJb+MNgX7PBx1+PRtEJzT/HQS R1xlM8pZOY7fjkyMuWo3o9v42yTaZtEgf4phFhrCrJ0fMnDYuTi6 X-Google-Smtp-Source: AGHT+IFBudSl4WgItCCnP5LfGTAa8cr1lbmtKS+NCmJH2FoXqV6FA1FwxGmVoAyBzdfwCVxMYS/NSg== X-Received: by 2002:a17:90b:817:b0:2c2:f2d6:60d4 with SMTP id 98e67ed59e1d1-2c2f2d664aamr3884336a91.8.1718020940887; Mon, 10 Jun 2024 05:02:20 -0700 (PDT) Received: from LancedeMBP.lan.lan ([2403:2c80:6::304c]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2c302f0ebdcsm2478478a91.23.2024.06.10.05.02.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jun 2024 05:02:20 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: willy@infradead.org, sj@kernel.org, baolin.wang@linux.alibaba.com, maskray@google.com, ziy@nvidia.com, ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v7 0/4] Reclaim lazyfree THP without splitting Date: Mon, 10 Jun 2024 20:02:05 +0800 Message-Id: <20240610120209.66311-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 46A364001C X-Stat-Signature: fmdou19t9rcgiwasbjase1ufuhmfu9pe X-HE-Tag: 1718020942-192370 X-HE-Meta: U2FsdGVkX18pBPieov2MkQe/15tGvnNuiryom3m0mvW1+Vwz6TMQJJ9wT19cfqOMXsAJOdWdH+3k3KjcStIdkIwnyWfx/AVNgXS6KVcrQMpe/xzDeyxgRGBNrQvMloYoqcA1nltE75nC4FBLBV1pGI6qLu5cDK7lbEjQ9o4gAswHUhz3WeLdry2dBx3NhB6GPJD2BYxP6/d1neqhSRmSSa7cqEsYd2ndijTnASoctN1Gws2rwFbhdfPjas/DEzbgv8/Druoas2B32cn2xu6h08kJ2UOcRgu5uFK4KNu3Ol+uLsVuy3cIZFTzfpO3JjMPmHPna3QBibquSzMCSQ75/ZW/qkHjCtemmIMPj/d6rGCymhnw+wNMUcVtIXj2lyYxsaFrPsRI1LNZrnGdQNxbUO3Nle8MruZIid7iBcoo5umiimsyrXtt1xhVOGKYC5kSU1tbS5h89f2GUsrEBho8bGNf0F/rqYs0w513kP4FarDekF7cAjaFJod1YRnh2fLxWE34r1dyOiofZAAgU88fuOJF+QQCmxN4ugl02O/k8qeZsGS6C7MqTeoEL+TC4A/jCGGAFWDzvuimYnkcFNkXPPkWeFNFulEDKkSmdW7aIteH/C9lmEJH0BbEQqlF4J8mdIeXmgt48uphvFUdR83x8Rm2AZe99mFImmA7fWQm4R6gNkqNjAh8UNwkSaf94xZme3KD3jxKghiWVrp9vnFC0PKgzPdQOJTrIkFzJJcBr/4BtoGjYNMgmj+Hgci2S08l9KDgdK6WIEniUlSE5tfRWGg5M5ZUaAS6aLRf0Y30F3fg7eAwW42/QQR4vr8+FX1pFdBkx3/zq2OmOrp9IDDNhWitexG2AKSkqXVLFh1QhauajuYpguKlyLaDvYZ9GQCCWbvfW20YXMYu8S866TYYecKgRNGoDrLDHDtveCfvRvUZ3R7XKkWgOb3CqiHHznIkLDs7Y/7ND52pSxgVACX hBa8d2pt lRRljrS58bdcTNqODA4IIyOxD/Dy4paXbiHWSeoF3i7br47eT/GhM9g87CjeAtDwGWP375DCFuyMAX28qGZY5pfTxf5WInAlNYpr7U9IhZUCamPlN4L3yIjFrGR5nfRgnw4E8fLtbh5auaKEqlxJs01EHditc/KgMM1VAXGPlWnHWsySWwAXY1CogdEdGzDLWoJs/BJGpvThT3Z0lxFdL48cSz1+00gE8EB3F/F3vwrtLcex6eDtG8rjJR3y+kEelanHGnET+DxbOVEO8fEgnD8hfF9U+N7eIvj5q1p5f9/OAIadB/QCPwNDaMPI3vxLmTATy1FX1GTqv4yKQ1aP/rx8adIU4IpQRGPDSgWSWknWGWnkrvkJ++6N2VSV0Z9Sbk2Dpd7DDOymFFue18oTsrxQ26Fy/S55+6p71IpgqyvdawM+D0yGCJKLZZbd75i/losjGivt3m9M9z0CbQERd1z+xGBMWPAobJprMvMjZF5ytisDvhD1t0oxwcblR90xI8i3b X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, This series adds support for reclaiming PMD-mapped THP marked as lazyfree without needing to first split the large folio via split_huge_pmd_address(). When the user no longer requires the pages, they would use madvise(MADV_FREE) to mark the pages as lazy free. Subsequently, they typically would not re-write to that memory again. During memory reclaim, if we detect that the large folio and its PMD are both still marked as clean and there are no unexpected references(such as GUP), so we can just discard the memory lazily, improving the efficiency of memory reclamation in this case. Performance Testing =================== On an Intel i5 CPU, reclaiming 1GiB of lazyfree THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better): -------------------------------------------- | Old | New | Change | -------------------------------------------- | 0.683426 | 0.049197 | -92.80% | -------------------------------------------- --- Changes since v6 [6] ==================== - mm/rmap: remove duplicated exit code in pagewalk loop - Pick RB from David - thanks! - mm/rmap: add helper to restart pgtable walk on changes - Add the page_vma_mapped_walk_restart() helper to handle scenarios where the page table walk needs to be restarted due to changes in the page table (suggested by David) - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Pass 'pvmw.address' to split_huge_pmd_locked() (per David) - Drop the check for PMD-mapped THP that is missing the mlock (per David) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - Rename the function __discard_trans_pmd_locked() to __discard_anon_folio_pmd_locked() (per David) Changes since v5 [5] ==================== - mm/rmap: remove duplicated exit code in pagewalk loop - Pick RB from Baolin Wang - thanks! - mm/mlock: check for THP missing the mlock in try_to_unmap_one() - Merge this patch into patch 2 (per Baolin Wang) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - Mark a folio as being backed by swap space if the folio or its PMD was redirtied (per Baolin Wang) - Use pmdp_huge_clear_flush() to get and flush a PMD entry (per Baolin Wang) Changes since v4 [4] ==================== - mm/rmap: remove duplicated exit code in pagewalk loop - Pick RB from Zi Yan - thanks! - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Remove the redundant alignment (per Baolin Wang) - Set pvmw.ptl to NULL after unlocking the PTL (per Baolin Wang) - mm/mlock: check for THP missing the mlock in try_to_unmap_one() - Check whether the mlock of PMD-mapped THP was missed (suggested by Baolin Wang) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - No need to check the TTU_SPLIT_HUGE_PMD flag for unmap_huge_pmd_locked() (per Zi Yan) - Drain the local mlock batch after folio_remove_rmap_pmd() (per Baolin Wang) Changes since v3 [3] ==================== - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Resolve compilation errors by handling the case where CONFIG_PGTABLE_HAS_HUGE_LEAVES is undefined (thanks to SeongJae Park) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - Remove the unnecessary conditional compilation directives (thanks to Barry Song) - Resolve compilation errors due to undefined references to unmap_huge_pmd_locked and split_huge_pmd_locked (thanks to Barry) Changes since v2 [2] ==================== - Update the changelog (thanks to David Hildenbrand) - Support try_to_unmap_one() to unmap PMD-mapped folios (thanks a lot to David Hildenbrand and Zi Yan) Changes since v1 [1] ==================== - Update the changelog - Follow the exact same logic as in try_to_unmap_one() (per David Hildenbrand) - Remove the extra code from rmap.c (per Matthew Wilcox) [1] https://lore.kernel.org/linux-mm/20240417141111.77855-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240422055213.60231-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240429132308.38794-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240501042700.83974-1-ioworker0@gmail.com [5] https://lore.kernel.org/linux-mm/20240513074712.7608-1-ioworker0@gmail.com [6] https://lore.kernel.org/linux-mm/20240521040244.48760-1-ioworker0@gmail.com Lance Yang (4): mm/rmap: remove duplicated exit code in pagewalk loop mm/rmap: add helper to restart pgtable walk on changes mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop mm/vmscan: avoid split lazyfree THP during shrink_folio_list() include/linux/huge_mm.h | 15 +++++ include/linux/rmap.h | 22 ++++++++ mm/huge_memory.c | 122 +++++++++++++++++++++++++++++++++------- mm/rmap.c | 77 ++++++++++++++----------- 4 files changed, 184 insertions(+), 52 deletions(-) base-commit: 6b77d01728f219a091f55ca3142bbc4e4057ca50