From patchwork Mon May 13 07:47:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13663095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1096AC25B78 for ; Mon, 13 May 2024 07:47:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6889B6B0273; Mon, 13 May 2024 03:47:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 636DE6B0274; Mon, 13 May 2024 03:47:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D7576B0275; Mon, 13 May 2024 03:47:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2FE3F6B0273 for ; Mon, 13 May 2024 03:47:51 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AD138140D1B for ; Mon, 13 May 2024 07:47:50 +0000 (UTC) X-FDA: 82112593500.09.FA12DA8 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf05.hostedemail.com (Postfix) with ESMTP id 00FAE100010 for ; Mon, 13 May 2024 07:47:48 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=S6jvoObt; spf=pass (imf05.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715586469; a=rsa-sha256; cv=none; b=uM4L6fx3GTs+WGA+5v6BjUUuYo5q2SJ9uRJcJBoshqRgGCd625IiU8nsETB1qs91XXJKXf E56QuTVkzkLTEZLloawdTC6ACz4Lnqc4QJmhYrJO3lsa/EbyS1XmHHQlVkMkUucc/ZOQ26 nQfonhA7fUxV6RBc5xCIneRimImDo2s= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=S6jvoObt; spf=pass (imf05.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715586469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=CMsOuSvzf4aVI6LsRrkYGtxxdKvIAWtBa8DssIl0GDM=; b=i1kBEkhJ6cyKrdLqx5AT98bmF+tyflysY2aXi9BQxGvmTOPrpmZeSXgNwxQz1BMW4YyLrE bOkxARxPBBl63piLml+/PUKVfvAwQm4blFZ3s08lgi68dpLt4JEqBZ8ZmSzAHKJL8WOZpj 9z99hQyj39TXqZv17wvc3PUgPWwrXb4= Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-53fa455cd94so3274593a12.2 for ; Mon, 13 May 2024 00:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715586468; x=1716191268; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=CMsOuSvzf4aVI6LsRrkYGtxxdKvIAWtBa8DssIl0GDM=; b=S6jvoObtDCacttJc9CFwOsh1tti7iqLnEnUZRH2gPX+DWQRLdeA5yalNh6MVj7SCTV vBT1kbzJ8HCTFmhadCyE+46FfUBYJNLmVZ6J7tDb0rFSdbUYCTdFAoHDDE7jhEwVIRPI 9eU+uC/4y/kSxFVL5VXYgGIqd8QhyFwbBFs9aVTSsklTVbjPxRvMAhUI7frtG5VXMTnZ 6owDYwokKsO9mD/ElCX9sNxrVC9WLybEFhH1h1jziBaLLZl+22m6QCnmc6qHMWY0cW7C EXbPKaPsdkG6KvAvaS4tlBYk06svIr97QbkAXSfaBdp+VbobUKErTCfAz4PscdcqztYd ORjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715586468; x=1716191268; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CMsOuSvzf4aVI6LsRrkYGtxxdKvIAWtBa8DssIl0GDM=; b=JIroYoTOTHD3pI/4AjsU3qMFEQzsjEx1pUpNqs7zxEC7YRXIMYsqfHBqzv+5L6yZlO agQ9wmylKLrBS0H1nyg2fswTqY9oOgvbb7S++ir4CGXaNuAltL+ZZMKhXh3Xg3vinnh2 T7uLxv7U4yBzeG1oAZsmM/I3VhoI0CNkblEeli/ISgqdc0eL4EOd9SAT+CX1rTH64ax8 jXygnofTJZiaICCkUOnsJaC0YWPL2CeEH9XS3kOEHqyrWg0bxByaGoAiuMHHnyoSLQS4 3yjnqreLuqLGuZ6x1Rhp4h7AhToPbibE3COw01Kgc+rCJcTj3yVGpC+celjY+zlguKjh s6QQ== X-Forwarded-Encrypted: i=1; AJvYcCXMKzC8mcuNB+cYZe4479oHZGrp6oVabIhP7c3Wpqzz43qV9Y+h50t9vaIGcWiLtwKF+KBsAVt38I2lz5CMoqbk6cc= X-Gm-Message-State: AOJu0YzXwSvVhuj5uQnk613YeOtzEGDIsXlgrKGD5QpEM/lH68pv+JpA iPHRr7PpyKcJgBkHYetri3rMt1tgMDQYglKde6GyLNMz52tjSSYX X-Google-Smtp-Source: AGHT+IELxEpVMigw41d+USsvJh6XNnuiakSF+kkZsKY2vnA7653ykgBEkQSFtd6+p793bowb5qsR4w== X-Received: by 2002:a05:6a21:99aa:b0:1af:cc48:3e25 with SMTP id adf61e73a8af0-1afde0af471mr9450020637.10.1715586467513; Mon, 13 May 2024 00:47:47 -0700 (PDT) Received: from LancedeMBP.lan.lan ([2403:2c80:6::304b]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-6f4d2a66621sm7008967b3a.13.2024.05.13.00.47.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 May 2024 00:47:47 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: willy@infradead.org, sj@kernel.org, baolin.wang@linux.alibaba.com, maskray@google.com, ziy@nvidia.com, ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH RESEND v5 0/4] Reclaim lazyfree THP without splitting Date: Mon, 13 May 2024 15:47:08 +0800 Message-Id: <20240513074712.7608-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 00FAE100010 X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: 4iuxt1rqhbatfpnmhan1dsh73y5bney1 X-HE-Tag: 1715586468-651439 X-HE-Meta: U2FsdGVkX190ACBH4Su54M/z9rnCTf1eM+C4GD6bvyyd46lMz3lyhBl9gRTqAwoqr33QFMjFsxNmurmTEt0ZHcgHwtZGOxULdtIDQ9/7WVq+LUElHWnLPJaH6onOanGZluT6WaYdlfCw8bErjLwkA0huS5TD26Kqe8y2yICbuOPL9SOvqf1OzXdCfY4ppbjjDaa3aGh9u59sJJlPdjZ1h0Oxqj5jnyE3pd0omqrv/lnxAStQFGAWnT+vvjCD6qpiD6PL/fy/1M3YJCFNooOgaFKxeZgeHn6wcOpBSUgvQL0s5R4c5YcEA9tESm0fdi2XGaze0VJ/kZQmoFvtYHQTQaJWnxJF/RHFY0wpe9PpxQCOFo8GYkAXUyz6VGebSqKaTg3vrqghmyCN94lT0GHyzuyumVDFihqV/wPHUmUtnalW+oxYiTF15t5vBvVkV+Ii8qwBFOLI4NjDqRtTbXupSbZGZwxiam77pHQda3eWWfc3t7UX8DYAaK/gVAUVm6RIiIDhYZyu2ZcKq3nrTSYnez011asL8kgHlPzKDCos8edQTTVNv6QswBC7XJiyW0NhqJIB9WTlex2uFDPH9EARAz/+1t/6RptQ4khv6PGlpKo/L0Udj75SyR1ptgpYnHx8YeGUQonOe43mhaYozKdJXlHRER7H+epsPgd2Z5iBoLVcWA9+HzvALN1ux6h8TdfXdH2b8D5sYxmYGVNfkhvD+/3JVW//v0QFvMY4hKEefbtpc0sTjoGLpwvDqJrTTGsOBV7MYK7wCk+2e0qHNRxgXweTOQjeXK2IZLdd1NoPaY2+VxsZ3/YcwEvkeDOxB5J38PSKUDHnZ63UjndE6bRJI4ULc+BsNeKrufmWrum9T+sB0QTz34niDnJIEtKVPGnFcYESx0szxOOr7dMQf9g1Tfk8IYMJiu19xsngUbWgr9MXqQswLqyWbojkfmoUuVDCRTCkGDsSt2p+E+bCTTn jkwyJLhQ L2bF8WmdxOCdr+7rXyTi0aJUqW6Ey1mVgOBiilL43o3vnpLjtNyBI05RNJEPasL2AQUFqriY0YfV8U6ZnUMZswlTHxh7pANAcOyZ6AuY/EIsUi44UIGXhnqppu+ykPIxTsP1OC4wJrXgrRrFqw3dtO63LMmQChLHVnVe20EebL4SSxt6cLA18sm38RkOi8mqfJ0Dfxar+GJBxL5tbO+7euAnqyWYGedGyJUPlcnZqqsyPrMrxS4A7NMN/b3F76sjxPmmXqQlaFQdMrfxCsqnAFHV1ar12mbShiuvAV4xAMZMzBhxvAW7pVNeM7Jgp7j6Kk/xh1sxt5X4AEN6tWJQlrRtkbCZ6715oHP+RS3hi9hcscW1OmWoAib5b1RHwyqtssxMxFww79EMBPFW3oX44TFB84iQVtCwG9kD192m6zhGuN+GeHEVws4QAiLRk7lcSTZ7Fapa3di2CR5NdafYLeh/BgNOB8xW8+gBXyhudL7J66WH3CaTf4zmicoikJHxp1bgI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, This series adds support for reclaiming PMD-mapped THP marked as lazyfree without needing to first split the large folio via split_huge_pmd_address(). When the user no longer requires the pages, they would use madvise(MADV_FREE) to mark the pages as lazy free. Subsequently, they typically would not re-write to that memory again. During memory reclaim, if we detect that the large folio and its PMD are both still marked as clean and there are no unexpected references(such as GUP), so we can just discard the memory lazily, improving the efficiency of memory reclamation in this case. Performance Testing =================== On an Intel i5 CPU, reclaiming 1GiB of lazyfree THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better): -------------------------------------------- | Old | New | Change | -------------------------------------------- | 0.683426 | 0.049197 | -92.80% | -------------------------------------------- --- Changes since v4 [4] ==================== - mm/rmap: remove duplicated exit code in pagewalk loop - Pick RB from Zi Yan - thanks! - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Remove the redundant alignment (per Baolin Wang) - Set pvmw.ptl to NULL after unlocking the PTL (per Baolin Wang) - mm/mlock: check for THP missing the mlock in try_to_unmap_one() - Check whether the mlock of PMD-mapped THP was missed (suggested by Baolin Wang) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - No need to check the TTU_SPLIT_HUGE_PMD flag for unmap_huge_pmd_locked() (per Zi Yan) - Drain the local mlock batch after folio_remove_rmap_pmd() (per Baolin Wang) Changes since v3 [3] ==================== - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Resolve compilation errors by handling the case where CONFIG_PGTABLE_HAS_HUGE_LEAVES is undefined (thanks to SeongJae Park) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - Remove the unnecessary conditional compilation directives (thanks to Barry Song) - Resolve compilation errors due to undefined references to unmap_huge_pmd_locked and split_huge_pmd_locked (thanks to Barry) Changes since v2 [2] ==================== - Update the changelog (thanks to David Hildenbrand) - Support try_to_unmap_one() to unmap PMD-mapped folios (thanks a lot to David Hildenbrand and Zi Yan) Changes since v1 [1] ==================== - Update the changelog - Follow the exact same logic as in try_to_unmap_one() (per David Hildenbrand) - Remove the extra code from rmap.c (per Matthew Wilcox) [1] https://lore.kernel.org/linux-mm/20240417141111.77855-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240422055213.60231-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240429132308.38794-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240501042700.83974-1-ioworker0@gmail.com Lance Yang (4): mm/rmap: remove duplicated exit code in pagewalk loop mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop mm/mlock: check for THP missing the mlock in try_to_unmap_one() mm/vmscan: avoid split lazyfree THP during shrink_folio_list() include/linux/huge_mm.h | 15 ++++++ mm/huge_memory.c | 117 +++++++++++++++++++++++++++++++++------- mm/rmap.c | 73 ++++++++++++++----------- 3 files changed, 153 insertions(+), 52 deletions(-)