From patchwork Tue Dec 10 21:37:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13902101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D4A6E7717F for ; Tue, 10 Dec 2024 21:38:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF5BA8D0012; Tue, 10 Dec 2024 16:38:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA6168D000B; Tue, 10 Dec 2024 16:38:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C45FD8D0012; Tue, 10 Dec 2024 16:38:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A5FAA8D000B for ; Tue, 10 Dec 2024 16:38:11 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4E4FC120EC6 for ; Tue, 10 Dec 2024 21:38:11 +0000 (UTC) X-FDA: 82880361900.07.2ECFF6F Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf18.hostedemail.com (Postfix) with ESMTP id 520051C000A for ; Tue, 10 Dec 2024 21:37:59 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=jmlxFVz7; spf=pass (imf18.hostedemail.com: domain of gourry@gourry.net designates 209.85.160.179 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733866674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=TZdq/UsD7TyngKHwHzlXM4cYkZjAFz4x6rdkvANzCaw=; b=guaKaMiA/XOW5IuN35f3sI9FjcTXzzEHlunmlG3wqvzqNUblMRRfET8yMdwDQhAqqSlSmh EnLNOcu6BANf6KzlxBQNIZq+4MDdTzxpZVo03lRX1ddqFqqOt3/mD0OHEjcsKPmqqusN43 V8v+krly0EpBsRk5R93y6aUWsZD89G0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733866674; a=rsa-sha256; cv=none; b=o/fA0WrXQQ1JrVPVknKKzTo19kEwz5hG2ZiWrpDDDpFOXWKXLcMGSVmhShJwLWs83Pdqe6 AHnWAOK7HB7wyzYj7thFRV3HCFFqpxxOiqePzZMfnPE7qxt7sZnCRZ+5qbpHgucWvicmHV favPCTxpUw+cZezjoXvTxIq6rE5KRFk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=jmlxFVz7; spf=pass (imf18.hostedemail.com: domain of gourry@gourry.net designates 209.85.160.179 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-4677675abd5so8189971cf.1 for ; Tue, 10 Dec 2024 13:38:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1733866688; x=1734471488; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=TZdq/UsD7TyngKHwHzlXM4cYkZjAFz4x6rdkvANzCaw=; b=jmlxFVz78D6J8coUO3DJ/Ifz3VwfBvxJdktFZl8wUPZfQX2zD4QLhixEvZptqhBo6D 44xLEJFBsfjotUysQn7aiZ/dTRV1GR0njW8gPcZpzmAf+CnzXko8TokOB8ISA3bHPl/V eI7OicAGd5IKd8yz2N5zJODEgkDXf3CgDGn5DxGMRnnOuH+hhIOIb1HdfEgh3JqUipZH jUq6VmIKb1jVfD+P1OuOzeiOSX1O03Bot3s1AbShV0ZFOFElqbeM6atscFdaTDjRVP0+ bGuYOTH0afWwCW+2QJ4ZMGOuW1Ud71jQc+O3mq4M7dsBJhJov+dSIXqFDorex3FhhDpR Twbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733866688; x=1734471488; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TZdq/UsD7TyngKHwHzlXM4cYkZjAFz4x6rdkvANzCaw=; b=Aktst31DD6lCwmXsBQGAGCX5vsw1qznzwq70LRy8oDg8vcXmVjPxnfgX2XXUHXidKi jNOIQyvj7lVl4cEOjwuwunMZzN1BAhoZrVvT1tRRHLPHp/myjs0kAi6fHKuH6Egsh7Zg M9BBdFqZaKqGK8lV1EleGh7N6xqPaEa1ShD+vjJu7eITRvVV9fIki/le6BdcotC/45M6 xYCInCiPECkdxxFGL73lZ4kABMk4QkVZbtDMNiNi6vI3DBs9VKmJCOraVM3iRql7YM8G B0OiuIyQuYuAWi34jLcDTWxY3xTYYjEzWvt6i41YEPboVSUUoEaXTATRFT8+qN8z41fA PG5Q== X-Gm-Message-State: AOJu0Yy3LygemdzPsJ19lCTT96HT/YWvAdaDpRfOrzJ0mIlbacdG+pDh VHUv9cOwlz6UKct4kL+z/81rk08Kr1bZNXelNpskw89vArfsdHdu7SY2VcH/qZlJi9CavM7eHrp e X-Gm-Gg: ASbGncttwXpZgjcH3mtryPNMgAWhIgvIFplIxFukKixxEOSIguvRt9x44l4VCmhqng6 ZZVAfyM2tY1E1beigWDvIzVrPGnruvx6zl82SMpiec6aBKwFVeGbxOzeX78p1HzN2/1ws8SBE1A u+ndDeSCouFz45cZhTM1WOwqjznX31cq0CuHxtKA6VLyw2Mivb7KxWaNcUPWH43UTwktEmYFwiI JV+ywC4i7n4JkAHOdIJ8wr0KbZQExuOMrG72PLWXe/0NL/k8BtbFQvVM0DXFms5F3fmhJL3tyaf mvaJvcLkHPgjBFG/C9rC9DHkAlHmIcXobnExrzs= X-Google-Smtp-Source: AGHT+IF3STy4JQEZRAPjNUq338yxYvMyJYYh65oB9kh5zvxnD+SeScIdjQ8RjjK1DIBynL8RO06p5Q== X-Received: by 2002:ac8:5dc8:0:b0:467:67db:ef7f with SMTP id d75a77b69052e-4678939d708mr11310171cf.39.1733866687948; Tue, 10 Dec 2024 13:38:07 -0800 (PST) Received: from PC2K9PVX.TheFacebook.com (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-467296f61f6sm65978991cf.43.2024.12.10.13.38.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 13:38:07 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, Feng Tang Subject: [RFC v2 PATCH 0/5] Promotion of Unmapped Page Cache Folios. Date: Tue, 10 Dec 2024 16:37:39 -0500 Message-ID: <20241210213744.2968-1-gourry@gourry.net> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 520051C000A X-Stat-Signature: swpncsofqp94afsweg4ez84ey583rdbk X-Rspam-User: X-HE-Tag: 1733866679-296046 X-HE-Meta: U2FsdGVkX18ALHRMyFyBS9ZzKnEcCV1SGveSxog7BD/GzOUquAt6+6Gq1sbsNlJzVpeodiwcVSI0c1TuwK2TgC47cXl2yycoiPwYR6fnloaQPDAngAv23yT54U/QgKHybqMrfbLDcZZSalmnrBFL2RgBpbZkvLaxzoprMRUReygN09DO6dM+zshgkQKOS1X4inYjVnOWt6jOgu+8GymimgsExUZXA1q+4d+hO7G1YdBETrMMu5qgas9gIsOOsKhoGgbexf6OPOsab2ni37EiGRgMt8pq18ZR9HBjwJQQaX1yzXhwBC6xSgaI957fgfduwHV6zTHS90osvXDdx+kd1lsyN8pE+JBGALjQtH5xlkm6TRoh2noHLb9Ioum4M+1BnVqgRsD8EiX213Yf4T/9N9zDhxi10BHJZeIIzR6/72qPzz6qMJOUgsXiiw/wO+Ev8rU6baWZ63WRvifYIoi5oV9zodxXhtrii8nGz2wJsAY83JOhHeRifo4jYQ2ZEgozLnuuGpb2OJiNseDCYnqKR/mmRuYNbWhHa6AHumIMXb1952G+KTUBnfPzOV3egtPu7lv7/jHTkjfZbmuCawgZvYLvlQDoHSIzjMA0TiTD1agx/yohlw9ZyPUGATgeZc3dWzJBPZY5NixXKsUyZkRNnCGn2jRZvAovJDuOL/4J9hVFaGNbIzwLdtoINiQAP0ktQzZq6EhQr6khbqnWIt1mCltXNQThx0rfYoJv6C+9e+rmJz53dnDbOxcbt1ZOfxuQZoGrWUuSTkgcqFRKaAwHmdHDZSinhO7g64frfU7QCOwzkJG8LrUlfh0lHLQEAwUg7lLRhdTGzwVchr9A+IB4Ly4iN1HPBF9dWsMMYs8JsvYOq4krtSNOJ8X57+fcvpYHaV5779ytybuyt7RV95JUsPy+qI0P7H7paA+Gd66doh1DXEeLTlKbn+E5Jaxvitrs9fKkAwCc6djuck1dOT9 meSPMgas nSRf3ZA1Q/cgfdYDPKa/PJ5JfByIscgQ8PSGRj8/kF18ZUDl+6olWsn8u/bzLF6qnoUB/hdWysI4guUag1fSmIvgR2mVsvblGNG1rwWa16lX1jUktRtNnrn38DPOhAcSbCJlbRZ6VVRgrDXlbtGM50AOqZP76dN9qdc8Nl0OCpYkYc7XmBZcgOzZDwPy7qR5jxzVt94mGOH7BYqh441TahuBRiLbB1uWH6qHpNGi2gG6EnbKCrSVgqB7AKLAiRXteJLFYQWn+4b+OBGAoBAXocEwVusPPXB6JJbX37t+7IQDdfetPkQY6gVN6yBNTcmrkUDv3GnQYm7k6Oo1uKSloSS3Gi5s2vERKjrzR49SZDSa7ub14aMhwXSAsmGTxhNNddeMXfhP2hER6K+HFn8pteABFra99dIyAxqTf81yZJRyogMwhrJp03PeXSopdk3/aV8b1P2sSiSFhmAxxSGEVBWiI0eZo30oWPryBSQ9Q/AucqoZ7bNHoyquOAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Unmapped page cache pages can be demoted to low-tier memory, but they can presently only be promoted in two conditions: 1) The page is fully swapped out and re-faulted 2) The page becomes mapped (and exposed to NUMA hint faults) This RFC proposes promoting unmapped page cache pages by using folio_mark_accessed as a hotness hint for unmapped pages. Patches 1-3 allow NULL as valid input to migration prep interfaces for vmf/vma - which is not present in unmapped folios. Patch 4 adds NUMA_HINT_PAGE_CACHE to vmstat Patch 5 adds the promotion mechanism, along with a sysfs extension which defaults the behavior to off. /sys/kernel/mm/numa/pagecache_promotion_enabled Functional test showed that we are able to reclaim some performance in canned scenarios (a file gets demoted and becomes hot with relatively little contention). See test/overhead section below. v2 - cleanup first commit to be accurate and take Ying's feedback - cleanup NUMA_HINT_ define usage - add NUMA_HINT_ type selection macro to keep code clean - mild comment updates Open Questions: ====== 1) Should we also add a limit to how much can be forced onto a single task's promotion list at any one time? This might piggy-back on the existing TPP promotion limit (256MB?) and would simply add something like task->promo_count. Technically we are limited by the batch read-rate before a TASK_RESUME occurs. 2) Should we exempt certain forms of folios, or add additional knobs/levers in to deal with things like large folios? 3) We added NUMA_HINT_PAGE_CACHE to differentiate hint faults so we could validate the behavior works as intended. Should we just call this a NUMA_HINT_FAULT and not add a new hint? 4) Benchmark suggestions that can pressure 1TB memory. This is not my typical wheelhouse, so if folks know of a useful benchmark that can pressure my 1TB (768 DRAM / 256 CXL) setup, I'd like to add additional measurements here. Development Notes ================= During development, we explored the following proposals: 1) directly promoting within folio_mark_accessed (FMA) Originally suggested by Johannes Weiner https://lore.kernel.org/all/20240803094715.23900-1-gourry@gourry.net/ This caused deadlocks due to the fact that the PTL was held in a variety of cases - but in particular during task exit. It also is incredibly inflexible and causes promotion-on-fault. It was discussed that a deferral mechanism was preferred. 2) promoting in filemap.c locations (calls of FMA) Originally proposed by Feng Tang and Ying Huang https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/patch/?id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329 First, we saw this as less problematic than directly hooking FMA, but we realized this has the potential to miss data in a variety of locations: swap.c, memory.c, gup.c, ksm.c, paddr.c - etc. Second, we discovered that the lock state of pages is very subtle, and that these locations in filemap.c can be called in an atomic context. Prototypes lead to a variety of stalls and lockups. 3) a new LRU - originally proposed by Keith Busch https://git.kernel.org/pub/scm/linux/kernel/git/kbusch/linux.git/patch/?id=6616afe9a722f6ebedbb27ade3848cf07b9a3af7 There are two issues with this approach: PG_promotable and reclaim. First - PG_promotable has generally be discouraged. Second - Attach this mechanism to an LRU is both backwards and counter-intutive. A promotable list is better served by a MOST recently used list, and since LRUs are generally only shrank when exposed to pressure it would require implementing a new promotion list shrinker that runs separate from the existing reclaim logic. 4) Adding a separate kthread - suggested by many This is - to an extent - a more general version of the LRU proposal. We still have to track the folios - which likely requires the addition of a page flag. Additionally, this method would actually contend pretty heavily with LRU behavior - i.e. we'd want to throttle addition to the promotion candidate list in some scenarios. 5) Doing it in task work This seemed to be the most realistic after considering the above. We observe the following: - FMA is an ideal hook for this and isolation is safe here - the new promotion_candidate function is an ideal hook for new filter logic (throttling, fairness, etc). - isolated folios are either promoted or putback on task resume, there are no additional concurrency mechanics to worry about - The mechanic can be made optional via a sysfs hook to avoid overhead in degenerate scenarios (thrashing). We also piggy-backed on the numa_hint_fault_latency timestamp to further throttle promotions to help avoid promotions on one or two time accesses to a particular page. Test: ====== Environment: 1.5-3.7GHz CPU, ~4000 BogoMIPS, 1TB Machine with 768GB DRAM and 256GB CXL A 64GB file being linearly read by 6-7 Python processes Goal: Generate promotions. Demonstrate stability and measure overhead. System Settings: echo 1 > /sys/kernel/mm/numa/demotion_enabled echo 1 > /sys/kernel/mm/numa/pagecache_promotion_enabled echo 2 > /proc/sys/kernel/numa_balancing Each process took up ~128GB each, with anonymous memory growing and shrinking as python filled and released buffers with the 64GB data. This causes DRAM pressure to generate demotions, and file pages to "become hot" - and therefore be selected for promotion. First we ran with promotion disabled to show consistent overhead as a result of forcing a file out to CXL memory. We first ran a single reader to see uncontended performance, launched many readers to force demotions, then droppedb back to a single reader to observe. Single-reader DRAM: ~16.0-16.4s Single-reader CXL (after demotion): ~16.8-17s Next we turned promotion on with only a single reader running. Before promotions: Node 0 MemFree: 636478112 kB Node 0 FilePages: 59009156 kB Node 1 MemFree: 250336004 kB Node 1 FilePages: 14979628 kB After promotions: Node 0 MemFree: 632267268 kB Node 0 FilePages: 72204968 kB Node 1 MemFree: 262567056 kB Node 1 FilePages: 2918768 kB Single-reader (after_promotion): ~16.5s Turning the promotion mechanism on when nothing had been demoted produced no appreciable overhead (memory allocation noise overpowers it) Read time did not change after turning promotion off after promotion occurred, which implies that the additional overhead is not coming from the promotion system itself - but likely other pages still trapped on the low tier. Either way, this at least demonstrates the mechanism is not particularly harmful when there are no pages to promote - and the mechanism is valuable when a file actually is quite hot. Notability, it takes some time for the average read loop to come back down, and there still remains unpromoted file pages trapped in pagecache. This isn't entirely unexpected, there are many files which may have been demoted, and they may not be very hot. Overhead ====== When promotion was tured on we saw a loop-runtime increate temporarily before: 16.8s during: 17.606216192245483 17.375206470489502 17.722095489501953 18.230552434921265 18.20712447166443 18.008254528045654 17.008427381515503 16.851454257965088 16.715774059295654 stable: ~16.5s We measured overhead with a separate patch that simply measured the rdtsc value before/after calls in promotion_candidate and task work. e.g.: + start = rdtsc(); list_for_each_entry_safe(folio, tmp, promo_list, lru) { list_del_init(&folio->lru); migrate_misplaced_folio(folio, NULL, nid); + count++; } + atomic_long_add(rdtsc()-start, &promo_time); + atomic_long_add(count, &promo_count); numa_migrate_prep: 93 - time(3969867917) count(42576860) migrate_misplaced_folio_prepare: 491 - time(3433174319) count(6985523) migrate_misplaced_folio: 1635 - time(11426529980) count(6985523) Thoughts on a good throttling heuristic would be appreciated here. Suggested-by: Huang Ying Suggested-by: Johannes Weiner Suggested-by: Keith Busch Suggested-by: Feng Tang Signed-off-by: Gregory Price Gregory Price (5): migrate: Allow migrate_misplaced_folio_prepare() to accept a NULL VMA. memory: move conditionally defined enums use inside ifdef tags memory: allow non-fault migration in numa_migrate_check path vmstat: add page-cache numa hints migrate,sysfs: add pagecache promotion .../ABI/testing/sysfs-kernel-mm-numa | 20 ++++++ include/linux/memory-tiers.h | 2 + include/linux/migrate.h | 2 + include/linux/sched.h | 3 + include/linux/sched/numa_balancing.h | 5 ++ include/linux/vm_event_item.h | 8 +++ init/init_task.c | 1 + kernel/sched/fair.c | 26 +++++++- mm/memory-tiers.c | 27 ++++++++ mm/memory.c | 32 +++++----- mm/mempolicy.c | 25 +++++--- mm/migrate.c | 61 ++++++++++++++++++- mm/swap.c | 3 + mm/vmstat.c | 2 + 14 files changed, 193 insertions(+), 24 deletions(-)