| Message ID | 20221214225123.2770216-1-yuanchu@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | mm: multi-gen LRU: working set extensions | expand |
Yuanchu Xie <yuanchu@google.com> writes: > Introduce a way of monitoring the working set of a workload, per page > type and per NUMA node, with granularity in minutes. It has page-level > granularity and minimal memory overhead by building on the > Multi-generational LRU framework, which already has most of the > infrastructure and is just missing a useful interface. > > MGLRU organizes pages in generations, where an older generation contains > colder pages, and aging promotes the recently used pages into the young > generation and creates a new one. The working set size is how much > memory an application needs to keep working, the amount of "hot" memory > that's frequently used. The only missing pieces between MGLRU > generations and working set estimation are a consistent aging cadence > and an interface; we introduce the two additions. So with kold kthread do we need aging in reclaim ? Should we switch reciam to wakeup up kold kthread to do aging instead of doing try_to_inc_max_seq? This would also help us to try different aging mechanism which can run better in a kthread. > > Periodic aging > ====== > MGLRU Aging is currently driven by reclaim, so the amount of time > between generations is non-deterministic. With memcgs being aged > regularly, MGLRU generations become time-based working set information. > > - memory.periodic_aging: a new root-level only file in cgroupfs > Writing to memory.periodic_aging sets the aging interval and opts into > periodic aging. > - kold: a new kthread that ages memcgs based on the set aging interval. > > Page idle age stats > ====== > - memory.page_idle_age: we group pages into idle age ranges, and present > the number of pages per node per pagetype in each range. This > aggregates the time information from MGLRU generations hierarchically. > > Use case: proactive reclaimer > ====== > The proactive reclaimer sets the aging interval, and periodically reads > the page idle age stats, forming a working set estimation, which it then > calculates an amount to write to memory.reclaim. > > With the page idle age stats, a proactive reclaimer could calculate a > precise amount of memory to reclaim without continuously probing and > inducing reclaim. > > A proactive reclaimer that uses a similar interface is used in the > Google data centers. > > Use case: workload introspection > ====== > A workload may use the working set estimates to adjust application > behavior as needed, e.g. preemptively killing some of its workers to > avoid its working set thrashing, or dropping caches to fit within a > limit. > It can also be valuable to application developers, who can benefit from > an out-of-the-box overview of the application's usage behaviors. > > TODO List > ====== > - selftests > - a userspace demonstrator combining periodic aging, page idle age > stats, memory.reclaim, and/or PSI > > Open questions > ====== > - MGLRU aging mechanism has a flag called force_scan. With > force_scan=false, invoking MGLRU aging when an lruvec has a maximum > number of generations does not actually perform aging. > However, with force_scan=true, MGLRU moves the pages in the oldest > generation to the second oldest generation. The force_scan=true flag > also disables some optimizations in MGLRU's page table walks. > The current patch sets force_scan=true, so that periodic aging would > work without a proactive reclaimer evicting the oldest generation. > > - The page idle age format uses a fixed set of time ranges in seconds. > I have considered having it be based on the aging interval, or just > compiling the raw timestamps. > With the age ranges based on the aging interval, a memcg that's > undergoing memcg reclaim might have its generations in the 10 > seconds range, and a much longer aging interval would obscure this > fact. > The raw timestamps from MGLRU could lead to a very large file when > aggregated hierarchically. > > Yuanchu Xie (2): > mm: multi-gen LRU: periodic aging > mm: multi-gen LRU: cgroup working set stats > > include/linux/kold.h | 44 ++++++++++ > include/linux/mmzone.h | 4 +- > mm/Makefile | 3 + > mm/kold.c | 150 ++++++++++++++++++++++++++++++++ > mm/memcontrol.c | 188 +++++++++++++++++++++++++++++++++++++++++ > mm/vmscan.c | 35 +++++++- > 6 files changed, 422 insertions(+), 2 deletions(-) > create mode 100644 include/linux/kold.h > create mode 100644 mm/kold.c > > -- > 2.39.0.314.g84b9a713c41-goog
On Wed, Dec 14, 2022 at 02:51:21PM -0800, Yuanchu Xie <yuanchu@google.com> wrote: > that's frequently used. The only missing pieces between MGLRU > generations and working set estimation are a consistent aging cadence > and an interface; we introduce the two additions. > > Periodic aging > ====== > MGLRU Aging is currently driven by reclaim, so the amount of time > between generations is non-deterministic. With memcgs being aged > regularly, MGLRU generations become time-based working set information. Is this periodic aging specific to memcgs? IOW, periodic aging isn't needed without memcgs (~with root only) (Perhaps similar question to Aneeh's.) > Use case: proactive reclaimer > ====== > The proactive reclaimer sets the aging interval, and periodically reads > the page idle age stats, forming a working set estimation, which it then > calculates an amount to write to memory.reclaim. > > With the page idle age stats, a proactive reclaimer could calculate a > precise amount of memory to reclaim without continuously probing and > inducing reclaim. Could the aging be also made per-memcg? (Similar to memory.reclaim, possibly without the new kthread (if global reclaim's aging is enough).) Thanks, Michal
On Wed, Jan 11, 2023 at 6:17 AM Michal Koutný <mkoutny@suse.com> wrote: > > On Wed, Dec 14, 2022 at 02:51:21PM -0800, Yuanchu Xie <yuanchu@google.com> wrote: > > that's frequently used. The only missing pieces between MGLRU > > generations and working set estimation are a consistent aging cadence > > and an interface; we introduce the two additions. > > > > Periodic aging > > ====== > > MGLRU Aging is currently driven by reclaim, so the amount of time > > between generations is non-deterministic. With memcgs being aged > > regularly, MGLRU generations become time-based working set information. > > Is this periodic aging specific to memcgs? IOW, periodic aging isn't > needed without memcgs (~with root only) > (Perhaps similar question to Aneeh's.) Originally, I didn't see much value in periodic aging without memcgs, as the main goal was to provide working set information. Periodic aging might lead to MGLRU making better reclaim decisions, but I don't have any benchmarks to back it up right now. > > > Use case: proactive reclaimer > > ====== > > The proactive reclaimer sets the aging interval, and periodically reads > > the page idle age stats, forming a working set estimation, which it then > > calculates an amount to write to memory.reclaim. > > > > With the page idle age stats, a proactive reclaimer could calculate a > > precise amount of memory to reclaim without continuously probing and > > inducing reclaim. > > Could the aging be also made per-memcg? (Similar to memory.reclaim, > possibly without the new kthread (if global reclaim's aging is enough).) It is possible. We can have hierarchical aging, invoked by writing to memory.aging with a time duration. For every child memcg, if its young generation is older than (current time - specified duration), do aging. However, now we need a userspace tool to drive the aging, invoking this interface every few seconds, since every memcg is aged at a different cadence. Having a kthread perform aging has the benefit of simplicity, gives a source of truth for the aging interval, and makes the feature more accessible. The application developers, if they want to take a look at the page idle age stats, could do so without needing additional ceremony. Thanks, Yuanchu
On Mon, Jan 9, 2023 at 10:25 PM Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> wrote: > > Yuanchu Xie <yuanchu@google.com> writes: > > > Introduce a way of monitoring the working set of a workload, per page > > type and per NUMA node, with granularity in minutes. It has page-level > > granularity and minimal memory overhead by building on the > > Multi-generational LRU framework, which already has most of the > > infrastructure and is just missing a useful interface. > > > > MGLRU organizes pages in generations, where an older generation contains > > colder pages, and aging promotes the recently used pages into the young > > generation and creates a new one. The working set size is how much > > memory an application needs to keep working, the amount of "hot" memory > > that's frequently used. The only missing pieces between MGLRU > > generations and working set estimation are a consistent aging cadence > > and an interface; we introduce the two additions. > > So with kold kthread do we need aging in reclaim ? Should we switch reciam > to wakeup up kold kthread to do aging instead of doing try_to_inc_max_seq? > This would also help us to try different aging mechanism which can run > better in a kthread. If I understand correctly, MGLRU tries to put off aging as much as possible for reclaim, and prefers aging uniformly in kswapd, so that already sort of happens. With periodic aging on, reclaim only triggers aging when it reclaims memory less than (MIN_NR_GENS * aging_interval) cold.