mbox series

[0/5] Transparent Page Placement for Tiered-Memory

Message ID cover.1637778851.git.hasanalmaruf@fb.com (mailing list archive)
Headers show
Series Transparent Page Placement for Tiered-Memory | expand

Message

Hasan Al Maruf Nov. 24, 2021, 6:58 p.m. UTC
[resend in proper format]

With the advent of new memory types and technologies, we can see different
types of memory together, e.g. DRAM, PMEM, CXL-enabled memory, etc. In
recent future, we can see CXL-Memory be available in the physical address-
space as a CPU-less NUMA node along with the native DDR memory channels.
As different types of memory have different level of performance impact,
how we manage pages across the NUMA nodes should be a matter of concern.

Dave Hansen's patchset on "Migrate Pages in lieu of discard" demotes
toptier pages to a slow tier node during the reclamation process.

    		https://lwn.net/Articles/860215/

However, that patchset does not include the features to promote pages on
slow tier memory node to the toptier one. As a result, pages demoted or
newly allocated on the slow tier node, experiences NUMA latency and hurt
application performance. In this patch set, we augment existing AutoNUMA
mechanism to promote pages from slow tier nodes to toptier nodes.

We decouple reclamation and allocation logics for the toptier node so that
reclamation gets triggered at a higher watermark and demotes colder pages
to the slow-tier memory. As a result, toptier nodes can maintain some free
space to accept both new allocation and promotion from slowtier nodes.
During promotion, we add hysteresis to page and only promote pages that
are less likely to be demoted within a short period of time. This reduces
the chance for a page being ping-ponged across the NUMA nodes due to
frequent demotion and promotion within a short period of time.

We tested this patchset on systems with CXL-enabled DRAM and PMEM tiers.
We find this patchset can bring hotter pages to the toptier node while
moving the colder pages to the slow-tier nodes for a good range of Meta
production workloads with live traffic. As a result, toptier nodes serve
more hot pages and the application performance improves.

Case Study of a Meta cache application with two NUMA nodes
==========================================================
Toptier node: DRAM directly attached to the CPU
Slowtier node: DRAM attached through CXL

Toptier vs Slowtier memory capacity ratio is 1:4

With default page placement policy, file caches fills up the toptier node
and anons get trapped in the slowtier node. Only 14% of the total anons
reside in toptier node. Remote NUMA read bandwidth is 80%. Throughput
regression is 18% compared to all memory being served from toptier node.

This patchset brings 80% of the anons to the toptier node. Anons on the
slowtier memory is mostly cold anons. As the toptier node can not host all
the hot memory, some hot files still remain on the slowtier node. Even
though, remote NUMA read bandwidth reduces from 80% to 40%. With this
patchset, throughput regression is only 5% compared to the baseline of
toptier node serving the whole working set.

Hasan Al Maruf (5):
  Promotion and demotion related statistics
  NUMA balancing for tiered-memory system
  Decouple reclaim and allocation for toptier nodes
  Reclaim to satisfy WMARK_DEMOTE on toptier nodes
  active LRU-based promotion to avoid ping-pong

 Documentation/admin-guide/sysctl/kernel.rst | 18 +++++
 Documentation/admin-guide/sysctl/vm.rst     | 12 ++++
 include/linux/mempolicy.h                   | 11 ++-
 include/linux/mm.h                          |  4 ++
 include/linux/mmzone.h                      |  5 ++
 include/linux/node.h                        |  7 ++
 include/linux/page-flags.h                  |  9 +++
 include/linux/page_ext.h                    |  3 +
 include/linux/sched/numa_balancing.h        | 63 ++++++++++++++++-
 include/linux/sched/sysctl.h                |  6 ++
 include/linux/vm_event_item.h               | 13 ++++
 include/trace/events/mmflags.h              | 10 ++-
 kernel/sched/core.c                         | 36 ++++++++--
 kernel/sched/fair.c                         | 23 ++++++-
 kernel/sched/sched.h                        |  2 +
 kernel/sysctl.c                             | 19 ++++--
 mm/huge_memory.c                            | 29 +++++---
 mm/memory.c                                 | 15 +++-
 mm/mempolicy.c                              | 30 +++++++-
 mm/migrate.c                                | 48 ++++++++++---
 mm/mprotect.c                               |  8 ++-
 mm/page_alloc.c                             | 34 ++++++++-
 mm/vmscan.c                                 | 76 +++++++++++++++++++--
 mm/vmstat.c                                 | 20 +++++-
 24 files changed, 451 insertions(+), 50 deletions(-)

--
2.30.2

Comments

Huang, Ying Nov. 25, 2021, 1:23 a.m. UTC | #1
Hasan Al Maruf <hasan3050@gmail.com> writes:

> [resend in proper format]
>
> With the advent of new memory types and technologies, we can see different
> types of memory together, e.g. DRAM, PMEM, CXL-enabled memory, etc. In
> recent future, we can see CXL-Memory be available in the physical address-
> space as a CPU-less NUMA node along with the native DDR memory channels.
> As different types of memory have different level of performance impact,
> how we manage pages across the NUMA nodes should be a matter of concern.
>
> Dave Hansen's patchset on "Migrate Pages in lieu of discard" demotes
> toptier pages to a slow tier node during the reclamation process.
>
>     		https://lwn.net/Articles/860215/
>
> However, that patchset does not include the features to promote pages on
> slow tier memory node to the toptier one. As a result, pages demoted or
> newly allocated on the slow tier node, experiences NUMA latency and hurt
> application performance. In this patch set, we augment existing AutoNUMA
> mechanism to promote pages from slow tier nodes to toptier nodes.
>
> We decouple reclamation and allocation logics for the toptier node so that
> reclamation gets triggered at a higher watermark and demotes colder pages
> to the slow-tier memory. As a result, toptier nodes can maintain some free
> space to accept both new allocation and promotion from slowtier nodes.
> During promotion, we add hysteresis to page and only promote pages that
> are less likely to be demoted within a short period of time. This reduces
> the chance for a page being ping-ponged across the NUMA nodes due to
> frequent demotion and promotion within a short period of time.
>
> We tested this patchset on systems with CXL-enabled DRAM and PMEM tiers.
> We find this patchset can bring hotter pages to the toptier node while
> moving the colder pages to the slow-tier nodes for a good range of Meta
> production workloads with live traffic. As a result, toptier nodes serve
> more hot pages and the application performance improves.
>
> Case Study of a Meta cache application with two NUMA nodes
> ==========================================================
> Toptier node: DRAM directly attached to the CPU
> Slowtier node: DRAM attached through CXL
>
> Toptier vs Slowtier memory capacity ratio is 1:4
>
> With default page placement policy, file caches fills up the toptier node
> and anons get trapped in the slowtier node. Only 14% of the total anons
> reside in toptier node. Remote NUMA read bandwidth is 80%. Throughput
> regression is 18% compared to all memory being served from toptier node.
>
> This patchset brings 80% of the anons to the toptier node. Anons on the
> slowtier memory is mostly cold anons. As the toptier node can not host all
> the hot memory, some hot files still remain on the slowtier node. Even
> though, remote NUMA read bandwidth reduces from 80% to 40%. With this
> patchset, throughput regression is only 5% compared to the baseline of
> toptier node serving the whole working set.

Hi, Hasan,

I found that quite some code in your patchset is exactly same as that in
my patchset as follows,

https://lore.kernel.org/lkml/20211116013522.140575-1-ying.huang@intel.com/

and patches in the following repo we used to publish some patchset that
hasn't been sent to community for review,

https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/log/?h=tiering-0.72

I am glad that more people have interest and worked on optimizing page
placement for tiering memory system.  How about we merge instead of
duplicate our effort?

Because I tried to make the patches above as simple as possible (at
least first 3), can you comment the most basic patches there to help
them to be improved.  And then we can build our more complex/advanced
patches on top of that?

Best Regards,
Huang, Ying
Hasan Al Maruf Nov. 30, 2021, 12:36 a.m. UTC | #2
Hi Huang,

We find the patches in the tiering series are well thought and helpful.
For our workloads, we initially started with that series and we find the
whole series is too complex and some features do not benefit as
expected. Therefore, we have come up with the current basic patches which
are essential and help achieve most of the intended behaviors while
reducing complexity as much as possible.

As we started with your tiering series (with 72 patches), there are
overlaps between our patches and the tiering series. We adopt the
functionalities from the tiering series, modify, and extend them to make
page placement mechanism simpler but workable. Here is the key points for
each of the patches in our Transparent Page Placement series.

Patch #1:
We combine all the promotion and demotion related statistics in this patch
Having statistics on both promotion, demotion, and failures help observe
the systems behavior and reason about performance behavior. Besides, anon
vs file breakdown in both promotion and demotion path help understand
application behavior on a tiered memory systems. As applications may have
different sensitivity toward the anon and file placements, this breakdown
in the migration path is often helpful to assess the effectiveness of the
page placement policy.

Patch #2:
This patch largely overlaps with your current series on NUMA Balancing.
https://lore.kernel.org/lkml/20211116013522.140575-1-ying.huang@intel.com/
This patch is a combination of your Patch #2 and Patch #3 except the
static 10MB free space in the top-tier node to maintain a free headroom
for new allocation and promotion. Rather, we find having a user defined
demote watermark would make it more generic that we include in our patch#3

Patch #3:
This patch has the logic for having a separate demote watermark per node.
In the tiering series, that demote watermark is somewhat bound to the
cgroup and triggered on per-application basis. Besides, It only supports
cgroup-v1. However, we think, instead of cgroup based soft reclamation,
a global per-node demote watermark is more meaningful and should be the
basic one to start with. In that case, the user does not have to think
about per-application setup.

Patch #4:
This patch includes the code for kswapd based reclamation. As I mentioned
earlier, instead of cgroup-based reclamation, here we look whether a node
is balanced during each kswapd invocation. For top-tier node, we check
whether kswapd reclaimed till DEMOTE_WMARK is satisfied, for other nodes
the default mechanism continues. The differences between tiering series
and this patch is the cgroup based reclamation vs per-node reclamation.

Patch #5:
In your patches for promotion, you consider re-fault time for promotion
candidate selection. Although the hot-threshold is tunable, from our
experiments, we find this not helpful to some extent. For example, if
different subset of pages have different re-access time, time-based
promotion should not be able to distinguish between them. If you make
the time window long enough, then any infrequently accessed pages will
also become the promotion candidate, and later be a candidate for the
demotion.

In this patch, we propose LRU based promotion, which would give anon and
files different promotion paths. If pages are used sporadically at high
frequency, irregular pages would be eventually moved from the active LRU
list. We find that our LRU based approach can reduce up to 11x promotion
traffic while retaining the same application throughput for multiple
workloads.

Besides, with promotion rate limit, if files largely get promoted to
top-tier, anon promotion rate often gets hampered as files are taking the
large portion of the total rate (which often happen for applications that
generates huge caches). In our LRU-based approach, each type has their own
separate LRU to check. So for workloads with smaller anons and large file
usage, with LRU-based approach, we can see more anons are being promoted
rather than the files.

I don't mind this patchset being merged to your current patchset under
discussion or any later ones. But, I think this series contains the very
basic functionalities to have a workable page placement mechanism for
tiered-memory. This can obviously be augmented by the other features in
you future tiering series.

Best,
Hasan
Huang, Ying Nov. 30, 2021, 1:29 a.m. UTC | #3
Hi, Hasan,

Hasan Al Maruf <hasan3050@gmail.com> writes:

> Hi Huang,
>
> We find the patches in the tiering series are well thought and helpful.
> For our workloads, we initially started with that series and we find the
> whole series is too complex and some features do not benefit as
> expected. Therefore, we have come up with the current basic patches which
> are essential and help achieve most of the intended behaviors while
> reducing complexity as much as possible.
> As we started with your tiering series (with 72 patches), there are
> overlaps between our patches and the tiering series. We adopt the
> functionalities from the tiering series, modify, and extend them to make
> page placement mechanism simpler but workable.

Thanks for the background!

> Here is the key points for
> each of the patches in our Transparent Page Placement series.
>
> Patch #1:
> We combine all the promotion and demotion related statistics in this patch
> Having statistics on both promotion, demotion, and failures help observe
> the systems behavior and reason about performance behavior. Besides, anon
> vs file breakdown in both promotion and demotion path help understand
> application behavior on a tiered memory systems. As applications may have
> different sensitivity toward the anon and file placements, this breakdown
> in the migration path is often helpful to assess the effectiveness of the
> page placement policy.
>
> Patch #2:
> This patch largely overlaps with your current series on NUMA Balancing.
> https://lore.kernel.org/lkml/20211116013522.140575-1-ying.huang@intel.com/
> This patch is a combination of your Patch #2 and Patch #3 except the
> static 10MB free space in the top-tier node to maintain a free headroom
> for new allocation and promotion. Rather, we find having a user defined
> demote watermark would make it more generic that we include in our patch#3
>
> Patch #3:
> This patch has the logic for having a separate demote watermark per node.
> In the tiering series, that demote watermark is somewhat bound to the
> cgroup and triggered on per-application basis. Besides, It only supports
> cgroup-v1. However, we think, instead of cgroup based soft reclamation,
> a global per-node demote watermark is more meaningful and should be the
> basic one to start with. In that case, the user does not have to think
> about per-application setup.
>
> Patch #4:
> This patch includes the code for kswapd based reclamation. As I mentioned
> earlier, instead of cgroup-based reclamation, here we look whether a node
> is balanced during each kswapd invocation. For top-tier node, we check
> whether kswapd reclaimed till DEMOTE_WMARK is satisfied, for other nodes
> the default mechanism continues. The differences between tiering series
> and this patch is the cgroup based reclamation vs per-node reclamation.
>
> Patch #5:
> In your patches for promotion, you consider re-fault time for promotion
> candidate selection. Although the hot-threshold is tunable, from our
> experiments, we find this not helpful to some extent. For example, if
> different subset of pages have different re-access time, time-based
> promotion should not be able to distinguish between them. If you make
> the time window long enough, then any infrequently accessed pages will
> also become the promotion candidate, and later be a candidate for the
> demotion.
>
> In this patch, we propose LRU based promotion, which would give anon and
> files different promotion paths. If pages are used sporadically at high
> frequency, irregular pages would be eventually moved from the active LRU
> list. We find that our LRU based approach can reduce up to 11x promotion
> traffic while retaining the same application throughput for multiple
> workloads.
>
> Besides, with promotion rate limit, if files largely get promoted to
> top-tier, anon promotion rate often gets hampered as files are taking the
> large portion of the total rate (which often happen for applications that
> generates huge caches). In our LRU-based approach, each type has their own
> separate LRU to check. So for workloads with smaller anons and large file
> usage, with LRU-based approach, we can see more anons are being promoted
> rather than the files.
>
> I don't mind this patchset being merged to your current patchset under
> discussion or any later ones. But, I think this series contains the very
> basic functionalities to have a workable page placement mechanism for
> tiered-memory. This can obviously be augmented by the other features in
> you future tiering series.

Thanks for detailed description!  After reading your patchset and the
description above, I found that the basic part ([1/6] - [3/6]) of the
promotion patchset as follows can be the base for your patchset too.

https://lore.kernel.org/lkml/20211116013522.140575-1-ying.huang@intel.com/

The main problem of that basic patchset is lacking review.  Can I ask
you to help to review that patchset, especially the common base [1/6] -
[3/6]?  If you think the rest of the patchset isn't good enough for you,
we can try to merge just [1/6] - [3/6] firstly.  Do you agree?

Best Regards,
Huang, Ying
Hasan Al Maruf Nov. 30, 2021, 4:49 a.m. UTC | #4
Hi Huang,

>Thanks for detailed description!  After reading your patchset and the
>description above, I found that the basic part ([1/6] - [3/6]) of the
>promotion patchset as follows can be the base for your patchset too.
>
>https://lore.kernel.org/lkml/20211116013522.140575-1-ying.huang@intel.com

Yeah, I agree with you. The first 3 patches of the above series are pretty
much the base of this series. We can add patch [3/5]-[5/5] after your 3
patches. Then add the patch [1/5] to include the statistics counters.

>The main problem of that basic patchset is lacking review.  Can I ask
>you to help to review that patchset, especially the common base [1/6] -
>[3/6]?  If you think the rest of the patchset isn't good enough for you,
>we can try to merge just [1/6] - [3/6] firstly.  Do you agree?

Sounds good to me. I would add my reviews within the next couple of days.

Thanks,
Hasan
luzhixing12345 July 1, 2024, 3:28 p.m. UTC | #5
Thanks,
Lu, Zhixing
luzhixing12345 July 1, 2024, 3:42 p.m. UTC | #6
Hi, I'm trying to apply these patches in linux kernel v5.15 but meet some conflicts,
could you please tell me which commit hash did you start?

Thanks,
Lu, Zhixing