| Message ID | 20210914013701.344956-2-ying.huang@intel.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | NUMA balancing: optimize memory placement for memory tiering system | expand |
On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: > > With the advent of various new memory types, some machines will have > multiple types of memory, e.g. DRAM and PMEM (persistent memory). The > memory subsystem of these machines can be called memory tiering > system, because the performance of the different types of memory are > usually different. > > In such system, because of the memory accessing pattern changing etc, > some pages in the slow memory may become hot globally. So in this > patch, the NUMA balancing mechanism is enhanced to optimize the page > placement among the different memory types according to hot/cold > dynamically. > > In a typical memory tiering system, there are CPUs, fast memory and > slow memory in each physical NUMA node. The CPUs and the fast memory > will be put in one logical node (called fast memory node), while the > slow memory will be put in another (faked) logical node (called slow > memory node). That is, the fast memory is regarded as local while the > slow memory is regarded as remote. So it's possible for the recently > accessed pages in the slow memory node to be promoted to the fast > memory node via the existing NUMA balancing mechanism. > > The original NUMA balancing mechanism will stop to migrate pages if the free > memory of the target node will become below the high watermark. This > is a reasonable policy if there's only one memory type. But this > makes the original NUMA balancing mechanism almost not work to optimize page > placement among different memory types. Details are as follows. > > It's the common cases that the working-set size of the workload is > larger than the size of the fast memory nodes. Otherwise, it's > unnecessary to use the slow memory at all. So in the common cases, > there are almost always no enough free pages in the fast memory nodes, > so that the globally hot pages in the slow memory node cannot be > promoted to the fast memory node. To solve the issue, we have 2 > choices as follows, > > a. Ignore the free pages watermark checking when promoting hot pages > from the slow memory node to the fast memory node. This will > create some memory pressure in the fast memory node, thus trigger > the memory reclaiming. So that, the cold pages in the fast memory > node will be demoted to the slow memory node. > > b. Make kswapd of the fast memory node to reclaim pages until the free > pages are a little more (about 10MB) than the high watermark. Then, > if the free pages of the fast memory node reaches high watermark, and > some hot pages need to be promoted, kswapd of the fast memory node > will be waken up to demote some cold pages in the fast memory node to > the slow memory node. This will free some extra space in the fast > memory node, so the hot pages in the slow memory node can be > promoted to the fast memory node. > > The choice "a" will create the memory pressure in the fast memory > node. If the memory pressure of the workload is high, the memory > pressure may become so high that the memory allocation latency of the > workload is influenced, e.g. the direct reclaiming may be triggered. > > The choice "b" works much better at this aspect. If the memory > pressure of the workload is high, the hot pages promotion will stop > earlier because its allocation watermark is higher than that of the > normal memory allocation. So in this patch, choice "b" is > implemented. > > In addition to the original page placement optimization among sockets, > the NUMA balancing mechanism is extended to be used to optimize page > placement according to hot/cold among different memory types. So the > sysctl user space interface (numa_balancing) is extended in a backward > compatible way as follow, so that the users can enable/disable these > functionality individually. > > The sysctl is converted from a Boolean value to a bits field. The > definition of the flags is, > > - 0x0: NUMA_BALANCING_DISABLED > - 0x1: NUMA_BALANCING_NORMAL > - 0x2: NUMA_BALANCING_MEMORY_TIERING Thanks for coming up with the patches. TBH the first question off the top of my head is all the complexity is really worthy for real life workload at the moment? And the interfaces (sysctl knob files exported to users) look complicated for the users. I don't know if the users know how to set an optimal value for their workloads. I don't disagree the NUMA balancing needs optimization and improvement for tiering memory, the question we need answer is how far we should go for now and what the interfaces should look like. Does it make sense to you? IMHO I'd prefer the most simple and straightforward approach at the moment. For example, we could just skip high water mark check for PMEM promotion. > > TODO: > > - Update ABI document: Documentation/sysctl/kernel.txt > > Signed-off-by: "Huang, Ying" <ying.huang@intel.com> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Michal Hocko <mhocko@suse.com> > Cc: Rik van Riel <riel@surriel.com> > Cc: Mel Gorman <mgorman@suse.de> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Dave Hansen <dave.hansen@linux.intel.com> > Cc: Yang Shi <shy828301@gmail.com> > Cc: Zi Yan <ziy@nvidia.com> > Cc: Wei Xu <weixugc@google.com> > Cc: osalvador <osalvador@suse.de> > Cc: Shakeel Butt <shakeelb@google.com> > Cc: linux-kernel@vger.kernel.org > Cc: linux-mm@kvack.org > --- > include/linux/sched/sysctl.h | 10 ++++++++++ > kernel/sched/core.c | 10 ++++------ > kernel/sysctl.c | 7 ++++--- > mm/migrate.c | 19 +++++++++++++++++-- > mm/vmscan.c | 16 ++++++++++++++++ > 5 files changed, 51 insertions(+), 11 deletions(-) > > diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h > index 304f431178fd..bc54c1d75d6d 100644 > --- a/include/linux/sched/sysctl.h > +++ b/include/linux/sched/sysctl.h > @@ -35,6 +35,16 @@ enum sched_tunable_scaling { > SCHED_TUNABLESCALING_END, > }; > > +#define NUMA_BALANCING_DISABLED 0x0 > +#define NUMA_BALANCING_NORMAL 0x1 > +#define NUMA_BALANCING_MEMORY_TIERING 0x2 > + > +#ifdef CONFIG_NUMA_BALANCING > +extern int sysctl_numa_balancing_mode; > +#else > +#define sysctl_numa_balancing_mode 0 > +#endif > + > /* > * control realtime throttling: > * > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > index 1bba4128a3e6..e61c2d415601 100644 > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -4228,6 +4228,8 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); > > #ifdef CONFIG_NUMA_BALANCING > > +int sysctl_numa_balancing_mode; > + > void set_numabalancing_state(bool enabled) > { > if (enabled) > @@ -4240,20 +4242,16 @@ void set_numabalancing_state(bool enabled) > int sysctl_numa_balancing(struct ctl_table *table, int write, > void *buffer, size_t *lenp, loff_t *ppos) > { > - struct ctl_table t; > int err; > - int state = static_branch_likely(&sched_numa_balancing); > > if (write && !capable(CAP_SYS_ADMIN)) > return -EPERM; > > - t = *table; > - t.data = &state; > - err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos); > + err = proc_dointvec_minmax(table, write, buffer, lenp, ppos); > if (err < 0) > return err; > if (write) > - set_numabalancing_state(state); > + set_numabalancing_state(*(int *)table->data); > return err; > } > #endif > diff --git a/kernel/sysctl.c b/kernel/sysctl.c > index 083be6af29d7..666c58455355 100644 > --- a/kernel/sysctl.c > +++ b/kernel/sysctl.c > @@ -115,6 +115,7 @@ static int sixty = 60; > > static int __maybe_unused neg_one = -1; > static int __maybe_unused two = 2; > +static int __maybe_unused three = 3; > static int __maybe_unused four = 4; > static unsigned long zero_ul; > static unsigned long one_ul = 1; > @@ -1803,12 +1804,12 @@ static struct ctl_table kern_table[] = { > #ifdef CONFIG_NUMA_BALANCING > { > .procname = "numa_balancing", > - .data = NULL, /* filled in by handler */ > - .maxlen = sizeof(unsigned int), > + .data = &sysctl_numa_balancing_mode, > + .maxlen = sizeof(int), > .mode = 0644, > .proc_handler = sysctl_numa_balancing, > .extra1 = SYSCTL_ZERO, > - .extra2 = SYSCTL_ONE, > + .extra2 = &three, > }, > #endif /* CONFIG_NUMA_BALANCING */ > { > diff --git a/mm/migrate.c b/mm/migrate.c > index a6a7743ee98f..a159a36dd412 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -50,6 +50,7 @@ > #include <linux/ptrace.h> > #include <linux/oom.h> > #include <linux/memory.h> > +#include <linux/sched/sysctl.h> > > #include <asm/tlbflush.h> > > @@ -2110,16 +2111,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) > { > int page_lru; > int nr_pages = thp_nr_pages(page); > + int order = compound_order(page); > > - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); > + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); > > /* Do not migrate THP mapped by multiple processes */ > if (PageTransHuge(page) && total_mapcount(page) > 1) > return 0; > > /* Avoid migrating to a node that is nearly full */ > - if (!migrate_balanced_pgdat(pgdat, nr_pages)) > + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { > + int z; > + > + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || > + !numa_demotion_enabled) > + return 0; > + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) > + return 0; > + for (z = pgdat->nr_zones - 1; z >= 0; z--) { > + if (populated_zone(pgdat->node_zones + z)) > + break; > + } > + wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE); > return 0; > + } > > if (isolate_lru_page(page)) > return 0; > diff --git a/mm/vmscan.c b/mm/vmscan.c > index f441c5946a4c..7fe737fd0e03 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -56,6 +56,7 @@ > > #include <linux/swapops.h> > #include <linux/balloon_compaction.h> > +#include <linux/sched/sysctl.h> > > #include "internal.h" > > @@ -3775,6 +3776,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) > return false; > } > > +/* > + * Keep the free pages on fast memory node a little more than the high > + * watermark to accommodate the promoted pages. > + */ > +#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT) > + > /* > * Returns true if there is an eligible zone balanced for the request order > * and highest_zoneidx > @@ -3796,6 +3803,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) > continue; > > mark = high_wmark_pages(zone); > + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && > + numa_demotion_enabled && > + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { > + unsigned long promote_mark; > + > + promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK, > + pgdat->node_present_pages >> 6); > + mark += promote_mark; > + } > if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) > return true; > } > -- > 2.30.2 >
Yang Shi <shy828301@gmail.com> writes: > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: >> >> With the advent of various new memory types, some machines will have >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The >> memory subsystem of these machines can be called memory tiering >> system, because the performance of the different types of memory are >> usually different. >> >> In such system, because of the memory accessing pattern changing etc, >> some pages in the slow memory may become hot globally. So in this >> patch, the NUMA balancing mechanism is enhanced to optimize the page >> placement among the different memory types according to hot/cold >> dynamically. >> >> In a typical memory tiering system, there are CPUs, fast memory and >> slow memory in each physical NUMA node. The CPUs and the fast memory >> will be put in one logical node (called fast memory node), while the >> slow memory will be put in another (faked) logical node (called slow >> memory node). That is, the fast memory is regarded as local while the >> slow memory is regarded as remote. So it's possible for the recently >> accessed pages in the slow memory node to be promoted to the fast >> memory node via the existing NUMA balancing mechanism. >> >> The original NUMA balancing mechanism will stop to migrate pages if the free >> memory of the target node will become below the high watermark. This >> is a reasonable policy if there's only one memory type. But this >> makes the original NUMA balancing mechanism almost not work to optimize page >> placement among different memory types. Details are as follows. >> >> It's the common cases that the working-set size of the workload is >> larger than the size of the fast memory nodes. Otherwise, it's >> unnecessary to use the slow memory at all. So in the common cases, >> there are almost always no enough free pages in the fast memory nodes, >> so that the globally hot pages in the slow memory node cannot be >> promoted to the fast memory node. To solve the issue, we have 2 >> choices as follows, >> >> a. Ignore the free pages watermark checking when promoting hot pages >> from the slow memory node to the fast memory node. This will >> create some memory pressure in the fast memory node, thus trigger >> the memory reclaiming. So that, the cold pages in the fast memory >> node will be demoted to the slow memory node. >> >> b. Make kswapd of the fast memory node to reclaim pages until the free >> pages are a little more (about 10MB) than the high watermark. Then, >> if the free pages of the fast memory node reaches high watermark, and >> some hot pages need to be promoted, kswapd of the fast memory node >> will be waken up to demote some cold pages in the fast memory node to >> the slow memory node. This will free some extra space in the fast >> memory node, so the hot pages in the slow memory node can be >> promoted to the fast memory node. >> >> The choice "a" will create the memory pressure in the fast memory >> node. If the memory pressure of the workload is high, the memory >> pressure may become so high that the memory allocation latency of the >> workload is influenced, e.g. the direct reclaiming may be triggered. >> >> The choice "b" works much better at this aspect. If the memory >> pressure of the workload is high, the hot pages promotion will stop >> earlier because its allocation watermark is higher than that of the >> normal memory allocation. So in this patch, choice "b" is >> implemented. >> >> In addition to the original page placement optimization among sockets, >> the NUMA balancing mechanism is extended to be used to optimize page >> placement according to hot/cold among different memory types. So the >> sysctl user space interface (numa_balancing) is extended in a backward >> compatible way as follow, so that the users can enable/disable these >> functionality individually. >> >> The sysctl is converted from a Boolean value to a bits field. The >> definition of the flags is, >> >> - 0x0: NUMA_BALANCING_DISABLED >> - 0x1: NUMA_BALANCING_NORMAL >> - 0x2: NUMA_BALANCING_MEMORY_TIERING > > Thanks for coming up with the patches. TBH the first question off the > top of my head is all the complexity is really worthy for real life > workload at the moment? And the interfaces (sysctl knob files exported > to users) look complicated for the users. I don't know if the users > know how to set an optimal value for their workloads. > > I don't disagree the NUMA balancing needs optimization and improvement > for tiering memory, the question we need answer is how far we should > go for now and what the interfaces should look like. Does it make > sense to you? > > IMHO I'd prefer the most simple and straightforward approach at the > moment. For example, we could just skip high water mark check for PMEM > promotion. Hi, Yang, Thanks for comments. I understand your concerns about complexity. I have tried to organize the patchset so that the initial patch is as simple as possible and the complexity is introduced step by step. But it seems that your simplest version is even simpler than my one :-) In this patch ([1/6]), I introduced 2 stuff. Firstly, a sysctl knob is provided to disable the NUMA balancing based promotion. Per my understanding, you suggest to remove this. If so, optimizing cross-socket access and promoting hot PMEM pages to DRAM must be enabled/disabled together. If a user wants to enable promoting the hot PMEM pages to DRAM but disable optimizing cross-socket access because they have already bound the CPU of the workload so that there's no much cross-socket access, how can they do? Secondly, we add a promote watermark to the DRAM node so that we can demote/promote pages between the high and promote watermark. Per my understanding, you suggest just to ignore the high watermark checking for promoting. The problem is that this may make the free pages of the DRAM node too few. If many pages are promoted in short time, the free pages will be kept near the min watermark for a while, so that the page allocation from the application will trigger direct reclaiming. We have observed page allocation failure in a test before with a similar policy. Best Regards, Huang, Ying
On Tue, Sep 14, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: > > Yang Shi <shy828301@gmail.com> writes: > > > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: > >> > >> With the advent of various new memory types, some machines will have > >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The > >> memory subsystem of these machines can be called memory tiering > >> system, because the performance of the different types of memory are > >> usually different. > >> > >> In such system, because of the memory accessing pattern changing etc, > >> some pages in the slow memory may become hot globally. So in this > >> patch, the NUMA balancing mechanism is enhanced to optimize the page > >> placement among the different memory types according to hot/cold > >> dynamically. > >> > >> In a typical memory tiering system, there are CPUs, fast memory and > >> slow memory in each physical NUMA node. The CPUs and the fast memory > >> will be put in one logical node (called fast memory node), while the > >> slow memory will be put in another (faked) logical node (called slow > >> memory node). That is, the fast memory is regarded as local while the > >> slow memory is regarded as remote. So it's possible for the recently > >> accessed pages in the slow memory node to be promoted to the fast > >> memory node via the existing NUMA balancing mechanism. > >> > >> The original NUMA balancing mechanism will stop to migrate pages if the free > >> memory of the target node will become below the high watermark. This > >> is a reasonable policy if there's only one memory type. But this > >> makes the original NUMA balancing mechanism almost not work to optimize page > >> placement among different memory types. Details are as follows. > >> > >> It's the common cases that the working-set size of the workload is > >> larger than the size of the fast memory nodes. Otherwise, it's > >> unnecessary to use the slow memory at all. So in the common cases, > >> there are almost always no enough free pages in the fast memory nodes, > >> so that the globally hot pages in the slow memory node cannot be > >> promoted to the fast memory node. To solve the issue, we have 2 > >> choices as follows, > >> > >> a. Ignore the free pages watermark checking when promoting hot pages > >> from the slow memory node to the fast memory node. This will > >> create some memory pressure in the fast memory node, thus trigger > >> the memory reclaiming. So that, the cold pages in the fast memory > >> node will be demoted to the slow memory node. > >> > >> b. Make kswapd of the fast memory node to reclaim pages until the free > >> pages are a little more (about 10MB) than the high watermark. Then, > >> if the free pages of the fast memory node reaches high watermark, and > >> some hot pages need to be promoted, kswapd of the fast memory node > >> will be waken up to demote some cold pages in the fast memory node to > >> the slow memory node. This will free some extra space in the fast > >> memory node, so the hot pages in the slow memory node can be > >> promoted to the fast memory node. > >> > >> The choice "a" will create the memory pressure in the fast memory > >> node. If the memory pressure of the workload is high, the memory > >> pressure may become so high that the memory allocation latency of the > >> workload is influenced, e.g. the direct reclaiming may be triggered. > >> > >> The choice "b" works much better at this aspect. If the memory > >> pressure of the workload is high, the hot pages promotion will stop > >> earlier because its allocation watermark is higher than that of the > >> normal memory allocation. So in this patch, choice "b" is > >> implemented. > >> > >> In addition to the original page placement optimization among sockets, > >> the NUMA balancing mechanism is extended to be used to optimize page > >> placement according to hot/cold among different memory types. So the > >> sysctl user space interface (numa_balancing) is extended in a backward > >> compatible way as follow, so that the users can enable/disable these > >> functionality individually. > >> > >> The sysctl is converted from a Boolean value to a bits field. The > >> definition of the flags is, > >> > >> - 0x0: NUMA_BALANCING_DISABLED > >> - 0x1: NUMA_BALANCING_NORMAL > >> - 0x2: NUMA_BALANCING_MEMORY_TIERING > > > > Thanks for coming up with the patches. TBH the first question off the > > top of my head is all the complexity is really worthy for real life > > workload at the moment? And the interfaces (sysctl knob files exported > > to users) look complicated for the users. I don't know if the users > > know how to set an optimal value for their workloads. > > > > I don't disagree the NUMA balancing needs optimization and improvement > > for tiering memory, the question we need answer is how far we should > > go for now and what the interfaces should look like. Does it make > > sense to you? > > > > IMHO I'd prefer the most simple and straightforward approach at the > > moment. For example, we could just skip high water mark check for PMEM > > promotion. > > Hi, Yang, > > Thanks for comments. > > I understand your concerns about complexity. I have tried to organize > the patchset so that the initial patch is as simple as possible and the > complexity is introduced step by step. But it seems that your simplest > version is even simpler than my one :-) > > In this patch ([1/6]), I introduced 2 stuff. > > Firstly, a sysctl knob is provided to disable the NUMA balancing based > promotion. Per my understanding, you suggest to remove this. If so, > optimizing cross-socket access and promoting hot PMEM pages to DRAM must > be enabled/disabled together. If a user wants to enable promoting the > hot PMEM pages to DRAM but disable optimizing cross-socket access > because they have already bound the CPU of the workload so that there's no > much cross-socket access, how can they do? I should make myself clearer. Here I mean the whole series, not this specific patch. I'm concerned that the interfaces (hint fault latency and ratelimit) are hard to understand and configure for users and whether we go too far at the moment or not. I'm dealing with the end users, I'd admit I'm not even sure how to configure the knobs to achieve optimal performance for different real life workloads. For this specific patch I'm ok to a new promotion mode. There might be usecase that users just want to do promotion between tiered memory but not care about NUMA locality. > > Secondly, we add a promote watermark to the DRAM node so that we can > demote/promote pages between the high and promote watermark. Per my > understanding, you suggest just to ignore the high watermark checking > for promoting. The problem is that this may make the free pages of the > DRAM node too few. If many pages are promoted in short time, the free > pages will be kept near the min watermark for a while, so that the page > allocation from the application will trigger direct reclaiming. We have > observed page allocation failure in a test before with a similar policy. The question is, applicable to the hint fault latency and ratelimit too, we already have some NUMA balancing knobs to control scan period and scan size and watermark knobs to tune how aggressively kswapd works, can they do the same jobs instead of introducing any new knobs? > > Best Regards, > Huang, Ying
Yang Shi <shy828301@gmail.com> writes: > On Tue, Sep 14, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: >> >> Yang Shi <shy828301@gmail.com> writes: >> >> > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: >> >> >> >> With the advent of various new memory types, some machines will have >> >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The >> >> memory subsystem of these machines can be called memory tiering >> >> system, because the performance of the different types of memory are >> >> usually different. >> >> >> >> In such system, because of the memory accessing pattern changing etc, >> >> some pages in the slow memory may become hot globally. So in this >> >> patch, the NUMA balancing mechanism is enhanced to optimize the page >> >> placement among the different memory types according to hot/cold >> >> dynamically. >> >> >> >> In a typical memory tiering system, there are CPUs, fast memory and >> >> slow memory in each physical NUMA node. The CPUs and the fast memory >> >> will be put in one logical node (called fast memory node), while the >> >> slow memory will be put in another (faked) logical node (called slow >> >> memory node). That is, the fast memory is regarded as local while the >> >> slow memory is regarded as remote. So it's possible for the recently >> >> accessed pages in the slow memory node to be promoted to the fast >> >> memory node via the existing NUMA balancing mechanism. >> >> >> >> The original NUMA balancing mechanism will stop to migrate pages if the free >> >> memory of the target node will become below the high watermark. This >> >> is a reasonable policy if there's only one memory type. But this >> >> makes the original NUMA balancing mechanism almost not work to optimize page >> >> placement among different memory types. Details are as follows. >> >> >> >> It's the common cases that the working-set size of the workload is >> >> larger than the size of the fast memory nodes. Otherwise, it's >> >> unnecessary to use the slow memory at all. So in the common cases, >> >> there are almost always no enough free pages in the fast memory nodes, >> >> so that the globally hot pages in the slow memory node cannot be >> >> promoted to the fast memory node. To solve the issue, we have 2 >> >> choices as follows, >> >> >> >> a. Ignore the free pages watermark checking when promoting hot pages >> >> from the slow memory node to the fast memory node. This will >> >> create some memory pressure in the fast memory node, thus trigger >> >> the memory reclaiming. So that, the cold pages in the fast memory >> >> node will be demoted to the slow memory node. >> >> >> >> b. Make kswapd of the fast memory node to reclaim pages until the free >> >> pages are a little more (about 10MB) than the high watermark. Then, >> >> if the free pages of the fast memory node reaches high watermark, and >> >> some hot pages need to be promoted, kswapd of the fast memory node >> >> will be waken up to demote some cold pages in the fast memory node to >> >> the slow memory node. This will free some extra space in the fast >> >> memory node, so the hot pages in the slow memory node can be >> >> promoted to the fast memory node. >> >> >> >> The choice "a" will create the memory pressure in the fast memory >> >> node. If the memory pressure of the workload is high, the memory >> >> pressure may become so high that the memory allocation latency of the >> >> workload is influenced, e.g. the direct reclaiming may be triggered. >> >> >> >> The choice "b" works much better at this aspect. If the memory >> >> pressure of the workload is high, the hot pages promotion will stop >> >> earlier because its allocation watermark is higher than that of the >> >> normal memory allocation. So in this patch, choice "b" is >> >> implemented. >> >> >> >> In addition to the original page placement optimization among sockets, >> >> the NUMA balancing mechanism is extended to be used to optimize page >> >> placement according to hot/cold among different memory types. So the >> >> sysctl user space interface (numa_balancing) is extended in a backward >> >> compatible way as follow, so that the users can enable/disable these >> >> functionality individually. >> >> >> >> The sysctl is converted from a Boolean value to a bits field. The >> >> definition of the flags is, >> >> >> >> - 0x0: NUMA_BALANCING_DISABLED >> >> - 0x1: NUMA_BALANCING_NORMAL >> >> - 0x2: NUMA_BALANCING_MEMORY_TIERING >> > >> > Thanks for coming up with the patches. TBH the first question off the >> > top of my head is all the complexity is really worthy for real life >> > workload at the moment? And the interfaces (sysctl knob files exported >> > to users) look complicated for the users. I don't know if the users >> > know how to set an optimal value for their workloads. >> > >> > I don't disagree the NUMA balancing needs optimization and improvement >> > for tiering memory, the question we need answer is how far we should >> > go for now and what the interfaces should look like. Does it make >> > sense to you? >> > >> > IMHO I'd prefer the most simple and straightforward approach at the >> > moment. For example, we could just skip high water mark check for PMEM >> > promotion. >> >> Hi, Yang, >> >> Thanks for comments. >> >> I understand your concerns about complexity. I have tried to organize >> the patchset so that the initial patch is as simple as possible and the >> complexity is introduced step by step. But it seems that your simplest >> version is even simpler than my one :-) >> >> In this patch ([1/6]), I introduced 2 stuff. >> >> Firstly, a sysctl knob is provided to disable the NUMA balancing based >> promotion. Per my understanding, you suggest to remove this. If so, >> optimizing cross-socket access and promoting hot PMEM pages to DRAM must >> be enabled/disabled together. If a user wants to enable promoting the >> hot PMEM pages to DRAM but disable optimizing cross-socket access >> because they have already bound the CPU of the workload so that there's no >> much cross-socket access, how can they do? > > I should make myself clearer. Here I mean the whole series, not this > specific patch. I'm concerned that the interfaces (hint fault latency > and ratelimit) are hard to understand and configure for users and > whether we go too far at the moment or not. I'm dealing with the end > users, I'd admit I'm not even sure how to configure the knobs to > achieve optimal performance for different real life workloads. Sorry, I misunderstand your original idea. I understand that the knob isn't user-friendly. But sometimes, we cannot avoid it completely :-( In this patchset, I try to introduce the complexity and knobs one by one, and show the performance benefit of each step for people to judge whether the newly added complexity and knob can be complemented by the performance increment. If the benefit of some patches cannot complement its complexity, I am OK to merge just part of the patchset firstly. So how about be more specific? For example, if you are general OK about the complexity and knob introduced by [1-3/6], but have concerns about [4/6], then we can discuss about that specifically? > For this specific patch I'm ok to a new promotion mode. There might be > usecase that users just want to do promotion between tiered memory but > not care about NUMA locality. Yes. >> Secondly, we add a promote watermark to the DRAM node so that we can >> demote/promote pages between the high and promote watermark. Per my >> understanding, you suggest just to ignore the high watermark checking >> for promoting. The problem is that this may make the free pages of the >> DRAM node too few. If many pages are promoted in short time, the free >> pages will be kept near the min watermark for a while, so that the page >> allocation from the application will trigger direct reclaiming. We have >> observed page allocation failure in a test before with a similar policy. > > The question is, applicable to the hint fault latency and ratelimit > too, we already have some NUMA balancing knobs to control scan period > and scan size and watermark knobs to tune how aggressively kswapd > works, can they do the same jobs instead of introducing any new knobs? In this specific patch, we don't introduce a new knob for the page demotion. For other knobs, how about discuss them in the patch that introduce them and one by one? Best Regards, Huang, Ying
On Tue, Sep 14, 2021 at 8:58 PM Huang, Ying <ying.huang@intel.com> wrote: > > Yang Shi <shy828301@gmail.com> writes: > > > On Tue, Sep 14, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: > >> > >> Yang Shi <shy828301@gmail.com> writes: > >> > >> > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: > >> >> > >> >> With the advent of various new memory types, some machines will have > >> >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The > >> >> memory subsystem of these machines can be called memory tiering > >> >> system, because the performance of the different types of memory are > >> >> usually different. > >> >> > >> >> In such system, because of the memory accessing pattern changing etc, > >> >> some pages in the slow memory may become hot globally. So in this > >> >> patch, the NUMA balancing mechanism is enhanced to optimize the page > >> >> placement among the different memory types according to hot/cold > >> >> dynamically. > >> >> > >> >> In a typical memory tiering system, there are CPUs, fast memory and > >> >> slow memory in each physical NUMA node. The CPUs and the fast memory > >> >> will be put in one logical node (called fast memory node), while the > >> >> slow memory will be put in another (faked) logical node (called slow > >> >> memory node). That is, the fast memory is regarded as local while the > >> >> slow memory is regarded as remote. So it's possible for the recently > >> >> accessed pages in the slow memory node to be promoted to the fast > >> >> memory node via the existing NUMA balancing mechanism. > >> >> > >> >> The original NUMA balancing mechanism will stop to migrate pages if the free > >> >> memory of the target node will become below the high watermark. This > >> >> is a reasonable policy if there's only one memory type. But this > >> >> makes the original NUMA balancing mechanism almost not work to optimize page > >> >> placement among different memory types. Details are as follows. > >> >> > >> >> It's the common cases that the working-set size of the workload is > >> >> larger than the size of the fast memory nodes. Otherwise, it's > >> >> unnecessary to use the slow memory at all. So in the common cases, > >> >> there are almost always no enough free pages in the fast memory nodes, > >> >> so that the globally hot pages in the slow memory node cannot be > >> >> promoted to the fast memory node. To solve the issue, we have 2 > >> >> choices as follows, > >> >> > >> >> a. Ignore the free pages watermark checking when promoting hot pages > >> >> from the slow memory node to the fast memory node. This will > >> >> create some memory pressure in the fast memory node, thus trigger > >> >> the memory reclaiming. So that, the cold pages in the fast memory > >> >> node will be demoted to the slow memory node. > >> >> > >> >> b. Make kswapd of the fast memory node to reclaim pages until the free > >> >> pages are a little more (about 10MB) than the high watermark. Then, > >> >> if the free pages of the fast memory node reaches high watermark, and > >> >> some hot pages need to be promoted, kswapd of the fast memory node > >> >> will be waken up to demote some cold pages in the fast memory node to > >> >> the slow memory node. This will free some extra space in the fast > >> >> memory node, so the hot pages in the slow memory node can be > >> >> promoted to the fast memory node. > >> >> > >> >> The choice "a" will create the memory pressure in the fast memory > >> >> node. If the memory pressure of the workload is high, the memory > >> >> pressure may become so high that the memory allocation latency of the > >> >> workload is influenced, e.g. the direct reclaiming may be triggered. > >> >> > >> >> The choice "b" works much better at this aspect. If the memory > >> >> pressure of the workload is high, the hot pages promotion will stop > >> >> earlier because its allocation watermark is higher than that of the > >> >> normal memory allocation. So in this patch, choice "b" is > >> >> implemented. > >> >> > >> >> In addition to the original page placement optimization among sockets, > >> >> the NUMA balancing mechanism is extended to be used to optimize page > >> >> placement according to hot/cold among different memory types. So the > >> >> sysctl user space interface (numa_balancing) is extended in a backward > >> >> compatible way as follow, so that the users can enable/disable these > >> >> functionality individually. > >> >> > >> >> The sysctl is converted from a Boolean value to a bits field. The > >> >> definition of the flags is, > >> >> > >> >> - 0x0: NUMA_BALANCING_DISABLED > >> >> - 0x1: NUMA_BALANCING_NORMAL > >> >> - 0x2: NUMA_BALANCING_MEMORY_TIERING > >> > > >> > Thanks for coming up with the patches. TBH the first question off the > >> > top of my head is all the complexity is really worthy for real life > >> > workload at the moment? And the interfaces (sysctl knob files exported > >> > to users) look complicated for the users. I don't know if the users > >> > know how to set an optimal value for their workloads. > >> > > >> > I don't disagree the NUMA balancing needs optimization and improvement > >> > for tiering memory, the question we need answer is how far we should > >> > go for now and what the interfaces should look like. Does it make > >> > sense to you? > >> > > >> > IMHO I'd prefer the most simple and straightforward approach at the > >> > moment. For example, we could just skip high water mark check for PMEM > >> > promotion. > >> > >> Hi, Yang, > >> > >> Thanks for comments. > >> > >> I understand your concerns about complexity. I have tried to organize > >> the patchset so that the initial patch is as simple as possible and the > >> complexity is introduced step by step. But it seems that your simplest > >> version is even simpler than my one :-) > >> > >> In this patch ([1/6]), I introduced 2 stuff. > >> > >> Firstly, a sysctl knob is provided to disable the NUMA balancing based > >> promotion. Per my understanding, you suggest to remove this. If so, > >> optimizing cross-socket access and promoting hot PMEM pages to DRAM must > >> be enabled/disabled together. If a user wants to enable promoting the > >> hot PMEM pages to DRAM but disable optimizing cross-socket access > >> because they have already bound the CPU of the workload so that there's no > >> much cross-socket access, how can they do? > > > > I should make myself clearer. Here I mean the whole series, not this > > specific patch. I'm concerned that the interfaces (hint fault latency > > and ratelimit) are hard to understand and configure for users and > > whether we go too far at the moment or not. I'm dealing with the end > > users, I'd admit I'm not even sure how to configure the knobs to > > achieve optimal performance for different real life workloads. > > Sorry, I misunderstand your original idea. I understand that the knob > isn't user-friendly. But sometimes, we cannot avoid it completely :-( > In this patchset, I try to introduce the complexity and knobs one by > one, and show the performance benefit of each step for people to judge > whether the newly added complexity and knob can be complemented by the > performance increment. If the benefit of some patches cannot complement > its complexity, I am OK to merge just part of the patchset firstly. Understood. But I really hesitate to go that far at this moment since the picture is not that clear yet IMHO. We have to support them (maybe forever) once we merge them. So I'd prefer to work on the simplest and most necessary stuff for now. Just like how we dealt with demotion. > > So how about be more specific? For example, if you are general OK about > the complexity and knob introduced by [1-3/6], but have concerns about > [4/6], then we can discuss about that specifically? Yeah, we could. > > > For this specific patch I'm ok to a new promotion mode. There might be > > usecase that users just want to do promotion between tiered memory but > > not care about NUMA locality. > > Yes. > > >> Secondly, we add a promote watermark to the DRAM node so that we can > >> demote/promote pages between the high and promote watermark. Per my > >> understanding, you suggest just to ignore the high watermark checking > >> for promoting. The problem is that this may make the free pages of the > >> DRAM node too few. If many pages are promoted in short time, the free > >> pages will be kept near the min watermark for a while, so that the page > >> allocation from the application will trigger direct reclaiming. We have > >> observed page allocation failure in a test before with a similar policy. > > > > The question is, applicable to the hint fault latency and ratelimit > > too, we already have some NUMA balancing knobs to control scan period > > and scan size and watermark knobs to tune how aggressively kswapd > > works, can they do the same jobs instead of introducing any new knobs? > > In this specific patch, we don't introduce a new knob for the page > demotion. For other knobs, how about discuss them in the patch that > introduce them and one by one? That comment is applicable to the watermark hack in this patch too. Per your above description, the problem is the significant amount of promotion in short period of time may deplete free memory. So I'm wondering if the amount of promotion could be ratelimited by NUMA balancing scan period and scan size. I understand this may have some hot pages stay on PMEM for a longer time, but does it really matter? In addition, the gap between low <--> min <--> high could be adjusted by watermark_scale_factor, so kswapd could work more aggressively to keep free memory. > > Best Regards, > Huang, Ying
Yang Shi <shy828301@gmail.com> writes: > On Tue, Sep 14, 2021 at 8:58 PM Huang, Ying <ying.huang@intel.com> wrote: >> >> Yang Shi <shy828301@gmail.com> writes: >> >> > On Tue, Sep 14, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: >> >> >> >> Yang Shi <shy828301@gmail.com> writes: >> >> >> >> > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: >> >> >> >> >> >> With the advent of various new memory types, some machines will have >> >> >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The >> >> >> memory subsystem of these machines can be called memory tiering >> >> >> system, because the performance of the different types of memory are >> >> >> usually different. >> >> >> >> >> >> In such system, because of the memory accessing pattern changing etc, >> >> >> some pages in the slow memory may become hot globally. So in this >> >> >> patch, the NUMA balancing mechanism is enhanced to optimize the page >> >> >> placement among the different memory types according to hot/cold >> >> >> dynamically. >> >> >> >> >> >> In a typical memory tiering system, there are CPUs, fast memory and >> >> >> slow memory in each physical NUMA node. The CPUs and the fast memory >> >> >> will be put in one logical node (called fast memory node), while the >> >> >> slow memory will be put in another (faked) logical node (called slow >> >> >> memory node). That is, the fast memory is regarded as local while the >> >> >> slow memory is regarded as remote. So it's possible for the recently >> >> >> accessed pages in the slow memory node to be promoted to the fast >> >> >> memory node via the existing NUMA balancing mechanism. >> >> >> >> >> >> The original NUMA balancing mechanism will stop to migrate pages if the free >> >> >> memory of the target node will become below the high watermark. This >> >> >> is a reasonable policy if there's only one memory type. But this >> >> >> makes the original NUMA balancing mechanism almost not work to optimize page >> >> >> placement among different memory types. Details are as follows. >> >> >> >> >> >> It's the common cases that the working-set size of the workload is >> >> >> larger than the size of the fast memory nodes. Otherwise, it's >> >> >> unnecessary to use the slow memory at all. So in the common cases, >> >> >> there are almost always no enough free pages in the fast memory nodes, >> >> >> so that the globally hot pages in the slow memory node cannot be >> >> >> promoted to the fast memory node. To solve the issue, we have 2 >> >> >> choices as follows, >> >> >> >> >> >> a. Ignore the free pages watermark checking when promoting hot pages >> >> >> from the slow memory node to the fast memory node. This will >> >> >> create some memory pressure in the fast memory node, thus trigger >> >> >> the memory reclaiming. So that, the cold pages in the fast memory >> >> >> node will be demoted to the slow memory node. >> >> >> >> >> >> b. Make kswapd of the fast memory node to reclaim pages until the free >> >> >> pages are a little more (about 10MB) than the high watermark. Then, >> >> >> if the free pages of the fast memory node reaches high watermark, and >> >> >> some hot pages need to be promoted, kswapd of the fast memory node >> >> >> will be waken up to demote some cold pages in the fast memory node to >> >> >> the slow memory node. This will free some extra space in the fast >> >> >> memory node, so the hot pages in the slow memory node can be >> >> >> promoted to the fast memory node. >> >> >> >> >> >> The choice "a" will create the memory pressure in the fast memory >> >> >> node. If the memory pressure of the workload is high, the memory >> >> >> pressure may become so high that the memory allocation latency of the >> >> >> workload is influenced, e.g. the direct reclaiming may be triggered. >> >> >> >> >> >> The choice "b" works much better at this aspect. If the memory >> >> >> pressure of the workload is high, the hot pages promotion will stop >> >> >> earlier because its allocation watermark is higher than that of the >> >> >> normal memory allocation. So in this patch, choice "b" is >> >> >> implemented. >> >> >> >> >> >> In addition to the original page placement optimization among sockets, >> >> >> the NUMA balancing mechanism is extended to be used to optimize page >> >> >> placement according to hot/cold among different memory types. So the >> >> >> sysctl user space interface (numa_balancing) is extended in a backward >> >> >> compatible way as follow, so that the users can enable/disable these >> >> >> functionality individually. >> >> >> >> >> >> The sysctl is converted from a Boolean value to a bits field. The >> >> >> definition of the flags is, >> >> >> >> >> >> - 0x0: NUMA_BALANCING_DISABLED >> >> >> - 0x1: NUMA_BALANCING_NORMAL >> >> >> - 0x2: NUMA_BALANCING_MEMORY_TIERING >> >> > >> >> > Thanks for coming up with the patches. TBH the first question off the >> >> > top of my head is all the complexity is really worthy for real life >> >> > workload at the moment? And the interfaces (sysctl knob files exported >> >> > to users) look complicated for the users. I don't know if the users >> >> > know how to set an optimal value for their workloads. >> >> > >> >> > I don't disagree the NUMA balancing needs optimization and improvement >> >> > for tiering memory, the question we need answer is how far we should >> >> > go for now and what the interfaces should look like. Does it make >> >> > sense to you? >> >> > >> >> > IMHO I'd prefer the most simple and straightforward approach at the >> >> > moment. For example, we could just skip high water mark check for PMEM >> >> > promotion. >> >> >> >> Hi, Yang, >> >> >> >> Thanks for comments. >> >> >> >> I understand your concerns about complexity. I have tried to organize >> >> the patchset so that the initial patch is as simple as possible and the >> >> complexity is introduced step by step. But it seems that your simplest >> >> version is even simpler than my one :-) >> >> >> >> In this patch ([1/6]), I introduced 2 stuff. >> >> >> >> Firstly, a sysctl knob is provided to disable the NUMA balancing based >> >> promotion. Per my understanding, you suggest to remove this. If so, >> >> optimizing cross-socket access and promoting hot PMEM pages to DRAM must >> >> be enabled/disabled together. If a user wants to enable promoting the >> >> hot PMEM pages to DRAM but disable optimizing cross-socket access >> >> because they have already bound the CPU of the workload so that there's no >> >> much cross-socket access, how can they do? >> > >> > I should make myself clearer. Here I mean the whole series, not this >> > specific patch. I'm concerned that the interfaces (hint fault latency >> > and ratelimit) are hard to understand and configure for users and >> > whether we go too far at the moment or not. I'm dealing with the end >> > users, I'd admit I'm not even sure how to configure the knobs to >> > achieve optimal performance for different real life workloads. >> >> Sorry, I misunderstand your original idea. I understand that the knob >> isn't user-friendly. But sometimes, we cannot avoid it completely :-( >> In this patchset, I try to introduce the complexity and knobs one by >> one, and show the performance benefit of each step for people to judge >> whether the newly added complexity and knob can be complemented by the >> performance increment. If the benefit of some patches cannot complement >> its complexity, I am OK to merge just part of the patchset firstly. > > Understood. But I really hesitate to go that far at this moment since > the picture is not that clear yet IMHO. We have to support them (maybe > forever) once we merge them. OK. The [1-3/6] is the simplest implementation. We can start with that firstly? > So I'd prefer to work on the simplest and most necessary stuff for > now. Just like how we dealt with demotion. > >> >> So how about be more specific? For example, if you are general OK about >> the complexity and knob introduced by [1-3/6], but have concerns about >> [4/6], then we can discuss about that specifically? > > Yeah, we could. > >> >> > For this specific patch I'm ok to a new promotion mode. There might be >> > usecase that users just want to do promotion between tiered memory but >> > not care about NUMA locality. >> >> Yes. >> >> >> Secondly, we add a promote watermark to the DRAM node so that we can >> >> demote/promote pages between the high and promote watermark. Per my >> >> understanding, you suggest just to ignore the high watermark checking >> >> for promoting. The problem is that this may make the free pages of the >> >> DRAM node too few. If many pages are promoted in short time, the free >> >> pages will be kept near the min watermark for a while, so that the page >> >> allocation from the application will trigger direct reclaiming. We have >> >> observed page allocation failure in a test before with a similar policy. >> > >> > The question is, applicable to the hint fault latency and ratelimit >> > too, we already have some NUMA balancing knobs to control scan period >> > and scan size and watermark knobs to tune how aggressively kswapd >> > works, can they do the same jobs instead of introducing any new knobs? >> >> In this specific patch, we don't introduce a new knob for the page >> demotion. For other knobs, how about discuss them in the patch that >> introduce them and one by one? > > That comment is applicable to the watermark hack in this patch too. > Per your above description, the problem is the significant amount of > promotion in short period of time may deplete free memory. So I'm > wondering if the amount of promotion could be ratelimited by NUMA > balancing scan period and scan size. I understand this may have some > hot pages stay on PMEM for a longer time, but does it really matter? > In addition, the gap between low <--> min <--> high could be adjusted > by watermark_scale_factor, so kswapd could work more aggressively to > keep free memory. We can control the NUMA balancing scan speed, but we cannot control the speed of the hint page faults. For example, we scanned a large portion of PMEM without many hint page faults because the pages are really cold, but suddenly a large amount of cold pages become hot, so they will be promoted to DRAM. This will create heavy memory pressure on DRAM node, make it hard for the normal page allocation from the applications. And, for some workloads, we need to promote the hot pages to DRAM quickly, otherwise, the pages will become cold. We should make it possible to support these users too. Do you agree? Best Regards, Huang, Ying
On Wed, Sep 15, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: > > Yang Shi <shy828301@gmail.com> writes: > > > On Tue, Sep 14, 2021 at 8:58 PM Huang, Ying <ying.huang@intel.com> wrote: > >> > >> Yang Shi <shy828301@gmail.com> writes: > >> > >> > On Tue, Sep 14, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: > >> >> > >> >> Yang Shi <shy828301@gmail.com> writes: > >> >> > >> >> > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: > >> >> >> > >> >> >> With the advent of various new memory types, some machines will have > >> >> >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The > >> >> >> memory subsystem of these machines can be called memory tiering > >> >> >> system, because the performance of the different types of memory are > >> >> >> usually different. > >> >> >> > >> >> >> In such system, because of the memory accessing pattern changing etc, > >> >> >> some pages in the slow memory may become hot globally. So in this > >> >> >> patch, the NUMA balancing mechanism is enhanced to optimize the page > >> >> >> placement among the different memory types according to hot/cold > >> >> >> dynamically. > >> >> >> > >> >> >> In a typical memory tiering system, there are CPUs, fast memory and > >> >> >> slow memory in each physical NUMA node. The CPUs and the fast memory > >> >> >> will be put in one logical node (called fast memory node), while the > >> >> >> slow memory will be put in another (faked) logical node (called slow > >> >> >> memory node). That is, the fast memory is regarded as local while the > >> >> >> slow memory is regarded as remote. So it's possible for the recently > >> >> >> accessed pages in the slow memory node to be promoted to the fast > >> >> >> memory node via the existing NUMA balancing mechanism. > >> >> >> > >> >> >> The original NUMA balancing mechanism will stop to migrate pages if the free > >> >> >> memory of the target node will become below the high watermark. This > >> >> >> is a reasonable policy if there's only one memory type. But this > >> >> >> makes the original NUMA balancing mechanism almost not work to optimize page > >> >> >> placement among different memory types. Details are as follows. > >> >> >> > >> >> >> It's the common cases that the working-set size of the workload is > >> >> >> larger than the size of the fast memory nodes. Otherwise, it's > >> >> >> unnecessary to use the slow memory at all. So in the common cases, > >> >> >> there are almost always no enough free pages in the fast memory nodes, > >> >> >> so that the globally hot pages in the slow memory node cannot be > >> >> >> promoted to the fast memory node. To solve the issue, we have 2 > >> >> >> choices as follows, > >> >> >> > >> >> >> a. Ignore the free pages watermark checking when promoting hot pages > >> >> >> from the slow memory node to the fast memory node. This will > >> >> >> create some memory pressure in the fast memory node, thus trigger > >> >> >> the memory reclaiming. So that, the cold pages in the fast memory > >> >> >> node will be demoted to the slow memory node. > >> >> >> > >> >> >> b. Make kswapd of the fast memory node to reclaim pages until the free > >> >> >> pages are a little more (about 10MB) than the high watermark. Then, > >> >> >> if the free pages of the fast memory node reaches high watermark, and > >> >> >> some hot pages need to be promoted, kswapd of the fast memory node > >> >> >> will be waken up to demote some cold pages in the fast memory node to > >> >> >> the slow memory node. This will free some extra space in the fast > >> >> >> memory node, so the hot pages in the slow memory node can be > >> >> >> promoted to the fast memory node. > >> >> >> > >> >> >> The choice "a" will create the memory pressure in the fast memory > >> >> >> node. If the memory pressure of the workload is high, the memory > >> >> >> pressure may become so high that the memory allocation latency of the > >> >> >> workload is influenced, e.g. the direct reclaiming may be triggered. > >> >> >> > >> >> >> The choice "b" works much better at this aspect. If the memory > >> >> >> pressure of the workload is high, the hot pages promotion will stop > >> >> >> earlier because its allocation watermark is higher than that of the > >> >> >> normal memory allocation. So in this patch, choice "b" is > >> >> >> implemented. > >> >> >> > >> >> >> In addition to the original page placement optimization among sockets, > >> >> >> the NUMA balancing mechanism is extended to be used to optimize page > >> >> >> placement according to hot/cold among different memory types. So the > >> >> >> sysctl user space interface (numa_balancing) is extended in a backward > >> >> >> compatible way as follow, so that the users can enable/disable these > >> >> >> functionality individually. > >> >> >> > >> >> >> The sysctl is converted from a Boolean value to a bits field. The > >> >> >> definition of the flags is, > >> >> >> > >> >> >> - 0x0: NUMA_BALANCING_DISABLED > >> >> >> - 0x1: NUMA_BALANCING_NORMAL > >> >> >> - 0x2: NUMA_BALANCING_MEMORY_TIERING > >> >> > > >> >> > Thanks for coming up with the patches. TBH the first question off the > >> >> > top of my head is all the complexity is really worthy for real life > >> >> > workload at the moment? And the interfaces (sysctl knob files exported > >> >> > to users) look complicated for the users. I don't know if the users > >> >> > know how to set an optimal value for their workloads. > >> >> > > >> >> > I don't disagree the NUMA balancing needs optimization and improvement > >> >> > for tiering memory, the question we need answer is how far we should > >> >> > go for now and what the interfaces should look like. Does it make > >> >> > sense to you? > >> >> > > >> >> > IMHO I'd prefer the most simple and straightforward approach at the > >> >> > moment. For example, we could just skip high water mark check for PMEM > >> >> > promotion. > >> >> > >> >> Hi, Yang, > >> >> > >> >> Thanks for comments. > >> >> > >> >> I understand your concerns about complexity. I have tried to organize > >> >> the patchset so that the initial patch is as simple as possible and the > >> >> complexity is introduced step by step. But it seems that your simplest > >> >> version is even simpler than my one :-) > >> >> > >> >> In this patch ([1/6]), I introduced 2 stuff. > >> >> > >> >> Firstly, a sysctl knob is provided to disable the NUMA balancing based > >> >> promotion. Per my understanding, you suggest to remove this. If so, > >> >> optimizing cross-socket access and promoting hot PMEM pages to DRAM must > >> >> be enabled/disabled together. If a user wants to enable promoting the > >> >> hot PMEM pages to DRAM but disable optimizing cross-socket access > >> >> because they have already bound the CPU of the workload so that there's no > >> >> much cross-socket access, how can they do? > >> > > >> > I should make myself clearer. Here I mean the whole series, not this > >> > specific patch. I'm concerned that the interfaces (hint fault latency > >> > and ratelimit) are hard to understand and configure for users and > >> > whether we go too far at the moment or not. I'm dealing with the end > >> > users, I'd admit I'm not even sure how to configure the knobs to > >> > achieve optimal performance for different real life workloads. > >> > >> Sorry, I misunderstand your original idea. I understand that the knob > >> isn't user-friendly. But sometimes, we cannot avoid it completely :-( > >> In this patchset, I try to introduce the complexity and knobs one by > >> one, and show the performance benefit of each step for people to judge > >> whether the newly added complexity and knob can be complemented by the > >> performance increment. If the benefit of some patches cannot complement > >> its complexity, I am OK to merge just part of the patchset firstly. > > > > Understood. But I really hesitate to go that far at this moment since > > the picture is not that clear yet IMHO. We have to support them (maybe > > forever) once we merge them. > > OK. The [1-3/6] is the simplest implementation. We can start with that > firstly? Sure. > > > So I'd prefer to work on the simplest and most necessary stuff for > > now. Just like how we dealt with demotion. > > > >> > >> So how about be more specific? For example, if you are general OK about > >> the complexity and knob introduced by [1-3/6], but have concerns about > >> [4/6], then we can discuss about that specifically? > > > > Yeah, we could. > > > >> > >> > For this specific patch I'm ok to a new promotion mode. There might be > >> > usecase that users just want to do promotion between tiered memory but > >> > not care about NUMA locality. > >> > >> Yes. > >> > >> >> Secondly, we add a promote watermark to the DRAM node so that we can > >> >> demote/promote pages between the high and promote watermark. Per my > >> >> understanding, you suggest just to ignore the high watermark checking > >> >> for promoting. The problem is that this may make the free pages of the > >> >> DRAM node too few. If many pages are promoted in short time, the free > >> >> pages will be kept near the min watermark for a while, so that the page > >> >> allocation from the application will trigger direct reclaiming. We have > >> >> observed page allocation failure in a test before with a similar policy. > >> > > >> > The question is, applicable to the hint fault latency and ratelimit > >> > too, we already have some NUMA balancing knobs to control scan period > >> > and scan size and watermark knobs to tune how aggressively kswapd > >> > works, can they do the same jobs instead of introducing any new knobs? > >> > >> In this specific patch, we don't introduce a new knob for the page > >> demotion. For other knobs, how about discuss them in the patch that > >> introduce them and one by one? > > > > That comment is applicable to the watermark hack in this patch too. > > Per your above description, the problem is the significant amount of > > promotion in short period of time may deplete free memory. So I'm > > wondering if the amount of promotion could be ratelimited by NUMA > > balancing scan period and scan size. I understand this may have some > > hot pages stay on PMEM for a longer time, but does it really matter? > > In addition, the gap between low <--> min <--> high could be adjusted > > by watermark_scale_factor, so kswapd could work more aggressively to > > keep free memory. > > We can control the NUMA balancing scan speed, but we cannot control the > speed of the hint page faults. For example, we scanned a large portion Could adjusting scan size help out? > of PMEM without many hint page faults because the pages are really cold, > but suddenly a large amount of cold pages become hot, so they will be > promoted to DRAM. This will create heavy memory pressure on DRAM node, > make it hard for the normal page allocation from the applications. > > And, for some workloads, we need to promote the hot pages to DRAM > quickly, otherwise, the pages will become cold. We should make it > possible to support these users too. Do you agree? I agree there may be such workloads. But do we have to achieve very good support for them right now? We don't even know how common such workload is. > > Best Regards, > Huang, Ying
Yang Shi <shy828301@gmail.com> writes: > On Wed, Sep 15, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: >> >> Yang Shi <shy828301@gmail.com> writes: >> >> > On Tue, Sep 14, 2021 at 8:58 PM Huang, Ying <ying.huang@intel.com> wrote: >> >> >> >> Yang Shi <shy828301@gmail.com> writes: >> >> >> >> > On Tue, Sep 14, 2021 at 6:45 PM Huang, Ying <ying.huang@intel.com> wrote: >> >> >> >> >> >> Yang Shi <shy828301@gmail.com> writes: >> >> >> >> >> >> > On Mon, Sep 13, 2021 at 6:37 PM Huang Ying <ying.huang@intel.com> wrote: >> >> >> >> >> >> >> >> With the advent of various new memory types, some machines will have >> >> >> >> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The >> >> >> >> memory subsystem of these machines can be called memory tiering >> >> >> >> system, because the performance of the different types of memory are >> >> >> >> usually different. >> >> >> >> >> >> >> >> In such system, because of the memory accessing pattern changing etc, >> >> >> >> some pages in the slow memory may become hot globally. So in this >> >> >> >> patch, the NUMA balancing mechanism is enhanced to optimize the page >> >> >> >> placement among the different memory types according to hot/cold >> >> >> >> dynamically. >> >> >> >> >> >> >> >> In a typical memory tiering system, there are CPUs, fast memory and >> >> >> >> slow memory in each physical NUMA node. The CPUs and the fast memory >> >> >> >> will be put in one logical node (called fast memory node), while the >> >> >> >> slow memory will be put in another (faked) logical node (called slow >> >> >> >> memory node). That is, the fast memory is regarded as local while the >> >> >> >> slow memory is regarded as remote. So it's possible for the recently >> >> >> >> accessed pages in the slow memory node to be promoted to the fast >> >> >> >> memory node via the existing NUMA balancing mechanism. >> >> >> >> >> >> >> >> The original NUMA balancing mechanism will stop to migrate pages if the free >> >> >> >> memory of the target node will become below the high watermark. This >> >> >> >> is a reasonable policy if there's only one memory type. But this >> >> >> >> makes the original NUMA balancing mechanism almost not work to optimize page >> >> >> >> placement among different memory types. Details are as follows. >> >> >> >> >> >> >> >> It's the common cases that the working-set size of the workload is >> >> >> >> larger than the size of the fast memory nodes. Otherwise, it's >> >> >> >> unnecessary to use the slow memory at all. So in the common cases, >> >> >> >> there are almost always no enough free pages in the fast memory nodes, >> >> >> >> so that the globally hot pages in the slow memory node cannot be >> >> >> >> promoted to the fast memory node. To solve the issue, we have 2 >> >> >> >> choices as follows, >> >> >> >> >> >> >> >> a. Ignore the free pages watermark checking when promoting hot pages >> >> >> >> from the slow memory node to the fast memory node. This will >> >> >> >> create some memory pressure in the fast memory node, thus trigger >> >> >> >> the memory reclaiming. So that, the cold pages in the fast memory >> >> >> >> node will be demoted to the slow memory node. >> >> >> >> >> >> >> >> b. Make kswapd of the fast memory node to reclaim pages until the free >> >> >> >> pages are a little more (about 10MB) than the high watermark. Then, >> >> >> >> if the free pages of the fast memory node reaches high watermark, and >> >> >> >> some hot pages need to be promoted, kswapd of the fast memory node >> >> >> >> will be waken up to demote some cold pages in the fast memory node to >> >> >> >> the slow memory node. This will free some extra space in the fast >> >> >> >> memory node, so the hot pages in the slow memory node can be >> >> >> >> promoted to the fast memory node. >> >> >> >> >> >> >> >> The choice "a" will create the memory pressure in the fast memory >> >> >> >> node. If the memory pressure of the workload is high, the memory >> >> >> >> pressure may become so high that the memory allocation latency of the >> >> >> >> workload is influenced, e.g. the direct reclaiming may be triggered. >> >> >> >> >> >> >> >> The choice "b" works much better at this aspect. If the memory >> >> >> >> pressure of the workload is high, the hot pages promotion will stop >> >> >> >> earlier because its allocation watermark is higher than that of the >> >> >> >> normal memory allocation. So in this patch, choice "b" is >> >> >> >> implemented. >> >> >> >> >> >> >> >> In addition to the original page placement optimization among sockets, >> >> >> >> the NUMA balancing mechanism is extended to be used to optimize page >> >> >> >> placement according to hot/cold among different memory types. So the >> >> >> >> sysctl user space interface (numa_balancing) is extended in a backward >> >> >> >> compatible way as follow, so that the users can enable/disable these >> >> >> >> functionality individually. >> >> >> >> >> >> >> >> The sysctl is converted from a Boolean value to a bits field. The >> >> >> >> definition of the flags is, >> >> >> >> >> >> >> >> - 0x0: NUMA_BALANCING_DISABLED >> >> >> >> - 0x1: NUMA_BALANCING_NORMAL >> >> >> >> - 0x2: NUMA_BALANCING_MEMORY_TIERING >> >> >> > >> >> >> > Thanks for coming up with the patches. TBH the first question off the >> >> >> > top of my head is all the complexity is really worthy for real life >> >> >> > workload at the moment? And the interfaces (sysctl knob files exported >> >> >> > to users) look complicated for the users. I don't know if the users >> >> >> > know how to set an optimal value for their workloads. >> >> >> > >> >> >> > I don't disagree the NUMA balancing needs optimization and improvement >> >> >> > for tiering memory, the question we need answer is how far we should >> >> >> > go for now and what the interfaces should look like. Does it make >> >> >> > sense to you? >> >> >> > >> >> >> > IMHO I'd prefer the most simple and straightforward approach at the >> >> >> > moment. For example, we could just skip high water mark check for PMEM >> >> >> > promotion. >> >> >> >> >> >> Hi, Yang, >> >> >> >> >> >> Thanks for comments. >> >> >> >> >> >> I understand your concerns about complexity. I have tried to organize >> >> >> the patchset so that the initial patch is as simple as possible and the >> >> >> complexity is introduced step by step. But it seems that your simplest >> >> >> version is even simpler than my one :-) >> >> >> >> >> >> In this patch ([1/6]), I introduced 2 stuff. >> >> >> >> >> >> Firstly, a sysctl knob is provided to disable the NUMA balancing based >> >> >> promotion. Per my understanding, you suggest to remove this. If so, >> >> >> optimizing cross-socket access and promoting hot PMEM pages to DRAM must >> >> >> be enabled/disabled together. If a user wants to enable promoting the >> >> >> hot PMEM pages to DRAM but disable optimizing cross-socket access >> >> >> because they have already bound the CPU of the workload so that there's no >> >> >> much cross-socket access, how can they do? >> >> > >> >> > I should make myself clearer. Here I mean the whole series, not this >> >> > specific patch. I'm concerned that the interfaces (hint fault latency >> >> > and ratelimit) are hard to understand and configure for users and >> >> > whether we go too far at the moment or not. I'm dealing with the end >> >> > users, I'd admit I'm not even sure how to configure the knobs to >> >> > achieve optimal performance for different real life workloads. >> >> >> >> Sorry, I misunderstand your original idea. I understand that the knob >> >> isn't user-friendly. But sometimes, we cannot avoid it completely :-( >> >> In this patchset, I try to introduce the complexity and knobs one by >> >> one, and show the performance benefit of each step for people to judge >> >> whether the newly added complexity and knob can be complemented by the >> >> performance increment. If the benefit of some patches cannot complement >> >> its complexity, I am OK to merge just part of the patchset firstly. >> > >> > Understood. But I really hesitate to go that far at this moment since >> > the picture is not that clear yet IMHO. We have to support them (maybe >> > forever) once we merge them. >> >> OK. The [1-3/6] is the simplest implementation. We can start with that >> firstly? > > Sure. > >> >> > So I'd prefer to work on the simplest and most necessary stuff for >> > now. Just like how we dealt with demotion. >> > >> >> >> >> So how about be more specific? For example, if you are general OK about >> >> the complexity and knob introduced by [1-3/6], but have concerns about >> >> [4/6], then we can discuss about that specifically? >> > >> > Yeah, we could. >> > >> >> >> >> > For this specific patch I'm ok to a new promotion mode. There might be >> >> > usecase that users just want to do promotion between tiered memory but >> >> > not care about NUMA locality. >> >> >> >> Yes. >> >> >> >> >> Secondly, we add a promote watermark to the DRAM node so that we can >> >> >> demote/promote pages between the high and promote watermark. Per my >> >> >> understanding, you suggest just to ignore the high watermark checking >> >> >> for promoting. The problem is that this may make the free pages of the >> >> >> DRAM node too few. If many pages are promoted in short time, the free >> >> >> pages will be kept near the min watermark for a while, so that the page >> >> >> allocation from the application will trigger direct reclaiming. We have >> >> >> observed page allocation failure in a test before with a similar policy. >> >> > >> >> > The question is, applicable to the hint fault latency and ratelimit >> >> > too, we already have some NUMA balancing knobs to control scan period >> >> > and scan size and watermark knobs to tune how aggressively kswapd >> >> > works, can they do the same jobs instead of introducing any new knobs? >> >> >> >> In this specific patch, we don't introduce a new knob for the page >> >> demotion. For other knobs, how about discuss them in the patch that >> >> introduce them and one by one? >> > >> > That comment is applicable to the watermark hack in this patch too. >> > Per your above description, the problem is the significant amount of >> > promotion in short period of time may deplete free memory. So I'm >> > wondering if the amount of promotion could be ratelimited by NUMA >> > balancing scan period and scan size. I understand this may have some >> > hot pages stay on PMEM for a longer time, but does it really matter? >> > In addition, the gap between low <--> min <--> high could be adjusted >> > by watermark_scale_factor, so kswapd could work more aggressively to >> > keep free memory. >> >> We can control the NUMA balancing scan speed, but we cannot control the >> speed of the hint page faults. For example, we scanned a large portion > > Could adjusting scan size help out? I don't think adjusting scan size helps here. "scan size" just changes how many pages are scanned for one task_work(), it even doesn't change the scan speed. How can it help? >> of PMEM without many hint page faults because the pages are really cold, >> but suddenly a large amount of cold pages become hot, so they will be >> promoted to DRAM. This will create heavy memory pressure on DRAM node, >> make it hard for the normal page allocation from the applications. >> >> And, for some workloads, we need to promote the hot pages to DRAM >> quickly, otherwise, the pages will become cold. We should make it >> possible to support these users too. Do you agree? > > I agree there may be such workloads. But do we have to achieve very > good support for them right now? We don't even know how common such > workload is. The performance of PMEM is much lower than that of DRAM now. If some pages in PMEM becomes hot, the quicker we move these hot PMEM pages to DRAM, the better the performance. So I think this is a common problem. And it's not too hard to implement. This is a small patch anyway. Best Regards, Huang, Ying
diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index 304f431178fd..bc54c1d75d6d 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -35,6 +35,16 @@ enum sched_tunable_scaling { SCHED_TUNABLESCALING_END, }; +#define NUMA_BALANCING_DISABLED 0x0 +#define NUMA_BALANCING_NORMAL 0x1 +#define NUMA_BALANCING_MEMORY_TIERING 0x2 + +#ifdef CONFIG_NUMA_BALANCING +extern int sysctl_numa_balancing_mode; +#else +#define sysctl_numa_balancing_mode 0 +#endif + /* * control realtime throttling: * diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1bba4128a3e6..e61c2d415601 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4228,6 +4228,8 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); #ifdef CONFIG_NUMA_BALANCING +int sysctl_numa_balancing_mode; + void set_numabalancing_state(bool enabled) { if (enabled) @@ -4240,20 +4242,16 @@ void set_numabalancing_state(bool enabled) int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { - struct ctl_table t; int err; - int state = static_branch_likely(&sched_numa_balancing); if (write && !capable(CAP_SYS_ADMIN)) return -EPERM; - t = *table; - t.data = &state; - err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos); + err = proc_dointvec_minmax(table, write, buffer, lenp, ppos); if (err < 0) return err; if (write) - set_numabalancing_state(state); + set_numabalancing_state(*(int *)table->data); return err; } #endif diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 083be6af29d7..666c58455355 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -115,6 +115,7 @@ static int sixty = 60; static int __maybe_unused neg_one = -1; static int __maybe_unused two = 2; +static int __maybe_unused three = 3; static int __maybe_unused four = 4; static unsigned long zero_ul; static unsigned long one_ul = 1; @@ -1803,12 +1804,12 @@ static struct ctl_table kern_table[] = { #ifdef CONFIG_NUMA_BALANCING { .procname = "numa_balancing", - .data = NULL, /* filled in by handler */ - .maxlen = sizeof(unsigned int), + .data = &sysctl_numa_balancing_mode, + .maxlen = sizeof(int), .mode = 0644, .proc_handler = sysctl_numa_balancing, .extra1 = SYSCTL_ZERO, - .extra2 = SYSCTL_ONE, + .extra2 = &three, }, #endif /* CONFIG_NUMA_BALANCING */ { diff --git a/mm/migrate.c b/mm/migrate.c index a6a7743ee98f..a159a36dd412 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -50,6 +50,7 @@ #include <linux/ptrace.h> #include <linux/oom.h> #include <linux/memory.h> +#include <linux/sched/sysctl.h> #include <asm/tlbflush.h> @@ -2110,16 +2111,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; int nr_pages = thp_nr_pages(page); + int order = compound_order(page); - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); /* Do not migrate THP mapped by multiple processes */ if (PageTransHuge(page) && total_mapcount(page) > 1) return 0; /* Avoid migrating to a node that is nearly full */ - if (!migrate_balanced_pgdat(pgdat, nr_pages)) + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_demotion_enabled) + return 0; + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) + return 0; + for (z = pgdat->nr_zones - 1; z >= 0; z--) { + if (populated_zone(pgdat->node_zones + z)) + break; + } + wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE); return 0; + } if (isolate_lru_page(page)) return 0; diff --git a/mm/vmscan.c b/mm/vmscan.c index f441c5946a4c..7fe737fd0e03 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include <linux/swapops.h> #include <linux/balloon_compaction.h> +#include <linux/sched/sysctl.h> #include "internal.h" @@ -3775,6 +3776,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) return false; } +/* + * Keep the free pages on fast memory node a little more than the high + * watermark to accommodate the promoted pages. + */ +#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT) + /* * Returns true if there is an eligible zone balanced for the request order * and highest_zoneidx @@ -3796,6 +3803,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) continue; mark = high_wmark_pages(zone); + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && + numa_demotion_enabled && + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { + unsigned long promote_mark; + + promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK, + pgdat->node_present_pages >> 6); + mark += promote_mark; + } if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) return true; }
With the advent of various new memory types, some machines will have multiple types of memory, e.g. DRAM and PMEM (persistent memory). The memory subsystem of these machines can be called memory tiering system, because the performance of the different types of memory are usually different. In such system, because of the memory accessing pattern changing etc, some pages in the slow memory may become hot globally. So in this patch, the NUMA balancing mechanism is enhanced to optimize the page placement among the different memory types according to hot/cold dynamically. In a typical memory tiering system, there are CPUs, fast memory and slow memory in each physical NUMA node. The CPUs and the fast memory will be put in one logical node (called fast memory node), while the slow memory will be put in another (faked) logical node (called slow memory node). That is, the fast memory is regarded as local while the slow memory is regarded as remote. So it's possible for the recently accessed pages in the slow memory node to be promoted to the fast memory node via the existing NUMA balancing mechanism. The original NUMA balancing mechanism will stop to migrate pages if the free memory of the target node will become below the high watermark. This is a reasonable policy if there's only one memory type. But this makes the original NUMA balancing mechanism almost not work to optimize page placement among different memory types. Details are as follows. It's the common cases that the working-set size of the workload is larger than the size of the fast memory nodes. Otherwise, it's unnecessary to use the slow memory at all. So in the common cases, there are almost always no enough free pages in the fast memory nodes, so that the globally hot pages in the slow memory node cannot be promoted to the fast memory node. To solve the issue, we have 2 choices as follows, a. Ignore the free pages watermark checking when promoting hot pages from the slow memory node to the fast memory node. This will create some memory pressure in the fast memory node, thus trigger the memory reclaiming. So that, the cold pages in the fast memory node will be demoted to the slow memory node. b. Make kswapd of the fast memory node to reclaim pages until the free pages are a little more (about 10MB) than the high watermark. Then, if the free pages of the fast memory node reaches high watermark, and some hot pages need to be promoted, kswapd of the fast memory node will be waken up to demote some cold pages in the fast memory node to the slow memory node. This will free some extra space in the fast memory node, so the hot pages in the slow memory node can be promoted to the fast memory node. The choice "a" will create the memory pressure in the fast memory node. If the memory pressure of the workload is high, the memory pressure may become so high that the memory allocation latency of the workload is influenced, e.g. the direct reclaiming may be triggered. The choice "b" works much better at this aspect. If the memory pressure of the workload is high, the hot pages promotion will stop earlier because its allocation watermark is higher than that of the normal memory allocation. So in this patch, choice "b" is implemented. In addition to the original page placement optimization among sockets, the NUMA balancing mechanism is extended to be used to optimize page placement according to hot/cold among different memory types. So the sysctl user space interface (numa_balancing) is extended in a backward compatible way as follow, so that the users can enable/disable these functionality individually. The sysctl is converted from a Boolean value to a bits field. The definition of the flags is, - 0x0: NUMA_BALANCING_DISABLED - 0x1: NUMA_BALANCING_NORMAL - 0x2: NUMA_BALANCING_MEMORY_TIERING TODO: - Update ABI document: Documentation/sysctl/kernel.txt Signed-off-by: "Huang, Ying" <ying.huang@intel.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Rik van Riel <riel@surriel.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Yang Shi <shy828301@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Wei Xu <weixugc@google.com> Cc: osalvador <osalvador@suse.de> Cc: Shakeel Butt <shakeelb@google.com> Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/sched/sysctl.h | 10 ++++++++++ kernel/sched/core.c | 10 ++++------ kernel/sysctl.c | 7 ++++--- mm/migrate.c | 19 +++++++++++++++++-- mm/vmscan.c | 16 ++++++++++++++++ 5 files changed, 51 insertions(+), 11 deletions(-)