| Message ID | 20211207022757.2523359-3-ying.huang@intel.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | NUMA balancing: optimize memory placement for memory tiering system | expand |
Hi Huang, >+void set_numabalancing_state(bool enabled) >+{ >+ if (enabled) >+ sysctl_numa_balancing_mode = NUMA_BALANCING_NORMAL; >+ else >+ sysctl_numa_balancing_mode = NUMA_BALANCING_DISABLED; >+ __set_numabalancing_state(enabled); >+} >+ One of the properties of optimized NUMA Balancing for tiered memory is we are not going to scan top-tier nodes as promotion doesn't make sense there (implemented in the next patch [3/6]). However, if a system has only single memory node with CPU, does it make sense to run `NUMA_BALANCING_NORMAL` mode there? What do you think about downgrading to `NUMA_BALANCING_MEMORY_TIERING` mode if a user setup NUMA Balancing on the default mode of `NUMA_BALANCING_NORMAL` on a single toptier memory node? >diff --git a/mm/vmscan.c b/mm/vmscan.c >index c266e64d2f7e..5edb5dfa8900 100644 >--- a/mm/vmscan.c >+++ b/mm/vmscan.c >@@ -56,6 +56,7 @@ > > #include <linux/swapops.h> > #include <linux/balloon_compaction.h> >+#include <linux/sched/sysctl.h> > > #include "internal.h" > >@@ -3919,6 +3920,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) > return false; > } > >+/* >+ * Keep the free pages on fast memory node a little more than the high >+ * watermark to accommodate the promoted pages. >+ */ >+#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT) >+ > /* > * Returns true if there is an eligible zone balanced for the request order > * and highest_zoneidx >@@ -3940,6 +3947,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) > continue; > > mark = high_wmark_pages(zone); >+ if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && >+ numa_demotion_enabled && >+ next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { >+ unsigned long promote_mark; >+ >+ promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK, >+ pgdat->node_present_pages >> 6); >+ mark += promote_mark; >+ } > if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) > return true; > } This can be moved to a different patch. I think, this patch [2/6] can be splitted into two basic patches -- 1. NUMA Balancing interface for tiered memory and 2. maintaining a headroom for promotion. Instead of having a static value for `NUMA_BALANCING_PROMOTE_WATERMARK` what about decoupling the allocation and reclamation and add a user-space interface for controling them? Do you think patch [2/5] and [3/5] of this series can be merged to your current patchset? https://lore.kernel.org/all/cover.1637778851.git.hasanalmaruf@fb.com/ Best, Hasan
Hasan Al Maruf <hasan3050@gmail.com> writes: > Hi Huang, > >>+void set_numabalancing_state(bool enabled) >>+{ >>+ if (enabled) >>+ sysctl_numa_balancing_mode = NUMA_BALANCING_NORMAL; >>+ else >>+ sysctl_numa_balancing_mode = NUMA_BALANCING_DISABLED; >>+ __set_numabalancing_state(enabled); >>+} >>+ > > One of the properties of optimized NUMA Balancing for tiered memory is we > are not going to scan top-tier nodes as promotion doesn't make sense there > (implemented in the next patch [3/6]). However, if a system has only > single memory node with CPU, does it make sense to run > `NUMA_BALANCING_NORMAL` mode there? What do you think about downgrading to > `NUMA_BALANCING_MEMORY_TIERING` mode if a user setup NUMA Balancing on > the default mode of `NUMA_BALANCING_NORMAL` on a single toptier memory > node? Consider a system with only 1 NUMA node and no PMEM, should we refuse NUMA balancing to be enabled at all? Per my understanding, the philosophy behind is to keep thing as small as possible instead of as smart as possible. Do you agree? >>diff --git a/mm/vmscan.c b/mm/vmscan.c >>index c266e64d2f7e..5edb5dfa8900 100644 >>--- a/mm/vmscan.c >>+++ b/mm/vmscan.c >>@@ -56,6 +56,7 @@ >> >> #include <linux/swapops.h> >> #include <linux/balloon_compaction.h> >>+#include <linux/sched/sysctl.h> >> >> #include "internal.h" >> >>@@ -3919,6 +3920,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) >> return false; >> } >> >>+/* >>+ * Keep the free pages on fast memory node a little more than the high >>+ * watermark to accommodate the promoted pages. >>+ */ >>+#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT) >>+ >> /* >> * Returns true if there is an eligible zone balanced for the request order >> * and highest_zoneidx >>@@ -3940,6 +3947,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) >> continue; >> >> mark = high_wmark_pages(zone); >>+ if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && >>+ numa_demotion_enabled && >>+ next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { >>+ unsigned long promote_mark; >>+ >>+ promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK, >>+ pgdat->node_present_pages >> 6); >>+ mark += promote_mark; >>+ } >> if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) >> return true; >> } > > This can be moved to a different patch. I think, this patch [2/6] can be > splitted into two basic patches -- 1. NUMA Balancing interface for tiered > memory and 2. maintaining a headroom for promotion. Johannes has taught me that, if we introduce a new function, variable, or interface, it's better to introduce its user together. So that we can determine whether it's necessary to do that, whether the definition is suitable, etc. I think that makes sense. So I try to do that in this patchset too. As in [2/5] of your patchset below, another possibility is to make 1. NUMA balancing interface for tiered memory and 2. skip scanning top tier memory in NUMA balancing in one patch. One concern is that although this is an optimization, there's almost no measurable performance difference. This makes it hard to justify to extend the user space interface. Do you have better data to support this? > Instead of having a static value for `NUMA_BALANCING_PROMOTE_WATERMARK` > what about decoupling the allocation and reclamation and add a user-space > interface for controling them? This means to add a new user space ABI. Because we may need to support the new ABI forever, we should have strong justification to add it. I am not against to add an ABI to adjust promotion watermark in general. I think that the path could be, - Have a simplest solution that works without introducing new ABI, like something in this patch, or revised. - Then try to add a new ABI in a separate patch with enough justification, for example, with much improved performance data. Do you agree? > Do you think patch [2/5] and [3/5] of this series can be merged to your > current patchset? > > https://lore.kernel.org/all/cover.1637778851.git.hasanalmaruf@fb.com/ Best Regards, Huang, Ying
On 12/7/2021 10:27 AM, Huang Ying wrote: > With the advent of various new memory types, some machines will have > multiple types of memory, e.g. DRAM and PMEM (persistent memory). The > memory subsystem of these machines can be called memory tiering > system, because the performance of the different types of memory are > usually different. > > In such system, because of the memory accessing pattern changing etc, > some pages in the slow memory may become hot globally. So in this > patch, the NUMA balancing mechanism is enhanced to optimize the page > placement among the different memory types according to hot/cold > dynamically. > > In a typical memory tiering system, there are CPUs, fast memory and > slow memory in each physical NUMA node. The CPUs and the fast memory > will be put in one logical node (called fast memory node), while the > slow memory will be put in another (faked) logical node (called slow > memory node). That is, the fast memory is regarded as local while the > slow memory is regarded as remote. So it's possible for the recently > accessed pages in the slow memory node to be promoted to the fast > memory node via the existing NUMA balancing mechanism. > > The original NUMA balancing mechanism will stop to migrate pages if the free > memory of the target node will become below the high watermark. This > is a reasonable policy if there's only one memory type. But this > makes the original NUMA balancing mechanism almost not work to optimize page > placement among different memory types. Details are as follows. > > It's the common cases that the working-set size of the workload is > larger than the size of the fast memory nodes. Otherwise, it's > unnecessary to use the slow memory at all. So in the common cases, > there are almost always no enough free pages in the fast memory nodes, > so that the globally hot pages in the slow memory node cannot be > promoted to the fast memory node. To solve the issue, we have 2 > choices as follows, > > a. Ignore the free pages watermark checking when promoting hot pages > from the slow memory node to the fast memory node. This will > create some memory pressure in the fast memory node, thus trigger > the memory reclaiming. So that, the cold pages in the fast memory > node will be demoted to the slow memory node. > > b. Make kswapd of the fast memory node to reclaim pages until the free > pages are a little more (about 10MB) than the high watermark. Then, > if the free pages of the fast memory node reaches high watermark, and > some hot pages need to be promoted, kswapd of the fast memory node > will be waken up to demote some cold pages in the fast memory node to > the slow memory node. This will free some extra space in the fast > memory node, so the hot pages in the slow memory node can be > promoted to the fast memory node. > > The choice "a" will create the memory pressure in the fast memory > node. If the memory pressure of the workload is high, the memory > pressure may become so high that the memory allocation latency of the > workload is influenced, e.g. the direct reclaiming may be triggered. > > The choice "b" works much better at this aspect. If the memory > pressure of the workload is high, the hot pages promotion will stop > earlier because its allocation watermark is higher than that of the > normal memory allocation. So in this patch, choice "b" is > implemented. > > In addition to the original page placement optimization among sockets, > the NUMA balancing mechanism is extended to be used to optimize page > placement according to hot/cold among different memory types. So the > sysctl user space interface (numa_balancing) is extended in a backward > compatible way as follow, so that the users can enable/disable these > functionality individually. > > The sysctl is converted from a Boolean value to a bits field. The > definition of the flags is, > > - 0x0: NUMA_BALANCING_DISABLED > - 0x1: NUMA_BALANCING_NORMAL > - 0x2: NUMA_BALANCING_MEMORY_TIERING > > Signed-off-by: "Huang, Ying" <ying.huang@intel.com> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Michal Hocko <mhocko@suse.com> > Cc: Rik van Riel <riel@surriel.com> > Cc: Mel Gorman <mgorman@techsingularity.net> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Dave Hansen <dave.hansen@linux.intel.com> > Cc: Yang Shi <shy828301@gmail.com> > Cc: Zi Yan <ziy@nvidia.com> > Cc: Wei Xu <weixugc@google.com> > Cc: osalvador <osalvador@suse.de> > Cc: Shakeel Butt <shakeelb@google.com> > Cc: Hasan Al Maruf <hasanalmaruf@fb.com> > Cc: linux-kernel@vger.kernel.org > Cc: linux-mm@kvack.org > --- > Documentation/admin-guide/sysctl/kernel.rst | 29 ++++++++++++++------- > include/linux/sched/sysctl.h | 10 +++++++ > kernel/sched/core.c | 21 ++++++++++++--- > kernel/sysctl.c | 3 ++- > mm/migrate.c | 19 ++++++++++++-- > mm/vmscan.c | 16 ++++++++++++ > 6 files changed, 82 insertions(+), 16 deletions(-) > > diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst > index 0e486f41185e..5502ea6083ba 100644 > --- a/Documentation/admin-guide/sysctl/kernel.rst > +++ b/Documentation/admin-guide/sysctl/kernel.rst > @@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst). > numa_balancing > ============== > > -Enables/disables automatic page fault based NUMA memory > -balancing. Memory is moved automatically to nodes > -that access it often. > +Enables/disables and configure automatic page fault based NUMA memory > +balancing. Memory is moved automatically to nodes that access it > +often. The value to set can be the result to OR the following, > > -Enables/disables automatic NUMA memory balancing. On NUMA machines, there > -is a performance penalty if remote memory is accessed by a CPU. When this > -feature is enabled the kernel samples what task thread is accessing memory > -by periodically unmapping pages and later trapping a page fault. At the > -time of the page fault, it is determined if the data being accessed should > -be migrated to a local memory node. > += ================================= > +0x0 NUMA_BALANCING_DISABLED > +0x1 NUMA_BALANCING_NORMAL > +0x2 NUMA_BALANCING_MEMORY_TIERING > += ================================= > + > +Or NUMA_BALANCING_NORMAL to optimize page placement among different > +NUMA nodes to reduce remote accessing. On NUMA machines, there is a > +performance penalty if remote memory is accessed by a CPU. When this > +feature is enabled the kernel samples what task thread is accessing > +memory by periodically unmapping pages and later trapping a page > +fault. At the time of the page fault, it is determined if the data > +being accessed should be migrated to a local memory node. > > The unmapping of pages and trapping faults incur additional overhead that > ideally is offset by improved memory locality but there is no universal > @@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, > numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, > numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. > > +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among > +different types of memory (represented as different NUMA nodes) to > +place the hot pages in the fast memory. This is implemented based on > +unmapping and page fault too. > > numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb > =============================================================================================================================== > diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h > index 304f431178fd..bc54c1d75d6d 100644 > --- a/include/linux/sched/sysctl.h > +++ b/include/linux/sched/sysctl.h > @@ -35,6 +35,16 @@ enum sched_tunable_scaling { > SCHED_TUNABLESCALING_END, > }; > > +#define NUMA_BALANCING_DISABLED 0x0 > +#define NUMA_BALANCING_NORMAL 0x1 > +#define NUMA_BALANCING_MEMORY_TIERING 0x2 > + > +#ifdef CONFIG_NUMA_BALANCING > +extern int sysctl_numa_balancing_mode; > +#else > +#define sysctl_numa_balancing_mode 0 > +#endif > + > /* > * control realtime throttling: > * > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > index 3c9b0fda64ac..5dcabc98432f 100644 > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -4265,7 +4265,9 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); > > #ifdef CONFIG_NUMA_BALANCING > > -void set_numabalancing_state(bool enabled) > +int sysctl_numa_balancing_mode; > + > +static void __set_numabalancing_state(bool enabled) > { > if (enabled) > static_branch_enable(&sched_numa_balancing); > @@ -4273,13 +4275,22 @@ void set_numabalancing_state(bool enabled) > static_branch_disable(&sched_numa_balancing); > } > > +void set_numabalancing_state(bool enabled) > +{ > + if (enabled) > + sysctl_numa_balancing_mode = NUMA_BALANCING_NORMAL; > + else > + sysctl_numa_balancing_mode = NUMA_BALANCING_DISABLED; > + __set_numabalancing_state(enabled); > +} > + > #ifdef CONFIG_PROC_SYSCTL > int sysctl_numa_balancing(struct ctl_table *table, int write, > void *buffer, size_t *lenp, loff_t *ppos) > { > struct ctl_table t; > int err; > - int state = static_branch_likely(&sched_numa_balancing); > + int state = sysctl_numa_balancing_mode; > > if (write && !capable(CAP_SYS_ADMIN)) > return -EPERM; > @@ -4289,8 +4300,10 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, > err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos); > if (err < 0) > return err; > - if (write) > - set_numabalancing_state(state); > + if (write) { > + sysctl_numa_balancing_mode = state; > + __set_numabalancing_state(state); > + } > return err; > } > #endif > diff --git a/kernel/sysctl.c b/kernel/sysctl.c > index 083be6af29d7..a1be94ea80ba 100644 > --- a/kernel/sysctl.c > +++ b/kernel/sysctl.c > @@ -115,6 +115,7 @@ static int sixty = 60; > > static int __maybe_unused neg_one = -1; > static int __maybe_unused two = 2; > +static int __maybe_unused three = 3; > static int __maybe_unused four = 4; > static unsigned long zero_ul; > static unsigned long one_ul = 1; > @@ -1808,7 +1809,7 @@ static struct ctl_table kern_table[] = { > .mode = 0644, > .proc_handler = sysctl_numa_balancing, > .extra1 = SYSCTL_ZERO, > - .extra2 = SYSCTL_ONE, > + .extra2 = &three, > }, > #endif /* CONFIG_NUMA_BALANCING */ > { > diff --git a/mm/migrate.c b/mm/migrate.c > index b7c27abb0e5c..286c84c014dd 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -50,6 +50,7 @@ > #include <linux/ptrace.h> > #include <linux/oom.h> > #include <linux/memory.h> > +#include <linux/sched/sysctl.h> > > #include <asm/tlbflush.h> > > @@ -2103,16 +2104,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) > { > int page_lru; > int nr_pages = thp_nr_pages(page); > + int order = compound_order(page); > > - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); > + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); > > /* Do not migrate THP mapped by multiple processes */ > if (PageTransHuge(page) && total_mapcount(page) > 1) > return 0; > > /* Avoid migrating to a node that is nearly full */ > - if (!migrate_balanced_pgdat(pgdat, nr_pages)) > + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { > + int z; > + > + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || > + !numa_demotion_enabled) > + return 0; > + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) > + return 0; > + for (z = pgdat->nr_zones - 1; z >= 0; z--) { > + if (populated_zone(pgdat->node_zones + z)) > + break; > + } > + wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE); > return 0; > + } > > if (isolate_lru_page(page)) > return 0; > diff --git a/mm/vmscan.c b/mm/vmscan.c > index c266e64d2f7e..5edb5dfa8900 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -56,6 +56,7 @@ > > #include <linux/swapops.h> > #include <linux/balloon_compaction.h> > +#include <linux/sched/sysctl.h> > > #include "internal.h" > > @@ -3919,6 +3920,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) > return false; > } > > +/* > + * Keep the free pages on fast memory node a little more than the high > + * watermark to accommodate the promoted pages. > + */ > +#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT) From our testing the fixed promote wartermark is not suitable for all scenarios, but as you said, I also agree with that we can start from a simplest solution that works. So please feel free to add: Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> > + > /* > * Returns true if there is an eligible zone balanced for the request order > * and highest_zoneidx > @@ -3940,6 +3947,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) > continue; > > mark = high_wmark_pages(zone); > + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && > + numa_demotion_enabled && > + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { > + unsigned long promote_mark; > + > + promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK, > + pgdat->node_present_pages >> 6); > + mark += promote_mark; > + } > if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) > return true; > }
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index 0e486f41185e..5502ea6083ba 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst). numa_balancing ============== -Enables/disables automatic page fault based NUMA memory -balancing. Memory is moved automatically to nodes -that access it often. +Enables/disables and configure automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes that access it +often. The value to set can be the result to OR the following, -Enables/disables automatic NUMA memory balancing. On NUMA machines, there -is a performance penalty if remote memory is accessed by a CPU. When this -feature is enabled the kernel samples what task thread is accessing memory -by periodically unmapping pages and later trapping a page fault. At the -time of the page fault, it is determined if the data being accessed should -be migrated to a local memory node. += ================================= +0x0 NUMA_BALANCING_DISABLED +0x1 NUMA_BALANCING_NORMAL +0x2 NUMA_BALANCING_MEMORY_TIERING += ================================= + +Or NUMA_BALANCING_NORMAL to optimize page placement among different +NUMA nodes to reduce remote accessing. On NUMA machines, there is a +performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing +memory by periodically unmapping pages and later trapping a page +fault. At the time of the page fault, it is determined if the data +being accessed should be migrated to a local memory node. The unmapping of pages and trapping faults incur additional overhead that ideally is offset by improved memory locality but there is no universal @@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among +different types of memory (represented as different NUMA nodes) to +place the hot pages in the fast memory. This is implemented based on +unmapping and page fault too. numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb =============================================================================================================================== diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index 304f431178fd..bc54c1d75d6d 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -35,6 +35,16 @@ enum sched_tunable_scaling { SCHED_TUNABLESCALING_END, }; +#define NUMA_BALANCING_DISABLED 0x0 +#define NUMA_BALANCING_NORMAL 0x1 +#define NUMA_BALANCING_MEMORY_TIERING 0x2 + +#ifdef CONFIG_NUMA_BALANCING +extern int sysctl_numa_balancing_mode; +#else +#define sysctl_numa_balancing_mode 0 +#endif + /* * control realtime throttling: * diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3c9b0fda64ac..5dcabc98432f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4265,7 +4265,9 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); #ifdef CONFIG_NUMA_BALANCING -void set_numabalancing_state(bool enabled) +int sysctl_numa_balancing_mode; + +static void __set_numabalancing_state(bool enabled) { if (enabled) static_branch_enable(&sched_numa_balancing); @@ -4273,13 +4275,22 @@ void set_numabalancing_state(bool enabled) static_branch_disable(&sched_numa_balancing); } +void set_numabalancing_state(bool enabled) +{ + if (enabled) + sysctl_numa_balancing_mode = NUMA_BALANCING_NORMAL; + else + sysctl_numa_balancing_mode = NUMA_BALANCING_DISABLED; + __set_numabalancing_state(enabled); +} + #ifdef CONFIG_PROC_SYSCTL int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { struct ctl_table t; int err; - int state = static_branch_likely(&sched_numa_balancing); + int state = sysctl_numa_balancing_mode; if (write && !capable(CAP_SYS_ADMIN)) return -EPERM; @@ -4289,8 +4300,10 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos); if (err < 0) return err; - if (write) - set_numabalancing_state(state); + if (write) { + sysctl_numa_balancing_mode = state; + __set_numabalancing_state(state); + } return err; } #endif diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 083be6af29d7..a1be94ea80ba 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -115,6 +115,7 @@ static int sixty = 60; static int __maybe_unused neg_one = -1; static int __maybe_unused two = 2; +static int __maybe_unused three = 3; static int __maybe_unused four = 4; static unsigned long zero_ul; static unsigned long one_ul = 1; @@ -1808,7 +1809,7 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = sysctl_numa_balancing, .extra1 = SYSCTL_ZERO, - .extra2 = SYSCTL_ONE, + .extra2 = &three, }, #endif /* CONFIG_NUMA_BALANCING */ { diff --git a/mm/migrate.c b/mm/migrate.c index b7c27abb0e5c..286c84c014dd 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -50,6 +50,7 @@ #include <linux/ptrace.h> #include <linux/oom.h> #include <linux/memory.h> +#include <linux/sched/sysctl.h> #include <asm/tlbflush.h> @@ -2103,16 +2104,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; int nr_pages = thp_nr_pages(page); + int order = compound_order(page); - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); /* Do not migrate THP mapped by multiple processes */ if (PageTransHuge(page) && total_mapcount(page) > 1) return 0; /* Avoid migrating to a node that is nearly full */ - if (!migrate_balanced_pgdat(pgdat, nr_pages)) + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_demotion_enabled) + return 0; + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) + return 0; + for (z = pgdat->nr_zones - 1; z >= 0; z--) { + if (populated_zone(pgdat->node_zones + z)) + break; + } + wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE); return 0; + } if (isolate_lru_page(page)) return 0; diff --git a/mm/vmscan.c b/mm/vmscan.c index c266e64d2f7e..5edb5dfa8900 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include <linux/swapops.h> #include <linux/balloon_compaction.h> +#include <linux/sched/sysctl.h> #include "internal.h" @@ -3919,6 +3920,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) return false; } +/* + * Keep the free pages on fast memory node a little more than the high + * watermark to accommodate the promoted pages. + */ +#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT) + /* * Returns true if there is an eligible zone balanced for the request order * and highest_zoneidx @@ -3940,6 +3947,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) continue; mark = high_wmark_pages(zone); + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && + numa_demotion_enabled && + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { + unsigned long promote_mark; + + promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK, + pgdat->node_present_pages >> 6); + mark += promote_mark; + } if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) return true; }
With the advent of various new memory types, some machines will have multiple types of memory, e.g. DRAM and PMEM (persistent memory). The memory subsystem of these machines can be called memory tiering system, because the performance of the different types of memory are usually different. In such system, because of the memory accessing pattern changing etc, some pages in the slow memory may become hot globally. So in this patch, the NUMA balancing mechanism is enhanced to optimize the page placement among the different memory types according to hot/cold dynamically. In a typical memory tiering system, there are CPUs, fast memory and slow memory in each physical NUMA node. The CPUs and the fast memory will be put in one logical node (called fast memory node), while the slow memory will be put in another (faked) logical node (called slow memory node). That is, the fast memory is regarded as local while the slow memory is regarded as remote. So it's possible for the recently accessed pages in the slow memory node to be promoted to the fast memory node via the existing NUMA balancing mechanism. The original NUMA balancing mechanism will stop to migrate pages if the free memory of the target node will become below the high watermark. This is a reasonable policy if there's only one memory type. But this makes the original NUMA balancing mechanism almost not work to optimize page placement among different memory types. Details are as follows. It's the common cases that the working-set size of the workload is larger than the size of the fast memory nodes. Otherwise, it's unnecessary to use the slow memory at all. So in the common cases, there are almost always no enough free pages in the fast memory nodes, so that the globally hot pages in the slow memory node cannot be promoted to the fast memory node. To solve the issue, we have 2 choices as follows, a. Ignore the free pages watermark checking when promoting hot pages from the slow memory node to the fast memory node. This will create some memory pressure in the fast memory node, thus trigger the memory reclaiming. So that, the cold pages in the fast memory node will be demoted to the slow memory node. b. Make kswapd of the fast memory node to reclaim pages until the free pages are a little more (about 10MB) than the high watermark. Then, if the free pages of the fast memory node reaches high watermark, and some hot pages need to be promoted, kswapd of the fast memory node will be waken up to demote some cold pages in the fast memory node to the slow memory node. This will free some extra space in the fast memory node, so the hot pages in the slow memory node can be promoted to the fast memory node. The choice "a" will create the memory pressure in the fast memory node. If the memory pressure of the workload is high, the memory pressure may become so high that the memory allocation latency of the workload is influenced, e.g. the direct reclaiming may be triggered. The choice "b" works much better at this aspect. If the memory pressure of the workload is high, the hot pages promotion will stop earlier because its allocation watermark is higher than that of the normal memory allocation. So in this patch, choice "b" is implemented. In addition to the original page placement optimization among sockets, the NUMA balancing mechanism is extended to be used to optimize page placement according to hot/cold among different memory types. So the sysctl user space interface (numa_balancing) is extended in a backward compatible way as follow, so that the users can enable/disable these functionality individually. The sysctl is converted from a Boolean value to a bits field. The definition of the flags is, - 0x0: NUMA_BALANCING_DISABLED - 0x1: NUMA_BALANCING_NORMAL - 0x2: NUMA_BALANCING_MEMORY_TIERING Signed-off-by: "Huang, Ying" <ying.huang@intel.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Rik van Riel <riel@surriel.com> Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Yang Shi <shy828301@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Wei Xu <weixugc@google.com> Cc: osalvador <osalvador@suse.de> Cc: Shakeel Butt <shakeelb@google.com> Cc: Hasan Al Maruf <hasanalmaruf@fb.com> Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- Documentation/admin-guide/sysctl/kernel.rst | 29 ++++++++++++++------- include/linux/sched/sysctl.h | 10 +++++++ kernel/sched/core.c | 21 ++++++++++++--- kernel/sysctl.c | 3 ++- mm/migrate.c | 19 ++++++++++++-- mm/vmscan.c | 16 ++++++++++++ 6 files changed, 82 insertions(+), 16 deletions(-)