| Message ID | 20231109045908.54996-1-byungchul@sk.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Reduce TLB flushes under some specific conditions | expand |
Byungchul Park <byungchul@sk.com> writes: > Hi everyone, > > While I'm working with CXL memory, I have been facing migration overhead > esp. TLB shootdown on promotion or demotion between different tiers. > Yeah.. most TLB shootdowns on migration through hinting fault can be > avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid > flushing TLB in batch if PTE is inaccessible"). > > However, it's only for ones using hinting fault. I thought it'd be much > better if we have a general mechanism to reduce # of TLB flushes and > TLB misses, that we can apply to any type of migration. I tried it only > for tiering migration for now tho. > > I'm suggesting a mechanism to reduce TLB flushes by keeping source and > destination of folios participated in the migrations until all TLB > flushes required are done, only if those folios are not mapped with > write permission PTE entries at all. I worked Based on v6.6-rc5. > > Can you believe it? I saw the number of TLB full flush reduced about > 80% and iTLB miss reduced about 50%, and the time wise performance > always shows at least 1% stable improvement with the workload I tested > with, XSBench. However, I believe that it would help more with other > ones or any real ones. It'd be appreciated to let me know if I'm missing > something. Can you help to test the effect of commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB") for your test case? To test it, you can revert it and compare the performance before and after the reverting. And, how do you trigger migration when testing XSBench? Use a tiered memory system, and migrate pages between DRAM and CXL memory back and forth? If so, how many pages will you migrate for each migration? -- Best Regards, Huang, Ying > > Byungchul > > --- > > Changes from v3: > > 1. Don't use the kconfig, CONFIG_MIGRC, and remove sysctl knob, > migrc_enable. (feedbacked by Nadav) > 2. Remove the optimization skipping CPUs that have already > performed TLB flushes needed by any reason when performing > TLB flushes by migrc because I can't tell the performance > difference between w/ the optimization and w/o that. > (feedbacked by Nadav) > 3. Minimize arch-specific code. While at it, move all the migrc > declarations and inline functions from include/linux/mm.h to > mm/internal.h (feedbacked by Dave Hansen, Nadav) > 4. Separate a part making migrc paused when the system is in > high memory pressure to another patch. (feedbacked by Nadav) > 5. Rename: > a. arch_tlbbatch_clean() to arch_tlbbatch_clear(), > b. tlb_ubc_nowr to tlb_ubc_ro, > c. migrc_try_flush_free_folios() to migrc_flush_free_folios(), > d. migrc_stop to migrc_pause. > (feedbacked by Nadav) > 6. Use ->lru list_head instead of introducing a new llist_head. > (feedbacked by Nadav) > 7. Use non-atomic operations of page-flag when it's safe. > (feedbacked by Nadav) > 8. Use stack instead of keeping a pointer of 'struct migrc_req' > in struct task, which is for manipulating it locally. > (feedbacked by Nadav) > 9. Replace a lot of simple functions to inline functions placed > in a header, mm/internal.h. (feedbacked by Nadav) > 10. Add additional sufficient comments. (feedbacked by Nadav) > 11. Remove a lot of wrapper functions. (feedbacked by Nadav) > > Changes from RFC v2: > > 1. Remove additional occupation in struct page. To do that, > unioned with lru field for migrc's list and added a page > flag. I know page flag is a thing that we don't like to add > but no choice because migrc should distinguish folios under > migrc's control from others. Instead, I force migrc to be > used only on 64 bit system to mitigate you guys from getting > angry. > 2. Remove meaningless internal object allocator that I > introduced to minimize impact onto the system. However, a ton > of tests showed there was no difference. > 3. Stop migrc from working when the system is in high memory > pressure like about to perform direct reclaim. At the > condition where the swap mechanism is heavily used, I found > the system suffered from regression without this control. > 4. Exclude folios that pte_dirty() == true from migrc's interest > so that migrc can work simpler. > 5. Combine several patches that work tightly coupled to one. > 6. Add sufficient comments for better review. > 7. Manage migrc's request in per-node manner (from globally). > 8. Add TLB miss improvement in commit message. > 9. Test with more CPUs(4 -> 16) to see bigger improvement. > > Changes from RFC: > > 1. Fix a bug triggered when a destination folio at the previous > migration becomes a source folio at the next migration, > before the folio gets handled properly so that the folio can > play with another migration. There was inconsistency in the > folio's state. Fixed it. > 2. Split the patch set into more pieces so that the folks can > review better. (Feedbacked by Nadav Amit) > 3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). > (Feedbacked by Nadav Amit) > 4. Tried to add sufficient comments to explain the patch set > better. (Feedbacked by Nadav Amit) > > Byungchul Park (3): > mm/rmap: Recognize read-only TLB entries during batched TLB flush > mm: Defer TLB flush by keeping both src and dst folios at migration > mm: Pause migrc mechanism at high memory pressure > > arch/x86/include/asm/tlbflush.h | 3 + > arch/x86/mm/tlb.c | 11 ++ > include/linux/mm_types.h | 21 +++ > include/linux/mmzone.h | 9 ++ > include/linux/page-flags.h | 4 + > include/linux/sched.h | 7 + > include/trace/events/mmflags.h | 3 +- > mm/internal.h | 78 ++++++++++ > mm/memory.c | 11 ++ > mm/migrate.c | 266 ++++++++++++++++++++++++++++++++ > mm/page_alloc.c | 30 +++- > mm/rmap.c | 35 ++++- > 12 files changed, 475 insertions(+), 3 deletions(-)
On 11/8/23 20:59, Byungchul Park wrote: > Can you believe it? I saw the number of TLB full flush reduced about > 80% and iTLB miss reduced about 50%, and the time wise performance > always shows at least 1% stable improvement with the workload I tested > with, XSBench. However, I believe that it would help more with other > ones or any real ones. It'd be appreciated to let me know if I'm missing > something. I see that you've moved a substantial amount of code out of arch/x86. That's great. But there doesn't appear to be any improvement in the justification or performance data. The page flag is also here, which is horribly frowned upon. It's an absolute no-go with this level of justification. I'd really suggest not sending any more of these out until those issues are rectified. I know I definitely won't be reviewing them in this state.
On Thu, Nov 09, 2023 at 06:26:08AM -0800, Dave Hansen wrote: > On 11/8/23 20:59, Byungchul Park wrote: > > Can you believe it? I saw the number of TLB full flush reduced about > > 80% and iTLB miss reduced about 50%, and the time wise performance > > always shows at least 1% stable improvement with the workload I tested > > with, XSBench. However, I believe that it would help more with other > > ones or any real ones. It'd be appreciated to let me know if I'm missing > > something. > > I see that you've moved a substantial amount of code out of arch/x86. > That's great. > > But there doesn't appear to be any improvement in the justification or > performance data. The page flag is also here, which is horribly frowned > upon. It's an absolute no-go with this level of justification. > > I'd really suggest not sending any more of these out until those issues > are rectified. I know I definitely won't be reviewing them in this state. Make sense. Lemme think it more and improve it. Byungchul
On Thu, Nov 09, 2023 at 01:20:29PM +0800, Huang, Ying wrote: > Byungchul Park <byungchul@sk.com> writes: > > > Hi everyone, > > > > While I'm working with CXL memory, I have been facing migration overhead > > esp. TLB shootdown on promotion or demotion between different tiers. > > Yeah.. most TLB shootdowns on migration through hinting fault can be > > avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid > > flushing TLB in batch if PTE is inaccessible"). > > > > However, it's only for ones using hinting fault. I thought it'd be much > > better if we have a general mechanism to reduce # of TLB flushes and > > TLB misses, that we can apply to any type of migration. I tried it only > > for tiering migration for now tho. > > > > I'm suggesting a mechanism to reduce TLB flushes by keeping source and > > destination of folios participated in the migrations until all TLB > > flushes required are done, only if those folios are not mapped with > > write permission PTE entries at all. I worked Based on v6.6-rc5. > > > > Can you believe it? I saw the number of TLB full flush reduced about > > 80% and iTLB miss reduced about 50%, and the time wise performance > > always shows at least 1% stable improvement with the workload I tested > > with, XSBench. However, I believe that it would help more with other > > ones or any real ones. It'd be appreciated to let me know if I'm missing > > something. > > Can you help to test the effect of commit 7e12beb8ca2a ("migrate_pages: > batch flushing TLB") for your test case? To test it, you can revert it > and compare the performance before and after the reverting. I will. > And, how do you trigger migration when testing XSBench? Use a tiered > memory system, and migrate pages between DRAM and CXL memory back and > forth? If so, how many pages will you migrate for each migration? Honestly I've been focusing on the migration # and TLB #. I will get back to you. Byungchul > -- > Best Regards, > Huang, Ying > > > > > Byungchul > > > > --- > > > > Changes from v3: > > > > 1. Don't use the kconfig, CONFIG_MIGRC, and remove sysctl knob, > > migrc_enable. (feedbacked by Nadav) > > 2. Remove the optimization skipping CPUs that have already > > performed TLB flushes needed by any reason when performing > > TLB flushes by migrc because I can't tell the performance > > difference between w/ the optimization and w/o that. > > (feedbacked by Nadav) > > 3. Minimize arch-specific code. While at it, move all the migrc > > declarations and inline functions from include/linux/mm.h to > > mm/internal.h (feedbacked by Dave Hansen, Nadav) > > 4. Separate a part making migrc paused when the system is in > > high memory pressure to another patch. (feedbacked by Nadav) > > 5. Rename: > > a. arch_tlbbatch_clean() to arch_tlbbatch_clear(), > > b. tlb_ubc_nowr to tlb_ubc_ro, > > c. migrc_try_flush_free_folios() to migrc_flush_free_folios(), > > d. migrc_stop to migrc_pause. > > (feedbacked by Nadav) > > 6. Use ->lru list_head instead of introducing a new llist_head. > > (feedbacked by Nadav) > > 7. Use non-atomic operations of page-flag when it's safe. > > (feedbacked by Nadav) > > 8. Use stack instead of keeping a pointer of 'struct migrc_req' > > in struct task, which is for manipulating it locally. > > (feedbacked by Nadav) > > 9. Replace a lot of simple functions to inline functions placed > > in a header, mm/internal.h. (feedbacked by Nadav) > > 10. Add additional sufficient comments. (feedbacked by Nadav) > > 11. Remove a lot of wrapper functions. (feedbacked by Nadav) > > > > Changes from RFC v2: > > > > 1. Remove additional occupation in struct page. To do that, > > unioned with lru field for migrc's list and added a page > > flag. I know page flag is a thing that we don't like to add > > but no choice because migrc should distinguish folios under > > migrc's control from others. Instead, I force migrc to be > > used only on 64 bit system to mitigate you guys from getting > > angry. > > 2. Remove meaningless internal object allocator that I > > introduced to minimize impact onto the system. However, a ton > > of tests showed there was no difference. > > 3. Stop migrc from working when the system is in high memory > > pressure like about to perform direct reclaim. At the > > condition where the swap mechanism is heavily used, I found > > the system suffered from regression without this control. > > 4. Exclude folios that pte_dirty() == true from migrc's interest > > so that migrc can work simpler. > > 5. Combine several patches that work tightly coupled to one. > > 6. Add sufficient comments for better review. > > 7. Manage migrc's request in per-node manner (from globally). > > 8. Add TLB miss improvement in commit message. > > 9. Test with more CPUs(4 -> 16) to see bigger improvement. > > > > Changes from RFC: > > > > 1. Fix a bug triggered when a destination folio at the previous > > migration becomes a source folio at the next migration, > > before the folio gets handled properly so that the folio can > > play with another migration. There was inconsistency in the > > folio's state. Fixed it. > > 2. Split the patch set into more pieces so that the folks can > > review better. (Feedbacked by Nadav Amit) > > 3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). > > (Feedbacked by Nadav Amit) > > 4. Tried to add sufficient comments to explain the patch set > > better. (Feedbacked by Nadav Amit) > > > > Byungchul Park (3): > > mm/rmap: Recognize read-only TLB entries during batched TLB flush > > mm: Defer TLB flush by keeping both src and dst folios at migration > > mm: Pause migrc mechanism at high memory pressure > > > > arch/x86/include/asm/tlbflush.h | 3 + > > arch/x86/mm/tlb.c | 11 ++ > > include/linux/mm_types.h | 21 +++ > > include/linux/mmzone.h | 9 ++ > > include/linux/page-flags.h | 4 + > > include/linux/sched.h | 7 + > > include/trace/events/mmflags.h | 3 +- > > mm/internal.h | 78 ++++++++++ > > mm/memory.c | 11 ++ > > mm/migrate.c | 266 ++++++++++++++++++++++++++++++++ > > mm/page_alloc.c | 30 +++- > > mm/rmap.c | 35 ++++- > > 12 files changed, 475 insertions(+), 3 deletions(-)
On Thu, Nov 09, 2023 at 01:20:29PM +0800, Huang, Ying wrote: > Byungchul Park <byungchul@sk.com> writes: > > > Hi everyone, > > > > While I'm working with CXL memory, I have been facing migration overhead > > esp. TLB shootdown on promotion or demotion between different tiers. > > Yeah.. most TLB shootdowns on migration through hinting fault can be > > avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid > > flushing TLB in batch if PTE is inaccessible"). > > > > However, it's only for ones using hinting fault. I thought it'd be much > > better if we have a general mechanism to reduce # of TLB flushes and > > TLB misses, that we can apply to any type of migration. I tried it only > > for tiering migration for now tho. > > > > I'm suggesting a mechanism to reduce TLB flushes by keeping source and > > destination of folios participated in the migrations until all TLB > > flushes required are done, only if those folios are not mapped with > > write permission PTE entries at all. I worked Based on v6.6-rc5. > > > > Can you believe it? I saw the number of TLB full flush reduced about > > 80% and iTLB miss reduced about 50%, and the time wise performance > > always shows at least 1% stable improvement with the workload I tested > > with, XSBench. However, I believe that it would help more with other > > ones or any real ones. It'd be appreciated to let me know if I'm missing > > something. > > Can you help to test the effect of commit 7e12beb8ca2a ("migrate_pages: > batch flushing TLB") for your test case? To test it, you can revert it > and compare the performance before and after the reverting. > > And, how do you trigger migration when testing XSBench? Use a tiered > memory system, and migrate pages between DRAM and CXL memory back and > forth? If so, how many pages will you migrate for each migration It was not an actual CXL memory but a cpuless remote numa node's DRAM recognized as a slow tier (node_is_toptier() == false) by the kernel. It's been okay to me because I've been focusing on TLB # and migration # while working with numa tiering mechanism and, I think, the time wise performance will be followed, big or little depending on the system configuration. So it migrates pages between the two DRAMs back and forth - promotion by hinting fault and demotion by page reclaim. I tested what you asked me with another slower system to make TLB miss overhead stand out. Unfortunately I got even worse result with vanilla v6.6-rc5 than v6.6-rc5 with 7e12beb8ca2a reverted, while the 'v6.6-rc5 + migrc' definitely shows far better result. Thoughts? Byungchul --- Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Kernel - v6.6-rc5, NUMA_BALANCING_MEMORY_TIERING, demotion enabled Benchmark - XSBench -p 50000000 (-p option makes the runtime longer) CASE1 - mainline v6.6-rc5 + 7e12beb8ca2a reverted ------------------------------------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 50000000 Performance counter stats for 'system wide': 190247118 itlb.itlb_flush 716182438 tlb_flush.dtlb_thread 327051673 tlb_flush.stlb_any 119542331968 dTLB-load-misses 724072795 dTLB-store-misses 3054343419 iTLB-load-misses 1172.580552728 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 5968431 pgmigrate_success 12484773 nr_tlb_remote_flush 6614459 nr_tlb_remote_flush_received 96022799 nr_tlb_local_flush_all 50869 nr_tlb_local_flush_one 785597 ... CASE2 - mainline v6.6-rc5 (vanilla) ------------------------------------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 50000000 Performance counter stats for 'system wide': 55139061 itlb.itlb_flush 286725687 tlb_flush.dtlb_thread 199687660 tlb_flush.stlb_any 119497951269 dTLB-load-misses 358434759 dTLB-store-misses 1867135967 iTLB-load-misses 1181.311084373 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 8190027 pgmigrate_success 17098994 nr_tlb_remote_flush 1955114 nr_tlb_remote_flush_received 29028093 nr_tlb_local_flush_all 140921 nr_tlb_local_flush_one 740767 ... CASE3 - mainline v6.6-rc5 + migrc ------------------------------------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 50000000 Performance counter stats for 'system wide': 6337091 itlb.itlb_flush 157229778 tlb_flush.dtlb_thread 148240163 tlb_flush.stlb_any 117701381319 dTLB-load-misses 231212468 dTLB-store-misses 973083466 iTLB-load-misses 1105.756705157 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 8791934 pgmigrate_success 18276174 nr_tlb_remote_flush 311146 nr_tlb_remote_flush_received 4387708 nr_tlb_local_flush_all 143883 nr_tlb_local_flush_one 740953 ...
On Thu, Nov 09, 2023 at 06:26:08AM -0800, Dave Hansen wrote: > On 11/8/23 20:59, Byungchul Park wrote: > > Can you believe it? I saw the number of TLB full flush reduced about > > 80% and iTLB miss reduced about 50%, and the time wise performance > > always shows at least 1% stable improvement with the workload I tested > > with, XSBench. However, I believe that it would help more with other > > ones or any real ones. It'd be appreciated to let me know if I'm missing > > something. > > I see that you've moved a substantial amount of code out of arch/x86. > That's great. > > But there doesn't appear to be any improvement in the justification or > performance data. The page flag is also here, which is horribly frowned > upon. It's an absolute no-go with this level of justification. > > I'd really suggest not sending any more of these out until those issues > are rectified. I know I definitely won't be reviewing them in this state. As I expected, I got a fair better result when I tested migrc with a system with a slower DRAM to make TLB miss overhead stand out. 1. XSBench execution time was reduced about 7%. 2. iTLB flush # was reduced stably about 90% while running XSBench. 3. iTLB miss # was reduced stably about 50% while running XSBench. https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/ Of course, I can reimplement migrc to replace PG_migrc with another thing like hash table but, IMHO, it's worth having the page flag if it gives such a good performance. Lemme know if not so that I'll change the way to implement. I'd like to note that no doubt migrc significantly reduces TLB miss and the impact depends on TLB miss overhead that varies according to the system configuration. Byungchul
On Thu, Nov 09, 2023 at 06:26:08AM -0800, Dave Hansen wrote: > On 11/8/23 20:59, Byungchul Park wrote: > > Can you believe it? I saw the number of TLB full flush reduced about > > 80% and iTLB miss reduced about 50%, and the time wise performance > > always shows at least 1% stable improvement with the workload I tested > > with, XSBench. However, I believe that it would help more with other > > ones or any real ones. It'd be appreciated to let me know if I'm missing > > something. > > I see that you've moved a substantial amount of code out of arch/x86. > That's great. > > But there doesn't appear to be any improvement in the justification or > performance data. The page flag is also here, which is horribly frowned > upon. It's an absolute no-go with this level of justification. Okay. I won't use an additional page flag anymore from migrc v5. Thanks. Byungchul
Hi everyone, While I'm working with CXL memory, I have been facing migration overhead esp. TLB shootdown on promotion or demotion between different tiers. Yeah.. most TLB shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE is inaccessible"). However, it's only for ones using hinting fault. I thought it'd be much better if we have a general mechanism to reduce # of TLB flushes and TLB misses, that we can apply to any type of migration. I tried it only for tiering migration for now tho. I'm suggesting a mechanism to reduce TLB flushes by keeping source and destination of folios participated in the migrations until all TLB flushes required are done, only if those folios are not mapped with write permission PTE entries at all. I worked Based on v6.6-rc5. Can you believe it? I saw the number of TLB full flush reduced about 80% and iTLB miss reduced about 50%, and the time wise performance always shows at least 1% stable improvement with the workload I tested with, XSBench. However, I believe that it would help more with other ones or any real ones. It'd be appreciated to let me know if I'm missing something. Byungchul --- Changes from v3: 1. Don't use the kconfig, CONFIG_MIGRC, and remove sysctl knob, migrc_enable. (feedbacked by Nadav) 2. Remove the optimization skipping CPUs that have already performed TLB flushes needed by any reason when performing TLB flushes by migrc because I can't tell the performance difference between w/ the optimization and w/o that. (feedbacked by Nadav) 3. Minimize arch-specific code. While at it, move all the migrc declarations and inline functions from include/linux/mm.h to mm/internal.h (feedbacked by Dave Hansen, Nadav) 4. Separate a part making migrc paused when the system is in high memory pressure to another patch. (feedbacked by Nadav) 5. Rename: a. arch_tlbbatch_clean() to arch_tlbbatch_clear(), b. tlb_ubc_nowr to tlb_ubc_ro, c. migrc_try_flush_free_folios() to migrc_flush_free_folios(), d. migrc_stop to migrc_pause. (feedbacked by Nadav) 6. Use ->lru list_head instead of introducing a new llist_head. (feedbacked by Nadav) 7. Use non-atomic operations of page-flag when it's safe. (feedbacked by Nadav) 8. Use stack instead of keeping a pointer of 'struct migrc_req' in struct task, which is for manipulating it locally. (feedbacked by Nadav) 9. Replace a lot of simple functions to inline functions placed in a header, mm/internal.h. (feedbacked by Nadav) 10. Add additional sufficient comments. (feedbacked by Nadav) 11. Remove a lot of wrapper functions. (feedbacked by Nadav) Changes from RFC v2: 1. Remove additional occupation in struct page. To do that, unioned with lru field for migrc's list and added a page flag. I know page flag is a thing that we don't like to add but no choice because migrc should distinguish folios under migrc's control from others. Instead, I force migrc to be used only on 64 bit system to mitigate you guys from getting angry. 2. Remove meaningless internal object allocator that I introduced to minimize impact onto the system. However, a ton of tests showed there was no difference. 3. Stop migrc from working when the system is in high memory pressure like about to perform direct reclaim. At the condition where the swap mechanism is heavily used, I found the system suffered from regression without this control. 4. Exclude folios that pte_dirty() == true from migrc's interest so that migrc can work simpler. 5. Combine several patches that work tightly coupled to one. 6. Add sufficient comments for better review. 7. Manage migrc's request in per-node manner (from globally). 8. Add TLB miss improvement in commit message. 9. Test with more CPUs(4 -> 16) to see bigger improvement. Changes from RFC: 1. Fix a bug triggered when a destination folio at the previous migration becomes a source folio at the next migration, before the folio gets handled properly so that the folio can play with another migration. There was inconsistency in the folio's state. Fixed it. 2. Split the patch set into more pieces so that the folks can review better. (Feedbacked by Nadav Amit) 3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). (Feedbacked by Nadav Amit) 4. Tried to add sufficient comments to explain the patch set better. (Feedbacked by Nadav Amit) Byungchul Park (3): mm/rmap: Recognize read-only TLB entries during batched TLB flush mm: Defer TLB flush by keeping both src and dst folios at migration mm: Pause migrc mechanism at high memory pressure arch/x86/include/asm/tlbflush.h | 3 + arch/x86/mm/tlb.c | 11 ++ include/linux/mm_types.h | 21 +++ include/linux/mmzone.h | 9 ++ include/linux/page-flags.h | 4 + include/linux/sched.h | 7 + include/trace/events/mmflags.h | 3 +- mm/internal.h | 78 ++++++++++ mm/memory.c | 11 ++ mm/migrate.c | 266 ++++++++++++++++++++++++++++++++ mm/page_alloc.c | 30 +++- mm/rmap.c | 35 ++++- 12 files changed, 475 insertions(+), 3 deletions(-)