| Message ID | cover.1613661472.git.charante@codeaurora.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | mm: balancing the node zones occupancy | expand |
On 18.02.21 18:24, Charan Teja Reddy wrote: > I would like to start discussion about balancing the occupancy of > memory zones in a node in the system whose imabalance may be caused by > migration of pages to other zones during hotremove and then hotadding > same memory. In this case there is a lot of free memory in newly hotadd > memory which can be filled up by the previous migrated pages(as part of > offline/hotremove) thus may free up some pressure in other zones of the > node. Why is this specific to memory hot(un)plug? I think the problem is more generic: Assume 1. Application 1 allocates a lot of memory and gets ZONE_MOVABLE. 2. Application 2 allocates a lot of memory and gets ZONE_NORMAL. 3. Application 1 quits. Same problem, no?
On 2/18/21 6:24 PM, Charan Teja Reddy wrote: > I would like to start discussion about balancing the occupancy of > memory zones in a node in the system whose imabalance may be caused by > migration of pages to other zones during hotremove and then hotadding > same memory. In this case there is a lot of free memory in newly hotadd > memory which can be filled up by the previous migrated pages(as part of > offline/hotremove) thus may free up some pressure in other zones of the > node. Can you share the use case for doing this? If it's to replace a failed RAM, then it's probably extremely rare, right. > We have the proof-of-concept code tried on the Snapdragon systems with > the system configuration, single memory node of just 2 zones, 6GB normal > zone and 2GB movable zone. And this Movable zone is such that hot-added > once and there after offline/online based on the need. Hm, snapdragon... so is this some kind of power saving thing? Anyway, shouln't auto NUMA balancing help here, and especially "Migrate Pages in lieu of discard" (CC'd Dave) as a generic mechanism, so we wouldn't need to have hotplug-specific actions?
Thanks David for the review comments!! On 2/18/2021 11:46 PM, David Hildenbrand wrote: >> I would like to start discussion about balancing the occupancy of >> memory zones in a node in the system whose imabalance may be caused by >> migration of pages to other zones during hotremove and then hotadding >> same memory. In this case there is a lot of free memory in newly hotadd >> memory which can be filled up by the previous migrated pages(as part of >> offline/hotremove) thus may free up some pressure in other zones of the >> node. > > Why is this specific to memory hot(un)plug? I think the problem is more > generic: > > Assume > > 1. Application 1 allocates a lot of memory and gets ZONE_MOVABLE. > 2. Application 2 allocates a lot of memory and gets ZONE_NORMAL. > 3. Application 1 quits. > > Same problem, no? Thanks for simplifying this problem. Yeah, this looks more generic problem. But for these type of problems, user/system administrator has clear view about the state of the system and thus may need to take some decisions to maintain the the node zones balancing e.g. like this change where migrate the eligible pages to other zones.
Thanks Vlastimil for the review comments!! On 2/19/2021 4:56 PM, Vlastimil Babka wrote: > Can you share the use case for doing this? If it's to replace a failed RAM, then > it's probably extremely rare, right. > >> We have the proof-of-concept code tried on the Snapdragon systems with >> the system configuration, single memory node of just 2 zones, 6GB normal >> zone and 2GB movable zone. And this Movable zone is such that hot-added >> once and there after offline/online based on the need. > Hm, snapdragon... so is this some kind of power saving thing? > You are correct. This is the power saving usecase which does the offline and online of the memory blocks in the system by the user. This is not a failed RAM. > Anyway, shouln't auto NUMA balancing help here, and especially "Migrate Pages in > lieu of discard" (CC'd Dave) as a generic mechanism, On the Snapdragon systems we have got only single memory node with Normal and movable zones. And my little understanding is that on most embedded systems we will just have single memory node. My limited understanding about this auto NUMA balancing is that there should be min. 2 nodes for this balancing to trigger. Please correct if I am wrong here. If I am correct then this approach is not suitable for us. Moreover the idea I would like to convey in this RFC patch is about __balancing the zones in a node but not across NUMA nodes__. > so we wouldn't need to have hotplug-specific actions? David has told a very simple view of this problem which is nothing todo with the hotplug specific actions. With just 2 zones(Normal and Movable) in a single node in the system, 1. Application 1 allocates a lot of memory and gets ZONE_MOVABLE. 2. Application 2 allocates a lot of memory and gets ZONE_NORMAL. 3. Application 1 quits. Then after step3, we can expect a lot free memory in the Movable zone but normal zone is under pressure. Applying the similar semantics of Auto numa balancing("Migrate pages in lieu of swap/discard"), we could migrate some eligible pages of Application 2 to Movable zone there by can relieve some pressure in Normal zone.
I would like to start discussion about balancing the occupancy of memory zones in a node in the system whose imabalance may be caused by migration of pages to other zones during hotremove and then hotadding same memory. In this case there is a lot of free memory in newly hotadd memory which can be filled up by the previous migrated pages(as part of offline/hotremove) thus may free up some pressure in other zones of the node. Say that in system has 2 zones(Normal and Movable), where Normal zone is almost filled up by the pages of the movable zone as part of offline operation and then we hot add a memory node as movable zone. At this moment, Movable zone is almost empty and Normal zone is almost filled up(by the migrated pages as part of previous offline/hot-remove operation). At this point of time, allocation requests from Normal zone may have to go through reclaim cycles thus cause some pressure. This problem is quite common in the system where they aggressively offline and online the memory blocks in the system where the former part do the migration of pages to the lower zones and the later part don't reverse them and as a result the offline operation may contribute to the pressure in other zones of the system. To overcome this problem, we can do the reverse of what offline operation did, after onlining the memory block i.e. **try to reverse migrate the pages from other zones which were migrated as part of the offline operation**. This may freeup some pressure built in the other zones because of offline operation. Since we are reverse migrating the pages in the system, we can name this as "reverse migration feature" or since we are actually balancing the occupancy of the zones in the system by migrating the pages we can name it as "balancing the system zones occupancy" or any name... We have the proof-of-concept code tried on the Snapdragon systems with the system configuration, single memory node of just 2 zones, 6GB normal zone and 2GB movable zone. And this Movable zone is such that hot-added once and there after offline/online based on the need. We run the below unit test to evalutate this: 1) Completely fill up the already hot added movable zone with anon pages 2) Offline this hot added movable zone. At this point there is no pressure in the normal zone yet, but this migration of pages can contribute to it in the future. 3) Now fill up the normal zone such that there is 300MB or less left in the system. 4) And now the user onlined the movable zone memory blocks in the system. 5) Run the tests of allocating 512MB of memory from the normal zone and in doing so try allocating the higher order pages first and then gradually fall back to lower orders. I took the help from ion system heap memory allocation which try to allocate the memory in available orders: 9, 4 and 0. 6) Repeat the above steps for 10 iterations and below is the average of the results. We did try to collect the time it takes to complete the tests and the distribution of anon pages(are the ones participated in the tests) in the system node zones. a) With out the reverse migration, it took an average of around 145msec to complete the test. b) With the reverse migration, it took an average of the 120msec to complete the tests. For distribution of the anon pages in the system we did try collect the anon pages left in the individual zone before and after the test: ------------------------------------- |------------------------------- Base | Reverse Migration --------------------------------------|------------------------------- Beforetest Aftertest | Beforetest Aftertest Normal zone(Anon) | Active 499825 45756 | 481441 203124 Inactive 46008 446687 | 51553 58602 Free 80350 224252 | 84877 **440586** Movable zone(Anon) | Active 2224 2626 | 2239 **484622** Inactive 8 8 | 9 7663 --------------------------------------|------------------------------- The above table shows that, On base case(left column), there exists a lot of anon pages in the system which can be migrated back to Movable zone(almost totally free), thus may freeup some space in the normal zone. With the reverse migration(Right coloumn), we see that the anon pages are evenly distributed in the system and lot of free memory left in the normal zones caused by the reverse migration. The code shows the PoC by assuming just 2 zones(normal and Movable) of a single node in the system. The number of pages to be reverse migrated is written on the sysfs interface from the userspace by monitoring the memory pressure events in the system. Charan Teja Reddy (1): mm: proof-of-concept for balancing node zones occupancy include/linux/migrate.h | 8 +- include/linux/mm.h | 3 + include/linux/mmzone.h | 2 + kernel/sysctl.c | 11 ++ mm/compaction.c | 4 +- mm/memory_hotplug.c | 265 ++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 290 insertions(+), 3 deletions(-)