| Message ID | 20181203080119.18989-2-xlpang@linux.alibaba.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [1/3] mm/memcg: Fix min/low usage in propagate_protected_usage() | expand |
On Mon 03-12-18 16:01:18, Xunlei Pang wrote: > There may be cgroup memory overcommitment, it will become > even common in the future. > > Let's enable kswapd to reclaim low-protected memory in case > of memory pressure, to mitigate the global direct reclaim > pressures which could cause jitters to the response time of > lantency-sensitive groups. Please be more descriptive about the problem you are trying to handle here. I haven't actually read the patch but let me emphasise that the low limit protection is important isolation tool. And allowing kswapd to reclaim protected memcgs is going to break the semantic as it has been introduced and designed. > > Signed-off-by: Xunlei Pang <xlpang@linux.alibaba.com> > --- > mm/vmscan.c | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 62ac0c488624..3d412eb91f73 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -3531,6 +3531,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) > > count_vm_event(PAGEOUTRUN); > > +retry: > do { > unsigned long nr_reclaimed = sc.nr_reclaimed; > bool raise_priority = true; > @@ -3622,6 +3623,13 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) > sc.priority--; > } while (sc.priority >= 1); > > + if (!sc.nr_reclaimed && sc.memcg_low_skipped) { > + sc.priority = DEF_PRIORITY; > + sc.memcg_low_reclaim = 1; > + sc.memcg_low_skipped = 0; > + goto retry; > + } > + > if (!sc.nr_reclaimed) > pgdat->kswapd_failures++; > > -- > 2.13.5 (Apple Git-94) >
On 2018/12/3 下午7:56, Michal Hocko wrote: > On Mon 03-12-18 16:01:18, Xunlei Pang wrote: >> There may be cgroup memory overcommitment, it will become >> even common in the future. >> >> Let's enable kswapd to reclaim low-protected memory in case >> of memory pressure, to mitigate the global direct reclaim >> pressures which could cause jitters to the response time of >> lantency-sensitive groups. > > Please be more descriptive about the problem you are trying to handle > here. I haven't actually read the patch but let me emphasise that the > low limit protection is important isolation tool. And allowing kswapd to > reclaim protected memcgs is going to break the semantic as it has been > introduced and designed. We have two types of memcgs: online groups(important business) and offline groups(unimportant business). Online groups are all configured with MAX low protection, while offline groups are not at all protected(with default 0 low). When offline groups are overcommitted, the global memory pressure suffers. This will cause the memory allocations from online groups constantly go to the slow global direct reclaim in order to reclaim online's page caches, as kswap is not able to reclaim low-protection memory. low is not hard limit, it's reasonable to be reclaimed by kswapd if there's no other reclaimable memory. > >> >> Signed-off-by: Xunlei Pang <xlpang@linux.alibaba.com> >> --- >> mm/vmscan.c | 8 ++++++++ >> 1 file changed, 8 insertions(+) >> >> diff --git a/mm/vmscan.c b/mm/vmscan.c >> index 62ac0c488624..3d412eb91f73 100644 >> --- a/mm/vmscan.c >> +++ b/mm/vmscan.c >> @@ -3531,6 +3531,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) >> >> count_vm_event(PAGEOUTRUN); >> >> +retry: >> do { >> unsigned long nr_reclaimed = sc.nr_reclaimed; >> bool raise_priority = true; >> @@ -3622,6 +3623,13 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) >> sc.priority--; >> } while (sc.priority >= 1); >> >> + if (!sc.nr_reclaimed && sc.memcg_low_skipped) { >> + sc.priority = DEF_PRIORITY; >> + sc.memcg_low_reclaim = 1; >> + sc.memcg_low_skipped = 0; >> + goto retry; >> + } >> + >> if (!sc.nr_reclaimed) >> pgdat->kswapd_failures++; >> >> -- >> 2.13.5 (Apple Git-94) >> >
On Mon 03-12-18 23:20:31, Xunlei Pang wrote: > On 2018/12/3 下午7:56, Michal Hocko wrote: > > On Mon 03-12-18 16:01:18, Xunlei Pang wrote: > >> There may be cgroup memory overcommitment, it will become > >> even common in the future. > >> > >> Let's enable kswapd to reclaim low-protected memory in case > >> of memory pressure, to mitigate the global direct reclaim > >> pressures which could cause jitters to the response time of > >> lantency-sensitive groups. > > > > Please be more descriptive about the problem you are trying to handle > > here. I haven't actually read the patch but let me emphasise that the > > low limit protection is important isolation tool. And allowing kswapd to > > reclaim protected memcgs is going to break the semantic as it has been > > introduced and designed. > > We have two types of memcgs: online groups(important business) > and offline groups(unimportant business). Online groups are > all configured with MAX low protection, while offline groups > are not at all protected(with default 0 low). > > When offline groups are overcommitted, the global memory pressure > suffers. This will cause the memory allocations from online groups > constantly go to the slow global direct reclaim in order to reclaim > online's page caches, as kswap is not able to reclaim low-protection > memory. low is not hard limit, it's reasonable to be reclaimed by > kswapd if there's no other reclaimable memory. I am sorry I still do not follow. What role do offline cgroups play. Those are certainly not low mem protected because mem_cgroup_css_offline will reset them to 0.
On 2018/12/4 AM 1:22, Michal Hocko wrote: > On Mon 03-12-18 23:20:31, Xunlei Pang wrote: >> On 2018/12/3 下午7:56, Michal Hocko wrote: >>> On Mon 03-12-18 16:01:18, Xunlei Pang wrote: >>>> There may be cgroup memory overcommitment, it will become >>>> even common in the future. >>>> >>>> Let's enable kswapd to reclaim low-protected memory in case >>>> of memory pressure, to mitigate the global direct reclaim >>>> pressures which could cause jitters to the response time of >>>> lantency-sensitive groups. >>> >>> Please be more descriptive about the problem you are trying to handle >>> here. I haven't actually read the patch but let me emphasise that the >>> low limit protection is important isolation tool. And allowing kswapd to >>> reclaim protected memcgs is going to break the semantic as it has been >>> introduced and designed. >> >> We have two types of memcgs: online groups(important business) >> and offline groups(unimportant business). Online groups are >> all configured with MAX low protection, while offline groups >> are not at all protected(with default 0 low). >> >> When offline groups are overcommitted, the global memory pressure >> suffers. This will cause the memory allocations from online groups >> constantly go to the slow global direct reclaim in order to reclaim >> online's page caches, as kswap is not able to reclaim low-protection >> memory. low is not hard limit, it's reasonable to be reclaimed by >> kswapd if there's no other reclaimable memory. > > I am sorry I still do not follow. What role do offline cgroups play. > Those are certainly not low mem protected because mem_cgroup_css_offline > will reset them to 0. > Oh, I meant "offline groups" to be "offline-business groups", memcgs refered to here are all "online state" from kernel's perspective.
On Tue 04-12-18 10:40:29, Xunlei Pang wrote: > On 2018/12/4 AM 1:22, Michal Hocko wrote: > > On Mon 03-12-18 23:20:31, Xunlei Pang wrote: > >> On 2018/12/3 下午7:56, Michal Hocko wrote: > >>> On Mon 03-12-18 16:01:18, Xunlei Pang wrote: > >>>> There may be cgroup memory overcommitment, it will become > >>>> even common in the future. > >>>> > >>>> Let's enable kswapd to reclaim low-protected memory in case > >>>> of memory pressure, to mitigate the global direct reclaim > >>>> pressures which could cause jitters to the response time of > >>>> lantency-sensitive groups. > >>> > >>> Please be more descriptive about the problem you are trying to handle > >>> here. I haven't actually read the patch but let me emphasise that the > >>> low limit protection is important isolation tool. And allowing kswapd to > >>> reclaim protected memcgs is going to break the semantic as it has been > >>> introduced and designed. > >> > >> We have two types of memcgs: online groups(important business) > >> and offline groups(unimportant business). Online groups are > >> all configured with MAX low protection, while offline groups > >> are not at all protected(with default 0 low). > >> > >> When offline groups are overcommitted, the global memory pressure > >> suffers. This will cause the memory allocations from online groups > >> constantly go to the slow global direct reclaim in order to reclaim > >> online's page caches, as kswap is not able to reclaim low-protection > >> memory. low is not hard limit, it's reasonable to be reclaimed by > >> kswapd if there's no other reclaimable memory. > > > > I am sorry I still do not follow. What role do offline cgroups play. > > Those are certainly not low mem protected because mem_cgroup_css_offline > > will reset them to 0. > > > > Oh, I meant "offline groups" to be "offline-business groups", memcgs > refered to here are all "online state" from kernel's perspective. What is offline-business group? Please try to explain the actual problem in much more details and do not let us guess.
On 2018/12/4 PM 3:25, Michal Hocko wrote: > On Tue 04-12-18 10:40:29, Xunlei Pang wrote: >> On 2018/12/4 AM 1:22, Michal Hocko wrote: >>> On Mon 03-12-18 23:20:31, Xunlei Pang wrote: >>>> On 2018/12/3 下午7:56, Michal Hocko wrote: >>>>> On Mon 03-12-18 16:01:18, Xunlei Pang wrote: >>>>>> There may be cgroup memory overcommitment, it will become >>>>>> even common in the future. >>>>>> >>>>>> Let's enable kswapd to reclaim low-protected memory in case >>>>>> of memory pressure, to mitigate the global direct reclaim >>>>>> pressures which could cause jitters to the response time of >>>>>> lantency-sensitive groups. >>>>> >>>>> Please be more descriptive about the problem you are trying to handle >>>>> here. I haven't actually read the patch but let me emphasise that the >>>>> low limit protection is important isolation tool. And allowing kswapd to >>>>> reclaim protected memcgs is going to break the semantic as it has been >>>>> introduced and designed. >>>> >>>> We have two types of memcgs: online groups(important business) >>>> and offline groups(unimportant business). Online groups are >>>> all configured with MAX low protection, while offline groups >>>> are not at all protected(with default 0 low). >>>> >>>> When offline groups are overcommitted, the global memory pressure >>>> suffers. This will cause the memory allocations from online groups >>>> constantly go to the slow global direct reclaim in order to reclaim >>>> online's page caches, as kswap is not able to reclaim low-protection >>>> memory. low is not hard limit, it's reasonable to be reclaimed by >>>> kswapd if there's no other reclaimable memory. >>> >>> I am sorry I still do not follow. What role do offline cgroups play. >>> Those are certainly not low mem protected because mem_cgroup_css_offline >>> will reset them to 0. >>> >> >> Oh, I meant "offline groups" to be "offline-business groups", memcgs >> refered to here are all "online state" from kernel's perspective. > > What is offline-business group? Please try to explain the actual problem > in much more details and do not let us guess. > Maybe I choosed the wrong word, let me rephase it, and here is an example. root 200GB / \ important(100GB) unimportant(100GB+DYNAMIC) / | \ / \ docker0 docker1... normal(100GB) oversold(DYNAMIC) / | \ / | \ j0 j1 ... w0 w1 ... "DYNAMIC" is controlled by the cluster job scheduler dynamically, it periodically samples the available system memory(/proc/meminfo "MemAvailable"), and use part of that to launch oversold jobs under some special conditions. When "oversold" is active, the whole system is put under heavy global memory pressure although memcgs are not. IOW "DYNAMIC" is primarily borrowed from "dockers" temporarily, oversold workers will be killed in a timely fashion if "dockers" needs their memory back suddenly which is rare. If kswapd doesn't reclaim low-protected memory configured among "important" dockers, memory allocations from dockers will trap into global direct reclaim constantly which harms their performance and response time. The inactive caches from dockers are allowed to be reclaimed although they are under low-protected(we used a simple MAX setting), we allow the inactive low-protected memory to be reclaimed immediately and asynchronously as long as there's no unprotected reclaimable memory. Its's also friendly to disk IO. For really latency-sensitive docker, memory.min is supposed to be used to guarantee its memory QoS. Thanks
diff --git a/mm/vmscan.c b/mm/vmscan.c index 62ac0c488624..3d412eb91f73 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3531,6 +3531,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) count_vm_event(PAGEOUTRUN); +retry: do { unsigned long nr_reclaimed = sc.nr_reclaimed; bool raise_priority = true; @@ -3622,6 +3623,13 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) sc.priority--; } while (sc.priority >= 1); + if (!sc.nr_reclaimed && sc.memcg_low_skipped) { + sc.priority = DEF_PRIORITY; + sc.memcg_low_reclaim = 1; + sc.memcg_low_skipped = 0; + goto retry; + } + if (!sc.nr_reclaimed) pgdat->kswapd_failures++;
There may be cgroup memory overcommitment, it will become even common in the future. Let's enable kswapd to reclaim low-protected memory in case of memory pressure, to mitigate the global direct reclaim pressures which could cause jitters to the response time of lantency-sensitive groups. Signed-off-by: Xunlei Pang <xlpang@linux.alibaba.com> --- mm/vmscan.c | 8 ++++++++ 1 file changed, 8 insertions(+)