| Message ID | 20231122100156.6568-3-ddrokosov@salutedevices.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | mm: memcg: improve vmscan tracepoints | expand |
| Context | Check | Description |
|---|---|---|
| netdev/tree_selection | success | Not a local patch |
On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote: > The shrink_memcg flow plays a crucial role in memcg reclamation. > Currently, it is not possible to trace this point from non-direct > reclaim paths. Is this really true? AFAICS we have mm_vmscan_lru_isolate mm_vmscan_lru_shrink_active mm_vmscan_lru_shrink_inactive which are in the vry core of the memory reclaim. Sure post processing those is some work. [...] > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 45780952f4b5..6d89b39d9a91 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -6461,6 +6461,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) > */ > cond_resched(); > > +#ifdef CONFIG_MEMCG > + trace_mm_vmscan_memcg_shrink_begin(sc->order, > + sc->gfp_mask, > + memcg); > +#endif this is a common code path for node and direct reclaim which means that we will have multiple begin/end tracepoints covering similar operations. To me that sounds excessive. If you are missing a cumulative kswapd alternative to mm_vmscan_direct_reclaim_begin mm_vmscan_direct_reclaim_end mm_vmscan_memcg_reclaim_begin mm_vmscan_memcg_reclaim_end mm_vmscan_memcg_softlimit_reclaim_begin mm_vmscan_memcg_softlimit_reclaim_end mm_vmscan_node_reclaim_begin mm_vmscan_node_reclaim_end then place it into kswapd path. But it would be really great to elaborate some more why this is really needed. Cannot you simply aggregate stats for kswapd from existing tracepoints?
Hello Michal, Thank you for the quick review! On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote: > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote: > > The shrink_memcg flow plays a crucial role in memcg reclamation. > > Currently, it is not possible to trace this point from non-direct > > reclaim paths. > > Is this really true? AFAICS we have > mm_vmscan_lru_isolate > mm_vmscan_lru_shrink_active > mm_vmscan_lru_shrink_inactive > > which are in the vry core of the memory reclaim. Sure post processing > those is some work. Sure, you are absolutely right. In the usual scenario, the memcg shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the tracepoints you mentioned and analyze them. However, there is one potential issue. Enabling these tracepoints will trigger the reclaim events show for all pages. Although we can filter them per pid, we cannot filter them per cgroup. Nevertheless, there are times when it would be extremely beneficial to comprehend the effectiveness of the reclaim process within the relevant cgroup. For this reason, I am adding the cgroup name to the memcg tracepoints and implementing a cumulative tracepoint for memcg shrink (LRU + slab)." > > [...] > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 45780952f4b5..6d89b39d9a91 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -6461,6 +6461,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) > > */ > > cond_resched(); > > > > +#ifdef CONFIG_MEMCG > > + trace_mm_vmscan_memcg_shrink_begin(sc->order, > > + sc->gfp_mask, > > + memcg); > > +#endif > > this is a common code path for node and direct reclaim which means that > we will have multiple begin/end tracepoints covering similar operations. > To me that sounds excessive. If you are missing a cumulative kswapd > alternative to > mm_vmscan_direct_reclaim_begin > mm_vmscan_direct_reclaim_end > mm_vmscan_memcg_reclaim_begin > mm_vmscan_memcg_reclaim_end > mm_vmscan_memcg_softlimit_reclaim_begin > mm_vmscan_memcg_softlimit_reclaim_end > mm_vmscan_node_reclaim_begin > mm_vmscan_node_reclaim_end > > then place it into kswapd path. But it would be really great to > elaborate some more why this is really needed. Cannot you simply > aggregate stats for kswapd from existing tracepoints? > > -- > Michal Hocko > SUSE Labs
On Wed 22-11-23 13:58:36, Dmitry Rokosov wrote: > Hello Michal, > > Thank you for the quick review! > > On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote: > > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote: > > > The shrink_memcg flow plays a crucial role in memcg reclamation. > > > Currently, it is not possible to trace this point from non-direct > > > reclaim paths. > > > > Is this really true? AFAICS we have > > mm_vmscan_lru_isolate > > mm_vmscan_lru_shrink_active > > mm_vmscan_lru_shrink_inactive > > > > which are in the vry core of the memory reclaim. Sure post processing > > those is some work. > > Sure, you are absolutely right. In the usual scenario, the memcg > shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the > tracepoints you mentioned and analyze them. However, there is one > potential issue. Enabling these tracepoints will trigger the reclaim > events show for all pages. Although we can filter them per pid, we > cannot filter them per cgroup. Nevertheless, there are times when it > would be extremely beneficial to comprehend the effectiveness of the > reclaim process within the relevant cgroup. For this reason, I am adding > the cgroup name to the memcg tracepoints and implementing a cumulative > tracepoint for memcg shrink (LRU + slab)." I can see how printing memcg in mm_vmscan_memcg_reclaim_begin makes it easier to postprocess per memcg reclaim. But you could do that just by adding that to mm_vmscan_memcg_reclaim_{begin, end}, no? Why exactly does this matter for kswapd and other global reclaim contexts?
On Wed, Nov 22, 2023 at 02:24:59PM +0100, Michal Hocko wrote: > On Wed 22-11-23 13:58:36, Dmitry Rokosov wrote: > > Hello Michal, > > > > Thank you for the quick review! > > > > On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote: > > > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote: > > > > The shrink_memcg flow plays a crucial role in memcg reclamation. > > > > Currently, it is not possible to trace this point from non-direct > > > > reclaim paths. > > > > > > Is this really true? AFAICS we have > > > mm_vmscan_lru_isolate > > > mm_vmscan_lru_shrink_active > > > mm_vmscan_lru_shrink_inactive > > > > > > which are in the vry core of the memory reclaim. Sure post processing > > > those is some work. > > > > Sure, you are absolutely right. In the usual scenario, the memcg > > shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the > > tracepoints you mentioned and analyze them. However, there is one > > potential issue. Enabling these tracepoints will trigger the reclaim > > events show for all pages. Although we can filter them per pid, we > > cannot filter them per cgroup. Nevertheless, there are times when it > > would be extremely beneficial to comprehend the effectiveness of the > > reclaim process within the relevant cgroup. For this reason, I am adding > > the cgroup name to the memcg tracepoints and implementing a cumulative > > tracepoint for memcg shrink (LRU + slab)." > > I can see how printing memcg in mm_vmscan_memcg_reclaim_begin makes it > easier to postprocess per memcg reclaim. But you could do that just by > adding that to mm_vmscan_memcg_reclaim_{begin, end}, no? Why exactly > does this matter for kswapd and other global reclaim contexts? From my point of view, kswapd and other non-direct reclaim paths are important for memcg analysis because they also influence the memcg reclaim statistics. The tracepoint mm_vmscan_memcg_reclaim_{begin, end} is called from the direct memcg reclaim flow, such as: - a direct write to the 'reclaim' node - changing 'max' and 'high' thresholds - raising the 'force_empty' mechanism - the charge path - etc. However, it doesn't cover global reclaim contexts, so it doesn't provide us with the full memcg reclaim statistics.
Michal, Shakeel, Sorry for pinging you here, but I don't quite understand your decision on this patchset. Is it a NAK or not? If it's not, should I consider redesigning something? For instance, introducing stub functions to remove ifdefs from shrink_node_memcgs(). Thank you for taking the time to look into this! On Wed, Nov 22, 2023 at 09:57:27PM +0300, Dmitry Rokosov wrote: > On Wed, Nov 22, 2023 at 02:24:59PM +0100, Michal Hocko wrote: > > On Wed 22-11-23 13:58:36, Dmitry Rokosov wrote: > > > Hello Michal, > > > > > > Thank you for the quick review! > > > > > > On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote: > > > > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote: > > > > > The shrink_memcg flow plays a crucial role in memcg reclamation. > > > > > Currently, it is not possible to trace this point from non-direct > > > > > reclaim paths. > > > > > > > > Is this really true? AFAICS we have > > > > mm_vmscan_lru_isolate > > > > mm_vmscan_lru_shrink_active > > > > mm_vmscan_lru_shrink_inactive > > > > > > > > which are in the vry core of the memory reclaim. Sure post processing > > > > those is some work. > > > > > > Sure, you are absolutely right. In the usual scenario, the memcg > > > shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the > > > tracepoints you mentioned and analyze them. However, there is one > > > potential issue. Enabling these tracepoints will trigger the reclaim > > > events show for all pages. Although we can filter them per pid, we > > > cannot filter them per cgroup. Nevertheless, there are times when it > > > would be extremely beneficial to comprehend the effectiveness of the > > > reclaim process within the relevant cgroup. For this reason, I am adding > > > the cgroup name to the memcg tracepoints and implementing a cumulative > > > tracepoint for memcg shrink (LRU + slab)." > > > > I can see how printing memcg in mm_vmscan_memcg_reclaim_begin makes it > > easier to postprocess per memcg reclaim. But you could do that just by > > adding that to mm_vmscan_memcg_reclaim_{begin, end}, no? Why exactly > > does this matter for kswapd and other global reclaim contexts? > > From my point of view, kswapd and other non-direct reclaim paths are > important for memcg analysis because they also influence the memcg > reclaim statistics. > > The tracepoint mm_vmscan_memcg_reclaim_{begin, end} is called from the > direct memcg reclaim flow, such as: > - a direct write to the 'reclaim' node > - changing 'max' and 'high' thresholds > - raising the 'force_empty' mechanism > - the charge path > - etc. > > However, it doesn't cover global reclaim contexts, so it doesn't provide > us with the full memcg reclaim statistics. > > -- > Thank you, > Dmitry
On Thu 23-11-23 14:26:29, Dmitry Rokosov wrote: > Michal, Shakeel, > > Sorry for pinging you here, but I don't quite understand your decision > on this patchset. > > Is it a NAK or not? If it's not, should I consider redesigning > something? For instance, introducing stub functions to > remove ifdefs from shrink_node_memcgs(). > > Thank you for taking the time to look into this! Sorry for a late reply. I have noticed you have posted a new version. Let me have a look and comment there.
Michal, On Mon, Nov 27, 2023 at 10:25:12AM +0100, Michal Hocko wrote: > On Thu 23-11-23 14:26:29, Dmitry Rokosov wrote: > > Michal, Shakeel, > > > > Sorry for pinging you here, but I don't quite understand your decision > > on this patchset. > > > > Is it a NAK or not? If it's not, should I consider redesigning > > something? For instance, introducing stub functions to > > remove ifdefs from shrink_node_memcgs(). > > > > Thank you for taking the time to look into this! > > Sorry for a late reply. I have noticed you have posted a new version. > Let me have a look and comment there. No problem! Thanks a lot for your time and attention! Let's continue in the next version thread.
diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h index 9b49cd120ae9..cb39b4f0dca9 100644 --- a/include/trace/events/vmscan.h +++ b/include/trace/events/vmscan.h @@ -182,6 +182,13 @@ DEFINE_EVENT(mm_vmscan_memcg_reclaim_begin_template, mm_vmscan_memcg_softlimit_r TP_ARGS(order, gfp_flags, memcg) ); +DEFINE_EVENT(mm_vmscan_memcg_reclaim_begin_template, mm_vmscan_memcg_shrink_begin, + + TP_PROTO(int order, gfp_t gfp_flags, const struct mem_cgroup *memcg), + + TP_ARGS(order, gfp_flags, memcg) +); + #endif /* CONFIG_MEMCG */ DECLARE_EVENT_CLASS(mm_vmscan_direct_reclaim_end_template, @@ -247,6 +254,13 @@ DEFINE_EVENT(mm_vmscan_memcg_reclaim_end_template, mm_vmscan_memcg_softlimit_rec TP_ARGS(nr_reclaimed, memcg) ); +DEFINE_EVENT(mm_vmscan_memcg_reclaim_end_template, mm_vmscan_memcg_shrink_end, + + TP_PROTO(unsigned long nr_reclaimed, const struct mem_cgroup *memcg), + + TP_ARGS(nr_reclaimed, memcg) +); + #endif /* CONFIG_MEMCG */ TRACE_EVENT(mm_shrink_slab_start, diff --git a/mm/vmscan.c b/mm/vmscan.c index 45780952f4b5..6d89b39d9a91 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6461,6 +6461,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) */ cond_resched(); +#ifdef CONFIG_MEMCG + trace_mm_vmscan_memcg_shrink_begin(sc->order, + sc->gfp_mask, + memcg); +#endif + mem_cgroup_calculate_protection(target_memcg, memcg); if (mem_cgroup_below_min(target_memcg, memcg)) { @@ -6491,6 +6497,11 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, sc->priority); +#ifdef CONFIG_MEMCG + trace_mm_vmscan_memcg_shrink_end(sc->nr_reclaimed - reclaimed, + memcg); +#endif + /* Record the group's reclaim efficiency */ if (!sc->proactive) vmpressure(sc->gfp_mask, memcg, false,
The shrink_memcg flow plays a crucial role in memcg reclamation. Currently, it is not possible to trace this point from non-direct reclaim paths. However, direct reclaim has its own tracepoint, so there is no issue there. In certain cases, when debugging memcg pressure, developers may need to identify all potential requests for memcg reclamation including kswapd(). The patchset introduces the tracepoints mm_vmscan_memcg_shrink_{begin|end}() to address this problem. Example of output in the kswapd context (non-direct reclaim): kswapd0-39 [001] ..... 240.356378: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test kswapd0-39 [001] ..... 240.356396: mm_vmscan_memcg_shrink_end: nr_reclaimed=0 memcg=test kswapd0-39 [001] ..... 240.356420: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test kswapd0-39 [001] ..... 240.356454: mm_vmscan_memcg_shrink_end: nr_reclaimed=1 memcg=test kswapd0-39 [001] ..... 240.356479: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test kswapd0-39 [001] ..... 240.356506: mm_vmscan_memcg_shrink_end: nr_reclaimed=4 memcg=test kswapd0-39 [001] ..... 240.356525: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test kswapd0-39 [001] ..... 240.356593: mm_vmscan_memcg_shrink_end: nr_reclaimed=11 memcg=test kswapd0-39 [001] ..... 240.356614: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test kswapd0-39 [001] ..... 240.356738: mm_vmscan_memcg_shrink_end: nr_reclaimed=25 memcg=test kswapd0-39 [001] ..... 240.356790: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test kswapd0-39 [001] ..... 240.357125: mm_vmscan_memcg_shrink_end: nr_reclaimed=53 memcg=test Signed-off-by: Dmitry Rokosov <ddrokosov@salutedevices.com> --- include/trace/events/vmscan.h | 14 ++++++++++++++ mm/vmscan.c | 11 +++++++++++ 2 files changed, 25 insertions(+)