| Message ID | 20230328061638.203420-6-yosryahmed@google.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | memcg: make rstat flushing irq and sleep friendly | expand |
On Tue, Mar 28, 2023 at 06:16:34AM +0000, Yosry Ahmed wrote: [...] > @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) > */ > static void flush_memcg_stats_dwork(struct work_struct *w); > static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); > -static DEFINE_SPINLOCK(stats_flush_lock); > static DEFINE_PER_CPU(unsigned int, stats_updates); > +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); > static atomic_t stats_flush_threshold = ATOMIC_INIT(0); > static u64 flush_next_time; > > @@ -636,15 +636,18 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) > > static void __mem_cgroup_flush_stats(void) > { > - unsigned long flag; > - > - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) > + /* > + * We always flush the entire tree, so concurrent flushers can just > + * skip. This avoids a thundering herd problem on the rstat global lock > + * from memcg flushers (e.g. reclaim, refault, etc). > + */ > + if (atomic_xchg(&stats_flush_ongoing, 1)) Have you profiled this? I wonder if we should replace the above with if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) to not always dirty the cacheline. This would not be an issue if there is no cacheline sharing but I suspect percpu stats_updates is sharing the cacheline with it and may cause false sharing with the parallel stat updaters (updaters only need to read the base percpu pointer). Other than that the patch looks good.
On Tue, Mar 28, 2023 at 06:16:34AM +0000, Yosry Ahmed wrote: > As Johannes notes in [1], stats_flush_lock is currently used to: > (a) Protect updated to stats_flush_threshold. > (b) Protect updates to flush_next_time. > (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. > > However: > > 1. stats_flush_threshold is already an atomic > > 2. flush_next_time is not atomic. The writer is locked, but the reader > is lockless. If the reader races with a flush, you could see this: > > if (time_after(jiffies, flush_next_time)) > spin_trylock() > flush_next_time = now + delay > flush() > spin_unlock() > spin_trylock() > flush_next_time = now + delay > flush() > spin_unlock() > > which means we already can get flushes at a higher frequency than > FLUSH_TIME during races. But it isn't really a problem. > > The reader could also see garbled partial updates, so it needs at > least READ_ONCE and WRITE_ONCE protection. > > 3. Serializing cgroup_rstat_flush() calls against the ratelimit > factors is currently broken because of the race in 2. But the race > is actually harmless, all we might get is the occasional earlier > flush. If there is no delta, the flush won't do much. And if there > is, the flush is justified. > > So the lock can be removed all together. However, the lock also served > the purpose of preventing a thundering herd problem for concurrent > flushers, see [2]. Use an atomic instead to serve the purpose of > unifying concurrent flushers. > > [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ > [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ > > Signed-off-by: Yosry Ahmed <yosryahmed@google.com> With Shakeel's suggestion: Acked-by: Johannes Weiner <hannes@cmpxchg.org>
On Tue, Mar 28, 2023 at 7:15 AM Shakeel Butt <shakeelb@google.com> wrote: > > On Tue, Mar 28, 2023 at 06:16:34AM +0000, Yosry Ahmed wrote: > [...] > > @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) > > */ > > static void flush_memcg_stats_dwork(struct work_struct *w); > > static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); > > -static DEFINE_SPINLOCK(stats_flush_lock); > > static DEFINE_PER_CPU(unsigned int, stats_updates); > > +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); > > static atomic_t stats_flush_threshold = ATOMIC_INIT(0); > > static u64 flush_next_time; > > > > @@ -636,15 +636,18 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) > > > > static void __mem_cgroup_flush_stats(void) > > { > > - unsigned long flag; > > - > > - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) > > + /* > > + * We always flush the entire tree, so concurrent flushers can just > > + * skip. This avoids a thundering herd problem on the rstat global lock > > + * from memcg flushers (e.g. reclaim, refault, etc). > > + */ > > + if (atomic_xchg(&stats_flush_ongoing, 1)) > > Have you profiled this? I wonder if we should replace the above with > > if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) I profiled the entire series with perf and I haven't noticed a notable difference between before and after the patch series -- but maybe some specific access patterns cause a regression, not sure. Does an atomic_cmpxchg() satisfy the same purpose? it's easier to read / more concise I guess. Something like if (atomic_cmpxchg(&stats_flush_ongoing, 0, 1)) WDYT? > > to not always dirty the cacheline. This would not be an issue if there > is no cacheline sharing but I suspect percpu stats_updates is sharing > the cacheline with it and may cause false sharing with the parallel stat > updaters (updaters only need to read the base percpu pointer). > > Other than that the patch looks good.
On Tue, Mar 28, 2023 at 11:53 AM Yosry Ahmed <yosryahmed@google.com> wrote: > [...] > > > + if (atomic_xchg(&stats_flush_ongoing, 1)) > > > > Have you profiled this? I wonder if we should replace the above with > > > > if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) > > I profiled the entire series with perf and I haven't noticed a notable > difference between before and after the patch series -- but maybe some > specific access patterns cause a regression, not sure. > > Does an atomic_cmpxchg() satisfy the same purpose? it's easier to read > / more concise I guess. > > Something like > > if (atomic_cmpxchg(&stats_flush_ongoing, 0, 1)) > > WDYT? > No, I don't think cmpxchg will be any different from xchg(). On x86, the cmpxchg will always write to stats_flush_ongoing and depending on the comparison result, it will either be 0 or 1 here. If you see the implementation of queued_spin_trylock(), it does the same as well.
On Tue, Mar 28, 2023 at 12:28 PM Shakeel Butt <shakeelb@google.com> wrote: > > On Tue, Mar 28, 2023 at 11:53 AM Yosry Ahmed <yosryahmed@google.com> wrote: > > > [...] > > > > + if (atomic_xchg(&stats_flush_ongoing, 1)) > > > > > > Have you profiled this? I wonder if we should replace the above with > > > > > > if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) > > > > I profiled the entire series with perf and I haven't noticed a notable > > difference between before and after the patch series -- but maybe some > > specific access patterns cause a regression, not sure. > > > > Does an atomic_cmpxchg() satisfy the same purpose? it's easier to read > > / more concise I guess. > > > > Something like > > > > if (atomic_cmpxchg(&stats_flush_ongoing, 0, 1)) > > > > WDYT? > > > > No, I don't think cmpxchg will be any different from xchg(). On x86, > the cmpxchg will always write to stats_flush_ongoing and depending on > the comparison result, it will either be 0 or 1 here. > > If you see the implementation of queued_spin_trylock(), it does the > same as well. Interesting. I thought cmpxchg by definition will compare first and only do the write if stats_flush_ongoing == 0 in this case. I thought queued_spin_trylock() was doing an atomic_read() first to avoid the LOCK instruction unnecessarily the lock is held by someone else.
On Tue, Mar 28, 2023 at 12:34 PM Yosry Ahmed <yosryahmed@google.com> wrote: > > On Tue, Mar 28, 2023 at 12:28 PM Shakeel Butt <shakeelb@google.com> wrote: > > > > On Tue, Mar 28, 2023 at 11:53 AM Yosry Ahmed <yosryahmed@google.com> wrote: > > > > > [...] > > > > > + if (atomic_xchg(&stats_flush_ongoing, 1)) > > > > > > > > Have you profiled this? I wonder if we should replace the above with > > > > > > > > if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) > > > > > > I profiled the entire series with perf and I haven't noticed a notable > > > difference between before and after the patch series -- but maybe some > > > specific access patterns cause a regression, not sure. > > > > > > Does an atomic_cmpxchg() satisfy the same purpose? it's easier to read > > > / more concise I guess. > > > > > > Something like > > > > > > if (atomic_cmpxchg(&stats_flush_ongoing, 0, 1)) > > > > > > WDYT? > > > > > > > No, I don't think cmpxchg will be any different from xchg(). On x86, > > the cmpxchg will always write to stats_flush_ongoing and depending on > > the comparison result, it will either be 0 or 1 here. > > > > If you see the implementation of queued_spin_trylock(), it does the > > same as well. > > Interesting. I thought cmpxchg by definition will compare first and > only do the write if stats_flush_ongoing == 0 in this case. > > I thought queued_spin_trylock() was doing an atomic_read() first to > avoid the LOCK instruction unnecessarily the lock is held by someone > else. Anyway, perhaps it's better to follow what queued_spin_trylock() is doing, even if only to avoid locking the cache line unnecessarily. (Although now that I think about it, I wonder why atomic_cmpxchg doesn't do this by default, food for thought)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ff39f78f962e..64ff33e02c96 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -636,15 +636,18 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * We always flush the entire tree, so concurrent flushers can just + * skip. This avoids a thundering herd problem on the rstat global lock + * from memcg flushers (e.g. reclaim, refault, etc). + */ + if (atomic_xchg(&stats_flush_ongoing, 1)) return; - flush_next_time = jiffies_64 + 2*FLUSH_TIME; + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) @@ -655,7 +658,7 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, flush_next_time)) + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) mem_cgroup_flush_stats(); }
As Johannes notes in [1], stats_flush_lock is currently used to: (a) Protect updated to stats_flush_threshold. (b) Protect updates to flush_next_time. (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. However: 1. stats_flush_threshold is already an atomic 2. flush_next_time is not atomic. The writer is locked, but the reader is lockless. If the reader races with a flush, you could see this: if (time_after(jiffies, flush_next_time)) spin_trylock() flush_next_time = now + delay flush() spin_unlock() spin_trylock() flush_next_time = now + delay flush() spin_unlock() which means we already can get flushes at a higher frequency than FLUSH_TIME during races. But it isn't really a problem. The reader could also see garbled partial updates, so it needs at least READ_ONCE and WRITE_ONCE protection. 3. Serializing cgroup_rstat_flush() calls against the ratelimit factors is currently broken because of the race in 2. But the race is actually harmless, all we might get is the occasional earlier flush. If there is no delta, the flush won't do much. And if there is, the flush is justified. So the lock can be removed all together. However, the lock also served the purpose of preventing a thundering herd problem for concurrent flushers, see [2]. Use an atomic instead to serve the purpose of unifying concurrent flushers. [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ Signed-off-by: Yosry Ahmed <yosryahmed@google.com> --- mm/memcontrol.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)