From patchwork Tue Jan 25 16:43:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 12724027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD931C433EF for ; Tue, 25 Jan 2022 16:43:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B25146B00AC; Tue, 25 Jan 2022 11:43:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DFE56B00AF; Tue, 25 Jan 2022 11:43:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E40D6B00AD; Tue, 25 Jan 2022 11:43:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 596CF6B00AB for ; Tue, 25 Jan 2022 11:43:46 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 050AD183CEED0 for ; Tue, 25 Jan 2022 16:43:46 +0000 (UTC) X-FDA: 79069380852.22.FFDCBF7 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf07.hostedemail.com (Postfix) with ESMTP id 182C540025 for ; Tue, 25 Jan 2022 16:43:44 +0000 (UTC) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1643129023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kssWja8AcLmfLvl1uaYz9JdYKOU1whUKqM7uXu9+L88=; b=GBnx7cgLFTcbVyQ/Q+alsUQWbr4UcDhBg23jZ4wgHrUwwuqtifn8uQAjXiwyNqIO/G0vTB QPeayPJKb62ciMf/bavQRmI/8LD8RAAouNXxogwe+7gXbnr4k1/dREOU0Xzf6fuwa6bLBp crQGWuELODMIe5AA5/LdVbmnoxZuVH/nOlJu41iaFBhBPgcAcOjCc2pyQUVZTHCHbln8Hr E9dsv+2Ds+H8cnONMxmPemztfW5H8rNr6vRo6LXhlO3Rkwfiui0mHMwzfVFg8mSwz63m+B bp8f+kQ16QQB1dz6QKMZUTOVN1iiDQbYA3SAq089CzYeU0n7JPdYKxvGRssniw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1643129023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kssWja8AcLmfLvl1uaYz9JdYKOU1whUKqM7uXu9+L88=; b=kfupKdh4Zcr2PH0nfRsWpBdXCc1KpYrqVL7NCWTcRk3XkXKdcdx1C905K5ReYv7OsPjosi kS7NZ+DXUfjU7yBA== To: cgroups@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , Johannes Weiner , Michal Hocko , =?utf-8?q?Michal_Koutn=C3=BD?= , Peter Zijlstra , Thomas Gleixner , Vladimir Davydov , Waiman Long , Sebastian Andrzej Siewior Subject: [PATCH 2/4] mm/memcg: Protect per-CPU counter by disabling preemption on PREEMPT_RT where needed. Date: Tue, 25 Jan 2022 17:43:35 +0100 Message-Id: <20220125164337.2071854-3-bigeasy@linutronix.de> In-Reply-To: <20220125164337.2071854-1-bigeasy@linutronix.de> References: <20220125164337.2071854-1-bigeasy@linutronix.de> MIME-Version: 1.0 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=GBnx7cgL; dkim=pass header.d=linutronix.de header.s=2020e header.b=kfupKdh4; spf=pass (imf07.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de X-Rspam-User: nil X-Rspamd-Queue-Id: 182C540025 X-Stat-Signature: 99m1rw8nsjwhtbw5kjgxpnexbdrk4rkc X-Rspamd-Server: rspam12 X-HE-Tag: 1643129024-426904 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The per-CPU counter are modified with the non-atomic modifier. The consistency is ensured by disabling interrupts for the update. On non PREEMPT_RT configuration this works because acquiring a spinlock_t typed lock with the _irq() suffix disables interrupts. On PREEMPT_RT configurations the RMW operation can be interrupted. Another problem is that mem_cgroup_swapout() expects to be invoked with disabled interrupts because the caller has to acquire a spinlock_t which is acquired with disabled interrupts. Since spinlock_t never disables interrupts on PREEMPT_RT the interrupts are never disabled at this point. The code is never called from in_irq() context on PREEMPT_RT therefore disabling preemption during the update is sufficient on PREEMPT_RT. The sections which explicitly disable interrupts can remain on PREEMPT_RT because the sections remain short and they don't involve sleeping locks (memcg_check_events() is doing nothing on PREEMPT_RT). Disable preemption during update of the per-CPU variables which do not explicitly disable interrupts. Signed-off-by: Sebastian Andrzej Siewior Acked-by: Vlastimil Babka Acked-by: Michal Hocko --- mm/memcontrol.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 36d27db673ca9..3d1b7cdd83db0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -667,6 +667,8 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); memcg = pn->memcg; + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + preempt_disable(); /* Update memcg */ __this_cpu_add(memcg->vmstats_percpu->state[idx], val); @@ -674,6 +676,8 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val); memcg_rstat_updated(memcg, val); + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + preempt_enable(); } /** @@ -756,8 +760,12 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, if (mem_cgroup_disabled()) return; + if (IS_ENABLED(PREEMPT_RT)) + preempt_disable(); __this_cpu_add(memcg->vmstats_percpu->events[idx], count); memcg_rstat_updated(memcg, count); + if (IS_ENABLED(PREEMPT_RT)) + preempt_enable(); } static unsigned long memcg_events(struct mem_cgroup *memcg, int event) @@ -7194,9 +7202,18 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry) * i_pages lock which is taken with interrupts-off. It is * important here to have the interrupts disabled because it is the * only synchronisation we have for updating the per-CPU variables. + * On PREEMPT_RT interrupts are never disabled and the updates to per-CPU + * variables are synchronised by keeping preemption disabled. */ - VM_BUG_ON(!irqs_disabled()); - mem_cgroup_charge_statistics(memcg, -nr_entries); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { + VM_BUG_ON(!irqs_disabled()); + mem_cgroup_charge_statistics(memcg, -nr_entries); + } else { + preempt_disable(); + mem_cgroup_charge_statistics(memcg, -nr_entries); + preempt_enable(); + } + memcg_check_events(memcg, page_to_nid(page)); css_put(&memcg->css);