From patchwork Sun Feb 9 22:30:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9363FC021A1 for ; Sun, 9 Feb 2025 22:30:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2500C280003; Sun, 9 Feb 2025 17:30:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D8626B0096; Sun, 9 Feb 2025 17:30:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F202C280003; Sun, 9 Feb 2025 17:30:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D0E306B0095 for ; Sun, 9 Feb 2025 17:30:33 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 881E4120D1F for ; Sun, 9 Feb 2025 22:30:33 +0000 (UTC) X-FDA: 83101851546.19.EB9707F Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf07.hostedemail.com (Postfix) with ESMTP id EB83340006 for ; Sun, 9 Feb 2025 22:30:31 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OiznD3kz; spf=pass (imf07.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b1s6u44EkwbMOEY9WgoVy3zZsrG3py7NrPuyt60qlMg=; b=2GvaickNu1Dff2t6W2FaubmXNhX/dmZSu1hJcDPsFz6Apk6GgSv1+8lJXEe6s5vEK9Enxh IqS0DmfQP+yZK+YmkPOmdCmkGChEdIfTVm7uOSPYbxxWEsXsZSe1yGd+n7OdyU0wVPQ40C 8CL7WY+f8pjkZAdQ2ohzP+EHLkkXXGU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140232; a=rsa-sha256; cv=none; b=Wo4UTGnm6M0mO3BKkaLOo2QhsjIBT5KeL6y5pU5ZuX6ZIEk0HyGnHf8DfQumg/xREjhNt8 uO1qLhcRO/KIMjTt8279EPOtZpdMVY0gNX1K5+BlznHyj6xBQgZJCcDJZvzmdO4e3A9B+Q 1lBHZ2IUCUcLbOXKLeATLaUif4gLMdo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OiznD3kz; spf=pass (imf07.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 9B2FF5C5978; Sun, 9 Feb 2025 22:29:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ADF1BC4CEDD; Sun, 9 Feb 2025 22:30:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140231; bh=jJC9e1iQUws78AISrOzytLZgqafLyxgOcIGGwKvAOvw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OiznD3kzJiFVXddt6RYzafH2Caz7uSr1dddZCrIu/u3/8mczDP7ZEKGEDjOVW/9yB btHfhsaIgyvr6xWqi13Em7VXQZrpIGjktay/irh2ACH0rVTRyVFuM/cwBj9P3YzoS2 HMYFygOMnWMB+fi00pTr3JpElh/O2dNQg8UHryyWUEt5erxMlSLx3vAWejp1Gl10L3 xA5YNFPcDwqlP5fBFm2Bie4loz+E0Sn1Zn5Yqu8KeIyjGU8ahnvAufLyvVx2ooEKYm kmKPw2Hf1FhX5g2//fFphPUfZoIcVTp+CopdTmBHa2lkSa3i/yegVGmp+lerJO4ZiS B3jytGQj0Lgew== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 6/6 v2] mm: Drain LRUs upon resume to userspace on nohz_full CPUs Date: Sun, 9 Feb 2025 23:30:04 +0100 Message-ID: <20250209223005.11519-7-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: EB83340006 X-Rspamd-Server: rspam07 X-Stat-Signature: nt335gk6gagkzh6gbinhrbey33g455ko X-HE-Tag: 1739140231-718244 X-HE-Meta: U2FsdGVkX1+uIBOlYGl7I7itZL7piOd/Wk58OfNgU1QJZbFQGCd8tTplo7R+0LMg/VglKXsXMut2TrUVP1N1zaSYbyyuQ+o/vPdrlOAcXftVsV3k8HzsrUpg6RnPmaBZGOyhhvJBiToouDQkpXpXcovUwdip8Dq1Ihvj/K/nTRRuACZQw0K2eLyQVKYgaqaTNUjr+8aCuGqccPgx6ttqTMegZ8kz7qVrMqP1SUd5v1+UuujHRLJIH/Y88axj5H9r3A3Yks982RmxUW3EX6665Bb4SZKi/uYvUd64SIdYTsSi4xzaARO0B6KfBfYaUQAlFhVxdZ8OfZTigLG3e9CN+f/uAllSENFJwekJNc80EQc8ysGjFnwL/JbCybdOEwBguRwMUnGp2+aPpxCsQRKD3R98jN9KU6PIkbfI5YWhEXvY/PYHO1FcNaj6AvscOTH1s7tZnBSW8f0gC7Rf2u51UFWfdmo4Bd7xrkyHGfhs+85XUoU6Y31IfCPK40eE4novCcgIoV/qsPkORPnnR4RuEhgfQOc1ZpYPhIglcXjv6PVFO+EV/HhE5+Kl//tkZBBhB9s5BjEFqUwUZCj/aXBN7aZCy5rCMeRPUTHydvbuyVpqTZcmVIF//6se2SwWeSXGd4+4tLM+B0/Padf9SrxvXWtLtYXPfbfmZQP5Ja4OlVp+XZ2TCHOIdfnsnkGpWSYiD/N/KodMqylsce1vFiK1ZULGPXMKwuLo4GzcDF4ed76BXiewCCugIqQR6WIbmjZe1RrHD2jyD2KR12rnSw7ABlDLE8ftZaeFQpC5uYSUscAmxi+OOwWH0E85KbCm14Ss9nppdiUVeZJTDeQo4Yp1aZbMG1AZxvhc4nM+JkfACurBb7wQ9jYhzRxLuDh/PiwMwVGD7XZbVmXdzw5g+Kci9JhyGJtpI+/cl/kRsAVnv7wEB+CCPnh2ayEUAXl8OKP4e+zhiY+YWm0bZIYRIDq b7XDNjTw SJmlNgOCzodP0wy1R+wDgewBWVzpXO8LHWV/jbIq1HdmOzhi+WtKQawLWlAZii/4++MYy1h91ERDEDFdpqxFtmFShfq7IMGn8HIzQFgdZ7h/2hjqLQcuWjDeBP6RZmNuera9vJO6jsJKn65f3Z55n5AL71Et/GjO3v3AAF16slGb3+2shpSikqL9NvtK/RdziGOS9Xpzs7toknF60taEQomNIthAbt+DZv2TzaUJO0KQ3tkbCvIg4YDM0OqInUy1wiHfefULppSUIKu5EttWWafM8NpkJhTJDxVPWqBjmAPGL9PBPrhByIgjIFn8HbDW3lTUu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LRUs can be drained through several ways. One of them may add disturbances to isolated workloads while queuing a work at any time to any target, whether running in nohz_full mode or not. Prevent from that on isolated tasks with defering LRUs drains upon resuming to userspace using the isolated task work framework. Signed-off-by: Frederic Weisbecker --- include/linux/swap.h | 1 + kernel/sched/isolation.c | 3 +++ mm/swap.c | 8 +++++++- 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index b13b72645db3..a6fdcc04403e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -406,6 +406,7 @@ extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); extern void lru_add_drain_all(void); +extern void lru_add_and_bh_lrus_drain(void); void folio_deactivate(struct folio *folio); void folio_mark_lazyfree(struct folio *folio); extern void swap_setup(void); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index f25a5cb33c0d..1f9ec201864c 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -8,6 +8,8 @@ * */ +#include + enum hk_flags { HK_FLAG_DOMAIN = BIT(HK_TYPE_DOMAIN), HK_FLAG_MANAGED_IRQ = BIT(HK_TYPE_MANAGED_IRQ), @@ -253,6 +255,7 @@ __setup("isolcpus=", housekeeping_isolcpus_setup); #if defined(CONFIG_NO_HZ_FULL) static void isolated_task_work(struct callback_head *head) { + lru_add_and_bh_lrus_drain(); } int __isolated_task_work_queue(void) diff --git a/mm/swap.c b/mm/swap.c index fc8281ef4241..da1e569ee3ce 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -37,6 +37,7 @@ #include #include #include +#include #include "internal.h" @@ -376,6 +377,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } local_unlock(&cpu_fbatches.lock); + + isolated_task_work_queue(); } #ifdef CONFIG_LRU_GEN @@ -738,7 +741,7 @@ void lru_add_drain(void) * the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain(void) +void lru_add_and_bh_lrus_drain(void) { local_lock(&cpu_fbatches.lock); lru_add_drain_cpu(smp_processor_id()); @@ -769,6 +772,9 @@ static bool cpu_needs_drain(unsigned int cpu) { struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu); + if (!housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE)) + return false; + /* Check these in order of likelihood that they're not zero */ return folio_batch_count(&fbatches->lru_add) || folio_batch_count(&fbatches->lru_move_tail) ||