From patchwork Thu Nov 7 11:13:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13866214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D7C6D43341 for ; Thu, 7 Nov 2024 11:18:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D22E66B0099; Thu, 7 Nov 2024 06:18:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD2676B009F; Thu, 7 Nov 2024 06:18:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B253A6B00A1; Thu, 7 Nov 2024 06:18:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 837476B0099 for ; Thu, 7 Nov 2024 06:18:28 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 26B59805F2 for ; Thu, 7 Nov 2024 11:18:28 +0000 (UTC) X-FDA: 82759049562.10.4EF9A90 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf08.hostedemail.com (Postfix) with ESMTP id B30EF16001D for ; Thu, 7 Nov 2024 11:18:02 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=hebjJzYt; dkim=pass header.d=linutronix.de header.s=2020e header.b=6rBlaRMX; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf08.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730978222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UREpr2g7dNESBp66vCKny1OLw5nYLTave8FGBQMbiqs=; b=nzAAgM5v0Q3YnEdc1uX+sN7NUKc03lSO+jSgSIuVp9P6W9owlDVNaFNoES8wq2XvhIrWAB 9cgxYRcIvNVOEUqHZvi5587u+04ymeM2ej9aMd/z1oOs9754f6Hx2YGkMTRxuSnSB5Eh2b IgCVPKj1pY5+SP4kW8OmVScOwWQPBFA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=hebjJzYt; dkim=pass header.d=linutronix.de header.s=2020e header.b=6rBlaRMX; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf08.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730978222; a=rsa-sha256; cv=none; b=oNKwun3T2buW4e3/blaE12dC0LZqS/V2AT7EPpU8kv/I/E1WB6g31RgxtQRdLiUM2YTS1f UnR+KzDuavSbbj4kIQMnva/ITdelmCSAPea1IMPgxHvd5I88bQu+wdMnmZgf55isHqUn7a f7dSUPX7DLN3g8K5AL5MB4FF1R2DSCM= From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1730978304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UREpr2g7dNESBp66vCKny1OLw5nYLTave8FGBQMbiqs=; b=hebjJzYt5XphUNmoFrKvrRey/8dSLOnZF+JVRGOlYlm0vHgkXUhLb2qING2kAiqHPYaesF +6JwaIot+DsI5wIYho72FiJO/MCvu1hhIXayNgicQBTDb0ZCIqVuCGeOFfd6RL4MuLKooF OAU7VSdzQqjrKU9J5xtOvdpqeaHAp/L7195AFiV1Gl0dErQSAwL8clasnyAe9NPqFTkUDS v0vhAOU+sDLxfPnDuLmgo9W2qsL+PJXX3L9FsXsXFDQ14Tin0BVsmDqjxSac4Kf+W2wXjj Otc1m0GaoC0cKcq0OCjJEZ8iSz2OppX9HYbpFR6BEfpr8LwQ876PK4XoIUgL+A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1730978304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UREpr2g7dNESBp66vCKny1OLw5nYLTave8FGBQMbiqs=; b=6rBlaRMXpg74iBIs0MX5d43tUWarZm/yzruVF2Ngma8aPTtKThQISSfBrY5vOrQGKd2fUK rvaNMEM3oa1+C9Aw== To: kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Paul E. McKenney" , Boqun Feng , Marco Elver , Peter Zijlstra , Tomas Gleixner , Vlastimil Babka , akpm@linux-foundation.org, cl@linux.com, iamjoonsoo.kim@lge.com, longman@redhat.com, penberg@kernel.org, rientjes@google.com, sfr@canb.auug.org.au, Sebastian Andrzej Siewior Subject: [PATCH v2 3/3] scftorture: Use a lock-less list to free memory. Date: Thu, 7 Nov 2024 12:13:08 +0100 Message-ID: <20241107111821.3417762-4-bigeasy@linutronix.de> In-Reply-To: <20241107111821.3417762-1-bigeasy@linutronix.de> References: <20241107111821.3417762-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B30EF16001D X-Stat-Signature: nj5a5s18r4mcnqcdi8nnc9j51ycbcucy X-HE-Tag: 1730978282-682587 X-HE-Meta: U2FsdGVkX18PKfMfsuP2/AFlOfSAE2crYPjt6FLMNl8cpFtSXgzYdPXhennPllx/65xumcDzaJcPQELIIXY5eZHjih+u9ekjpj77RI6slOFywRfuk7TpqWcei8Mz6BBh2xgHryIZK04mL6QxT0me50FgBEG4JiPzTwPr5VJS4yUoqTNS80JbmcucHi4rUd+bGdhQSSia4N//vmbR16Mmn4ZtkMBrxw7swM8d3Qt2EzdV4Fl4p4uJN8Hd4yJ+jVwnlYmIvwGIrXBbtsKSSGikSjOZf/n4kzVPGGlYItLHzR17J7cEQxMxp5OGLCA1s94/RcWJD+3nKqrVKCXIjQ2Wqu8Rw+LsJKzvKApDQfHjE1Ov8R5SKSE9zMHrW8VJixDlWvngD4jdkVifT28RP+SNqJQUYAzioGWjSNEU7O8SNJD3MMzPuqO5k8Z7pczEHnahFZc3hVPu5zLOvJN3Qcda+CdDTgJR2GfoBTylLA1LFTFSTw91B865Bz+EG31nV1vBQyZrCryYx9oSgeEh+f4C3Uj5zsrJEyFm7E7FmcygArHpj/C5HYKDIX4o9N3bWW+RHb9GyZVyJBXbW4J3UN7aBSJRxs7Y6/9FDu3vtcquuCPHgZ9GiDYE7Ax+A8+jWz9Bd1yHQk4tq62yJ0VhqbafpEvjZkiEcvGyncTnSN3UjWAVPe4Jb58HnUZywN8zziHoQf9HQ+FVIYnHoWCUvl+PpL6S6xXwdeLCBP2ZnOJrx+d54p5nu4VSiIIOhKDb29cfK0aDeJUjgms8WfkbgMUUvRSSD37Mw2R/VxsLK9qDsP8NKpwnrgRgsX/h1X8e5ska6mpbrqWi0WaIdDUKdq+CG5SqUHWoMkt28AJ0xJMMx+ZabxRE7pXFRSelQHz78Eli27UB2dfiwiE2xpp358uKJwlREmWp9Kzqa3aLdfoLtTNMe1Bvym5ygVylBu3NRv8k8k0Irw5pFwypXWh9h5Q LFRiD2bl ZUl4On6Rm3/1Arv132aoUpArnfQtJn3I9qZQdTdNTWEUJHxlUzYCwOt+YyZIN0bBbMsR4IULZ7I9MLGK9hUkKw+mkoCi7sctyLnNGa0j5i+Tuy2Og4EZZWGtUr5sfUjiikbQvXllwUTFhoTaNYLbLxiogUx+YvBUJQ32D6hR6muWCRJI6gR0/YyfnDf55L6FrAfYs6wXW+9Y/oVKnwJuYB/K44QZD6lK+Oi80yBCTflwDXxePIDPtwp5u4w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: scf_handler() is used as a SMP function call. This function is always invoked in IRQ-context even with forced-threading enabled. This function frees memory which not allowed on PREEMPT_RT because the locking underneath is using sleeping locks. Add a per-CPU scf_free_pool where each SMP functions adds its memory to be freed. This memory is then freed by scftorture_invoker() on each iteration. On the majority of invocations the number of items is less than five. If the thread sleeps/ gets delayed the number exceed 350 but did not reach 400 in testing. These were the spikes during testing. The bulk free of 64 pointers at once should improve the give-back if the list grows. The list size is ~1.3 items per invocations. Having one global scf_free_pool with one cleaning thread let the list grow to over 10.000 items with 32 CPUs (again, spikes not the average) especially if the CPU went to sleep. The per-CPU part looks like a good compromise. Reported-by: "Paul E. McKenney" Closes: https://lore.kernel.org/lkml/41619255-cdc2-4573-a360-7794fc3614f7@paulmck-laptop/ Signed-off-by: Sebastian Andrzej Siewior --- kernel/scftorture.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 555b3b10621fe..1268a91af5d88 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -97,6 +97,7 @@ struct scf_statistics { static struct scf_statistics *scf_stats_p; static struct task_struct *scf_torture_stats_task; static DEFINE_PER_CPU(long long, scf_invoked_count); +static DEFINE_PER_CPU(struct llist_head, scf_free_pool); // Data for random primitive selection #define SCF_PRIM_RESCHED 0 @@ -133,6 +134,7 @@ struct scf_check { bool scfc_wait; bool scfc_rpc; struct completion scfc_completion; + struct llist_node scf_node; }; // Use to wait for all threads to start. @@ -148,6 +150,31 @@ static DEFINE_TORTURE_RANDOM_PERCPU(scf_torture_rand); extern void resched_cpu(int cpu); // An alternative IPI vector. +static void scf_add_to_free_list(struct scf_check *scfcp) +{ + struct llist_head *pool; + unsigned int cpu; + + cpu = raw_smp_processor_id() % nthreads; + pool = &per_cpu(scf_free_pool, cpu); + llist_add(&scfcp->scf_node, pool); +} + +static void scf_cleanup_free_list(unsigned int cpu) +{ + struct llist_head *pool; + struct llist_node *node; + struct scf_check *scfcp; + + pool = &per_cpu(scf_free_pool, cpu); + node = llist_del_all(pool); + while (node) { + scfcp = llist_entry(node, struct scf_check, scf_node); + node = node->next; + kfree(scfcp); + } +} + // Print torture statistics. Caller must ensure serialization. static void scf_torture_stats_print(void) { @@ -296,7 +323,7 @@ static void scf_handler(void *scfc_in) if (scfcp->scfc_rpc) complete(&scfcp->scfc_completion); } else { - kfree(scfcp); + scf_add_to_free_list(scfcp); } } @@ -363,7 +390,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra scfp->n_single_wait_ofl++; else scfp->n_single_ofl++; - kfree(scfcp); + scf_add_to_free_list(scfcp); scfcp = NULL; } break; @@ -391,7 +418,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra preempt_disable(); } else { scfp->n_single_rpc_ofl++; - kfree(scfcp); + scf_add_to_free_list(scfcp); scfcp = NULL; } break; @@ -428,7 +455,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra pr_warn("%s: Memory-ordering failure, scfs_prim: %d.\n", __func__, scfsp->scfs_prim); atomic_inc(&n_mb_out_errs); // Leak rather than trash! } else { - kfree(scfcp); + scf_add_to_free_list(scfcp); } barrier(); // Prevent race-reduction compiler optimizations. } @@ -479,6 +506,8 @@ static int scftorture_invoker(void *arg) VERBOSE_SCFTORTOUT("scftorture_invoker %d started", scfp->cpu); do { + scf_cleanup_free_list(cpu); + scftorture_invoke_one(scfp, &rand); while (cpu_is_offline(cpu) && !torture_must_stop()) { schedule_timeout_interruptible(HZ / 5); @@ -538,6 +567,8 @@ static void scf_torture_cleanup(void) end: torture_cleanup_end(); + for (i = 0; i < nthreads; i++) + scf_cleanup_free_list(i); } static int __init scf_torture_init(void)