From patchwork Thu Oct 31 10:21:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Galbraith X-Patchwork-Id: 13857785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CD25E68973 for ; Thu, 31 Oct 2024 10:21:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DD5B6B0089; Thu, 31 Oct 2024 06:21:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 989A26B008A; Thu, 31 Oct 2024 06:21:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82A2B6B008C; Thu, 31 Oct 2024 06:21:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5BD196B0089 for ; Thu, 31 Oct 2024 06:21:39 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 08DADAD06B for ; Thu, 31 Oct 2024 10:21:39 +0000 (UTC) X-FDA: 82733505582.02.4276AED Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) by imf10.hostedemail.com (Postfix) with ESMTP id 13526C001F for ; Thu, 31 Oct 2024 10:21:24 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmx.de header.s=s31663417 header.b=dtn73n0r; spf=pass (imf10.hostedemail.com: domain of efault@gmx.de designates 212.227.15.15 as permitted sender) smtp.mailfrom=efault@gmx.de; dmarc=pass (policy=quarantine) header.from=gmx.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730369882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Vnti2AGSrPFnzLrZE2hR13Xa35l0I7YXOqGNiR8vqto=; b=CkwmM0V18T1HmYb6IhqrivScNMkuG8q+q2llgc4MB1Ix9r3xbNMHSUt9aD6eXARQ3YQVNi mYz53B95Q1gmHwmCBvhETxYTcHGNnbZE2TpdCXRtwoc2L7K0quEeD8MqX2ambdxJcANjEp 6Pvs2VzmjkjrooUH5ohN9BLXNCblJDw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmx.de header.s=s31663417 header.b=dtn73n0r; spf=pass (imf10.hostedemail.com: domain of efault@gmx.de designates 212.227.15.15 as permitted sender) smtp.mailfrom=efault@gmx.de; dmarc=pass (policy=quarantine) header.from=gmx.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730369882; a=rsa-sha256; cv=none; b=12xijdWfvIm9ICZgd28lCFHu0fu9XywA0Kz0DpE+nasV0agJX6Wd7/r4Ok7eeZglgHHJh1 4OCC4gQ8EnAjQowiqoPQR+6H4gqhfimbfmuvIAwA/TyrmXsDFIxGLVWwrMom56IEfu+ITs W2dinF1bqcKC4dl+YNaIVySLOvKw8k8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmx.de; s=s31663417; t=1730370095; x=1730974895; i=efault@gmx.de; bh=Vnti2AGSrPFnzLrZE2hR13Xa35l0I7YXOqGNiR8vqto=; h=X-UI-Sender-Class:Message-ID:Subject:From:To:Cc:Date:In-Reply-To: References:Content-Type:MIME-Version:cc:content-transfer-encoding: content-type:date:from:message-id:mime-version:reply-to:subject: to; b=dtn73n0r85uKvOZGDZX1YCCavlubO4V++dWU0s2o1tS4vFe0jadiVLdHH02aNo1u HcZPUSbjYeLwrHYKmjianhm0OtErYCmsS0FkCJG3BzL67XpmrHI+wEkppH1kOOrzP GX0SXlN3g1XkHtfRCA13pffvX708ckY+Q10kIGML7wcrTUCYfKSLgcVZAHGToc6dt dJj3mP/BE619Kqj8L+IR4E1ZlGRKtWtnDyG6mAa10gFzSulQ0hmDOFiQ2sQWSmTJn j2Dz3fO9nIF5mM0B5DQFy89SRd8SgHRVJF/a4q4ih66/nsIwScOqlwtoycnfN9v3q Nu0wc4yS9bYXKbPpzw== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from homer.fritz.box ([91.212.106.187]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M7b6l-1tDrMY2fZF-002Eg7; Thu, 31 Oct 2024 11:21:35 +0100 Message-ID: <235e470402ee86a6bf94cfe73543b0c0f5a42071.camel@gmx.de> Subject: RFC [PATCH1] mm: vmscan: Limit isolate_lru_folios() IRQ holdoff latency impact From: Mike Galbraith To: LKML Cc: linux-mm Date: Thu, 31 Oct 2024 11:21:34 +0100 In-Reply-To: <88ae633aa910c7e320b89e5f4779d472f7cffffb.camel@gmx.de> References: <88ae633aa910c7e320b89e5f4779d472f7cffffb.camel@gmx.de> User-Agent: Evolution 3.42.4 MIME-Version: 1.0 X-Provags-ID: V03:K1:sk2kb5Jd5sO2tHpqGdYyzvcw5MkUz5GbQwKTXYXHSx8X4XnGvsz m9hXQK4SkFI1JwNk5vYASrH0Rwugy3cCcZgIYhyMCnor4R6kURY/zfbzOvIRT+btQkxsxKd CX+JaIG2u4dmI3vszBdxpW/+Cb8y/k48nKCT3E5RPa0JaiAz5tWD/B6NUZ/+XLvI+39xFe3 8df+P9w9bJO2i0hxfIEbA== UI-OutboundReport: notjunk:1;M01:P0:JKVogsXjwqM=;LwqKlXR14RZewERScfnCPXmeg/c GwKc7LhThWgJRJ9AudDZbCOpBkLmNWzBEJ56jg20w+wJ5mvxZLFyFqacj6oBTIed62HBiAfd1 k3YRzcLScVW64zmgKfFcaSqMZ90U2QGkwedEG8lFPEWImFy0D9Za1z5dm8TVRJRMuQv0Zvkz2 xxz3y7u/DfElGfFepqMmCMxA8ra7YLJO8fwb88eLO0GLNEU666fOU7Y8y0Kby+fCDxvpeLAer eN9UbpfUCXvavnNha7wBd7rsJMXrYzrC76wYQIjaqioDvko+N48CqJVqw/0iCcS5GXYekAcM3 0bw66qovaZfhWB2tl9xPZ8Y7kLm+r5XVPLOHhUwdi9T3cSEaG0JP8HOnZ6Ef/inM9CMMmC1Ou jXIvFgGp+tbv9Yre3kEtawE/oAAVFjjw1MGI8H5UjS2GxKarHagyqAyQL9ldHPsnbV6bnpsLa RNVC7MvP/z4sspijQscS71/il0u5yMBZ/m81vKBGmlNocQfs6AI7XaqLvrgdjRHkVC+9XyK/a ojstxGphbpXGVl/iNYPUgC+1usaELYL77A891jPRzzViuEkxGUlB9F1hZOh/yQxcpbQUfDJK/ 7T46ayTK5dP4wf12HHUmHL8mTZ/cdym7CvEDhgwVfB2rQOxnKsT0R1O/uac7llk+sp2DzXL10 eRIM7hZfCnsXuEkMSZEaeBvMD322Qiy8HI1mO6J6FJJbV8P98G7wpSxGXmbmK9ALUo6urHA5o 0xWC5Gn5uFq12PVF3IpuSRkJ2WkytAcExextdezb4wRQeAeEnp7Sok/bvt1+tKWrYKkMMrLAN xkM5Un0kcC8LzWsMRamiH2Ow== X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 13526C001F X-Stat-Signature: y5az9ke44uuc44m6b3hffo4h7uagkgh9 X-Rspam-User: X-HE-Tag: 1730370084-113693 X-HE-Meta: U2FsdGVkX19fu7zEmrDLPJBjWzZlemdM5AgQ7kR6WyLbBMwKv1W8iZso7912crArSWHhwZm74c8+TsbYi01nrRicr9ynoPqb3jbTbKdWr1poMWGBMwbCXyN/gMkU0b/ynfnJE+WfF6zsUA+FqfxQWdj/B9MtXmBO7Aj/UItH55TnlyRhJxCWmh4xZct3UFyI/12NkFJmjW6hIc2MLulEMhqJ3EG4s08uyWCecYXNhkOeDm3X/gbKvIJduJvgVq8E/fwAqJRd4nULgDHzImTUwTwT/Amhd75wUEJaiOJ7jlPW1XayxYIVWoxvj94+4+cSpAgE8jr3Ov/L9kki4H8DnhWPmWnT06xsobrmPAnzyL/2su2mLKYVtDLkfvQ9PjTIxQOcfAx3IOiVBlUIKRu9qYx7rONkxLipf3C3De+ANpj7k7vgQjex3EH1G8NLOQ9Y1DE37Zy3IsoA+kSdjGXytNy/bk93gtDAMbLUZiWAueMbJCET9nqXVxUbgjqqVN+TLxaeQozTI+Uu2PD6TvL0VobJIARwGEDHhsb3cv9i6SWx//6C45OB0V3Km/uIxWgO7xKHYcvxRaJrXtPN+kuXoiuhmM14Mx1TBrMuRbD5XnV78AMz8KozgXnMcOKj5QbLD1FLK2c8KZmKonxswrAo2U8XZKCpoW6R20Netx164Sqgb92sBFcg1T0aVvjyT1X6j/NvIgi2A8F9LIKOT27bP8KOHLotXV2+7cI3w2D8mVOqXqYwfeow7T8B2iaP+/CMtFB5oYm0sEYlwp8Lf+tAFve+2myPvFMHZVgpcsZRhik/yxj6au8Rqg4kmN0m5W5MVFso2GLIPq6eeKCgG+dmTG1VxyY6zHEHiLTGwdhuM2yA1HydKaLqbBg2NEsdpQVjw/aRHo02FIrt/AjIkDsOID9idru89uiEYK27hbu93xsj30P0hFbXpCUtRHqgQNfJF+UcH/lXbjIJeHNCSOX M++EHre5 Xo8wme/3q0qgPv29noNxwsPB3PBcpI7qXg8uG4jba1Cd26uxiBnm8zAT7SWwPg4S67WexUddWfJL4f7kjHNIc88JyS8+e2vACSKNX3qo+pxyUHd9QqB575BuWeQVTaS31aFsZHfKJTpHP6ZiXTEfI8pa45wbk/6eVOV13eqa3u/0p4qN40nffuswmESoM8HYjvsdEYwayEdIB5ufh+TbcA+PaIErmOT3yAiSUncD1aiTcLArPWebau6DC8XPwr7UKNPmGZegXTjZkfWSxLJo0gRlE2iZ/gl1ESnwOniZm+fyS2b/aVUP9/DDbgxGMWSCDuEHLmYU8rvZrIzPOhfATkwzwthcXe56v6f6mT499u9P4igvk3sn6tWGBFvxJNArS9ck3gb1B9E0hIrjiEhQHoundFxElN6l64cHVNwuaA6iVe1iFHiIs2svfuEcHEoktJCvS2IVTEeBG6wsoOqClSjq/6ToV+XNAt4Pf9/CMF6tVfz4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: (was regression: mm: vmscan: - size XL irqoff time increase v6.10+) While examining a service latency regression beginning at v6.10, it was discovered that isolate_lru_folios() runtimes could cause IRQ holdoff spanning several ticks. Worse measured was 78ms. Given it's called with IRQs disabled, cap its runtme to around a ms. With this patch, and one to mitigate move_expired_inodes() induced wakeup latency hits up 66ms, the load that repeatably produced this: (6.12.0-rc5 running osc kernel package build + ram sized bonnie) T: 0 ( 4541) P:99 I:1000 C: 773124 Min: 1 Act: 6 Avg: 3 Max: 53381 T: 1 ( 4542) P:99 I:1500 C: 515381 Min: 1 Act: 7 Avg: 3 Max: 60119 T: 2 ( 4543) P:99 I:2000 C: 386465 Min: 1 Act: 5 Avg: 4 Max: 67469 T: 3 ( 4544) P:99 I:2500 C: 309182 Min: 1 Act: 7 Avg: 4 Max: 65534 T: 4 ( 4545) P:99 I:3000 C: 257682 Min: 1 Act: 6 Avg: 4 Max: 57776 T: 5 ( 4546) P:99 I:3500 C: 220874 Min: 1 Act: 4 Avg: 4 Max: 64737 T: 6 ( 4547) P:99 I:4000 C: 193289 Min: 1 Act: 6 Avg: 4 Max: 52567 T: 7 ( 4548) P:99 I:4500 C: 171799 Min: 1 Act: 6 Avg: 5 Max: 52606 latency went to this: T: 0 ( 6078) P:99 I:1000 C:1031230 Min: 1 Act: 7 Avg: 4 Max: 4449 T: 1 ( 6079) P:99 I:1500 C: 687505 Min: 1 Act: 5 Avg: 4 Max: 2075 T: 2 ( 6080) P:99 I:2000 C: 515627 Min: 1 Act: 334 Avg: 5 Max: 2987 T: 3 ( 6081) P:99 I:2500 C: 412503 Min: 1 Act: 8 Avg: 5 Max: 2425 T: 4 ( 6082) P:99 I:3000 C: 343752 Min: 1 Act: 11 Avg: 5 Max: 1466 T: 5 ( 6083) P:99 I:3500 C: 294643 Min: 1 Act: 282 Avg: 5 Max: 3844 T: 6 ( 6084) P:99 I:4000 C: 257813 Min: 1 Act: 7 Avg: 5 Max: 3291 T: 7 ( 6085) P:99 I:4500 C: 229167 Min: 1 Act: 9 Avg: 5 Max: 3516 Signed-off-by: Mike Galbraith --- mm/vmscan.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) # tracer: irqsoff # # irqsoff latency trace v1.1.5 on 6.12.0.g4236f913-master # -------------------------------------------------------------------- # latency: 77766 us, #12/12, CPU#5 | (M:unknown VP:0, KP:0, SP:0 HP:0 #P:8) # ----------------- # | task: kswapd0-90 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: _raw_spin_lock_irq+0x16/0x40 # => ended at: _raw_spin_unlock_irq+0x10/0x30 # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / kswapd0-90 5d.... 0us : _raw_spin_lock_irq+0x16/0x40 kswapd0-90 5d.... 0us : preempt_count_add+0x0/0x50 <-_raw_spin_lock_irq+0x20/0x40 kswapd0-90 5d..1. 0us*: isolate_lru_folios+0x0/0x3e0 <-shrink_inactive_list+0x10f/0x440 kswapd0-90 5d..1. 77765us : __rcu_read_lock+0x0/0x20 <-isolate_lru_folios+0x160/0x3e0 kswapd0-90 5d..1. 77765us : __rcu_read_unlock+0x0/0x180 <-isolate_lru_folios+0x199/0x3e0 kswapd0-90 5d..1. 77766us : __mod_node_page_state+0x0/0x70 <-isolate_lru_folios+0x2ee/0x3e0 kswapd0-90 5d..1. 77766us : __mod_zone_page_state+0x0/0x60 <-isolate_lru_folios+0x2fc/0x3e0 kswapd0-90 5d..1. 77766us : __mod_node_page_state+0x0/0x70 <-shrink_inactive_list+0x12a/0x440 kswapd0-90 5d..1. 77766us : _raw_spin_unlock_irq+0x0/0x30 <-shrink_inactive_list+0x16a/0x440 kswapd0-90 5d..1. 77766us : _raw_spin_unlock_irq+0x10/0x30 kswapd0-90 5d..1. 77767us : tracer_hardirqs_on+0xcb/0xe0 <-_raw_spin_unlock_irq+0x10/0x30 kswapd0-90 5d..1. 77774us : => shrink_inactive_list+0x16a/0x440 => shrink_lruvec+0x2fc/0x470 => shrink_node+0x1d6/0x4a0 => balance_pgdat+0x280/0x5c0 => kswapd+0x10f/0x350 => kthread+0xfa/0x130 => ret_from_fork+0x2d/0x50 => ret_from_fork_asm+0x11/0x20 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -63,6 +63,9 @@ #include #include #include +#ifndef CONFIG_PREEMPT_RT +#include +#endif #include "internal.h" #include "swap.h" @@ -1657,6 +1660,10 @@ static unsigned long isolate_lru_folios( unsigned long skipped = 0; unsigned long scan, total_scan, nr_pages; LIST_HEAD(folios_skipped); +#ifndef CONFIG_PREEMPT_RT + u64 then = local_clock(); + int batch = 0; +#endif total_scan = 0; scan = 0; @@ -1709,6 +1716,18 @@ static unsigned long isolate_lru_folios( move_to = dst; move: list_move(&folio->lru, move_to); + +#ifndef CONFIG_PREEMPT_RT + /* + * IRQs are disabled, cap holdoff at a millisecond or so. + */ + batch += nr_pages; + if (batch < 1024) + continue; + if (local_clock() - then >= NSEC_PER_MSEC) + break; + batch = 0; +#endif } /*