From patchwork Mon Aug 14 15:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "T.J. Mercier" X-Patchwork-Id: 13353002 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79558C04A94 for ; Mon, 14 Aug 2023 15:16:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDCCD6B0078; Mon, 14 Aug 2023 11:16:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8B116B007B; Mon, 14 Aug 2023 11:16:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B51918E0001; Mon, 14 Aug 2023 11:16:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A29956B0078 for ; Mon, 14 Aug 2023 11:16:57 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7313DC0656 for ; Mon, 14 Aug 2023 15:16:57 +0000 (UTC) X-FDA: 81123062874.20.E3DF5B6 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 9FF6718000B for ; Mon, 14 Aug 2023 15:16:55 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Syd749r+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3ZkXaZAkKCIc4uxp2ntp2rzzrwp.nzxwty58-xxv6lnv.z2r@flex--tjmercier.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ZkXaZAkKCIc4uxp2ntp2rzzrwp.nzxwty58-xxv6lnv.z2r@flex--tjmercier.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692026215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=/FNoZQB0IrLaKqGcAaCCQtd/SwINEujNen6PFYywNmQ=; b=O0VJ06XGOQZ5loXEpS3a2zzhOamoGKif+LZekdDaaUV/8JdT4r2EkXBBkTWMAChlL6tip+ yNk+bvB17k74AyUdt6D+TKvTufO1jH+ApSw+v9avl+U68dRWMa6Jw2zZ6h82xtmMgVyiXi pZeoTP7bNCyL7/FI6Vxcl/DjUVz6eOE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Syd749r+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3ZkXaZAkKCIc4uxp2ntp2rzzrwp.nzxwty58-xxv6lnv.z2r@flex--tjmercier.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ZkXaZAkKCIc4uxp2ntp2rzzrwp.nzxwty58-xxv6lnv.z2r@flex--tjmercier.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692026215; a=rsa-sha256; cv=none; b=XMIFU7lR9ZTpTY/hZ2Vf/rcjW+QAAN7BzC1HPd7o5j/KwFyPHIV10YmVHo6Yau6bqvuE+M cq7mjOGFExho2CYlI43QI11FSB1baj/sP+YuBFkst5tEERIcJ3qft+PFIBEeTC9GrZER5I IHPA2ZwJnm3Wff3Gl1jvMplSdiOWSFQ= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-589ebdea23fso25542037b3.0 for ; Mon, 14 Aug 2023 08:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692026214; x=1692631014; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=/FNoZQB0IrLaKqGcAaCCQtd/SwINEujNen6PFYywNmQ=; b=Syd749r+wn/31GfcBZ2X1EmM8zOBT3WHN3gRME1+GWTQKeuS/r1PTSbEnPXHTwGHwe 6fnC4j8d1fVeqUEjKApqDPF/0xYsuJSLnk6UEBLeKjNU84fJOo9Qij8huJ/ZiRRBhPAO rmdzwAM+EdgClvgMsvs/I/fsVDh7o7JA5Vvtpq1gwsZjVds09RvWlssJNWOnc+s3ogCd xdJBuSqn95iMh+mtkpV37HzhOze6UfxJ3EqiJGh5GpMWFL/E5Fx/dD6eQpKlwfBTu+d3 uqbpDpJZHSBkqaADM23Swa5RWpd4ZoY3kiCu1eZUQetrUSaLIpoaHPvU7sEFqDSFXAj+ cflA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692026214; x=1692631014; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=/FNoZQB0IrLaKqGcAaCCQtd/SwINEujNen6PFYywNmQ=; b=aNSn/jKt0ay5hSvVL5C2Vv+FoXEBr7GqthvsrF9O4pme8rNBOQB33LpvuLMBM+jPOm Ek6TfVdnzHl1kyTzoVhO3KoO7TsswSyAgYvm8OdT6BG5sN2FJPH0eWs38i/ShL22pkbi 5VNARwbC6q5h1f/lRtkwI01UAxjyTL7a2qzKEGvRnR7PzGEO1rUGFalQdFuKKm60aJV+ E9LROIDT1Zgtn2ZvJzfBdWEFTl/c9Cu6n5N9y4d9viLXjqMpg+mjP9mNx/ci73mXwFY/ sdws1a47vDaCSFxz4olWJNV5jUJSA971XRf5tbwXWIhGocqZvjoUivmT5vceo7N1KqpM XICg== X-Gm-Message-State: AOJu0Yx1QTxd9ga8xtQgWGFJQClzZdaPr+LjOES3IfbDYHVOLo28GsoE X1bkbAEKoCP2XWi/a6ZNWttgkkwyJX+gGPU= X-Google-Smtp-Source: AGHT+IGNjXXgrbDzHO8XmjU8hvNW3ac8in3D1QL3hTJWGpwZ0u3oHyq6TITxY7sjk641/7DG+3aOn6ThNsHUDiw= X-Received: from tj.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:53a]) (user=tjmercier job=sendgmr) by 2002:a05:690c:300f:b0:589:6c60:f4a0 with SMTP id ey15-20020a05690c300f00b005896c60f4a0mr240385ywb.0.1692026214561; Mon, 14 Aug 2023 08:16:54 -0700 (PDT) Date: Mon, 14 Aug 2023 15:16:36 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230814151636.1639123-1-tjmercier@google.com> Subject: [PATCH mm-unstable] mm: multi-gen LRU: don't spin during memcg release From: "T.J. Mercier" To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org Cc: android-mm@google.com, "T.J. Mercier" , stable@vger.kernel.org X-Rspam-User: X-Stat-Signature: reztx11azrpjmqiqfru9o6uskngjc1au X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9FF6718000B X-HE-Tag: 1692026215-76879 X-HE-Meta: U2FsdGVkX197jRV0Tg7ukbnylE657GUms5iCuX5LEh36yJOj1yQKuIAbsLxqt2ud+32X1drYTW8Tq1ebVsLSgBBj1/a22R9HWCHIxZK19s2rXgxD9cgpyyhrIYvICkcdFS7BtFIMQBLy9U2i/z792YCt0fLrmL77wqEZQIGbO+GkEjd1jZdEjQJjN0dNf9gv6r1nZIQ56vwQ5wzLJJ7sxy1rdjxKBXa7wW8RFLTPFLPDCQvS8nZdPHBe0tntfPLfg19DJPSFSUejiFH2BZwU0jqECFmNUnzXNdvmCEXRdguWSzxSyqoOcmZLot5tc7idV4nevOzDxDactnRbsfiBGznbai8aboYAJhnrGz1hTjoR5qlJKRwVvG4XgOEcfowggZaSkOBSFZ0ov2pXemEazKXIiVNW0tnaCKF2+Gh46DTSQWw+YRCPQfGlNEOJG1oJUXE8TSk4A+/obvF7PGcR9rjxMaXn7VOQQHY1sWO2N3vlXYfJ6saRXKzkKJN+m+542dsdVVQ4/SLpsArIShHSGuYlKoa9D+hcANfywjVNOssVRcsuvCzSFBz23TAEB6T8LLXJbmtS+1c/5KSFPMlknE8HfXdyqM+Ktzt2m3uIutpMtLThjpHKwZv+MN3LxnBKmI2d7rMBR0/gxfwAMPVxVPrqsAQD2I31bJcdtTs9Pzi3QNnc5yF/wgoinhgsGkOvEQWFaUnUQbDKuyC2iBho68YrxrUdCRFXfJPw9NmDdPdox6bBMfQlwnI/D02uSXun4V6aolqPLu+P85/6tF8ZO5Qmvb4BOREXNXbl2pKP4XIcF6d9vhYmJ4ub+y+aDMnua1ZCfvxDEzwnwTGaWh/QiQVCoo1frr/xnuzX7/WSdk+a0IhystdGFV6Pi5VXEXb7Yg7PFKOQzhIKRQ3X0qPWHZgZYGgCx3ke3vais4Bg0wJWBntSKPXBMwHmewSWztdH82LToaPEK23YgT9H0h9 BHII1jtQ 8i9hGe1QNuNTQnxZ2bjnBpBQjypSEAzD3DJZRLtFXtSoyzsZ5ZuxZOfIpNr75cV5pYJLNyxNN1gyNvmUXNbPp/Y9ghNMv+rA3aVkmZL2D+PKileF8pL+Q2t7u7WXxUBjl5lfJNrX8e0vPYBbnIXouApl6AAW3vJkXTkIPlW9gG/1iCa7bgdp/NXD5n/yQtrySGbQQncQdtf+/98Bg/CamE/v0y1lD+4cce98oearVOUdndT+fa6SQLDQqdXkt9iCP2A4VZ5Yk/DFm2lVGDVAB4rsdCP+gtBx5q1CzJkG/z8n+g2YMh2sGuMJIcQELl1bAilTnlbixiMNhDOvX53Ec6yJpbntxcb7KZmTPvupLcrepn47K3oBEccgMIKEJog26oyVGmQuwSJ3xly5KXPswOLXNa0a1aU3ockm7Ca1g2i8UH9Jx3Pvbrk71i7d3hTWbzV5CewqJj1E8nmbUuHH5HREvLevfjTwb4SwA/gasbQrhnqwHQpHwKj/QbQstiCFeCcafYhRaVnpN5BZO+Igjz5n3WyO+vajqffw+ea9f7ooB0BgoxHnQev1qlKpoiaztFPwqdOz3sJl8wsWS9HqOKioJ0uj4rJDM3u+J17PZHub+mOuQGx7fklg9cSe2b2X+RFDifBs9DaP5RAEuZsHHtGvUBk93MyI/ymHKgQf4O7EcIgPKT51xle/VnQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.009275, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a memcg is in the process of being released mem_cgroup_tryget will fail because its reference count has already reached 0. This can happen during reclaim if the memcg has already been offlined, and we reclaim all remaining pages attributed to the offlined memcg. shrink_many attempts to skip the empty memcg in this case, and continue reclaiming from the remaining memcgs in the old generation. If there is only one memcg remaining, or if all remaining memcgs are in the process of being released then shrink_many will spin until all memcgs have finished being released. The release occurs through a workqueue, so it can take a while before kswapd is able to make any further progress. This fix results in reductions in kswapd activity and direct reclaim in a test where 28 apps (working set size > total memory) are repeatedly launched in a random sequence: A B delta ratio(%) allocstall_movable 5962 3539 -2423 -40.64 allocstall_normal 2661 2417 -244 -9.17 kswapd_high_wmark_hit_quickly 53152 7594 -45558 -85.71 pageoutrun 57365 11750 -45615 -79.52 Fixes: e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists") Cc: stable@vger.kernel.org Signed-off-by: T.J. Mercier Acked-by: Yu Zhao --- mm/vmscan.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 157ed68470ee..c7c149cb8d66 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4856,16 +4856,17 @@ void lru_gen_release_memcg(struct mem_cgroup *memcg) spin_lock_irq(&pgdat->memcg_lru.lock); - VM_WARN_ON_ONCE(hlist_nulls_unhashed(&lruvec->lrugen.list)); + if (hlist_nulls_unhashed(&lruvec->lrugen.list)) + goto unlock; gen = lruvec->lrugen.gen; - hlist_nulls_del_rcu(&lruvec->lrugen.list); + hlist_nulls_del_init_rcu(&lruvec->lrugen.list); pgdat->memcg_lru.nr_memcgs[gen]--; if (!pgdat->memcg_lru.nr_memcgs[gen] && gen == get_memcg_gen(pgdat->memcg_lru.seq)) WRITE_ONCE(pgdat->memcg_lru.seq, pgdat->memcg_lru.seq + 1); - +unlock: spin_unlock_irq(&pgdat->memcg_lru.lock); } } @@ -5447,8 +5448,10 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) rcu_read_lock(); hlist_nulls_for_each_entry_rcu(lrugen, pos, &pgdat->memcg_lru.fifo[gen][bin], list) { - if (op) + if (op) { lru_gen_rotate_memcg(lruvec, op); + op = 0; + } mem_cgroup_put(memcg); @@ -5456,7 +5459,7 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) memcg = lruvec_memcg(lruvec); if (!mem_cgroup_tryget(memcg)) { - op = 0; + lru_gen_release_memcg(memcg); memcg = NULL; continue; }