From patchwork Tue May 14 20:26:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13664417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 654EBC04FFE for ; Tue, 14 May 2024 20:26:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F21B26B02A0; Tue, 14 May 2024 16:26:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED2686B02A1; Tue, 14 May 2024 16:26:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D99806B02A2; Tue, 14 May 2024 16:26:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BCD036B02A0 for ; Tue, 14 May 2024 16:26:50 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6DA881205B3 for ; Tue, 14 May 2024 20:26:50 +0000 (UTC) X-FDA: 82118134980.30.5C5D98C Received: from mail-ot1-f44.google.com (mail-ot1-f44.google.com [209.85.210.44]) by imf16.hostedemail.com (Postfix) with ESMTP id 6AB5E180021 for ; Tue, 14 May 2024 20:26:48 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=RwKCnB78; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf16.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.44 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715718408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=0O5XCYkGYyzGIUkvLmvAAXCAYXLNncUqXZsXZSUBTyQ=; b=jZzAl8thtOvcixzjv4fbxQbtCcJQPM73TvI62WPmJX41e5/YajEa4iEPYcA8TqO89aPUPn hnQDIcaKggnvrIBRYPDw3e3hjgElgzzdZ0CTP0CWaNGzWZ8nX2f61u1XgK0F1TFg/TEhHh DU+2KBE1wPTzn0VVOCj+Ca/34z+4Dbs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=RwKCnB78; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf16.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.44 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715718408; a=rsa-sha256; cv=none; b=Y9cNSD+ffgE66mil8z3LWcGU2KADMHOGTYzUbCnwQ91O//E7sCSDiN6ZxLpcRp75E9HI5Y usR+ZqDCOzEgFl0RY7nlpeLcWiUYeeRL5kmUq3oJ4difP03GDzQV5ZSjiExRP8f7BKeaO9 5IXnZXX8vI5kxg8Nd365bXsaPPiFzTY= Received: by mail-ot1-f44.google.com with SMTP id 46e09a7af769-6f0e3b45706so3770085a34.0 for ; Tue, 14 May 2024 13:26:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1715718407; x=1716323207; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=0O5XCYkGYyzGIUkvLmvAAXCAYXLNncUqXZsXZSUBTyQ=; b=RwKCnB78KckdV/z6MF8tP816SNiDWQKkj0bWliwPEu6vOYeVJ46XfxX59m0wSGbjXo XqFxnNpu6iL91lCEDq2PfpGY7bnhc4nNJcM/DMD6X+2PQhHTAxz2HwEmybfy7Rzdsp6f AjgDla+AYrKRlsNKupG9MHUOueOG3LCtC+JCghHeZJNNTA767A11el6v6K+qNgU0VWgY NpaQsUo4y/FvkPpZ6Lhwsv+gp+x7rKsKMnhm3F4hhLo9zpYh3kq8+hmp8RFumAoEqC53 pzcehsTZq5hPa0ehboid09zy9n2xso669Lxdh6T9gLCVsSjQCL7dXpn8ZwRpVoM2DRwH qJAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715718407; x=1716323207; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0O5XCYkGYyzGIUkvLmvAAXCAYXLNncUqXZsXZSUBTyQ=; b=ZbT1wvbbiPhVs6YMLQVMZqt4Q5zxFSS1r2qVQxhR3PwCqfkBAe4QxrOCmVFw3XUFUm 5t45dyxslEEbkgZ16HmWt/Abkbg7jHG9sNpKkwXe6zsahVwag2C2WoKkqzX9u1ZHnssn yCjFkN+O79zajIlJTSJuu+IyEbJMOQmn5uiHm3JzPeWQ7Rt1VrQ/ke8dzGZCmRnitUdk B6TtKreplVeFtdXfew3h4g8KvATTR7cqxLZ6Ps1cYrmpFCJZUk1+eFCPR2YfyuIMVdme 8nOVVzYxb5ZYfqRX5OSRHKqpYcxrxXLRvK9AjX3x2WzaR/bfWYy2HJUCXZAK1hb0o5Ln Emxg== X-Forwarded-Encrypted: i=1; AJvYcCWnPe7fuO9T7O8qW9Hgp0+ySyBrfHpUGZcRWopnC3uvFg9eeSCjKTsnA8R+mMcGxVjXjIkjeESs7/UkOJ8rxNV/+1I= X-Gm-Message-State: AOJu0Yz2FWA6EBTDvf9qjHNFaNPm3dsUf0fMxLausGpjhPNfJelAoCfC LcFortUscZuhCcLZiZXdnZ7/GOMV8rIsazIzt/CVnQPPVXnU9pklPB80kfD1gLw= X-Google-Smtp-Source: AGHT+IGO+qAPmSeGOWuLJTewv+LxksrKrecgjOdKQj5wXGe1ZeHG3N4/yS0s7LykbAFJrCVucSG0eA== X-Received: by 2002:a9d:3e4d:0:b0:6ef:8634:f5f1 with SMTP id 46e09a7af769-6f0e92e9ec5mr16038071a34.31.1715718407030; Tue, 14 May 2024 13:26:47 -0700 (PDT) Received: from localhost ([2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id af79cd13be357-792bf31168csm598943285a.115.2024.05.14.13.26.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 13:26:46 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Michal Hocko , Roman Gushchin , Shakeel Butt , Rik van Riel , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH] mm: vmscan: restore incremental cgroup iteration Date: Tue, 14 May 2024 16:26:41 -0400 Message-ID: <20240514202641.2821494-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.45.0 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 6AB5E180021 X-Stat-Signature: 1xh8n1myjc83t53a11jrdbk18zfrzjn8 X-HE-Tag: 1715718408-759171 X-HE-Meta: U2FsdGVkX1/HvCDUNwcJV12AZYx44NOIsGylef1q/lguJSM4ANRWUyUjIlzySxMgo0gbZGy08MHVQ0AqEcy83HgFW3YMeq708UVgs38i9ukqKryIGlkV8rEVzlxnfKQeX1DboglnU+QPPeLhBs1A8zObWKnP2uY2exAav7jdcDvqDWAqpMNt+EY21s7wSjgzYjObVNs/fi2soY/JYRHLzP5mej2Y8dlBo8b09eZTxNxwJrdRmqp1aeEtQuWxgUsEdl8M4pmj95ocTwYFYvQI7pafudfbq308hqk3pNuT0jPhj8xrJFkKisc58SeYVglvjWhJ4CzaQSPS8KFen/VtCz56NdFNvwXhQgeTLBJwH/wt03HER1rkSFXvjcbNdiU1VUm3WiiVD51hA9g7Ihctk3Yt40H3AJwDDm540ACd96HF4FyyQ7VhhluMweOjYtxWcmhd5c67McUFNu377tnRtMLQJIk8Kex8oSahccfwEgma2dPW4JM5+VAluy3thEQgl7oFn7Phbdxvhq2ZWwjsRBXg1JBqf83MaykdZim0nuGPkFSI7VA+GCGx5n0zAP+TnbmKieMYxVJxM4x8HcYxiuiSJIa01qGLgTWzo/c8egpH9CU3T+vNTpscxRboS9+Eb5MVk49eI4AFZamYFDZq9Cw2GHwwquaEe7HwQmY6XW69X3acOtFnAlD0B6QKjfHNCcuF/eFmeU1l4a0w/hCx2VSRR422J6Cs8B2LVIVVfaJ2QtczIxWkqWDVBXQN/vHmxoEP6wndmbzk5Hb2xbWOHD+BEyhc9pVVgXNTW3ZA7r+CYBRtwurRoRCV9va+mgZcwSGofJSp+0PGL4YDqk6wohFRCMCRqLcayAHIC4mbCle5OCPzKbMuuE7rOvWOCmoT7sAu/BqsQiSutLdudiPliTewPLU0Pkm4XPTiAVmsYzEBbDqaSKrzUX70Ln8vDDwRlcV41BYXqpULNJ1ZyTf atbcsh1e bEDz48JMo64XT4tzMsofLh6d1GlKzfsNFpHRiwwXgewfsdfq+iDAiUUtLHXJWcFpk2cUA39Yfb7GSmhAnOCq5WyuBKCXizU2lWOEQwo5AZgaQlqRSXakC9mQImwECwWmtAisz3fOnWLC3GJec3HB9I3SCNWxpK5nwhzAZh4eebdknjvk3VOBezugi9zSectwd+gOkRBLchnSBt4vVeM7I4IucbcH/5p4hDOzIruKL+s+V2kK0O9/Di8sCF1c+yneNuO1hCM1arirvGYl+VKmM3en83NfwuxfUMHX89hyYjzE1hHBJXH4q/PMUbf/b+AOfodofgQCc36EqSfyNNPGst1eWl0Z2vmF8/2MBUymzKG9NVcL6smouMppA2mVkyhSQF3+ie6ayChD8dsPTbUTynv4VoNiyvcsPfKtPAZhZlmf+Z8Sdjb8GmO2DRb1CBu2c6t0pWCsevatDb+IG2ctcAzzSufZEy872nWBUsXdfPYWZ5jMBJMM4zqPX/swmel/Gs9hQ51OyvSKgpEX8W8s/sqOQOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, reclaim always walks the entire cgroup tree in order to ensure fairness between groups. While overreclaim is limited in shrink_lruvec(), many of our systems have a sizable number of active groups, and an even bigger number of idle cgroups with cache left behind by previous jobs; the mere act of walking all these cgroups can impose significant latency on direct reclaimers. In the past, we've used a save-and-restore iterator that enabled incremental tree walks over multiple reclaim invocations. This ensured fairness, while keeping the work of individual reclaimers small. However, in edge cases with a lot of reclaim concurrency, individual reclaimers would sometimes not see enough of the cgroup tree to make forward progress and (prematurely) declare OOM. Consequently we switched to comprehensive walks in 1ba6fc9af35b ("mm: vmscan: do not share cgroup iteration between reclaimers"). To address the latency problem without bringing back the premature OOM issue, reinstate the shared iteration, but with a restart condition to do the full walk in the OOM case - similar to what we do for memory.low enforcement and active page protection. In the worst case, we do one more full tree walk before declaring OOM. But the vast majority of direct reclaim scans can then finish much quicker, while fairness across the tree is maintained: - Before this patch, we observed that direct reclaim always takes more than 100us and most direct reclaim time is spent in reclaim cycles lasting between 1ms and 1 second. Almost 40% of direct reclaim time was spent on reclaim cycles exceeding 100ms. - With this patch, almost all page reclaim cycles last less than 10ms, and a good amount of direct page reclaim finishes in under 100us. No page reclaim cycles lasting over 100ms were observed anymore. The shared iterator state is maintaned inside the target cgroup, so fair and incremental walks are performed during both global reclaim and cgroup limit reclaim of complex subtrees. Reported-by: Rik van Riel Signed-off-by: Johannes Weiner Signed-off-by: Rik van Riel Reviewed-by: Shakeel Butt Reviewed-by: Roman Gushchin --- mm/vmscan.c | 42 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6981a71c8ef0..fc22704fbbe1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -133,6 +133,9 @@ struct scan_control { unsigned int memcg_low_reclaim:1; unsigned int memcg_low_skipped:1; + /* Shared cgroup tree walk failed, rescan the whole tree */ + unsigned int memcg_full_walk:1; + unsigned int hibernation_mode:1; /* One of the zones is ready for compaction */ @@ -5862,9 +5865,25 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) { struct mem_cgroup *target_memcg = sc->target_mem_cgroup; + struct mem_cgroup_reclaim_cookie reclaim = { + .pgdat = pgdat, + }; + struct mem_cgroup_reclaim_cookie *partial = &reclaim; struct mem_cgroup *memcg; - memcg = mem_cgroup_iter(target_memcg, NULL, NULL); + /* + * In most cases, direct reclaimers can do partial walks + * through the cgroup tree, using an iterator state that + * persists across invocations. This strikes a balance between + * fairness and allocation latency. + * + * For kswapd, reliable forward progress is more important + * than a quick return to idle. Always do full walks. + */ + if (current_is_kswapd() || sc->memcg_full_walk) + partial = NULL; + + memcg = mem_cgroup_iter(target_memcg, NULL, partial); do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); unsigned long reclaimed; @@ -5914,7 +5933,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) sc->nr_scanned - scanned, sc->nr_reclaimed - reclaimed); - } while ((memcg = mem_cgroup_iter(target_memcg, memcg, NULL))); + /* If partial walks are allowed, bail once goal is reached */ + if (partial && sc->nr_reclaimed >= sc->nr_to_reclaim) { + mem_cgroup_iter_break(target_memcg, memcg); + break; + } + } while ((memcg = mem_cgroup_iter(target_memcg, memcg, partial))); } static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) @@ -6287,6 +6311,20 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist, if (sc->compaction_ready) return 1; + /* + * In most cases, direct reclaimers can do partial walks + * through the cgroup tree to meet the reclaim goal while + * keeping latency low. Since the iterator state is shared + * among all direct reclaim invocations (to retain fairness + * among cgroups), though, high concurrency can result in + * individual threads not seeing enough cgroups to make + * meaningful forward progress. Avoid false OOMs in this case. + */ + if (!sc->memcg_full_walk) { + sc->memcg_full_walk = 1; + goto retry; + } + /* * We make inactive:active ratio decisions based on the node's * composition of memory, but a restrictive reclaim_idx or a