From patchwork Tue Feb 6 17:52:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "T.J. Mercier" X-Patchwork-Id: 13547683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DF48C48297 for ; Tue, 6 Feb 2024 17:52:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFA3B6B007D; Tue, 6 Feb 2024 12:52:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA9DC6B007E; Tue, 6 Feb 2024 12:52:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C98E26B0080; Tue, 6 Feb 2024 12:52:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BA6056B007D for ; Tue, 6 Feb 2024 12:52:57 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6FCA8A019B for ; Tue, 6 Feb 2024 17:52:57 +0000 (UTC) X-FDA: 81762124794.14.E6BE992 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf24.hostedemail.com (Postfix) with ESMTP id B377C18000D for ; Tue, 6 Feb 2024 17:52:55 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jZp+TfEz; spf=pass (imf24.hostedemail.com: domain of 39nHCZQkKCBkI8B3G173G5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--tjmercier.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=39nHCZQkKCBkI8B3G173G5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--tjmercier.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707241975; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=NnkI5UQ3se7RbAiVyXACobU8giPRA6UB/GtRqT8enc4=; b=lMmAXGbuV9gGtUtFlhNHx9hjrb7Ma5NtIpzmVWWY2RuEfm9ppMjhW0JinJXx9PBLeU7PBf DWJohe6WM6aZ93LZsJLFizMQDc0oA9w1kQeglaojanRg8TKIcNZxfiv4mYDBe+hJ+vH6M3 pjWYfmBm1AqKf6XjImD6yRi7hZ0EfFE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707241975; a=rsa-sha256; cv=none; b=p5gt/zQ/ApEjECWgYKtjO96SzuCghygAHdTPSUgnnJxoXMmmOU2LRltVBxQTQ3dw1muWlQ OYVz/ooRqmFNZmbeOWN83ONdkL6ivv+uyet+MA52DM08vSByHS86ETOdlwe01BgqA2X6dg SQd0H9yMoCC8WmytHFwTqT5pID6kNz8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jZp+TfEz; spf=pass (imf24.hostedemail.com: domain of 39nHCZQkKCBkI8B3G173G5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--tjmercier.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=39nHCZQkKCBkI8B3G173G5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--tjmercier.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-6e05f5ca521so594722b3a.2 for ; Tue, 06 Feb 2024 09:52:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707241974; x=1707846774; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id :mime-version:date:from:to:cc:subject:date:message-id:reply-to; bh=NnkI5UQ3se7RbAiVyXACobU8giPRA6UB/GtRqT8enc4=; b=jZp+TfEzlTZ8mno0fk8KY+KV3dR4RWMAx6jNH6r2QLP0WjrT/Yye+nqqGebE5nhdGE qtuoMLaCOW0h669q+KNGY5qWB2OqjMgWOjVgZGtA8/3Dz3CNc+KH1G+9M9kpznYPnOiF EQGKhs88mggPiVSjGQMwi7b91wiOOndWon0Q2LeCPncrB4l9iKJPFgmBB9Qn8ji/yQ1A TO+Bu01Y5JlT4ZGFnFbQW9kiWt8i41Z6ijENucoR+lJHkE7KDcYyhLB/rtZyjPtxIt7o YOfl/XDqlz+V1gA8pDRQ7/fjAjC3BIOIhH6fmc/Kc98ZYWjOvpbyxCe1wVc02IBp0xQs bRSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707241974; x=1707846774; h=content-transfer-encoding:cc:to:from:subject:message-id :mime-version:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=NnkI5UQ3se7RbAiVyXACobU8giPRA6UB/GtRqT8enc4=; b=bd0ny6IGuy4SF+9IFA81FAKS6Zgit68U80a0lUbYq4ojVZHt46XhUFhm08TSPJVv5W jFH4gVVBv+Dt2wd4dLMTlWxod1WL7gn+5Se+flZcn+LeawI/oVAOTWUNA1m1JZ+3sqfp tKFxPLzyKs1ErlN6fgN29WIIv7m36BXpolFCbayGWu2MOA9MFePJaCWczX1nMKhCIBHG msX9oGkjpOQrBReT5/9PJ/LvPpftqvJcFYphvKAp8vLCbo56KSZUWaPQrD2mOgnBbMCi T/2pKUFKrMpxRFlfYI9SbZAaH/Xi0CYR6eg9zmLOpCpt8TRXuY+Iyb1xofPBUvCqiHiE hGew== X-Gm-Message-State: AOJu0YxTC2tKnLQOLS2wy8urc4VQfy2h3Fg2ON09ljtMWdOFlI+SG1XJ 8vxqR+plzZevkg2AVOPAN4imUW0Yb4dDmh5yAwZzlBvKWQ64xjycnZ+EqBYegbPetu0YhZ9X+x8 WU2HEtmSFo/q1jQ== X-Google-Smtp-Source: AGHT+IFfvyuPQfLG3R8pxlTzGNbrQ4aI0ovDtf3rTDW0vN+S5SToqxRa433lLgtzsnmPoI64wOwc7OjmECXxhZo= X-Received: from tj-virt.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5683]) (user=tjmercier job=sendgmr) by 2002:a05:6a00:17aa:b0:6df:ef44:90bb with SMTP id s42-20020a056a0017aa00b006dfef4490bbmr11498pfg.5.1707241974509; Tue, 06 Feb 2024 09:52:54 -0800 (PST) Date: Tue, 6 Feb 2024 17:52:50 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.594.gd9cf4e227d-goog Message-ID: <20240206175251.3364296-1-tjmercier@google.com> Subject: [PATCH v4] mm: memcg: Use larger batches for proactive reclaim From: "T.J. Mercier" To: tjmercier@google.com, Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Efly Young Cc: android-mm@google.com, yuzhao@google.com, mkoutny@suse.com, Yosry Ahmed , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Stat-Signature: mi5ktnsmmabj4jwstp53ukjzu9dotxpp X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B377C18000D X-Rspam-User: X-HE-Tag: 1707241975-233546 X-HE-Meta: U2FsdGVkX1/3S/+J9fPSZsjiNy7eB/uFKGSBBvPgzA0uyJ3K9Xdu4eS/y4gG33tmcn5fGxzZSkLHkJLjzcqKsd96Z4+SgoAuzpPvQmgNks1MpqyMdiHpUIHa7U3IZJyvSZyzZPPccEfXJkXWP0Fwzp5ONOuzm37rElBVNivk3nhqlm8sNJTQU7yELIGItewIXQ6I1kRjjNcwRGxxPW+YQWgPu+BENXZc28VhAxj+DCgTsYLwbvsK/ryeAEB42C6eil7ChZwQqhI/8jJ+foaqi/Ce5TsCXb4IipyawMxCqAdRpNJx0dUv6Uxmv4BC6yVQmMWgrsLSx08PWPUd7V6+71280uEYb9ltb9Nc0Q9Mehg6tTYkZEyk6jG3IYr20eouYrGWL9y2v6gDWAVXSlJSJrFPN6muvRiZtOnmWVygBfk1b3wZuiHRp1nezUY2Ehsad/tNPj1W8vuYdxAewSREnpKmIslBtCSEaOxbLzetjQiZ4vX9V3UXoMZ9uZc40V0dSldoGKSq5fH5F/Tf4Yp484V6NCo7gGbvv/5jQa9CfP1hyB5/vCCzEDxCEMtLU7x4F0DkZ7o8Pc33uAr2T1zcMugbdNimhNCLQxWZI08mXmr20FFJjzFfih5/GBrg9hhj2p4x4WQfUp5BG87VfbM/O18Mm9c5+yQJlKKuNn/ViQee5CBCDn2S9p8XBO3Jbq6OKLf/HlVDNAX9q65UdT/dX3HpvOvkK54tuxH0YOtwiacjoBJY3f0rJ8q0XOcgIkCjibrYLbvtEGKyQAol0TXsmgZpIMhEecCODcXStecJbJLsI1IRjOPc8p8jvMzGaYksVK4HZMk+RVySloU6HOQmBTGrsoCs4jCqqEwA7CKgBSk+mbT5kpBopwaeHDCO9kx80fmzPf+lfdz0Hl4vuuHg7q/DdypLD3ngGYvpm2q60nnI2ldwYIwExelxyDYIDSzRIfOQRf+tNHdB/2j2fAM ApzAslOp 8ZHJBeezO4oe4Q7RYm6cYl35F1YLoFKCf6yj0iWjHMq5ZbGuiSV6dJwfgmZHSC52jRnoHefbEnH1j8CBKi8uyPwnJNF/j70Y6MKU/nZoqX4Rosm2y1HxDVfaY+AotCXjsj4zZbGgdmdcwJ4DsLJAcK7gNn837KbVz374K56cbCWVs8DS3TB2HIZWhrmoYNfLbDOsB+fs4nOmhvJFM15yuDJQjLXAwxiuyY0G894NYVrr5NeCHotbo9IhHWtChFPjZd6NrFhRSgyAoKiBjskcG/XPiYIDsfz6jyFSMkHHLpv7WRHbePQPP7MNwKHWsMht99p8j3ucKDOd4GLT84Gwcd28chNvyjSs5K9fiKaPkc7LfqH91bI/ss3RzP7nR0c7buFqrnUcDLwxu/j//ATvW0LxA0UTeTB/aJGpN+aISQFj/RmkPPpSghBDm2CsdGgmjBpb7PzywFRGcGvCxqMAK1vQmVzAmx1qOuGvlBXq3ASxSOZTM1AuSVJF+8qM1WKcKExkEJ5fWK0ScVEXC8uuYJ5/lAdK2BKuBCrJdLwDDeGi+A8NNmY0XcQds8GtblQTBlNjf2/Luw2yLMOPfSYBlWR+tmg7FVeKhHAfiBu5gL3xChLV9wrlGc9p19Sj4rmVmH57Dk7lLpUg/+3MirgW4EvIchYR5VnAzqRujvpvQfffJIDmbWZ7GloUarVKwPUW4RH0MhiACrBYMFtm+Sj4umHjw42K1e8guQ0y32Tbqju2GpO156S6aEq+YVspamPW4pJ8RdNSzC9w9ChE290/DBavRwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Before 388536ac291 ("mm:vmscan: fix inaccurate reclaim during proactive reclaim") we passed the number of pages for the reclaim request directly to try_to_free_mem_cgroup_pages, which could lead to significant overreclaim. After 0388536ac291 the number of pages was limited to a maximum 32 (SWAP_CLUSTER_MAX) to reduce the amount of overreclaim. However such a small batch size caused a regression in reclaim performance due to many more reclaim start/stop cycles inside memory_reclaim. The restart cost is amortized over more pages with larger batch sizes, and becomes a significant component of the runtime if the batch size is too small. Reclaim tries to balance nr_to_reclaim fidelity with fairness across nodes and cgroups over which the pages are spread. As such, the bigger the request, the bigger the absolute overreclaim error. Historic in-kernel users of reclaim have used fixed, small sized requests to approach an appropriate reclaim rate over time. When we reclaim a user request of arbitrary size, use decaying batch sizes to manage error while maintaining reasonable throughput. MGLRU enabled - memcg LRU used root - full reclaim pages/sec time (sec) pre-0388536ac291 : 68047 10.46 post-0388536ac291 : 13742 inf (reclaim-reclaimed)/4 : 67352 10.51 MGLRU enabled - memcg LRU not used /uid_0 - 1G reclaim pages/sec time (sec) overreclaim (MiB) pre-0388536ac291 : 258822 1.12 107.8 post-0388536ac291 : 105174 2.49 3.5 (reclaim-reclaimed)/4 : 233396 1.12 -7.4 MGLRU enabled - memcg LRU not used /uid_0 - full reclaim pages/sec time (sec) pre-0388536ac291 : 72334 7.09 post-0388536ac291 : 38105 14.45 (reclaim-reclaimed)/4 : 72914 6.96 Fixes: 0388536ac291 ("mm:vmscan: fix inaccurate reclaim during proactive reclaim") Signed-off-by: T.J. Mercier Reviewed-by: Yosry Ahmed Acked-by: Johannes Weiner Acked-by: Shakeel Butt Reviewed-by: Michal Koutný Acked-by: Michal Hocko --- v4: Add additional info to commit message and move definition of batch_size per Michal Hocko. No functional changes. v3: Formatting fixes per Yosry Ahmed and Johannes Weiner. No functional changes. v2: Simplify the request size calculation per Johannes Weiner and Michal Koutný mm/memcontrol.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 46d8d02114cf..02b054a316d3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6963,6 +6963,8 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, reclaim_options = MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE; while (nr_reclaimed < nr_to_reclaim) { + /* Will converge on zero, but reclaim enforces a minimum */ + unsigned long batch_size = (nr_to_reclaim - nr_reclaimed) / 4; unsigned long reclaimed; if (signal_pending(current)) @@ -6977,8 +6979,7 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, lru_add_drain_all(); reclaimed = try_to_free_mem_cgroup_pages(memcg, - min(nr_to_reclaim - nr_reclaimed, SWAP_CLUSTER_MAX), - GFP_KERNEL, reclaim_options); + batch_size, GFP_KERNEL, reclaim_options); if (!reclaimed && !nr_retries--) return -EAGAIN;