From patchwork Thu Jan 11 18:33:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13517705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EB56C47077 for ; Thu, 11 Jan 2024 18:33:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABDDA6B009A; Thu, 11 Jan 2024 13:33:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A46A26B009E; Thu, 11 Jan 2024 13:33:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E6E76B00A0; Thu, 11 Jan 2024 13:33:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 792106B009A for ; Thu, 11 Jan 2024 13:33:39 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4D4AFA0B47 for ; Thu, 11 Jan 2024 18:33:39 +0000 (UTC) X-FDA: 81667878558.24.9384E35 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf19.hostedemail.com (Postfix) with ESMTP id A5E0E1A0028 for ; Thu, 11 Jan 2024 18:33:37 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e2kqwQ0+; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704998017; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=45PrX3tP7VGxtv3hFQc544iEjjZ3CI9ChuG4EQT6Q8E=; b=nDpR1DomIB+not1cBjJ8UzvkcSEcUno0hN9I2bVtgY1Eh4PxcZ52yauxPunZPxS8ONQXFT QES//5owirZGX4qC8auBg5J5GhTe/19sGRJsWKFtBvhdbC8K37eTMrlzXPO0DTYJrPB2tH oOwWpRLJeqcfXxdGI/yyDdh055MW4k0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704998017; a=rsa-sha256; cv=none; b=ZV/9gMLqWCp75DQNIWBzlKyC/uy4Ey2oWr+lompeCldTFZes0ddkxZrZBYxdm9kqDRjNPQ GEHglzQc7h9PZMhveuuNZtQL43gKN7+vam3pRQWP4cYhzZMbqnbpAO+tBrUc82eAcIrpFI +k0qnc/ZpdaL2UNnPuiQEb7FoHTbxdE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e2kqwQ0+; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d3eb299e2eso35978485ad.2 for ; Thu, 11 Jan 2024 10:33:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704998015; x=1705602815; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=45PrX3tP7VGxtv3hFQc544iEjjZ3CI9ChuG4EQT6Q8E=; b=e2kqwQ0+PInjSJHE55X8ebmb/+wvyIpn3VOoYSGF4cR1Pb/dc3nJBB6xxzkoqKtm2w OxaIsSrpUoWf/nQ6vPams7OQumwnIoyP8vZr4SsBVameVq4TuDO6ZxYQRgbG8iOd2LrX 4+j5mjrWEoQle3LegBHO7BiqW/iWQ6Mn3HE6TLTnQmZsSjQJuK9Psnsj1HJMRrEvUNHK 5babGSuMxGGZugcI+8RVg9QcAjQi8vQTqMcIdl46LQKQo4V/GlytHQBwGmC8Su0rJA8b 8uEsYCPXII9p9rObyUq2iMNQHc32UIoTL2d1ZzPrri4pYHeSu777Ev6yavnrn+BtyXYK RKhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704998015; x=1705602815; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=45PrX3tP7VGxtv3hFQc544iEjjZ3CI9ChuG4EQT6Q8E=; b=faiV0IOh3yxE6iB1mQH1ErjWQREpeLtSKOjWNtD+TIjQf2KGLNNbrBZiV8hsCT26Br CrZinu+6NIfHJspCxmZdFtyS1MWSj4OnVIoRcSGBDh8xo29O8n+liCnNC2U0FZvqYUwh 1431ONHzj4kD+pj4ld+IG9aWduGMBcQVuzl0wSA5JMQNV1quQA6b+I/R536qr0Cfu2a2 tCHLA6d66stO0EY3JGixou1Uo4XeZimLsuftXBIBvM6gYk+3ZolMqopu5SNIYL8nG8tR Ik5Z1BLhrVOKyeEEPQfMRqNsa5Dhio4mlB4Q3Zbvct4TRbPwq/+KelOO9b+4/jjFmCwY 1j3A== X-Gm-Message-State: AOJu0YzghtP7ELO7J54fSsNk6zRNdgsIyzKkLi8kUXK0Dtu+hRVldmyw xRl6u1wV1UZqCrkJ5kVFxR8ut6brlpKo9nVN X-Google-Smtp-Source: AGHT+IH6jPbpjavXfb2XhjF9Qj7SJga+lz1w/NpKn2d06Fh3VXdOL8AoHZAdLauknrtPRNDLMJd9YA== X-Received: by 2002:a17:902:d346:b0:1d4:60b1:27af with SMTP id l6-20020a170902d34600b001d460b127afmr145599plk.97.1704998015184; Thu, 11 Jan 2024 10:33:35 -0800 (PST) Received: from KASONG-MB2.tencent.com ([1.203.117.98]) by smtp.gmail.com with ESMTPSA id mf3-20020a170902fc8300b001d08e080042sm1483267plb.43.2024.01.11.10.33.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Jan 2024 10:33:34 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , Chris Li , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 0/3] mm, lru_gen: batch update pages when aging Date: Fri, 12 Jan 2024 02:33:18 +0800 Message-ID: <20240111183321.19984-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: A5E0E1A0028 X-Rspam-User: X-Stat-Signature: eqiemuwhyhzhgigyp8s6iw1ofqwpsjpo X-Rspamd-Server: rspam03 X-HE-Tag: 1704998017-474433 X-HE-Meta: U2FsdGVkX1/Z9BN32aujKQKFNiJ5xYKLTZhPm0ZrSe1SFbhsMXUYtbvyykhLctjIamvGr470y7g7NfQLsuJ5WNuVR1X/cEfCpaBElDLGmark2jW5mwnRZyb18sHVGaInwvYiq4iWzAZJgxWif5C/auQQ7Qegx/5NPtcvZQDsOlqGO/mbg4FxTOb5AAZRFaydIfEVoVsLeQNJwrBnFg3Q+SvrVofkenu1M+fZQo4FUAowN9xMJFnfxlsPFez5IfT3hO4cZchTrfsXIYIjdnwPzYziCANLrBd4REuEopOHCdafZ4AE9+pxf8PHly3I5HvYG1nf9JfenurbhMJgxgBFLlO7Zjgm0NLqPQdXkX96aA0Sdaj0jDoXU3qRjj0oYwZBTo/qXFMu8zbXTsEWjLrgJataTDDSdkDbh+zrIMqmWapll9wL/qC8hLWdx/ZTcIooWBJTBLEYymu0yFS3t6afJNGjFxvz0eePvJseHn4iwzYW8pwfNp9Cg2svPgaIYEGpa7MY0KxxVCfsnLCFCVrWLDh5kgSrNlYbZ7vGJOqxM/DXUUjAkwd9EgUDeODcz3zVnSUhoKGesTjZ9bvDjyNsF3F7xHTHLwJbcExkjBR9E5RXguhQAsALae7rV/BNVNmGLpozeZcSj74KnqeC0225apmjb6oTuEaY2q3Lq68bYWs9iGU1/pAfZSd8151XrDXRJ/9lJMtrzdnQwKp+Jgmceg1iScdis2wjuhPkmB/CxF8Xp6wqSB7svvdpqQXK0JUNsSkx2QBdDKREdR5NLqS6DOfhlvrA9w5604jOKzmDBmERUaWK5w01j43FN2aUiBRQQQwaAvnbyKMokRjpvaMRRey3oo88R58i218tnZjJ0p69fmdk9rHN5iaWNJcQ4PLyIUZjCUP3/YnskpvRSNrodXzm0+szGoGJ/8mBQYFVZc6aFqwIvzMA5PB/404wLIg2x3hXwyS3pS9E50OGbV9 dlg9uByt QbNN2zK8eP45AVFetEPQ4FpaZcDeZWOn5lBVebYM7vLtP8i6goVBNDLpX7Ivc5+YYgz2Kjp6x/SkCB9J3MMVCnfnsQwaBbKxXmt+Rpuc5vvUeiKT1KoCn113v8Jp4csJg9fpqA0XfK6tatQYxbx/3ktLyXMWFqxeHyb+UYWKyt81LSMzoxpOBUkqdx2ERc32nR1ZzYGfdV9UTHhlF4A6N9zvFmjqf2s769Pn2Y2BRg+VT2FklgSUsXgyagbWRuV7L9KZx5ecsaIFwGCVNduBPUz9PvusAgrILFcbDArmA42ZdkVer1jJugQeZag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001787, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Hi, this is updated version of previous series: https://lore.kernel.org/linux-mm/20231222102255.56993-1-ryncsn@gmail.com/ Currently when MGLRU ages, it moves the pages one by one and updates mm counter page by page, which is correct but the overhead can be optimized by batching these operations. In pervious series I only test with memtier which didn't show a good enough improment. Acutally in-mem fio benifits the most from patch 3: Ramdisk fio test in a 4G memcg on a EPYC 7K62 with: fio -name=mglru --numjobs=16 --directory=/mnt --size=960m \ --buffered=1 --ioengine=io_uring --iodepth=128 \ --iodepth_batch_submit=32 --iodepth_batch_complete=32 \ --rw=randread --random_distribution=zipf:0.5 --norandommap \ --time_based --ramp_time=1m --runtime=5m --group_reporting Before this series: bw ( MiB/s): min= 7644, max= 9293, per=100.00%, avg=8777.77, stdev=16.59, samples=9568 iops : min=1956954, max=2379053, avg=2247108.51, stdev=4247.22, samples=9568 After this series (+7.5%): bw ( MiB/s): min= 8462, max= 9902, per=100.00%, avg=9444.77, stdev=16.43, samples=9568 iops : min=2166433, max=2535135, avg=2417858.23, stdev=4205.15, samples=9568 However it's highly related to the actual timing and use case. Besides, batch moving also has a good effect on LRU ordering. Currently when MGLRU ages, it walks the LRU backward, and the protected pages are moved to the tail of newer gen one by one, which reverses the order of pages in LRU. Moving them in batches can help keep their order, only in a small scope though due to the scan limit of MAX_LRU_BATCH pages. I noticed a higher performance gain if there are a lot of pages getting protected, but hard to reproduce, so instead I tested using a simpler benchmark, memtier, also for a more generic result. The main overhead here is not aging but the result is also looking good: Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-3: 44890.50 Ops/sec (+1.8%) Some more test result in commit messages. Kairui Song (3): mm, lru_gen: batch update counters on againg mm, lru_gen: move pages in bulk when aging mm, lru_gen: try to prefetch next page when canning LRU mm/vmscan.c | 140 ++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 16 deletions(-)