From patchwork Thu Jan 11 18:33:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13517706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07273C47077 for ; Thu, 11 Jan 2024 18:33:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8EA8D6B009E; Thu, 11 Jan 2024 13:33:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8993F6B00A0; Thu, 11 Jan 2024 13:33:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EC6C6B00A1; Thu, 11 Jan 2024 13:33:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 59B3D6B009E for ; Thu, 11 Jan 2024 13:33:42 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2C2E412065A for ; Thu, 11 Jan 2024 18:33:42 +0000 (UTC) X-FDA: 81667878684.22.0409BBD Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf03.hostedemail.com (Postfix) with ESMTP id 6B00020007 for ; Thu, 11 Jan 2024 18:33:40 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HdasRBFn; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704998020; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2NX9KnQsza9Uz6QRxAoPzwzDEngTlQbn6WQFeW15RSs=; b=sJmy/dDiUJ4NbaN42B8kf1n8cLFrvDmI69BUbOeSMKzcZwfJXDUC3OEL3AtORO4L7BaaW+ fKaqhqlugANEGl376hQ352ARmw2h7b6iPR8pvO8PMEW5BjnVUW5kZrmiR7/wxbMUiVwWbu cjIxO4rWKTy07y8suisiMbj37Ibfri0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704998020; a=rsa-sha256; cv=none; b=clAP3vHP/JPGG8TX5zY8jXiqqfuiOPwVGwu340Ees4UxTg5iIoHjb4qaF0cgCTn13+/CUD o771qZZvG3y6hrhgBncOBtrFyiAvRfz3utQ0BHOWUVy3HPrpIfL16m+6ODfERXdwwC2RkG VhmKPT+O+4DTfwT5HPeH96jmSnZPuM0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HdasRBFn; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d4414ec9c7so29538315ad.0 for ; Thu, 11 Jan 2024 10:33:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704998018; x=1705602818; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=2NX9KnQsza9Uz6QRxAoPzwzDEngTlQbn6WQFeW15RSs=; b=HdasRBFnuj3u5rB3VmiMInohvkt9BlC5oeJ/Y+cT94UYn+VfdgLNmQdkJYelGCasmG TqF6EUlR5IV7LOMgcEBfHw5/2hXuy5H8rfaCGSIjYlidg0BERtqqzuNMUzxK/b0SGrnj MbyqUJFNf2rWdG4E2WHUzfl7tnEjbtvCdt0vTPLNl0mRyxqYWdxZ5n1J5YKntZQz7bUq slZue8rxJUoqC3LYuhiAVZTd/8Xv++Yq0VgbozAn2Te5ZV2xO9dzbiugALMCtiRkmD8f B2Wdqx0CSPy7nQwxw6ABd2hWSZvjhw5Vn1QozVCQfbjofT0SeFyjrrDPM4q9JaDz/CFj ZAbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704998018; x=1705602818; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=2NX9KnQsza9Uz6QRxAoPzwzDEngTlQbn6WQFeW15RSs=; b=MB/wc6Mw8ZWgddag2jzW83q+fkaNc/NgwhH19+gbDTrdt2aOtV/tvFIbeB7Ne8Xkdu l1gfB/6lzeMdprstdSb64YT3Yc6xa/JnpQQ2kY4GisOnLxJGZ6LilJ+F5YWd8rg/3aAg cwPFFtttPeqhnt6amHbpHNTDZ8800pOQjtWoXUJltpPj/e8t+p+izTNVrI52QpfyGw66 6N8qpF1lY1RZdkvqQxcDaFCCjv31aWOsvB82+X59R651ZFocB9dPo1AflNkm5Ij0C3f/ fblxgAGHWmpYIA7OA9R/WOwxSFRntG7hVcMwXNZDFOzJwwmz4cVF2RocOerORHFtX9mn Adiw== X-Gm-Message-State: AOJu0YwbmTw4SoOFdVCeVzWYerbGcHlNWXIKlqx7E2Lp09+OsmyEsjUI NWXunowxCCBYxuDUIQfh+O2Q+xdSoCRmR/Xh X-Google-Smtp-Source: AGHT+IHnDtOq2y94Dh8PFjaWgtWLEz5nVEvXR0aC1RzvZIlhpBXuXDqh89s31F4som1CtytG5C2rzA== X-Received: by 2002:a17:902:6b82:b0:1d4:7685:90df with SMTP id p2-20020a1709026b8200b001d4768590dfmr161590plk.31.1704998018116; Thu, 11 Jan 2024 10:33:38 -0800 (PST) Received: from KASONG-MB2.tencent.com ([1.203.117.98]) by smtp.gmail.com with ESMTPSA id mf3-20020a170902fc8300b001d08e080042sm1483267plb.43.2024.01.11.10.33.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Jan 2024 10:33:37 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , Chris Li , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 1/3] mm, lru_gen: batch update counters on againg Date: Fri, 12 Jan 2024 02:33:19 +0800 Message-ID: <20240111183321.19984-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240111183321.19984-1-ryncsn@gmail.com> References: <20240111183321.19984-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: bthsuddz3a8mbq4qnc9uwd1aketk1mwp X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6B00020007 X-Rspam-User: X-HE-Tag: 1704998020-167323 X-HE-Meta: U2FsdGVkX1+nntuu3GoMnaRigN38/nk/LXu2JOEWkEYQbZXVYFaU3wfXHMu0Oz1R91jSHyvBZQMVDCZGYiV6ogNcXCMkDIOiOZDT9GHYSQJ6BqH+Xof16Ck2pgzJuYVTV8vbLVNePeZqKkORnGdj9xXXWn3xERYTvoJnj1vucZ3E7SNjW5oY5MdDZKPlGdI5r6tsI6RDTPEDECEsFzKevjnSYxTxsUX6/MSzFarm1j1XCWmjf6T32hqk0wDdkJ7M1LfXjpzJXjaUoq2BhRhF9ZnXr2vNMQmW4oybmQOowxrFv57Mw6hjClb7VCg6Fjw6iGjoPC0Z4ZVc+xbL5Ma1wcOpQr0ed1kDLUpVKwUL29/P1Ckx2rQpPiykaylPmDLZYxOmxhX6KQOMxFRJER1fHzZac+hqphGG9X/M8h2wMaGt7Zc7Z02pMabiVqpgiBzPR/pF69mu3VCvLEWK3ir+XDlx8kujuKzHqRZ4ZPVo2j5cwsEuNRI/INevfYQpj0xuFDLq+7TyIGNHk+R8NUFf3KfFgPuF24KkpLAS+aN5QXLYI0LFgI+BbjHYwMBjS6lpoMru/4CJ23CPoBli7jiOzN/+pxcYsk3LMBG98Z7ckJ3GU2braukoYgxXvZ7I6mJ2PsjQDGWvEVeGgg013PcpxsI92dKmOMTRJoB3QuKw5rDtD9qPInyLdutzlKA/83fk4FnRcwFlPsOrOOeFCPEkpIhBvtKHdAX1mmlaqJ+IKxrsh8jLkbtZIzmh7AY7OGtZqYe00CZ6lfExHInaRFffiWDtol2SJ4NuqBvcnuXhXTFmPnqyTs4NiUNCUt/eKQV4cCLGaR+FKYaZUEpOqr2CZiPFDa+mGn2aGoWu1ZaJ94cEcE+kQiS02g4IEahTAeSNmzUa8o1cYqDqUgA3OF+SrBdeex8GOwXctA5W4kpRenrHAUI+Yyl6tzqBIzq2XEFbYWMYHYOwSA5W4VSK416 5W8tGVeT w7vCOGqzncOv30Qx/EwNDGcnDAGmat3MJSKI2ytdlIT/pgi4FCsYVHt0LNZhkkOUhOdxSTlu+vdaIM8tUGYMSHMcQmu4TcMusOA5Hvb3OfKzGkUK1Kvf1vVFgEzhjnsZqJRug2pbf4FXzQptpcUnLDCFcC7DebYRts87X6UiuaYqXXKFVfUupRdhrWNWHNNXIZDdLTwBNlsdbGxJKw97kn1YkYYO0hvK7MtOB/N9hJCi/xbiaUCRNGtU9QVA5LW79jh7qSsl3yaVFEjlBh45ERTX6ygFoXP7OR2g1UVEspSDr+Wy/y/lKpNP7Iw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.011855, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When lru_gen is aging, it will update mm counters page by page, which causes a higher overhead if age happens frequently or there are a lot of pages in one generation getting moved. Optimize this by doing the counter update in batch. Although most __mod_*_state has its own caches the overhead is still observable. Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After: 44687.08 Ops/sec (+1.5%) Signed-off-by: Kairui Song --- mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 4f9c854ce6cc..185d53607c7e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3113,9 +3113,47 @@ static int folio_update_gen(struct folio *folio, int gen) return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; } +/* + * Update LRU gen in batch for each lru_gen LRU list. The batch is limited to + * each gen / type / zone level LRU. Batch is applied after finished or aborted + * scanning one LRU list. + */ +struct gen_update_batch { + int delta[MAX_NR_GENS]; +}; + +static void lru_gen_update_batch(struct lruvec *lruvec, int type, int zone, + struct gen_update_batch *batch) +{ + int gen; + int promoted = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + enum lru_list lru = type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + + for (gen = 0; gen < MAX_NR_GENS; gen++) { + int delta = batch->delta[gen]; + + if (!delta) + continue; + + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + promoted += delta; + } + + if (promoted) { + __update_lru_size(lruvec, lru, zone, -promoted); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, promoted); + } +} + /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, + bool reclaiming, struct gen_update_batch *batch) { + int delta = folio_nr_pages(folio); int type = folio_is_file_lru(folio); struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); @@ -3138,7 +3176,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai new_flags |= BIT(PG_reclaim); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + batch->delta[old_gen] -= delta; + batch->delta[new_gen] += delta; return new_gen; } @@ -3672,6 +3711,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { int zone; int remaining = MAX_LRU_BATCH; + struct gen_update_batch batch = { }; struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); @@ -3690,12 +3730,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - new_gen = folio_inc_gen(lruvec, folio, false); + new_gen = folio_inc_gen(lruvec, folio, false, &batch); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - if (!--remaining) + if (!--remaining) { + lru_gen_update_batch(lruvec, type, zone, &batch); return false; + } } + lru_gen_update_batch(lruvec, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4215,7 +4258,7 @@ void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) ******************************************************************************/ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, - int tier_idx) + int tier_idx, struct gen_update_batch *batch) { bool success; int gen = folio_lru_gen(folio); @@ -4257,7 +4300,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c if (tier > tier_idx || refs == BIT(LRU_REFS_WIDTH)) { int hist = lru_hist_from_seq(lrugen->min_seq[type]); - gen = folio_inc_gen(lruvec, folio, false); + gen = folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], @@ -4267,7 +4310,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { - gen = folio_inc_gen(lruvec, folio, false); + gen = folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4275,7 +4318,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* waiting for writeback */ if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { - gen = folio_inc_gen(lruvec, folio, true); + gen = folio_inc_gen(lruvec, folio, true, batch); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4341,6 +4384,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, for (i = MAX_NR_ZONES; i > 0; i--) { LIST_HEAD(moved); int skipped_zone = 0; + struct gen_update_batch batch = { }; int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head = &lrugen->folios[gen][type][zone]; @@ -4355,7 +4399,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, scanned += delta; - if (sort_folio(lruvec, folio, sc, tier)) + if (sort_folio(lruvec, folio, sc, tier, &batch)) sorted += delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4375,6 +4419,8 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, skipped += skipped_zone; } + lru_gen_update_batch(lruvec, type, zone, &batch); + if (!remaining || isolated >= MIN_LRU_BATCH) break; } From patchwork Thu Jan 11 18:33:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13517707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B2ABC47258 for ; Thu, 11 Jan 2024 18:33:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 969546B00A2; Thu, 11 Jan 2024 13:33:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E8E76B00A3; Thu, 11 Jan 2024 13:33:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 739EE6B00A4; Thu, 11 Jan 2024 13:33:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5B67C6B00A2 for ; Thu, 11 Jan 2024 13:33:45 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 306101A0CAD for ; Thu, 11 Jan 2024 18:33:45 +0000 (UTC) X-FDA: 81667878810.21.C0BD868 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf23.hostedemail.com (Postfix) with ESMTP id 63D4B14001E for ; Thu, 11 Jan 2024 18:33:43 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TflZKVXF; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704998023; a=rsa-sha256; cv=none; b=O594zRTcjvYXiuE9Xtqfc3/ZaGE99EFY9QVyflfB7tUdMLQTy5NntuHidyplH3T+KU0bU+ IcWIKstg/Mx7PX7zEaBqq2Ohjwx2Id8H3gzXFM4TpdOLJv243q6EOpTJlpFQDjai1tmOt6 Bv0xYMs+FTnQrvPsnkv0W8idK8c87o8= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TflZKVXF; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704998023; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ooTkGE2rW3pIuHvnebpbhiU0Zel5DUvb4P5ZSh1D2Dw=; b=KNYIw5KpKlh7xZXk1bx0Rewq6wTf80j3SYtGfR4FM6jtPNtXY//F7xC6dawCj6qcIDBOcV p+1ZuciO4BRx1AzXsqiPYXhZwfilKUTgaw28PCEpQLEYUd+pm8fSQYwA+V1DuNo4qY0fN5 w4hs5XEtXVMnD/FXf1NT0/Zu9PqPB3c= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1d51ba18e1bso46461265ad.0 for ; Thu, 11 Jan 2024 10:33:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704998021; x=1705602821; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ooTkGE2rW3pIuHvnebpbhiU0Zel5DUvb4P5ZSh1D2Dw=; b=TflZKVXFmbNmvhFoaTvdQZhXHhco2DwnC3IDTO7Ogg0jfjco5/Ky2zgtol6wN92PVi nSw64IQswOVctX4QiD5ev9RPmhpMPFOL59AZCMN1/h2zSxFkWYrjHVvuxiKqSgxGsZjU bjdeI6a1/HJ3D49tfHTsSfQOkLyqdUN3EaPPYqbnwrFqyeVBC5+QGPZQYsHnN2GV/INH xAG703loVZSKEweffJdJpqDGqdsd75j6Az6AUHF6t/sGR1WZsV33AuiBnoq2UTYCGmWq Zgqr2hcCelbQWXgazG2S1UD0XTPQIlGsylwgQqala572QhfF06g2OwwcMnA161+HlRuE EAlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704998021; x=1705602821; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ooTkGE2rW3pIuHvnebpbhiU0Zel5DUvb4P5ZSh1D2Dw=; b=Nrv4nAYZSmUuQLPbJ5LZ/QRPbnLctJC2LeNix98kythgjBaugiqmFx1AketfNwCVBJ P2rBlArmtqb02rSnTDERgsp8MJ3ZzH9s7cGJQKxZvm6EiuhNZXuKahm3+OvMUKdX4bne ylJnhwEHs0GjfvJ0HE6xkzRCAw82LsxSJdpyjoKNX3VfIkHXXJfSIRm3M/Z3VGiVUjZC oEZZpajw8yJobSpYZoRGLvky6cj6uazHSSbJTAiaBEscNpPuk4EM95Bj2ptM1SOa+wxu D/8bUI3sPt85HbhHxKvAxxB4IxVoToKpytG9y0nDxNKPBki7oNQAdFXxYZtaSi6G+7gB feqQ== X-Gm-Message-State: AOJu0YxY+UAucmEx4FkbYF3Wx/dWB9ntkmaJVwPi14G+pkL+PYwqlYOU AJjwI8Mh2GT16/NDZNO1zDWFLR4AlM28gzT9 X-Google-Smtp-Source: AGHT+IEDHbwlcQ2ocO41xF0rCg4oiBOl0m3y0fRQIMYq2UBZBjcvr8e3NyI9T83Xh8KcFvM5suVFWw== X-Received: by 2002:a17:902:690c:b0:1d4:1b4e:ebf5 with SMTP id j12-20020a170902690c00b001d41b4eebf5mr171681plk.10.1704998020964; Thu, 11 Jan 2024 10:33:40 -0800 (PST) Received: from KASONG-MB2.tencent.com ([1.203.117.98]) by smtp.gmail.com with ESMTPSA id mf3-20020a170902fc8300b001d08e080042sm1483267plb.43.2024.01.11.10.33.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Jan 2024 10:33:40 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , Chris Li , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 2/3] mm, lru_gen: move pages in bulk when aging Date: Fri, 12 Jan 2024 02:33:20 +0800 Message-ID: <20240111183321.19984-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240111183321.19984-1-ryncsn@gmail.com> References: <20240111183321.19984-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 63D4B14001E X-Stat-Signature: xw54dqubyzs1agnkz73wbm39pe5aaqaa X-Rspam-User: X-HE-Tag: 1704998023-313043 X-HE-Meta: U2FsdGVkX19OOHwpZcOyeeFv2ObNlmvWCntM+q/R/MPIcU8oYUIw1W1+GU9x2EHuM1oknPLUyFmxrqc+OVMj9qhSxAEnfI/XekOXpNKTRWBVK/lSRcX1BUdYUQaGeo4ZmwHh63s6Pbjs5HE83XsCLvo14LkCIHzkhBoyPSy4Q1xZzXFm4yCxi6Dbep2Xl33xheNfXP5BRxL/NpP7g1rJmtkaoLCfezLQhfNImAqbclHA06TeY1Kom829sSgM7t0R3Jk2hg/nkFRSdczT3tHCPaEmCzQeKaGnJI7aGNh8XqR3nQHOXyryhiYOJMLSUB3835ol4/yOwYOj+9KUl1ZacRE1R2EXzk0lZpg2DNTtswB0Zp6FgChHzqj0oKxQFkQopSfoTAImU3iihoayaxqAWGYajG4G5Ui7wFQu4pPTaiYBP2U+UZTPE1ddjmkQLpzvFrwHi+b2PbBnnfVFXLHqf1PgZBl29zq+snEw/H2lq7czhMRU5pnEmNBlEDanFRPMgW28Xn70pml2Avf1/yRIbs3gAG/v5PVpsHf3FN88eRMc4AwZHfxRwaUvg4PbuwuiyufZtnnmKVhfS1HIANvgjKA3jm7fsZsjgJg27PEboBV6KRCwu6KUhk1tT83YQF6TRt1eVLDABNI0jnc8C+OIUOccUdrriQzDVU+AwL32UCB8pZUYrVYMbgxjscYeij0da6nz3xE+TVDVazv55EPf6kPfK+wwe8K02gqJIjGzYqztUMQA/V9kIYOAQfKler86nEQ7F12aMTMoV9yUe3fHBg4rVIe+9XOk8DIjIhMTxMA2hEIkIrhUZ9QAma3nj7w850o9rXm+IENxn1ATOXUC4Gz8pDMn6IQlyptFavNHv5rXIt5f3Sy6YuT24JfQZurUNT620A2OTDbaXpezgTtXrezF8PnWk1raBQcnd4lZuLFaIIMxfgt7opcXKtyq19ZI9Fsp56/2ttkzRm/tLF5 7dizrWNk yYPZdxZpYj6ms+m5OB76CDcNOl+ZYP2EsRJkIheGVzlg7B4xZ+FJ+57WomdYgXDUcRIF6h5NB1pqiM+2NlaiuckeaC4k+9Yw+9SlN7ZFSGfJI2B+rrFjb3/M2OARb2QEgp8IkE4TJkzXkgtJnk0K4LtntYYw9FObZie0TIuO7jjbAvHx4I6AnfzynD57Oe/Pfnz5sJnT80fQW0iOT+jl5V1Qm4O+PyPws9xajwlOv8baDGucZy1yt/SQbLZmJEvb3sjcajb4kKnZYbPyMF4OtZKknlqLqwuSXpyg4gM4mdMi+rJ2D8ztVP4F49w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001261, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Another overhead of aging is page moving. Actually, in most cases, pages are being moved to the same gen after folio_inc_gen is called, especially the protected pages. So it's better to move them in bulk. This also has a good effect on LRU order. Currently when MGLRU ages, it walks the LRU backwards, and the protected pages are moved to the tail of newer gen one by one, which actually reverses the order of pages in LRU. Moving them in batches can help keep their order, only in a small scope though, due to the scan limit of MAX_LRU_BATCH pages. After this commit, we can see a slight performance gain (with CONFIG_DEBUG_LIST=n): Test 1: of memcached in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-2: 44810.01 Ops/sec (+1.8%) Test 2: MySQL in 6G memcg with: echo 'set GLOBAL innodb_buffer_pool_size=16106127360;' | \ mysql -u USER -h localhost --password=PASS sysbench /usr/share/sysbench/oltp_read_only.lua \ --mysql-user=USER --mysql-password=PASS --mysql-db=sb\ --tables=48 --table-size=2000000 --threads=16 --time=1800\ --report-interval=5 run QPS of 6 test runs: Before: 134126.83 134352.13 134045.19 133985.12 134787.47 134554.43 After patch 1-2 (+0.4%): 134913.38 134695.35 134891.31 134662.66 135090.32 134901.14 Only about 10% CPU time is spent in kernel space for MySQL test so the improvement is very trivial. There could be a higher performance gain when pages are getting protected aggressively. Signed-off-by: Kairui Song --- mm/vmscan.c | 84 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 71 insertions(+), 13 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 185d53607c7e..57b6549946c3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3120,9 +3120,46 @@ static int folio_update_gen(struct folio *folio, int gen) */ struct gen_update_batch { int delta[MAX_NR_GENS]; + struct folio *head, *tail; }; -static void lru_gen_update_batch(struct lruvec *lruvec, int type, int zone, +static void inline lru_gen_inc_bulk_finish(struct lru_gen_folio *lrugen, + int bulk_gen, bool type, int zone, + struct gen_update_batch *batch) +{ + if (!batch->head) + return; + + list_bulk_move_tail(&lrugen->folios[bulk_gen][type][zone], + &batch->head->lru, + &batch->tail->lru); + + batch->head = NULL; +} + +/* + * When aging, protected pages will go to the tail of the same higher + * gen, so the can be moved in batches. Besides reduced overhead, this + * also avoids changing their LRU order in a small scope. + */ +static inline void lru_gen_try_inc_bulk(struct lru_gen_folio *lrugen, struct folio *folio, + int bulk_gen, int gen, bool type, int zone, + struct gen_update_batch *batch) +{ + /* + * If folio not moving to the bulk_gen, it's raced with promotion + * so it need to go to the head of another LRU. + */ + if (bulk_gen != gen) + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); + + if (!batch->head) + batch->tail = folio; + + batch->head = folio; +} + +static void lru_gen_update_batch(struct lruvec *lruvec, int bulk_gen, int type, int zone, struct gen_update_batch *batch) { int gen; @@ -3130,6 +3167,8 @@ static void lru_gen_update_batch(struct lruvec *lruvec, int type, int zone, struct lru_gen_folio *lrugen = &lruvec->lrugen; enum lru_list lru = type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + lru_gen_inc_bulk_finish(lrugen, bulk_gen, type, zone, batch); + for (gen = 0; gen < MAX_NR_GENS; gen++) { int delta = batch->delta[gen]; @@ -3714,6 +3753,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) struct gen_update_batch batch = { }; struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); + int bulk_gen = (old_gen + 1) % MAX_NR_GENS; if (type == LRU_GEN_ANON && !can_swap) goto done; @@ -3721,24 +3761,33 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) /* prevent cold/hot inversion if force_scan is true */ for (zone = 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head = &lrugen->folios[old_gen][type][zone]; + struct folio *prev = NULL; - while (!list_empty(head)) { - struct folio *folio = lru_to_folio(head); + if (!list_empty(head)) + prev = lru_to_folio(head); + while (prev) { + struct folio *folio = prev; VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); + if (unlikely(list_is_first(&folio->lru, head))) + prev = NULL; + else + prev = lru_to_folio(&folio->lru); + new_gen = folio_inc_gen(lruvec, folio, false, &batch); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new_gen, type, zone, &batch); if (!--remaining) { - lru_gen_update_batch(lruvec, type, zone, &batch); + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); return false; } } - lru_gen_update_batch(lruvec, type, zone, &batch); + + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4258,7 +4307,7 @@ void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) ******************************************************************************/ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, - int tier_idx, struct gen_update_batch *batch) + int tier_idx, int bulk_gen, struct gen_update_batch *batch) { bool success; int gen = folio_lru_gen(folio); @@ -4301,7 +4350,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c int hist = lru_hist_from_seq(lrugen->min_seq[type]); gen = folio_inc_gen(lruvec, folio, false, batch); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, zone, batch); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); @@ -4311,7 +4360,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { gen = folio_inc_gen(lruvec, folio, false, batch); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, zone, batch); return true; } @@ -4385,11 +4434,16 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, LIST_HEAD(moved); int skipped_zone = 0; struct gen_update_batch batch = { }; + int bulk_gen = (gen + 1) % MAX_NR_GENS; int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head = &lrugen->folios[gen][type][zone]; + struct folio *prev = NULL; - while (!list_empty(head)) { - struct folio *folio = lru_to_folio(head); + if (!list_empty(head)) + prev = lru_to_folio(head); + + while (prev) { + struct folio *folio = prev; int delta = folio_nr_pages(folio); VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); @@ -4398,8 +4452,12 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); scanned += delta; + if (unlikely(list_is_first(&folio->lru, head))) + prev = NULL; + else + prev = lru_to_folio(&folio->lru); - if (sort_folio(lruvec, folio, sc, tier, &batch)) + if (sort_folio(lruvec, folio, sc, tier, bulk_gen, &batch)) sorted += delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4419,7 +4477,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, skipped += skipped_zone; } - lru_gen_update_batch(lruvec, type, zone, &batch); + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); if (!remaining || isolated >= MIN_LRU_BATCH) break; From patchwork Thu Jan 11 18:33:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13517708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F9D8C47077 for ; Thu, 11 Jan 2024 18:33:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D09216B00A4; Thu, 11 Jan 2024 13:33:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C69DA6B00A5; Thu, 11 Jan 2024 13:33:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A96AD6B00A6; Thu, 11 Jan 2024 13:33:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8EA056B00A4 for ; Thu, 11 Jan 2024 13:33:48 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DDEBB140D16 for ; Thu, 11 Jan 2024 18:33:47 +0000 (UTC) X-FDA: 81667878894.25.F46F932 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf28.hostedemail.com (Postfix) with ESMTP id DD732C001A for ; Thu, 11 Jan 2024 18:33:45 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WNFNOKmH; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704998025; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FtBGRSINF6KI+RQqf15dVUHoZl7wlhzwJWjlp3npicI=; b=vTXLrNoPUevJcG3/eyrk7eR3V9XpafK5acwgYMYHZ7rEPHaTfjTREp/KmE5l8k48UjzZ7B tjEt1j08uO44IEZ+TluDF73VMHsXMB1NoRH813lvA6wkDS6QbMZ/9Ig4aCip4Nx3Uo+o3f tQha+SeNnIKpQY5Zj7u8PRdLjeDIlos= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WNFNOKmH; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704998025; a=rsa-sha256; cv=none; b=iY3QoEYrLG3W+0uzYbN9ZS1kJaGaOe9kkPCP7w8zyIUf2xSMt2mHohGICBNQp3Wh2IAPKb 50jUXll2LYXdbhyYWq/NH2hR+eLtpLNHhxiYG56/MqPtBa9Fe/GkaJ02mQw7O4gSLqK8ji HitwpNMWlgt8jghr8tM0O70eiP7NKFI= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d54b86538aso28936355ad.0 for ; Thu, 11 Jan 2024 10:33:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704998024; x=1705602824; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=FtBGRSINF6KI+RQqf15dVUHoZl7wlhzwJWjlp3npicI=; b=WNFNOKmHBtz9sgvB/YpUhwCstqKNWZnALcGAofrWrWdVqBsEJJ6jP7Vmqd4HQiCQUR 1v9rfx4jFHF/ZpG2uwoRE2hi96VsDMLT46DwJA/chPU23Y/ucvz4g9Y7OVhNiEbZoYSD NYZiagRFJl1946uAcD809+3rBLUYMH01zbyIhfSO9mcUGOEdXIUl169D7JncR80CYn9t XzgBHH8mWfjKIPg4n0etXVxciCo4gXHM56BLLp4Oc9Lx2uoRTulWQoblWwTgHMA2s/Lg lymbXEp+HF3YmJ5Nbi1WjGo9ZYNPsDgg9vdZl992TngwG+F+/3zKhfnZwI5L/Z8sRaLP MPAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704998024; x=1705602824; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=FtBGRSINF6KI+RQqf15dVUHoZl7wlhzwJWjlp3npicI=; b=iBowDl2T724QtpxT02oWL/fhKG2VtHWmaX3Hwvz+31eFRrI6C84PYOGZB4st3pnaY0 qu5fbH9CpWyOOXdI2ENSM1sZDKcBmAYMNBMovvyfVaPGPNvV2K0z3Ytn9F8/WpA2v4MU vqwfrqPCl1qTlkwSZ5ZALW1PthPTBwe5stuk27w6k5nKBm/73nzFKlYazUm1uuRiyibM cKlEg5arJoRQeD1KwvUoxxe0GYAE8eaPbAr0qmSP1SdoLmg5CjYyeFMbnJBNQeWlpwmB vWIEqUgQdnuVWeiWzVaZSb8jLtKI9saZnTM+4WLAL+r6o6WzrKvzskv6eTLCiJvzU19Q hjKg== X-Gm-Message-State: AOJu0YwvLSvR/zW5NcXOI8L2LIelilKCn0X0SZSRAOLyExLYFPDvJVXq rIIgz6mPI+6Lcqnx7+6w15L7RACWDX3/mUBP X-Google-Smtp-Source: AGHT+IFBEjyW6RvYROEJiwKiMLp56a9P4SPF8Qzgo7qO9Q/w+d/Lqk+mkTK5izHiT22OVPgAbeFUzw== X-Received: by 2002:a17:902:ea91:b0:1d4:ca2e:2bfb with SMTP id x17-20020a170902ea9100b001d4ca2e2bfbmr149921plb.42.1704998023664; Thu, 11 Jan 2024 10:33:43 -0800 (PST) Received: from KASONG-MB2.tencent.com ([1.203.117.98]) by smtp.gmail.com with ESMTPSA id mf3-20020a170902fc8300b001d08e080042sm1483267plb.43.2024.01.11.10.33.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Jan 2024 10:33:43 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , Chris Li , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 3/3] mm, lru_gen: try to prefetch next page when canning LRU Date: Fri, 12 Jan 2024 02:33:21 +0800 Message-ID: <20240111183321.19984-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240111183321.19984-1-ryncsn@gmail.com> References: <20240111183321.19984-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: DD732C001A X-Rspam-User: X-Stat-Signature: y8rdibzddcmcacm9fxc88xe55r1iseja X-Rspamd-Server: rspam01 X-HE-Tag: 1704998025-914538 X-HE-Meta: U2FsdGVkX18Ux/HouFheBplIs06u9Wgd50/p4lIQNZyTRcNHtCl3q89Em52/GtHPOu6ctk1ZGEl8k1kBG8fBYJSynQ1+4/bHB3G0fowL2xy8DHY8YraWi0T0NYbEQLM7VfsIlilBBhzVR2FxTwBbzq+L9IwtDjpS2lhEz/GSjGoooZxt7yMq71UcfIhpE3FI8M/n/OYdqWSifDeLr79KZVc4KqRrZqN3J5iVNlUqDaBwCmkWWyBjouB+D7L54mZXKJO0lcgbx13IdolB/kkoCo7lKlWlEPbxYyoTrVH/FjwSUhHtUe70MwyEztxP9ujkHaoBhPZBkA/MDyk8YY/wUx3Uv6oILe8Eykh6Z9jQ7kQ4y0KIYCMj+4SI92jTqTC3K5e/574Ti/ClO8J2Qei6F0UTj5J+pVQzyDiXzoPSAF6H4LJ1aYRXrDHyJHsE1PDj7NprYf3GHvDdxjYJiR6n7loXz5Ne/9qRslarU8SsPNvfBvgPo6jVDbhjS8ADKjCe2O04PFLXznUWeFcCfSx7J0bahVShXzAkE6KHc5nzs8chZ8mj5vjshqCl1L7H2vs+fqPHo0EcGCzWtlf3YCpbndxLAtuXw80X3ALJ5dbMBfjfuMbj4ZUYeLdgan565wI6bPS/ORiT4Krh8IhxXL9QvWvkfJfJUz0j5GX7ucSPARaJXRK9OyRyId/PXnUTEXpbXen6F1BfooIEA6m+Stu6d7QJlk3HDoGlgvxFzD6ZU0IAJUzjeEEAK22VccK21DdFiCy4Nh+1PunjWz/WzZ5H8TXd9KCH0fCWp0PCYtXbe7AZU1USIu0ocFbB7qlGExy5zpVa+H2UVtgvUNwvYkSDKosY+z9t6bAJv6/LN2/h7DLWEHAE0XlCJqDS1jkN0YN7YhYvNd4+vsGKe/SoMq5SgOq+Xvh3ZakzeAU0rcIVd7w3wr86OJe+qTF45Lp0FixdT1S2RhZ3Frw/+ZS+5zU 32G720o7 eOHdSq1YYcxPpA/WARRhBe30jCGy7El2zgqoMMKFAKrQPNyCoBI/38YTYdwOrRm0mZC7GFSavrrw3dFjjX7r6Kr6HKqKx3MzLtHC0MdKLfqRlJaarok8v5v78b2cRJ09AJZsX/dq25hVN+4bujoFrCKd+FEQ66XNFEBGjmrhVK/knFQE+B+667kB9jYnlNG5us1VLHg7tokvxf7pL40G00IK8KBHrC/DYdnm3sAJdPmtNwNMW2x+ZyiQ6zvHzgo8/CpiuZgwnVmoBpYUPeVsX5YBkvvfzFlGnGnoJteqV8pQSohldxk15DzRlm92+aDn00LhT6ZWgxwyl5XerKY3uPj/aezW+qQ98lV/6vNOp9i7TMY7m2VahHPGKViNWd8ckE0CPeWKDif2lk39pQ5YT6zmIk+Z0IFGVHf5tMLAhFgaCkFk3pN3gscl2FZRtSiP14hJbS+i3CfCmZxGTWMH+ZRtilL6ZSCascQ/jfqgD43WZBunkuox+Myk5hkdTegUzHTeAaMtzVoBadoVEu1cohrqoSN+/dBIqUdTPrGEuBa3ghxU9uKDQbLHyGH66JXSYvVZGk1+U0L1RGspDf6k1lG95joWkmQ2SXybIwc8yV9hluq3SPDDfw6n8hXqZ0dlfcMoxdkhw+fkLi+Lfo42hPamDwWDPqxCG420gZB6fds6KgfA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Prefetch for inactive/active LRU have been long exiting, apply the same optimization for MGLRU. Ramdisk based swap test in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-3: 44890.50 Ops/sec (+1.8%) Ramdisk fio test in a 4G memcg on a EPYC 7K62 with: fio -name=mglru --numjobs=16 --directory=/mnt --size=960m \ --buffered=1 --ioengine=io_uring --iodepth=128 \ --iodepth_batch_submit=32 --iodepth_batch_complete=32 \ --rw=randread --random_distribution=zipf:0.5 --norandommap \ --time_based --ramp_time=1m --runtime=5m --group_reporting Before this patch: bw ( MiB/s): min= 7644, max= 9293, per=100.00%, avg=8777.77, stdev=16.59, samples=9568 iops : min=1956954, max=2379053, avg=2247108.51, stdev=4247.22, samples=9568 After this patch (+7.5%): bw ( MiB/s): min= 8462, max= 9902, per=100.00%, avg=9444.77, stdev=16.43, samples=9568 iops : min=2166433, max=2535135, avg=2417858.23, stdev=4205.15, samples=9568 Prefetch is highly related to timing and architecture so it may only help in certain cases, some extra test showed at least no regression here for the series: Ramdisk memtier test above in a 8G memcg on an Intel i7-9700: memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=36000000 --key-pattern=P:P -c 1 -t 12 \ --ratio 1:0 --pipeline 8 -d 1024 -x 4 Average result of 12 test runs: Before: 61241.96 Ops/sec After patch 1-3: 61268.53 Ops/sec (+0.0%) Signed-off-by: Kairui Song --- mm/vmscan.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 57b6549946c3..4ef83db40adb 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3773,10 +3773,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - if (unlikely(list_is_first(&folio->lru, head))) + if (unlikely(list_is_first(&folio->lru, head))) { prev = NULL; - else + } else { prev = lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } new_gen = folio_inc_gen(lruvec, folio, false, &batch); lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new_gen, type, zone, &batch); @@ -4452,10 +4454,12 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); scanned += delta; - if (unlikely(list_is_first(&folio->lru, head))) + if (unlikely(list_is_first(&folio->lru, head))) { prev = NULL; - else + } else { prev = lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } if (sort_folio(lruvec, folio, sc, tier, bulk_gen, &batch)) sorted += delta;