From patchwork Tue Dec 31 04:35:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E920E77194 for ; Tue, 31 Dec 2024 04:35:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D332F6B008C; Mon, 30 Dec 2024 23:35:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CBB336B0092; Mon, 30 Dec 2024 23:35:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B35E36B0095; Mon, 30 Dec 2024 23:35:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 86A986B008C for ; Mon, 30 Dec 2024 23:35:58 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 327BD479AC for ; Tue, 31 Dec 2024 04:35:58 +0000 (UTC) X-FDA: 82953990168.24.72D6A55 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf16.hostedemail.com (Postfix) with ESMTP id 26199180002 for ; Tue, 31 Dec 2024 04:35:07 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BjFitxHV; spf=pass (imf16.hostedemail.com: domain of 3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lq6dqRxgSDV3CyACy0+VqeWU26YT+ayL2zVnCSsSGnY=; b=03uvcpvH0hyK+V2rMccciNmn1u+TnejU1Vjd6/TcgstP0mlmvIxM/JawnPXCQOGG0OT2v5 8enW7dxz3C7r/VJ9zWY38RVdTqelH3v9w0jFUQ20Dbv8Bgrz6l2GtYUzYx+xahhZfF3wBf JTeuq2y8GB00Xx+VleVBOSyKc67TDno= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BjFitxHV; spf=pass (imf16.hostedemail.com: domain of 3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619734; a=rsa-sha256; cv=none; b=jsN5BtXf4WZ9rWRDeSmM96gbTl5fI+cVYfYOTVmyY3KhnyPkuggbyvEEfCUOfNLVG+baxF j989PwUnV+1DC2+EEAB3DV96nb0MAXZkgROjJ1rzT7N+N5qWDJFXh1DEnPa5xjbC6GdH9w wiB7NdfEqtJfYlVBJW4Qm1HsyZwFrVo= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef9da03117so20659268a91.1 for ; Mon, 30 Dec 2024 20:35:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619755; x=1736224555; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lq6dqRxgSDV3CyACy0+VqeWU26YT+ayL2zVnCSsSGnY=; b=BjFitxHVSboLhHSItW7LAyEzsQX9Wn4o0iMCv9yS7iUC+CoaREXNfy5fX9Qt9rDMNq /u/BaViI1VivEXDwnGE6Mh+zDXD6fg7P8XqWzCsZseAfoFBwUa9DuJu/FIwoK6gc2M7J y1tuZE7S/Teuc0ouKg2onZMECVK90FppNmsIuu23rB+zHJiiW0I6Dif9zeU9+hU6WGj1 yPANOzNBkXGwJHb0C7+0nsOxtuV5/qnwwhWBtuWSyWbZhP7wH88UGEH4PZglxGHWq2in Ln7HIg2mR6SrBxxRLQpfCVE13DAtbDYZMDgU5XMT1B3+ICLKfeQcSUoa9uKh69ltRO8Z zYBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619755; x=1736224555; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lq6dqRxgSDV3CyACy0+VqeWU26YT+ayL2zVnCSsSGnY=; b=gH1ypXTNYqq9gBkanr0Qvgaq/dj+WN60m1KT73WgMle7STcgm0pg2r9KVUs66dgDCN 2oFowwiab3Qubnk+KlQ2KlaGIruBC0eIGinZdqK/9qGzsAh+1o35o2s7UE8KdTYeBWD3 EH7/E+UB0DhfFGAUDRKFcb3bH5caMZng8A6HOSiqO2uD+xa90TFwIlzzsZK64SEm2B8P KJgLfVwpcj8n3K6uWw8M30mJPsXDsuaoPOlhc3qulnJ6Sy81STBnSxvD4aQvduZroTBF +g2/Q2lKd9ylWyMx5+7qPtsbW5uQrbXkuHhBHAM8xLBLip4BTC4FRCy4uTdQW1CIH48s Pf1Q== X-Gm-Message-State: AOJu0YwmSfIxHftCB4vg8fNcNzOjMue3Uh6EqPYWMvkqE6qqCGIcft2z nt/g110KCEYNiG64G8qbD2GB4Gapl6Tw8muZIXucUw6flrY83iK/FNlSY+/uitFWbLPefVmrjHt 2DA== X-Google-Smtp-Source: AGHT+IHeZyv9yGkkGSeakHyyxsc/LoAVc3WVdX/rKDrbygxdcMqZjnwE7jl/rLSwE6a6nOYHrpf3ke3Jfww= X-Received: from pfd7.prod.google.com ([2002:a05:6a00:a807:b0:727:2d74:d385]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:430d:b0:725:db34:6a8c with SMTP id d2e1a72fcca58-72abddbd4f7mr55420073b3a.13.1735619755260; Mon, 30 Dec 2024 20:35:55 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:35 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-5-yuzhao@google.com> Subject: [PATCH mm-unstable v4 4/7] mm/mglru: rework type selection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , David Stevens , Kalesh Singh X-Rspamd-Queue-Id: 26199180002 X-Stat-Signature: 1mcfbawzpto7eh9tjcszga3ucpcu4jam X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735619707-914493 X-HE-Meta: U2FsdGVkX1/ZH7JIpiXhbKUvwpbOer8TK3ceLE8VmWnyf9RGvSgQkWxNxqAxL9NTn/OiNur2ZTRQRO3zzM/Irg2VjnfBkCjjRL1Ad4JZ1o179UQUhvr+J+GdXVqriVwZ/TARp9YGp6BKDhjLXpETYcqN6aZ0n1y30IDVfHiBzRbOkg5QibaXCngRydxLIFwnahJap1fiXgWl0lSP0SWe3IpYi6/9SclFp4JmgGToOzvs1XbIwrPv5aIxYnaOfs6Xi2vT1ugkBs66lMWsBVGmPh/POwKmEbDRHRs4Urbz5TxYCU4/ogyOhqheAXdAKsk9eZsjldLu4LvZP9oClUY4F67Og3K9CY7SkynvneDh5I97bJaBiFY7QHKNHuCUr5KTbFI1KY1Zmj8RWKMdd5DlOR6oDd6XKrvoHAFQHRL/lEjskz7NoaB/GsnoN4UMwNt/uWF/pMTOoWf3vGXjj6sZ6KA/N1wllMyp8x0DBnr5dYcLY/AT85NVZX6bUi7Br1jvs5NUJGmvgZtNNWiS+MsvuodOZaMOAt/0cSJ7kiks0Ak4v6S4869iAnozYQVGd6xlMq0ET9pufEN8wKDheLe8nkhRfEV/xI1qcw2ndiJkBebQk9UG2woLdfKF2gsLOZ7adMb8aNNx6VwS0s9ZUPc0m/LNQE0dD6dtS1VfuHscVmpnqBzUkaC5JuR3TZFmTXxsS4Q9Fmz6SGTunZxKI4lsyLE+vS9D5Lh44cEaR3OE+4DDcahZSN7F59Bq6qXJz0F1IG77DR7ybZA2uDzXHYypYODbF/KL2yN3Lrb2y35aPlTsb9v7IwDmN82nVer0aq8AnRM5H2IYczXKXuwC0AVvc3sVNl7qDfqnneqQ1DeQjD4esRff3Pc0ZI9R9Z/8waUkMZtCmzeK5WhaXa5Wu2q7KXivdP6aM2cjPi5evxfoO9gi5WbI5z2b4r+U6OoEVxAHYo6nCwD4cfTReYr+JJ4 i2JR1hC/ C4XMqUfC8Xm9S0G5CKTfO62rUiu71iqqHpCYHicIYp7EsET9spOm9Wfa59B44jGGfAplT2/PKj9/8oA9rKmARtmNT5u83cNdJX4+KCzUH1/Zb70WntcJhG69lAUToRtp6qerjAL8kPJMHXS2LFCD25k+7pGYineka60ePt8a3AXH6KW57ILaI1g0VjoubppH/WDwjWjmOSNGJZoaPp9sCnCOxd51dbLTQSmWvW+sOeFEw5nNwaVj6Xjl99w2+2Pj+fF+pJ9RbaCvM6idOK4+XWcCZPSrdk+nzk2WNcWJL0l5J2B+wnLx9F1koF6w3GzxYAB/Ur6usW903X+XWqjJP4UgEbmiI/RQjYNy5QORhUes4xGM0Y5dv2NBoB8xVyytg54G7hQKSU5m8NuHwHzreHJkb8jojQ4Os8nrpo3yzNXzcZhzaTIs5KdiTwIx3G8tBBc5H0NT2Lx2xkduF/OoOrGF10Yt+8RRwp2zrYlCLD7UbxHbkZ0/c3e8nRpzMsJckgUhkQjgcGhgkkY8oGe5kucz7vsT0IpiI+jpQ83hle4uhAGAWJrYhpjgnfQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000068, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework type selection so that it is based on the total refaults from all tiers of each type. Also allow a type to be selected until that type reaches MIN_NR_GENS, regardless of whether that type has a larger min_seq or not, to accommodate extreme swappiness. Since some tiers of a selected type can have higher refaults than the first tier of the other type, use a less larger gain factor 2:3 instead of 1:2, in order for those tiers in the selected type to be better protected. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: David Stevens Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/vmscan.c | 82 +++++++++++++++++------------------------------------ 1 file changed, 26 insertions(+), 56 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f767e3d34e73..a33221298fd0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3093,15 +3093,20 @@ struct ctrl_pos { static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, struct ctrl_pos *pos) { + int i; struct lru_gen_folio *lrugen = &lruvec->lrugen; int hist = lru_hist_from_seq(lrugen->min_seq[type]); - pos->refaulted = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + - lrugen->protected[hist][type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); pos->gain = gain; + pos->refaulted = pos->total = 0; + + for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) { + pos->refaulted += lrugen->avg_refaulted[type][i] + + atomic_long_read(&lrugen->refaulted[hist][type][i]); + pos->total += lrugen->avg_total[type][i] + + lrugen->protected[hist][type][i] + + atomic_long_read(&lrugen->evicted[hist][type][i]); + } } static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) @@ -4501,13 +4506,13 @@ static int get_tier_idx(struct lruvec *lruvec, int type) struct ctrl_pos sp, pv; /* - * To leave a margin for fluctuations, use a larger gain factor (1:2). + * To leave a margin for fluctuations, use a larger gain factor (2:3). * This value is chosen because any other tier would have at least twice * as many refaults as the first tier. */ - read_ctrl_pos(lruvec, type, 0, 1, &sp); + read_ctrl_pos(lruvec, type, 0, 2, &sp); for (tier = 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, 2, &pv); + read_ctrl_pos(lruvec, type, tier, 3, &pv); if (!positive_ctrl_err(&sp, &pv)) break; } @@ -4515,68 +4520,34 @@ static int get_tier_idx(struct lruvec *lruvec, int type) return tier - 1; } -static int get_type_to_scan(struct lruvec *lruvec, int swappiness, int *tier_idx) +static int get_type_to_scan(struct lruvec *lruvec, int swappiness) { - int type, tier; struct ctrl_pos sp, pv; - int gain[ANON_AND_FILE] = { swappiness, MAX_SWAPPINESS - swappiness }; + if (!swappiness) + return LRU_GEN_FILE; + + if (swappiness == MAX_SWAPPINESS) + return LRU_GEN_ANON; /* - * Compare the first tier of anon with that of file to determine which - * type to scan. Also need to compare other tiers of the selected type - * with the first tier of the other type to determine the last tier (of - * the selected type) to evict. + * Compare the sum of all tiers of anon with that of file to determine + * which type to scan. */ - read_ctrl_pos(lruvec, LRU_GEN_ANON, 0, gain[LRU_GEN_ANON], &sp); - read_ctrl_pos(lruvec, LRU_GEN_FILE, 0, gain[LRU_GEN_FILE], &pv); - type = positive_ctrl_err(&sp, &pv); + read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp); + read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv); - read_ctrl_pos(lruvec, !type, 0, gain[!type], &sp); - for (tier = 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, gain[type], &pv); - if (!positive_ctrl_err(&sp, &pv)) - break; - } - - *tier_idx = tier - 1; - - return type; + return positive_ctrl_err(&sp, &pv); } static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, int *type_scanned, struct list_head *list) { int i; - int type; - int tier = -1; - DEFINE_MIN_SEQ(lruvec); - - /* - * Try to make the obvious choice first, and if anon and file are both - * available from the same generation, - * 1. Interpret swappiness 1 as file first and MAX_SWAPPINESS as anon - * first. - * 2. If !__GFP_IO, file first since clean pagecache is more likely to - * exist than clean swapcache. - */ - if (!swappiness) - type = LRU_GEN_FILE; - else if (min_seq[LRU_GEN_ANON] < min_seq[LRU_GEN_FILE]) - type = LRU_GEN_ANON; - else if (swappiness == 1) - type = LRU_GEN_FILE; - else if (swappiness == MAX_SWAPPINESS) - type = LRU_GEN_ANON; - else if (!(sc->gfp_mask & __GFP_IO)) - type = LRU_GEN_FILE; - else - type = get_type_to_scan(lruvec, swappiness, &tier); + int type = get_type_to_scan(lruvec, swappiness); for_each_evictable_type(i, swappiness) { int scanned; - - if (tier < 0) - tier = get_tier_idx(lruvec, type); + int tier = get_tier_idx(lruvec, type); *type_scanned = type; @@ -4585,7 +4556,6 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; type = !type; - tier = -1; } return 0;