From patchwork Sat Sep 14 06:37:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13804274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B496AFC619D for ; Sat, 14 Sep 2024 06:38:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0496F6B007B; Sat, 14 Sep 2024 02:38:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3A916B0082; Sat, 14 Sep 2024 02:38:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E02706B0083; Sat, 14 Sep 2024 02:38:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C4C446B007B for ; Sat, 14 Sep 2024 02:38:02 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 67A81A92B5 for ; Sat, 14 Sep 2024 06:38:02 +0000 (UTC) X-FDA: 82562388804.11.641BA84 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf18.hostedemail.com (Postfix) with ESMTP id A26511C0007 for ; Sat, 14 Sep 2024 06:37:59 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lFNANM8Y; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726295772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=h0XGXnTz50hOJaeNhqtXIh7+Vq+6DDbhWS4ntxPilS0=; b=6XBjePeDFnXSEAn03SM71J7lMRHoriMHDKwfdzlQyYyAJpokiX1sIBS1BoOJRxB22z8yJw E29Jsi1FxtQQernX3+Cxz+ofmfEaNgmUef04ddvRc56mL0N0crRx2d4N9K//tQzhLZuEXO PTCfdT8JMsaTAmeLv53hqwprO60IHlc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726295772; a=rsa-sha256; cv=none; b=fxB3HpZ8b7EJeiPv8O6xMz41UVwrwDO9+dzvZ8XlI4Qhz/uI64o14mkqkOgplIxVfgwoxU +0T0iWlb09+rg8E12dq/isXHKtFqAIeEGRWJQLBJbduH0OmEs2aJ0owXyaK6lHyaFCvuhH XIBkq7BWlO689mSsu9rV3oDdWNbLuzA= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lFNANM8Y; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2068acc8a4fso15367475ad.1 for ; Fri, 13 Sep 2024 23:37:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726295878; x=1726900678; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=h0XGXnTz50hOJaeNhqtXIh7+Vq+6DDbhWS4ntxPilS0=; b=lFNANM8Y/b4hQrizFJ1aVq/WRcIGIZno1ewoKDy7pDZcVhODy4QksTtjnDCLHPK1jy PxQcLeclhhZiPOKK8qy+Dir518gkneG21UsJflFFv1T28mntbXzFpYHBUGH/OsnxOyWQ McSqOAMKy+cJKegwNxo7dBhhW/VkNbSzXpVZ1C1gj0EHIfAdAdkp1u7NvlV/vQKjha9x +Py3bS/s6sZhezVb+r8rYk52BjxsKsFqfa24gWMsd4Lo2zQq0AkTNVQzJbTywE5Rvqek qStiE+EKyV+AEkbK8OGYLWysQCsMC1aypSi9c7odRRicDirK1Y8D+lVzXbWzMVpCTeNa KObg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726295878; x=1726900678; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=h0XGXnTz50hOJaeNhqtXIh7+Vq+6DDbhWS4ntxPilS0=; b=EC2BLgDd/xrXtMb79nTtaAu6/6EPGr5Zkys239iLGv6CLy2ch/u8JzZjJFq36Q/7DW o7SFl9ht64CvoecJ5ITIBZteTbO47fWWP7Oj127zR4cOES/GRjcQ6+J6WCBB7flnEj9R aEnVaDFXS/XUrBOgzhC2NBD0WK222rNmfre98KV20Omk74RHVFG8u9ixB1NpsjgnG2/V gKwFcJtS+/FSxWI7QN1MnCekGPtxJnwQ8uetcJZJT0QIqz59+VB8Oqb4hO9dH8X4QGBW bC/TjqvKX5h5J2t5t3aj25sKvGdjaw0Z9ljdi8rSfXVhWFCLiloWCT70o44jCPKYXa2C twwQ== X-Forwarded-Encrypted: i=1; AJvYcCWS8PcT16KP/+eaZ6q5SVsM+jSScV6QjR2yd+rT0VT+Pf2xaY6RNNnULwWXicY37K5Fl9lTIZ4M1A==@kvack.org X-Gm-Message-State: AOJu0YyayOFV2lPQvXJCtQ+mbJZ2F3GfYOYno3m8cWX6FDAH9VQSNUWB pBqQeAVDA8jBmnHihNxIbd4OMTjasCLtUSF+GqrAgb5+lL47/wXF X-Google-Smtp-Source: AGHT+IHUSR5jD2duXHLe3BrgPTf5h7DhPZAFxKcDMcB2yTXSiHyvVeTo590mSkNIjdz80NvYheEK+w== X-Received: by 2002:a17:90b:388d:b0:2d8:f81f:87a0 with SMTP id 98e67ed59e1d1-2dbb9dee70cmr6516049a91.9.1726295878119; Fri, 13 Sep 2024 23:37:58 -0700 (PDT) Received: from Barrys-MBP.hub ([2407:7000:8942:5500:6927:3898:c182:14f1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-207946d2928sm4835395ad.120.2024.09.13.23.37.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 13 Sep 2024 23:37:57 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: mhocko@suse.com, fengbaopeng@honor.com, gaoxu2@honor.com, hailong.liu@oppo.com, kaleshsingh@google.com, linux-kernel@vger.kernel.org, lokeshgidra@google.com, ngeoffray@google.com, shli@fb.com, surenb@google.com, yipengxiang@honor.com, david@redhat.com, yuzhao@google.com, minchan@kernel.org, Barry Song Subject: [PATCH RFC] mm: mglru: provide a separate list for lazyfree anon folios Date: Sat, 14 Sep 2024 18:37:46 +1200 Message-Id: <20240914063746.46290-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A26511C0007 X-Stat-Signature: d1stwi4k8wxamae4oz9nrk6z8enoo59i X-HE-Tag: 1726295879-162763 X-HE-Meta: U2FsdGVkX19hJrxteE6wXNIUda3AVNLjYYvmJ9dYYbNcCa+13R/Xwd8t4l89oQzOvEjwRVtOf1jyytX6EYXIf2xfEXiOmzSzRc0LUzpPvjewnabd0t0QVaeiXxV7xqDpSFnkDIRufs3QRdHh9aTM+qKlZJuCP9DFr7MyGwapqeCfgTx0BXgBt+n+B1d/OyYz3X3ZL53k5NXwMbCRl6/0ux2nND2VRXCtbfegy/lGhkjwTZ4JmTpZWRBHmN2zmMt9gTPM49zk7Rdafb6cCQczdC+DwewAwpzut99jdQN/hdW3Za4nxhO5B0ptEBcsAeDQmy3yJDrz3d2mAsg3+QmPMeiQ5/cgXCZ15DQ3u8K/taALN1+TwmRP/1+5Qat+LZE56FhKHRbugZdX1MKUzubYPGm/f449W5lyBdaMCLBwULQQKYPmp0NzlRdRlYRvI0TIdcx0gTI3KnNdLIKf64XV/BDDLMAY8iZcQlJNpVxyvZcCHE0gPy5rqzGBxr2uwhTAHv4zwd2pVjZAs8va1vReVlET1tck8HfS2vqNRMcdK2hKf2FJa1hilFE1xy7vCmghIb1f2g++uwLR1oku9Nv58m54mTuBWR2h2hN8ksTqBMIzaUo5CN3GCCCF88NFBmkKa1mIf7wLfId89bpywtb/DhvAf9N7j6Zy5D3lQGOL+p2AvLYFQT8CKYClkIlf/NEapXfTa1dQxgwy0503VImxVErUlyP4HjZyQN/hSQ0nMKaOJiKhFi88ZhZ5k4AHMHJLJ4wOYU3D4VtwDmvG6P0FQmn0GrT5EabW5WrjrE4YvsN08DeTrcdJk9BrbOKdOadYLK3JjSwI43zqAXg4jI5oArqpgoMIUoY4SWxT4HcEzF/CWxoouS6TvhgTu5EL46S4zWBv39Vb+tSspLNqZkf1zhyRZsqi6fJEgKkkD4WcssWkbyxxhDx0Tx8iM8KxJwnnbhybwYyhOgey1FsDDvY CosB0DOi 16gZIVPitvf4EnvtoVYL70AUBcIgWhoFxT+4JmiAVIfx12qhCijKb6oFxPsKTfeNOCe9YmKd030Fvn0VrFxD04FH0ntn9rOMvKAWFIpDnfunnW8cUGGHLnjODJi1/NqjGU0lhn8hBC2yOPucPIn9XRSFabG6S3xH34ujnjLWLOzokPdI+/CKSzMAxaQlocjUQzTZNgs15iJqnTxnJPnDTLyFVsA1arG2VuNB6xfn3fVgdDVPGSOsY8kzyp4DPusuRetbJh7eo5euxbFPXFGyakBAZ7MfcAOdNpVv0lyxQlC3PUmrqY5myB9P/2VMvWmXgk7asDIYmvqd0OHV0lOB+ej7r/oud4wwKgzTrtcGyNdy41chsPpI9eikTZVEMQP4A2wbkxzBfSTbyip65Y2D+EKF/JJBvTz24roF0vsP5FlNGUU/lR/Ta0ZpihVib+VZHOOmzRIKE+KEt7rOOgVknX+YhcwsXDABgPICWXpRc4siGDpByg1CrrRMr/ydqxitaYz6l X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song This follows up on the discussion regarding Gaoxu's work[1]. It's unclear if there's still interest in implementing a separate LRU list for lazyfree folios, but I decided to explore it out of curiosity. According to Lokesh, MADV_FREE'd anon folios are expected to be released earlier than file folios. One option, as implemented by Gao Xu, is to place lazyfree anon folios at the tail of the file's `min_seq` generation. However, this approach results in lazyfree folios being released in a LIFO manner, which conflicts with LRU behavior, as noted by Michal. To address this, this patch proposes maintaining a separate list for lazyfree anon folios while keeping them classified under the "file" LRU type to minimize code changes. These lazyfree anon folios will still be counted as file folios and share the same generation with regular files. In the eviction path, the lazyfree list will be prioritized for scanning before the actual file LRU list. [1] https://lore.kernel.org/linux-mm/f29f64e29c08427b95e3df30a5770056@honor.com/ Signed-off-by: Barry Song Tested-by: Gao Xu --- include/linux/mm_inline.h | 5 +- include/linux/mmzone.h | 2 +- mm/vmscan.c | 97 +++++++++++++++++++++++---------------- 3 files changed, 61 insertions(+), 43 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index f4fe593c1400..118d70ed3120 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -225,6 +225,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); + int lazyfree = type ? folio_test_anon(folio) : 0; struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE_FOLIO(gen != -1, folio); @@ -262,9 +263,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ if (reclaiming) - list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + list_add_tail(&folio->lru, &lrugen->folios[gen][type + lazyfree][zone]); else - list_add(&folio->lru, &lrugen->folios[gen][type][zone]); + list_add(&folio->lru, &lrugen->folios[gen][type + lazyfree][zone]); return true; } diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 17506e4a2835..5d2331778528 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -434,7 +434,7 @@ struct lru_gen_folio { /* the birth time of each generation in jiffies */ unsigned long timestamps[MAX_NR_GENS]; /* the multi-gen LRU lists, lazily sorted on eviction */ - struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + struct list_head folios[MAX_NR_GENS][ANON_AND_FILE + 1][MAX_NR_ZONES]; /* the multi-gen LRU sizes, eventually consistent */ long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the exponential moving average of refaulted */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 96abf4a52382..9dc665dc6ba9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3725,21 +3725,25 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) /* prevent cold/hot inversion if force_scan is true */ for (zone = 0; zone < MAX_NR_ZONES; zone++) { - struct list_head *head = &lrugen->folios[old_gen][type][zone]; + int list_num = type ? 2 : 1; + struct list_head *head; - while (!list_empty(head)) { - struct folio *folio = lru_to_folio(head); + for (int i = list_num - 1; i >= 0; i--) { + head = &lrugen->folios[old_gen][type + i][zone]; + while (!list_empty(head)) { + struct folio *folio = lru_to_folio(head); - VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); - VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); - VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); - VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); + VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - new_gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + new_gen = folio_inc_gen(lruvec, folio, false); + list_move_tail(&folio->lru, &lrugen->folios[new_gen][type + i][zone]); - if (!--remaining) - return false; + if (!--remaining) + return false; + } } } done: @@ -4291,6 +4295,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c int refs = folio_lru_refs(folio); int tier = lru_tier_from_refs(refs); struct lru_gen_folio *lrugen = &lruvec->lrugen; + int lazyfree = type ? folio_test_anon(folio) : 0; VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio); @@ -4306,7 +4311,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* promoted */ if (gen != lru_gen_from_seq(lrugen->min_seq[type])) { - list_move(&folio->lru, &lrugen->folios[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type + lazyfree][zone]); return true; } @@ -4315,7 +4320,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c int hist = lru_hist_from_seq(lrugen->min_seq[type]); gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + list_move_tail(&folio->lru, &lrugen->folios[gen][type + lazyfree][zone]); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); @@ -4325,7 +4330,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* ineligible */ if (!folio_test_lru(folio) || zone > sc->reclaim_idx) { gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + list_move_tail(&folio->lru, &lrugen->folios[gen][type + lazyfree][zone]); return true; } @@ -4333,7 +4338,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { gen = folio_inc_gen(lruvec, folio, true); - list_move(&folio->lru, &lrugen->folios[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type + lazyfree][zone]); return true; } @@ -4377,7 +4382,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, int type, int tier, struct list_head *list) { - int i; + int i, j; int gen; enum vm_event_item item; int sorted = 0; @@ -4399,33 +4404,38 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, LIST_HEAD(moved); int skipped_zone = 0; int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; - struct list_head *head = &lrugen->folios[gen][type][zone]; - - while (!list_empty(head)) { - struct folio *folio = lru_to_folio(head); - int delta = folio_nr_pages(folio); - - VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); - VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); - VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); - VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - - scanned += delta; + int list_num = type ? 2 : 1; + struct list_head *head; + + for (j = list_num - 1; j >= 0; j--) { + head = &lrugen->folios[gen][type + j][zone]; + while (!list_empty(head)) { + struct folio *folio = lru_to_folio(head); + int delta = folio_nr_pages(folio); + + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); + VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); + + scanned += delta; + + if (sort_folio(lruvec, folio, sc, tier)) + sorted += delta; + else if (isolate_folio(lruvec, folio, sc)) { + list_add(&folio->lru, list); + isolated += delta; + } else { + list_move(&folio->lru, &moved); + skipped_zone += delta; + } - if (sort_folio(lruvec, folio, sc, tier)) - sorted += delta; - else if (isolate_folio(lruvec, folio, sc)) { - list_add(&folio->lru, list); - isolated += delta; - } else { - list_move(&folio->lru, &moved); - skipped_zone += delta; + if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH) + goto isolate_done; } - - if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH) - break; } +isolate_done: if (skipped_zone) { list_splice(&moved, head); __count_zid_vm_events(PGSCAN_SKIP, zone, skipped_zone); @@ -5586,8 +5596,15 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) for (i = 0; i <= MIN_NR_GENS + 1; i++) lrugen->timestamps[i] = jiffies; - for_each_gen_type_zone(gen, type, zone) + for_each_gen_type_zone(gen, type, zone) { INIT_LIST_HEAD(&lrugen->folios[gen][type][zone]); + /* + * lazyfree anon folios have a separate list while using + * file as type + */ + if (type) + INIT_LIST_HEAD(&lrugen->folios[gen][type + 1][zone]); + } if (mm_state) mm_state->seq = MIN_NR_GENS;