From patchwork Tue Dec 31 04:35:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3890BE77188 for ; Tue, 31 Dec 2024 04:35:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD7526B0083; Mon, 30 Dec 2024 23:35:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B5B486B0085; Mon, 30 Dec 2024 23:35:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F9FA6B0088; Mon, 30 Dec 2024 23:35:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 779E36B0083 for ; Mon, 30 Dec 2024 23:35:50 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ED525A0D42 for ; Tue, 31 Dec 2024 04:35:49 +0000 (UTC) X-FDA: 82953988866.01.AAD56FC Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf30.hostedemail.com (Postfix) with ESMTP id 2B51980002 for ; Tue, 31 Dec 2024 04:34:16 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="y6v/Ota/"; spf=pass (imf30.hostedemail.com: domain of 3o3RzZwYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3o3RzZwYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619726; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xw2CHOQueA9zkMREwjqXXK1goFWTBSTiadQe8+3ME3M=; b=PHsOCKAO/pQ+2Rjz7tWh7vR+BWROsU1olViFv+ZLZVzBLLaYndFkBZm0XZROebw9SLx2PM RBhkncLXykOgzlXy6TEOkIzNBmSv0e+Or0Znf/VT6cvb6JSy7AtDxuYksdeAI8jqiOxPrf 6BHMpV63fUa0y3DBjZ7UHyc14mnKcew= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619726; a=rsa-sha256; cv=none; b=k4ujzX9eXTJBrINcsy5LmcCW/6LTaluGY+63jAGck0UkhmEEkMZsg2/jqG5RoNrM6tRJL5 G4QyGr4Jufha8ARnt+/72urzwyTCLcnaP0PzMOJXp7AgEFOwilH3IDoBoGAc9h/Kb3NkFc u8ebaS0HQODgu+2S+cAlr1omdA57t1w= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="y6v/Ota/"; spf=pass (imf30.hostedemail.com: domain of 3o3RzZwYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3o3RzZwYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ee5616e986so20974421a91.2 for ; Mon, 30 Dec 2024 20:35:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619747; x=1736224547; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Xw2CHOQueA9zkMREwjqXXK1goFWTBSTiadQe8+3ME3M=; b=y6v/Ota/PSi0B1865z63EzjBBsMl3mwvADqULr4syp//+aMW6iU6F+GVeLPBaNUCXp 9yyWvJHekOXIymUBxe4WVzfWuuOOIv1h0cAlTQ5hIg4jbNXsI48ntYXu87VmoiPX7VRX XidcEiAvtTMuliMDLp+DF/bFrzUVSZNNNJQ3RCLQgWQ4XzsXhaB018Djco0xL8Tlox00 fVzBIahGyollG4y6y9ASwtrhEeaUx1opppJfjQbePDBIKRaNyBfjMBZ+N6gkILnCedrX YK7ukFQf3+wq1afVoeVJG0SJ4Pj0wMqn4cUoDYWQxssBDFRjECyn6rISymI9Jm6EWZ08 u8kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619747; x=1736224547; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xw2CHOQueA9zkMREwjqXXK1goFWTBSTiadQe8+3ME3M=; b=awJIzuJSIm02YdEgZbixUseuu/MRykTMRztEwnSZ4v1ebgm4G8IDYwCJ8i5abcgZkT yug5R2OrgJALtQFxxru/iVPqZO2jXWlOM+XKKEd/ks6eIog/YtvTZRCyvvInYFJInKOU YMfyZnPtQ2uGEfnihj6x6McRD/kJW0mylH1EUhZUk2198srDFwzA4B7KP+PVzqhiqg0I 0HZ2tiz18KO04mKtodFGE2tsJAjdOS8fIDNuCoWmIg1onZR5JeKWwz/b0UJ1OaViAqeZ 2WCkPF9lijiTNMTPgeeppSSxM7mlCxzOYWqF8W1hZ25nY5Epub1GQhgSHl4zAznE4gmX Nk1w== X-Gm-Message-State: AOJu0YyApvyfQ7N0SIkvAfBFTsGS6t9cKuLvE1TWK114A/9gZMQgbYro IwVIVKbABgrrxxM3dvZTiWLlDfNclgEfxo81dM+i9G2mmm6CPR3hJDyMHG3L1zXkKF0WZieGnh0 NHw== X-Google-Smtp-Source: AGHT+IFqFvRA79cO3VpI5IoQ9FD/iueUtKICiJreOXklTdeZ0vkOif4CjJ0SLo91oWp/vZ2d3gnF2M2cOO0= X-Received: from pjbpw17.prod.google.com ([2002:a17:90b:2791:b0:2ef:94c6:5048]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:524b:b0:2ee:9d49:3ae6 with SMTP id 98e67ed59e1d1-2f452e1d12fmr56370216a91.10.1735619747003; Mon, 30 Dec 2024 20:35:47 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:32 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-2-yuzhao@google.com> Subject: [PATCH mm-unstable v4 1/7] mm/mglru: clean up workingset From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kalesh Singh X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2B51980002 X-Stat-Signature: 99wh5nosomhc3pu1pznosa5rradaiuij X-Rspam-User: X-HE-Tag: 1735619656-202659 X-HE-Meta: U2FsdGVkX18e4W1tBWdjH7X1Fvuzr+De1sohTFDpTW4xoRbczP4hSSDSVYTQ0u4NT1isbv2XgP/KTDG5DT6RRbJyBqlxjvmZXepeSaHF458oPluMSUnYQR0yx37DNYUkC/amg3OClRD8RTOeNoDMCtyZaYqj4UIFPYBEKzJeL446oCl9ENVbkYfP4Xcz+w4R/H1BIWeNQyhDaN8wjwdZWMd/1oStFZwrWkIk1FG3swTeEf6rlFNW68xWHIgL4GeT/sdsaHWdpWEIhfOE4DW6PAhVgBmQ9lNL+XeHlFb38ps4q/Q6vJfWvagCIHdY81gxymZAlMbuBvFzR7SexyJ4jBvzlTsAgS5kMsgpMK2CYIxz26CPk7wv0BbI8HFWgAZ+MZKg44sud1IoWRspRu7mCU9AkSxj6beWjlbjqWjL2wXOjNUL2mHzSvWOGoKWD/3TR2FFiCvIgXsR7xkRFY0C6we3imTytSUiT5BhOm5ULmWEsJ79+MLxFe67fCPqBIUeMbESa3IVmSnDKy7OrU+UidzuvZPmajthNqcOuO3iDK5dN6eZWXd/AqNM0N/qnZwRHIySoQETWjAwoJ/rh5kGz8WmB2nbZtyRQCTjU5Mn1UUMxosvx9ocLHhyIok+vMacFXufyr7ka5bOgV7rIEn4MBdm3lJO1YTFuw9vHY8UG5hm+HhlcrS3whHfQYFNJoYgBR8YI3Bng+DbnAqB9SnRI3fYX6MlyKRxzPHcwzgfz+3DG81tW1+RaA73JYiJMKycXKQ+nyTSLHNQKInOYl+tVc3f5lX31TNsippSi6pq6HQ0zbn3EZ8MJZc2l01ir9PTKxpQobhb3ZxwZ6KT17iEh+WDk3mm6F1XTcTfneEWyoGg9k6Ri4JuU6EvSJhCOMjnEyhS3PLhPifFm4pGdcM/vxliprQzNWzL1H0P86U0r2m9f+2isO5LwXsgLAh2FM6duSAYlgK2Bh9kRgvJScS w4cYmqnG 172MC85EpTPAENtSy0bzMzN5Pv4RzYtN/DJa+eHyHMAW/W/JgPYS8akCFNrnCY5MfY946CKDWr4jMJWTp5flQ1dZF65KvpkqXoAvnyuxLPhw+OjUe6jNwwr3aGKi/+6b4GQlGCEDuiUCp2zaI6OIEYY6DiWWRn0bJ51ase5s5qHAiddTk9Nego73RD9pAm0D74jfTyMc3cZBTKFfsk8JNCzCmXKUyncSq9x413Ntm5xGzkWU0oHZeWcq+gTFU8fxqNTwju1uyC82qtSTitlB1JO/A2nHKR3beOsod95e1zOhtaWKIZHSuTGa3Xnwra1AbKm6HQtalLSBQPcIz+jTaMoopRNfDVVfELS+ROvgTLoZRuZ1WC8wtXT6NSCYMCNXWPmWx0bZ6IR/qaIVeZq8jeAAJN37QeX0AhGcVdYGhlZF++Y8d5L7fIBMp6N+3T3WtLi4px5oSXM7BxjgYkls9l+dqkWBX6OotmzidJa3597BkPSq0o5bIRraFBzKvipMN+v2gTQj0cbHEcnD23d2IfbhUBxqNQJ14JNZi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000769, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move VM_BUG_ON_FOLIO() to cover both the default and MGLRU paths. Also use a pair of rcu_read_lock() and rcu_read_unlock() within each path, to improve readability. This change should not have any side effects. Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/workingset.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index a4705e196545..ad181d1b8cf1 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -428,17 +428,17 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, struct pglist_data *pgdat; unsigned long eviction; - rcu_read_lock(); - if (lru_gen_enabled()) { - bool recent = lru_gen_test_recent(shadow, file, - &eviction_lruvec, &eviction, workingset); + bool recent; + rcu_read_lock(); + recent = lru_gen_test_recent(shadow, file, &eviction_lruvec, + &eviction, workingset); rcu_read_unlock(); return recent; } - + rcu_read_lock(); unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -459,14 +459,12 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, * configurations instead. */ eviction_memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && - (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { - rcu_read_unlock(); + if (!mem_cgroup_tryget(eviction_memcg)) + eviction_memcg = NULL; + rcu_read_unlock(); + + if (!mem_cgroup_disabled() && !eviction_memcg) return false; - } - - rcu_read_unlock(); - /* * Flush stats (and potentially sleep) outside the RCU read section. * @@ -544,6 +542,8 @@ void workingset_refault(struct folio *folio, void *shadow) bool workingset; long nr; + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (lru_gen_enabled()) { lru_gen_refault(folio, shadow); return; @@ -558,7 +558,6 @@ void workingset_refault(struct folio *folio, void *shadow) * is actually experiencing the refault event. Make sure the folio is * locked to guarantee folio_memcg() stability throughout. */ - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); From patchwork Tue Dec 31 04:35:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3016CE7718B for ; Tue, 31 Dec 2024 04:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 800D06B0085; Mon, 30 Dec 2024 23:35:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78ACE6B0088; Mon, 30 Dec 2024 23:35:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 567806B0089; Mon, 30 Dec 2024 23:35:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2A1556B0085 for ; Mon, 30 Dec 2024 23:35:53 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E9C21160D15 for ; Tue, 31 Dec 2024 04:35:52 +0000 (UTC) X-FDA: 82953988992.01.A338488 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 28632100008 for ; Tue, 31 Dec 2024 04:34:19 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rz5x2o9M; spf=pass (imf05.hostedemail.com: domain of 3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n2jVLy+uRLHye3kRN9bJLrDt1hriY6TngJrphV7c4w0=; b=DaaWIiTZ0A8aHs6VoNoGhopSYz2JXYPiNb9e2KtwqI3inJRVUvczJe5f/fRR5wnCNvewXo x91GWOCTc3qcvH+NwcOJPM9C9yDLhDuVE9EeHniIbSall0ojsDIycjJi3ceSzWEaBauO4h nJhVTJ2Ho7+r7bPn6Yo1rkZ298vjg2A= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rz5x2o9M; spf=pass (imf05.hostedemail.com: domain of 3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619715; a=rsa-sha256; cv=none; b=BQnVZrOCDcvmgeFolF0/+ELJKgYEmWD1nA1J/FdzAFKhmKQfle+omHTbqRSTjRg4rezRWW k0RuhVInATHgiw92v/fZZUWMKyY7580KI/+n1q6kQz6GQwPZln+qMxuRvKOwFnJNprgWR1 +3IfHHKuC4bRilt/AhdBD73SOjf6l/w= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-216405eea1fso122044725ad.0 for ; Mon, 30 Dec 2024 20:35:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619750; x=1736224550; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n2jVLy+uRLHye3kRN9bJLrDt1hriY6TngJrphV7c4w0=; b=rz5x2o9Mv8SsgYlHC/5vbaD2FFhAGSqftqf2JHSOtm+mfAcOQu+c1ecIXKZqsj3ZSG hCNKZO5CCJFlvFjqKuTWCYgbJUFPZhg2uUKXIK0HBZTfRD3nFIJMSgZWrXx61V/sG/5L giGfWChKzB6JZUGv9Cbc4sYiKoRotAMkZW2K7vrteNw4YQ3Sk4SSPyw502VK6pk+1aiu 2zHW76N//eZUDwqcf33LcVWu3n2ZZNLmgziWBxbW+mkqW7EnVlIETSO1gb3FYya3Nh3Y SxTjUDTAKQYYkQoKSJdfP7H2I54Yl8cKCXzABiSSufa4iPpcr/sHijRBAhNxlRQmRJF8 wzxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619750; x=1736224550; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n2jVLy+uRLHye3kRN9bJLrDt1hriY6TngJrphV7c4w0=; b=aSGZcZIK0MYhHOs+2rKKgCdr8S0Vr9JI6lK3mabVV9BAQ6YPI3zu1s4lEz9lejwG1F kR9EbIVakCvD1osEOUcXOynlkKht/3Adn26xEeIXihCvOo93fyDIcbmnFHFQ5u86QhqY 3tK+2qbp8yylQ8mlF1Y6erhS+6BHEkz+5jFS5dUcbhqVU5D+qP8Be02QtQaSoZYmPFb3 zbJ467/VDSHsq1aue7hUsN41Z/uyG1hnAXZBfKqBkkB0SLp3/Uo89SK8ovbRn0MXCltN UZjS5j+Zgd+pYwCWLJ6XoKOzCrZpQZcLOzD75arrJEE5fTdzmWXnvpn7vWbc48zUewdI eAVA== X-Gm-Message-State: AOJu0Yxrbi4/6Tqv9xGJVwUckI9JY2HSh3jpb4cmrQhPKUFf3s3VDkL+ qTQHwcQ8X0KOEl1C0OV4VddJiekrDpkQUBvvU0n2Iap/fNqHo7aYBxIOj+OJ4AlVkbOFx1FO7lh kvw== X-Google-Smtp-Source: AGHT+IEyFV9cXf/CY6ZxXUeDDPF7xVQGYuDgmn8dUSrnoiU1eyKeiE9Z7M5a9yhOQj+r6LPmuJijye0+/+0= X-Received: from pgtq9.prod.google.com ([2002:a65:6849:0:b0:845:b983:9e34]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ebc6:b0:216:501e:e321 with SMTP id d9443c01a7336-219e6f2700cmr576287415ad.56.1735619749657; Mon, 30 Dec 2024 20:35:49 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:33 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-3-yuzhao@google.com> Subject: [PATCH mm-unstable v4 2/7] mm/mglru: optimize deactivation From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Bharata B Rao , Kalesh Singh X-Rspamd-Queue-Id: 28632100008 X-Rspamd-Server: rspam12 X-Stat-Signature: rsj11znwikngrf94gpogos16wsffm66c X-Rspam-User: X-HE-Tag: 1735619659-986033 X-HE-Meta: U2FsdGVkX19q2w9yUcG/9TaBCAXg6qUcwUVo6oDlqawq3RPAwse/aEF6Yv4Oj3LuMJcFFoKbMGt3KGmuppEA0yIR2nJeU7QY6HeXgjxGq1YPGNc+JuoJsDcTDHLK8AfKU1O6cJs1mJxBJm233DbHeX5RzLvzpMRxJuSNW4/aCe+MrUSbwQXC0KeQskiK1mHfjueDnfLNZjGT91cPnB3dVFCI9P7rWFoQC58Nxfd9iR01JMUhWAPu9P5j4fKTkfgZYT9XabQYKfyilNFrMeGYq3OGiLfkaIFDV6z32mMKEgx7qfj6qhg6kuaC32YtWUy+yKUoLOYXfSg5jp7o8g7GgZufC+HGlZdtv8+BTNlgNztmru0HRlzNfiRm+wdwSD0ETWoes7VskApuNTZKwinjBtEBB9dSzdNnRaXJbErP7XyFsm/Ivh+erM6HUx4skQEcHGAlYT8ISwN7/IcGmRwi5e033HuGtt1GDMjZdigRCT/XJ4r24XKu1A7OIBp91YuITndth8CgmCHlPcMmhHmKtzZeld07GYtDmqCvMTyJSopALCv40lLcsaV6OEhwJZmzgGR06RNDmvbk/NrDk2EXf4oXrht8T4yqcbR3aXTK8DbXTTaEIWa800XRY6YRUkhEiQEswS/dgPDjGy55y4miLw6cN+wZj/KcTEznVNqy8sOat430V996/f2gT8S5Zt9LIMWF4RZWgJkV38XEyVsRV8ZK04VfmC0YxF7YYTPQEyv6kLeO6WNfhP4WbQpl7pNvy/3RZ71DG4qhqqYDZw+631TD7A2D8JDbUeiZKXUBwENsL4f6vc0m7LTwqA336B32Y16dQ3hbBZlTaZnx04qAFG3c4dRH3MS9WaBydgQsY85KmMttuDGikzNf5RRKiGRN75YsDdB59GdkPhtn1DRT1sLEoAvf8TxUYCaPjIRXH6hYIY0KpNTuisLS8dSPbaSbBZDFEFfJmV3QF3gQmSL EpyOpPtF 3KPnM8FbI1JjJOJ3nbu+laH0godHQpTgWJFKXOg6N0WANbiVhSn6SUARGYia5NvR1zmlz2vbab5OJedT8qb3vNaJcFgKrdaQE1c5jzqLFLFoVZ2n+/ExQOHa7c2sjykaWAYxprYg9fC7MuXnResmmLoPwE1AjoNf6fTJR41VtnytE47LbjLBPcG3dXm9L62g0/E+aAQFRMLM6GIhL5gRkCnr0npoZHvlRmgiXgFgJsqqng05f+IO87w7+dbpt7tS3N/49VbX1Xpt4GKKWwWAxd0qWI/bk6Tq9WEzSSWKCiJdnICUOz8kKxZw2+1a5N6Ct1WfSoJrP/Oq7rMIdhkUBh5tSVG2rMb9OO3hr+5lpSiMb7aCdpu4Mm3AX0hmtYnJQ166FNAWbfeMnD3SsMTOGtXlFEFrzYirBRleLZpPc9vYzqNo1Pc/qtdOD8xIyhdVsuez6qwg2c5IpYsrvtu7Cu7F5F4zunRTxCYT7YluhzYvo3VswR4Bl7/XkjPNYgI7xS7u7nbTqhdbjVtbhZkhHG6zoV6tSpAcMELWjOVxU3iM8BGvcVLkWswVKMzqgZAjpdaU63yds6Ei7FzMN363UkuNPfas82rR0c/8qulhTEOvVkgg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not shuffle a folio in the deactivation paths if it is already in the oldest generation. This reduces the LRU lock contention. Before this patch, the contention is reproducible by FIO, e.g., fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \ -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync \ -bs=4k -numjobs=400 -runtime=25000 --time_based \ -group_reporting -name=mglru 98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl After this patch, deactivate_file_folio() bails out early without taking the LRU lock. A side effect is that a folio can be left at the head of the oldest generation, rather than the tail. If reclaim happens at the same time, it cannot reclaim this folio immediately. Since there is no known correlation between truncation and reclaim, this side effect is considered insignificant. Reported-by: Bharata B Rao Closes: https://lore.kernel.org/CAOUHufawNerxqLm7L9Yywp3HJFiYVrYO26ePUb1jH-qxNGWzyA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/swap.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3a01acfd5a89..649ef7f2b74b 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -379,7 +379,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } #ifdef CONFIG_LRU_GEN -static void folio_inc_refs(struct folio *folio) + +static void lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); @@ -406,10 +407,34 @@ static void folio_inc_refs(struct folio *folio) new_flags |= old_flags & ~LRU_REFS_MASK; } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } -#else -static void folio_inc_refs(struct folio *folio) + +static bool lru_gen_clear_refs(struct folio *folio) { + struct lru_gen_folio *lrugen; + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + + if (gen < 0) + return true; + + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + + lrugen = &folio_lruvec(folio)->lrugen; + /* whether can do without shuffling under the LRU lock */ + return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); } + +#else /* !CONFIG_LRU_GEN */ + +static void lru_gen_inc_refs(struct folio *folio) +{ +} + +static bool lru_gen_clear_refs(struct folio *folio) +{ + return false; +} + #endif /* CONFIG_LRU_GEN */ /** @@ -428,7 +453,7 @@ static void folio_inc_refs(struct folio *folio) void folio_mark_accessed(struct folio *folio) { if (lru_gen_enabled()) { - folio_inc_refs(folio); + lru_gen_inc_refs(folio); return; } @@ -524,7 +549,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) */ static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio) { - bool active = folio_test_active(folio); + bool active = folio_test_active(folio) || lru_gen_enabled(); long nr_pages = folio_nr_pages(folio); if (folio_test_unevictable(folio)) @@ -589,7 +614,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) lruvec_del_folio(lruvec, folio); folio_clear_active(folio); - folio_clear_referenced(folio); + if (lru_gen_enabled()) + lru_gen_clear_refs(folio); + else + folio_clear_referenced(folio); /* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal @@ -657,6 +685,9 @@ void deactivate_file_folio(struct folio *folio) if (folio_test_unevictable(folio)) return; + if (lru_gen_enabled() && lru_gen_clear_refs(folio)) + return; + folio_batch_add_and_move(folio, lru_deactivate_file, true); } @@ -670,7 +701,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled())) + if (folio_test_unevictable(folio)) + return; + + if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio)) return; folio_batch_add_and_move(folio, lru_deactivate, true); From patchwork Tue Dec 31 04:35:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F18B7E77188 for ; Tue, 31 Dec 2024 04:35:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 829086B0089; Mon, 30 Dec 2024 23:35:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B0746B008A; Mon, 30 Dec 2024 23:35:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DC436B008C; Mon, 30 Dec 2024 23:35:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 366A56B0089 for ; Mon, 30 Dec 2024 23:35:56 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A2BD280CEB for ; Tue, 31 Dec 2024 04:35:55 +0000 (UTC) X-FDA: 82953988152.17.9F34C52 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf14.hostedemail.com (Postfix) with ESMTP id 26F65100009 for ; Tue, 31 Dec 2024 04:35:00 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hI4sH8yC; spf=pass (imf14.hostedemail.com: domain of 3qHRzZwYKCKEZVaIBPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuzhao.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3qHRzZwYKCKEZVaIBPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619709; a=rsa-sha256; cv=none; b=8gV0B8sUxwYMQ8HRWv9vuWO+dhzJsh/qRqMszL94huO3DhOPYQkAM3TQ9ovyK4zNRZnzBM 6qOMcFbVOLLyt0NStAvwVLRI7gZ3kZMGYS22V1dtx8OvqMFLAo/Vvt0Qrw9IcP2Cgyk+3D XgzGhLcit7Aq3ssSiP/c5tj7GXSHax0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hI4sH8yC; spf=pass (imf14.hostedemail.com: domain of 3qHRzZwYKCKEZVaIBPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuzhao.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3qHRzZwYKCKEZVaIBPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sDhUReo2deRGp+tnJMIYJBVB2YFMdmBRAUrx9f/kOiA=; b=Y3DaxDC/sfj3IZa7aEfbZ5xv8yKlFtmlj/PXHEMjtC+soe3VZ/Eovf7eQUkT3/YHDqstNF tQzgWguNu8DgcG5qZBux/frAX9qfNPf6ID8q2ph+mRDBu/OqY0ZRpL2hXP258hYv8xebGL xgrynSyZc4TP6Yr1mofAiVTbKC3u15s= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ee5668e09bso13415044a91.3 for ; Mon, 30 Dec 2024 20:35:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619752; x=1736224552; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sDhUReo2deRGp+tnJMIYJBVB2YFMdmBRAUrx9f/kOiA=; b=hI4sH8yCVDY2L1KZVsW/NfwMxWYNVhB37+xGHmRD3W4R8wEQq75vdUbbKSnBh0Xprr 9UAxjL7EjZr03FdKhOf5GzkHUKgQF2ZmRGl4Y8jZXbqD9i12iJOlwPpJD1VDjstFE+PD 6qP3N6E8L0vZ4xcOecojXYoNxlbjcgAfGziWgt8FjRJdkscWOo8nh298c0Bx5/gj3ZXo IHNI7tR/OuLhRNRHPelTxrmBduPseOIvF7H/CZAMmijU5Kz8c/vVyeUIM1+MkOHiXoYD zItAEqy3NVcW7zXDUIoTG5Jy+W2wksSBUQxuLsxSS04/4pkRuh4CXmhezKZFvtrhT3Xm SX5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619752; x=1736224552; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sDhUReo2deRGp+tnJMIYJBVB2YFMdmBRAUrx9f/kOiA=; b=pR4MUjfkER/q4WeNmmEJ2nHV5W/Uz4pM4ZsMKOhY4Yutqz1z26xFEln514Z07Jafrv SEzvf6MtWhSk4rp94XbhLZWy4RXl+WXa0SrHt0dhXwT/9qAPdxu2iZ0wO4EF2gLDGrjH jnKftjoKgN0jNc1DL1so8gHemECZt6snAweUOTPI/RaaeOwrevXIbCYr+/eaTwnYMZbB kXCCGB81+KyFha2W8V2KSfdD+b+poY/oqBvB779iVSDZlM4i+cTDMH7L1lIkR63h3nXM exGWtn1w57umwDIehhohLd9GoMEwHkitfdo+4pzwmeMguXi9DllUluxm8BT2p1RAfbVN mODA== X-Gm-Message-State: AOJu0YwtDaoUeCqlqvHvpfeTfJmiVD97btWgFlfGpep94hZAqLh0sLAi AMFucsopUKA98FPVDSvJwIbITM/T4b1cj+GYquRtbTL+Ikg9m2C76DWLc529sJSnDlYT8oJmwT8 0xA== X-Google-Smtp-Source: AGHT+IG6nxstyXgWqSu+L2n9Zxz+hgGy42XY3ZsLF+ONO7heKN9bb4QBB98vkqXP6rvyE1kIrVxQ7Qx/GdU= X-Received: from pfxa12.prod.google.com ([2002:a05:6a00:1d0c:b0:725:d350:a304]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:8d8d:b0:725:e37d:cd36 with SMTP id d2e1a72fcca58-72abdd3c476mr58090892b3a.2.1735619752645; Mon, 30 Dec 2024 20:35:52 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:34 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-4-yuzhao@google.com> Subject: [PATCH mm-unstable v4 3/7] mm/mglru: rework aging feedback From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , David Stevens , Kalesh Singh X-Rspamd-Queue-Id: 26F65100009 X-Stat-Signature: iwen94tbgeatw9adoqwcsqxi8p7z77xe X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735619700-136074 X-HE-Meta: U2FsdGVkX18t0rYB2jaJvMUV1z6D0mRMo/ViE/aYBioHyhFkBm5t0hWhAfHbZXgNsrS/inpfv5GMT5OfjE69WEZ7YdiEaVBNh2WMW7ojgixO6i/DAf4LKLXyxSElVi6bERvacPN8bTjP16hw9mK0MDzvP/VOM+JuFKaZvHCIhUyzyLnwLk7j8VbhW6YjGx6fniMzCyCcY5eS3zJkLmoU0X7F6wNbpOiP8jUT/LaXk6rI/QOsdK65O9/iKlvN8YMVpXcuXuM0csKJkWta0Jl3RM/b5j7fpS2+nEe3d09LLOuhRQCR6rAlmulW4Gn6mDnnx23YZH5LVm7kbvKm/xrrbJBK0e0cK/JdQAwzhG/RH4rRBktr552eY5m5tTpaXda2Ca2/djIrYTHZb1cxjeM2UCJV0fPyQ1okT+A1A2BX2V80ormCbPY7gWEf3P+Vzy1Dlh3sG14ptXRoBIKAr/96C9LME0p1D04mv1Ch0aT7TmB5/cP3ATm5m/Pu+NiFVp7Cd0mNXWJp9ql1+buk3zj0ZeNMAnWSHpFZhhWVjYpYeTQ5oNX9R0Y2fRUp0IxSC1+wn6HwFjJBl0twuaGBlhCW695j83GctNeV7VtXdmDURU8fBJz4HDFb6nq3t0rIzXt30BxLA+3TjcjQ9LohrHw283fMuxEicVoBj3Pg9XEtqNZTd9U147yvU2lomgzLDOcl1x17QNaJjbzt6VL3J+gm2q0BtSQLA+x1JnOv/Eps+IhjnFH9+GMk+A2UXFPePC3z7/hs0m9jiYFMBmK9wHogUHJROtbC3x/FcZ6XgbKLNi5K18AhN4pONeWwELT+/dcpZikOqKGt7mTTeAzhuTIfIynno7AtgEUjqpqtxhcaqAwTEHYdYNJLK+9VM9bsSYyGvh3iVztNsO3vYzhiuzesRWDVFb4gL75Kj62HjZfKEiKHhgFekGiK0jupa7REFAIpI4qdfdn3J4CUS35PHV0 YTVfbWm2 wxAFNeGwOEO+WXKgJ7pqsyu/9kLsDC9fFAKXEs/8AxBsjOMcWUjCPF8QJHJzQ2KbMks/KC2AVCwuofpgVa2Lyf8FGQRO774F17SEA9B+9G4OsIuUvfmC7A+JGXsvfOmiAyfkURyBlx6nVyPpjXm0T+XeWbFxThcFK8p7VAJS9hCXl0IWzo60C+MhplUzm1y9AxmHj4SqbpilthA6Sm4UWzQAveSjc4Y9sBU/RexZoW2MAyKsHhyV6n88qApwmARjkv86Ov17vxRuVa0SGF7Qcma6SmEu8SwbWyD8WQoUBL3z3ZGv7u/LjXXs9gsU5jtoDUXNW1mpLo8PHNL1wb3jQA2cZsvHEboag6oS6sJsn3sBmJkNS34XYDyZNdiKX0eqmU6UF5UVjRBPwY+Y3ctbmcjMgLIAMQwuIHlqHgSVXKbwjs7RCFWxCaK37VdCXf170HroZeslFpU1VQG5m5jiQGl/CsF6Jo7wSdqRO3RsYnCXSLAJ+tNw3Z2wqnCU00iTAW0LJK8uL/+TJNWrrdecXtqDEsHcYn3LieCwwrcHlac8RLD8SJSt79nkpCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The aging feedback is based on both the number of generations and the distribution of folios in each generation. The number of generations is currently the distance between max_seq and anon min_seq. This is because anon min_seq is not allowed to move past file min_seq. The rationale for that is that file is always evictable whereas anon is not. However, for use cases where anon is a lot cheaper than file: 1. Anon in the second oldest generation can be a better choice than file in the oldest generation. 2. A large amount of file in the oldest generation can skew the distribution, making should_run_aging() return false negative. Allow anon and file min_seq to move independently, and use solely the number of generations as the feedback for aging. Specifically, when both anon and file are evictable, anon min_seq can now be greater than file min_seq, and therefore the number of generations becomes the distance between max_seq and min(min_seq[0],min_seq[1]). And should_run_aging() returns true if and only if the number of generations is less than MAX_NR_GENS. As the first step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. The next twos patch will take advantage of this change; the last patch in this series will better distribute folios across MAX_NR_GENS. Reported-by: David Stevens Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- include/linux/mmzone.h | 17 ++-- mm/vmscan.c | 200 ++++++++++++++++++----------------------- 2 files changed, 96 insertions(+), 121 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..8245ecb0400b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -421,12 +421,11 @@ enum { /* * The youngest generation number is stored in max_seq for both anon and file * types as they are aged on an equal footing. The oldest generation numbers are - * stored in min_seq[] separately for anon and file types as clean file pages - * can be evicted regardless of swap constraints. - * - * Normally anon and file min_seq are in sync. But if swapping is constrained, - * e.g., out of swap space, file min_seq is allowed to advance and leave anon - * min_seq behind. + * stored in min_seq[] separately for anon and file types so that they can be + * incremented independently. Ideally min_seq[] are kept in sync when both anon + * and file types are evictable. However, to adapt to situations like extreme + * swappiness, they are allowed to be out of sync by at most + * MAX_NR_GENS-MIN_NR_GENS-1. * * The number of pages in each generation is eventually consistent and therefore * can be transiently negative when reset_batch_size() is pending. @@ -446,8 +445,8 @@ struct lru_gen_folio { unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS]; /* the exponential moving average of evicted+protected */ unsigned long avg_total[ANON_AND_FILE][MAX_NR_TIERS]; - /* the first tier doesn't need protection, hence the minus one */ - unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS - 1]; + /* can only be modified under the LRU lock */ + unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; /* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; @@ -498,7 +497,7 @@ struct lru_gen_mm_walk { int mm_stats[NR_MM_STATS]; /* total batched items */ int batched; - bool can_swap; + int swappiness; bool force_scan; }; diff --git a/mm/vmscan.c b/mm/vmscan.c index f236db86de8a..f767e3d34e73 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2627,11 +2627,17 @@ static bool should_clear_pmd_young(void) READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_FILE]), \ } +#define evictable_min_seq(min_seq, swappiness) \ + min((min_seq)[!(swappiness)], (min_seq)[(swappiness) != MAX_SWAPPINESS]) + #define for_each_gen_type_zone(gen, type, zone) \ for ((gen) = 0; (gen) < MAX_NR_GENS; (gen)++) \ for ((type) = 0; (type) < ANON_AND_FILE; (type)++) \ for ((zone) = 0; (zone) < MAX_NR_ZONES; (zone)++) +#define for_each_evictable_type(type, swappiness) \ + for ((type) = !(swappiness); (type) <= ((swappiness) != MAX_SWAPPINESS); (type)++) + #define get_memcg_gen(seq) ((seq) % MEMCG_NR_GENS) #define get_memcg_bin(bin) ((bin) % MEMCG_NR_BINS) @@ -2677,10 +2683,16 @@ static int get_nr_gens(struct lruvec *lruvec, int type) static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) { - /* see the comment on lru_gen_folio */ - return get_nr_gens(lruvec, LRU_GEN_FILE) >= MIN_NR_GENS && - get_nr_gens(lruvec, LRU_GEN_FILE) <= get_nr_gens(lruvec, LRU_GEN_ANON) && - get_nr_gens(lruvec, LRU_GEN_ANON) <= MAX_NR_GENS; + int type; + + for (type = 0; type < ANON_AND_FILE; type++) { + int n = get_nr_gens(lruvec, type); + + if (n < MIN_NR_GENS || n > MAX_NR_GENS) + return false; + } + + return true; } /****************************************************************************** @@ -3087,9 +3099,8 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, pos->refaulted = lrugen->avg_refaulted[type][tier] + atomic_long_read(&lrugen->refaulted[hist][type][tier]); pos->total = lrugen->avg_total[type][tier] + + lrugen->protected[hist][type][tier] + atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - pos->total += lrugen->protected[hist][type][tier - 1]; pos->gain = gain; } @@ -3116,17 +3127,15 @@ static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); sum = lrugen->avg_total[type][tier] + + lrugen->protected[hist][type][tier] + atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - sum += lrugen->protected[hist][type][tier - 1]; WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); } if (clear) { atomic_long_set(&lrugen->refaulted[hist][type][tier], 0); atomic_long_set(&lrugen->evicted[hist][type][tier], 0); - if (tier) - WRITE_ONCE(lrugen->protected[hist][type][tier - 1], 0); + WRITE_ONCE(lrugen->protected[hist][type][tier], 0); } } } @@ -3261,7 +3270,7 @@ static int should_skip_vma(unsigned long start, unsigned long end, struct mm_wal return true; if (vma_is_anonymous(vma)) - return !walk->can_swap; + return !walk->swappiness; if (WARN_ON_ONCE(!vma->vm_file || !vma->vm_file->f_mapping)) return true; @@ -3271,7 +3280,10 @@ static int should_skip_vma(unsigned long start, unsigned long end, struct mm_wal return true; if (shmem_mapping(mapping)) - return !walk->can_swap; + return !walk->swappiness; + + if (walk->swappiness == MAX_SWAPPINESS) + return true; /* to exclude special mappings like dax, etc. */ return !mapping->a_ops->read_folio; @@ -3359,7 +3371,7 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned } static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, - struct pglist_data *pgdat, bool can_swap) + struct pglist_data *pgdat) { struct folio *folio; @@ -3370,10 +3382,6 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, if (folio_memcg(folio) != memcg) return NULL; - /* file VMAs can contain anon pages from COW */ - if (!folio_is_file_lru(folio) && !can_swap) - return NULL; - return folio; } @@ -3429,7 +3437,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (pfn == -1) continue; - folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); + folio = get_pfn_folio(pfn, memcg, pgdat); if (!folio) continue; @@ -3514,7 +3522,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area if (pfn == -1) goto next; - folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); + folio = get_pfn_folio(pfn, memcg, pgdat); if (!folio) goto next; @@ -3726,22 +3734,26 @@ static void clear_mm_walk(void) kfree(walk); } -static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) +static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) { int zone; int remaining = MAX_LRU_BATCH; struct lru_gen_folio *lrugen = &lruvec->lrugen; + int hist = lru_hist_from_seq(lrugen->min_seq[type]); int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); - if (type == LRU_GEN_ANON && !can_swap) + if (type ? swappiness == MAX_SWAPPINESS : !swappiness) goto done; - /* prevent cold/hot inversion if force_scan is true */ + /* prevent cold/hot inversion if the type is evictable */ for (zone = 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head = &lrugen->folios[old_gen][type][zone]; while (!list_empty(head)) { struct folio *folio = lru_to_folio(head); + int refs = folio_lru_refs(folio); + int tier = lru_tier_from_refs(refs); + int delta = folio_nr_pages(folio); VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); @@ -3751,6 +3763,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) new_gen = folio_inc_gen(lruvec, folio, false); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + delta); + if (!--remaining) return false; } @@ -3762,7 +3777,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) return true; } -static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) +static bool try_to_inc_min_seq(struct lruvec *lruvec, int swappiness) { int gen, type, zone; bool success = false; @@ -3772,7 +3787,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); /* find the oldest populated generation */ - for (type = !can_swap; type < ANON_AND_FILE; type++) { + for_each_evictable_type(type, swappiness) { while (min_seq[type] + MIN_NR_GENS <= lrugen->max_seq) { gen = lru_gen_from_seq(min_seq[type]); @@ -3788,13 +3803,17 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) } /* see the comment on lru_gen_folio */ - if (can_swap) { - min_seq[LRU_GEN_ANON] = min(min_seq[LRU_GEN_ANON], min_seq[LRU_GEN_FILE]); - min_seq[LRU_GEN_FILE] = max(min_seq[LRU_GEN_ANON], lrugen->min_seq[LRU_GEN_FILE]); + if (swappiness && swappiness != MAX_SWAPPINESS) { + unsigned long seq = lrugen->max_seq - MIN_NR_GENS; + + if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq) + min_seq[LRU_GEN_ANON] = seq; + else if (min_seq[LRU_GEN_FILE] > seq && min_seq[LRU_GEN_ANON] < seq) + min_seq[LRU_GEN_FILE] = seq; } - for (type = !can_swap; type < ANON_AND_FILE; type++) { - if (min_seq[type] == lrugen->min_seq[type]) + for_each_evictable_type(type, swappiness) { + if (min_seq[type] <= lrugen->min_seq[type]) continue; reset_ctrl_pos(lruvec, type, true); @@ -3805,8 +3824,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) return success; } -static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, - bool can_swap, bool force_scan) +static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness) { bool success; int prev, next; @@ -3824,13 +3842,11 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, if (!success) goto unlock; - for (type = ANON_AND_FILE - 1; type >= 0; type--) { + for (type = 0; type < ANON_AND_FILE; type++) { if (get_nr_gens(lruvec, type) != MAX_NR_GENS) continue; - VM_WARN_ON_ONCE(!force_scan && (type == LRU_GEN_FILE || can_swap)); - - if (inc_min_seq(lruvec, type, can_swap)) + if (inc_min_seq(lruvec, type, swappiness)) continue; spin_unlock_irq(&lruvec->lru_lock); @@ -3874,7 +3890,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, } static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, - bool can_swap, bool force_scan) + int swappiness, bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -3885,7 +3901,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, VM_WARN_ON_ONCE(seq > READ_ONCE(lrugen->max_seq)); if (!mm_state) - return inc_max_seq(lruvec, seq, can_swap, force_scan); + return inc_max_seq(lruvec, seq, swappiness); /* see the comment in iterate_mm_list() */ if (seq <= READ_ONCE(mm_state->seq)) @@ -3910,7 +3926,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, walk->lruvec = lruvec; walk->seq = seq; - walk->can_swap = can_swap; + walk->swappiness = swappiness; walk->force_scan = force_scan; do { @@ -3920,7 +3936,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, } while (mm); done: if (success) { - success = inc_max_seq(lruvec, seq, can_swap, force_scan); + success = inc_max_seq(lruvec, seq, swappiness); WARN_ON_ONCE(!success); } @@ -3961,13 +3977,13 @@ static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) { int gen, type, zone; unsigned long total = 0; - bool can_swap = get_swappiness(lruvec, sc); + int swappiness = get_swappiness(lruvec, sc); struct lru_gen_folio *lrugen = &lruvec->lrugen; struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); - for (type = !can_swap; type < ANON_AND_FILE; type++) { + for_each_evictable_type(type, swappiness) { unsigned long seq; for (seq = min_seq[type]; seq <= max_seq; seq++) { @@ -3987,6 +4003,7 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc { int gen; unsigned long birth; + int swappiness = get_swappiness(lruvec, sc); struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MIN_SEQ(lruvec); @@ -3996,8 +4013,7 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc if (!lruvec_is_sizable(lruvec, sc)) return false; - /* see the comment on lru_gen_folio */ - gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); + gen = lru_gen_from_seq(evictable_min_seq(min_seq, swappiness)); birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); return time_is_before_jiffies(birth + min_ttl); @@ -4064,7 +4080,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) unsigned long addr = pvmw->address; struct vm_area_struct *vma = pvmw->vma; struct folio *folio = pfn_folio(pvmw->pfn); - bool can_swap = !folio_is_file_lru(folio); struct mem_cgroup *memcg = folio_memcg(folio); struct pglist_data *pgdat = folio_pgdat(folio); struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); @@ -4117,7 +4132,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (pfn == -1) continue; - folio = get_pfn_folio(pfn, memcg, pgdat, can_swap); + folio = get_pfn_folio(pfn, memcg, pgdat); if (!folio) continue; @@ -4333,8 +4348,8 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c gen = folio_inc_gen(lruvec, folio, false); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); - WRITE_ONCE(lrugen->protected[hist][type][tier - 1], - lrugen->protected[hist][type][tier - 1] + delta); + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + delta); return true; } @@ -4533,7 +4548,6 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw { int i; int type; - int scanned; int tier = -1; DEFINE_MIN_SEQ(lruvec); @@ -4558,21 +4572,23 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw else type = get_type_to_scan(lruvec, swappiness, &tier); - for (i = !swappiness; i < ANON_AND_FILE; i++) { + for_each_evictable_type(i, swappiness) { + int scanned; + if (tier < 0) tier = get_tier_idx(lruvec, type); + *type_scanned = type; + scanned = scan_folios(lruvec, sc, type, tier, list); if (scanned) - break; + return scanned; type = !type; tier = -1; } - *type_scanned = type; - - return scanned; + return 0; } static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness) @@ -4588,6 +4604,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap struct reclaim_stat stat; struct lru_gen_mm_walk *walk; bool skip_retry = false; + struct lru_gen_folio *lrugen = &lruvec->lrugen; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -4597,7 +4614,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap scanned += try_to_inc_min_seq(lruvec, swappiness); - if (get_nr_gens(lruvec, !swappiness) == MIN_NR_GENS) + if (evictable_min_seq(lrugen->min_seq, swappiness) + MIN_NR_GENS > lrugen->max_seq) scanned = 0; spin_unlock_irq(&lruvec->lru_lock); @@ -4669,63 +4686,32 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap } static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, - bool can_swap, unsigned long *nr_to_scan) + int swappiness, unsigned long *nr_to_scan) { int gen, type, zone; - unsigned long old = 0; - unsigned long young = 0; - unsigned long total = 0; + unsigned long size = 0; struct lru_gen_folio *lrugen = &lruvec->lrugen; DEFINE_MIN_SEQ(lruvec); - /* whether this lruvec is completely out of cold folios */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { - *nr_to_scan = 0; + *nr_to_scan = 0; + /* have to run aging, since eviction is not possible anymore */ + if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; - } - for (type = !can_swap; type < ANON_AND_FILE; type++) { + for_each_evictable_type(type, swappiness) { unsigned long seq; for (seq = min_seq[type]; seq <= max_seq; seq++) { - unsigned long size = 0; - gen = lru_gen_from_seq(seq); for (zone = 0; zone < MAX_NR_ZONES; zone++) size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - - total += size; - if (seq == max_seq) - young += size; - else if (seq + MIN_NR_GENS == max_seq) - old += size; } } - *nr_to_scan = total; - - /* - * The aging tries to be lazy to reduce the overhead, while the eviction - * stalls when the number of generations reaches MIN_NR_GENS. Hence, the - * ideal number of generations is MIN_NR_GENS+1. - */ - if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - return false; - - /* - * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) - * of the total number of pages for each generation. A reasonable range - * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * aging cares about the upper bound of hot pages, while the eviction - * cares about the lower bound of cold pages. - */ - if (young * MIN_NR_GENS > total) - return true; - if (old * (MIN_NR_GENS + 2) < total) - return true; - - return false; + *nr_to_scan = size; + /* better to run aging even though eviction is still possible */ + return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq; } /* @@ -4733,7 +4719,7 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg * reclaim. */ -static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool can_swap) +static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, int swappiness) { bool success; unsigned long nr_to_scan; @@ -4743,7 +4729,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) return -1; - success = should_run_aging(lruvec, max_seq, can_swap, &nr_to_scan); + success = should_run_aging(lruvec, max_seq, swappiness, &nr_to_scan); /* try to scrape all its memory if this memcg was deleted */ if (nr_to_scan && !mem_cgroup_online(memcg)) @@ -4754,7 +4740,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool return nr_to_scan >> sc->priority; /* stop scanning this lruvec as it's low on cold folios */ - return try_to_inc_max_seq(lruvec, max_seq, can_swap, false) ? -1 : 0; + return try_to_inc_max_seq(lruvec, max_seq, swappiness, false) ? -1 : 0; } static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) @@ -5298,8 +5284,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, s = "rep"; n[0] = atomic_long_read(&lrugen->refaulted[hist][type][tier]); n[1] = atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - n[2] = READ_ONCE(lrugen->protected[hist][type][tier - 1]); + n[2] = READ_ONCE(lrugen->protected[hist][type][tier]); } for (i = 0; i < 3; i++) @@ -5354,7 +5339,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v) seq_printf(m, " node %5d\n", nid); if (!full) - seq = min_seq[LRU_GEN_ANON]; + seq = evictable_min_seq(min_seq, MAX_SWAPPINESS / 2); else if (max_seq >= MAX_NR_GENS) seq = max_seq - MAX_NR_GENS + 1; else @@ -5394,23 +5379,14 @@ static const struct seq_operations lru_gen_seq_ops = { }; static int run_aging(struct lruvec *lruvec, unsigned long seq, - bool can_swap, bool force_scan) + int swappiness, bool force_scan) { DEFINE_MAX_SEQ(lruvec); - DEFINE_MIN_SEQ(lruvec); - - if (seq < max_seq) - return 0; if (seq > max_seq) return -EINVAL; - if (!force_scan && min_seq[!can_swap] + MAX_NR_GENS - 1 <= max_seq) - return -ERANGE; - - try_to_inc_max_seq(lruvec, max_seq, can_swap, force_scan); - - return 0; + return try_to_inc_max_seq(lruvec, max_seq, swappiness, force_scan) ? 0 : -EEXIST; } static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_control *sc, @@ -5426,7 +5402,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co while (!signal_pending(current)) { DEFINE_MIN_SEQ(lruvec); - if (seq < min_seq[!swappiness]) + if (seq < evictable_min_seq(min_seq, swappiness)) return 0; if (sc->nr_reclaimed >= nr_to_reclaim) From patchwork Tue Dec 31 04:35:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E920E77194 for ; Tue, 31 Dec 2024 04:35:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D332F6B008C; Mon, 30 Dec 2024 23:35:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CBB336B0092; Mon, 30 Dec 2024 23:35:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B35E36B0095; Mon, 30 Dec 2024 23:35:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 86A986B008C for ; Mon, 30 Dec 2024 23:35:58 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 327BD479AC for ; Tue, 31 Dec 2024 04:35:58 +0000 (UTC) X-FDA: 82953990168.24.72D6A55 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf16.hostedemail.com (Postfix) with ESMTP id 26199180002 for ; Tue, 31 Dec 2024 04:35:07 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BjFitxHV; spf=pass (imf16.hostedemail.com: domain of 3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lq6dqRxgSDV3CyACy0+VqeWU26YT+ayL2zVnCSsSGnY=; b=03uvcpvH0hyK+V2rMccciNmn1u+TnejU1Vjd6/TcgstP0mlmvIxM/JawnPXCQOGG0OT2v5 8enW7dxz3C7r/VJ9zWY38RVdTqelH3v9w0jFUQ20Dbv8Bgrz6l2GtYUzYx+xahhZfF3wBf JTeuq2y8GB00Xx+VleVBOSyKc67TDno= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BjFitxHV; spf=pass (imf16.hostedemail.com: domain of 3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3q3RzZwYKCKQcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619734; a=rsa-sha256; cv=none; b=jsN5BtXf4WZ9rWRDeSmM96gbTl5fI+cVYfYOTVmyY3KhnyPkuggbyvEEfCUOfNLVG+baxF j989PwUnV+1DC2+EEAB3DV96nb0MAXZkgROjJ1rzT7N+N5qWDJFXh1DEnPa5xjbC6GdH9w wiB7NdfEqtJfYlVBJW4Qm1HsyZwFrVo= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef9da03117so20659268a91.1 for ; Mon, 30 Dec 2024 20:35:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619755; x=1736224555; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lq6dqRxgSDV3CyACy0+VqeWU26YT+ayL2zVnCSsSGnY=; b=BjFitxHVSboLhHSItW7LAyEzsQX9Wn4o0iMCv9yS7iUC+CoaREXNfy5fX9Qt9rDMNq /u/BaViI1VivEXDwnGE6Mh+zDXD6fg7P8XqWzCsZseAfoFBwUa9DuJu/FIwoK6gc2M7J y1tuZE7S/Teuc0ouKg2onZMECVK90FppNmsIuu23rB+zHJiiW0I6Dif9zeU9+hU6WGj1 yPANOzNBkXGwJHb0C7+0nsOxtuV5/qnwwhWBtuWSyWbZhP7wH88UGEH4PZglxGHWq2in Ln7HIg2mR6SrBxxRLQpfCVE13DAtbDYZMDgU5XMT1B3+ICLKfeQcSUoa9uKh69ltRO8Z zYBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619755; x=1736224555; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lq6dqRxgSDV3CyACy0+VqeWU26YT+ayL2zVnCSsSGnY=; b=gH1ypXTNYqq9gBkanr0Qvgaq/dj+WN60m1KT73WgMle7STcgm0pg2r9KVUs66dgDCN 2oFowwiab3Qubnk+KlQ2KlaGIruBC0eIGinZdqK/9qGzsAh+1o35o2s7UE8KdTYeBWD3 EH7/E+UB0DhfFGAUDRKFcb3bH5caMZng8A6HOSiqO2uD+xa90TFwIlzzsZK64SEm2B8P KJgLfVwpcj8n3K6uWw8M30mJPsXDsuaoPOlhc3qulnJ6Sy81STBnSxvD4aQvduZroTBF +g2/Q2lKd9ylWyMx5+7qPtsbW5uQrbXkuHhBHAM8xLBLip4BTC4FRCy4uTdQW1CIH48s Pf1Q== X-Gm-Message-State: AOJu0YwmSfIxHftCB4vg8fNcNzOjMue3Uh6EqPYWMvkqE6qqCGIcft2z nt/g110KCEYNiG64G8qbD2GB4Gapl6Tw8muZIXucUw6flrY83iK/FNlSY+/uitFWbLPefVmrjHt 2DA== X-Google-Smtp-Source: AGHT+IHeZyv9yGkkGSeakHyyxsc/LoAVc3WVdX/rKDrbygxdcMqZjnwE7jl/rLSwE6a6nOYHrpf3ke3Jfww= X-Received: from pfd7.prod.google.com ([2002:a05:6a00:a807:b0:727:2d74:d385]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:430d:b0:725:db34:6a8c with SMTP id d2e1a72fcca58-72abddbd4f7mr55420073b3a.13.1735619755260; Mon, 30 Dec 2024 20:35:55 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:35 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-5-yuzhao@google.com> Subject: [PATCH mm-unstable v4 4/7] mm/mglru: rework type selection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , David Stevens , Kalesh Singh X-Rspamd-Queue-Id: 26199180002 X-Stat-Signature: 1mcfbawzpto7eh9tjcszga3ucpcu4jam X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735619707-914493 X-HE-Meta: U2FsdGVkX1/ZH7JIpiXhbKUvwpbOer8TK3ceLE8VmWnyf9RGvSgQkWxNxqAxL9NTn/OiNur2ZTRQRO3zzM/Irg2VjnfBkCjjRL1Ad4JZ1o179UQUhvr+J+GdXVqriVwZ/TARp9YGp6BKDhjLXpETYcqN6aZ0n1y30IDVfHiBzRbOkg5QibaXCngRydxLIFwnahJap1fiXgWl0lSP0SWe3IpYi6/9SclFp4JmgGToOzvs1XbIwrPv5aIxYnaOfs6Xi2vT1ugkBs66lMWsBVGmPh/POwKmEbDRHRs4Urbz5TxYCU4/ogyOhqheAXdAKsk9eZsjldLu4LvZP9oClUY4F67Og3K9CY7SkynvneDh5I97bJaBiFY7QHKNHuCUr5KTbFI1KY1Zmj8RWKMdd5DlOR6oDd6XKrvoHAFQHRL/lEjskz7NoaB/GsnoN4UMwNt/uWF/pMTOoWf3vGXjj6sZ6KA/N1wllMyp8x0DBnr5dYcLY/AT85NVZX6bUi7Br1jvs5NUJGmvgZtNNWiS+MsvuodOZaMOAt/0cSJ7kiks0Ak4v6S4869iAnozYQVGd6xlMq0ET9pufEN8wKDheLe8nkhRfEV/xI1qcw2ndiJkBebQk9UG2woLdfKF2gsLOZ7adMb8aNNx6VwS0s9ZUPc0m/LNQE0dD6dtS1VfuHscVmpnqBzUkaC5JuR3TZFmTXxsS4Q9Fmz6SGTunZxKI4lsyLE+vS9D5Lh44cEaR3OE+4DDcahZSN7F59Bq6qXJz0F1IG77DR7ybZA2uDzXHYypYODbF/KL2yN3Lrb2y35aPlTsb9v7IwDmN82nVer0aq8AnRM5H2IYczXKXuwC0AVvc3sVNl7qDfqnneqQ1DeQjD4esRff3Pc0ZI9R9Z/8waUkMZtCmzeK5WhaXa5Wu2q7KXivdP6aM2cjPi5evxfoO9gi5WbI5z2b4r+U6OoEVxAHYo6nCwD4cfTReYr+JJ4 i2JR1hC/ C4XMqUfC8Xm9S0G5CKTfO62rUiu71iqqHpCYHicIYp7EsET9spOm9Wfa59B44jGGfAplT2/PKj9/8oA9rKmARtmNT5u83cNdJX4+KCzUH1/Zb70WntcJhG69lAUToRtp6qerjAL8kPJMHXS2LFCD25k+7pGYineka60ePt8a3AXH6KW57ILaI1g0VjoubppH/WDwjWjmOSNGJZoaPp9sCnCOxd51dbLTQSmWvW+sOeFEw5nNwaVj6Xjl99w2+2Pj+fF+pJ9RbaCvM6idOK4+XWcCZPSrdk+nzk2WNcWJL0l5J2B+wnLx9F1koF6w3GzxYAB/Ur6usW903X+XWqjJP4UgEbmiI/RQjYNy5QORhUes4xGM0Y5dv2NBoB8xVyytg54G7hQKSU5m8NuHwHzreHJkb8jojQ4Os8nrpo3yzNXzcZhzaTIs5KdiTwIx3G8tBBc5H0NT2Lx2xkduF/OoOrGF10Yt+8RRwp2zrYlCLD7UbxHbkZ0/c3e8nRpzMsJckgUhkQjgcGhgkkY8oGe5kucz7vsT0IpiI+jpQ83hle4uhAGAWJrYhpjgnfQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000068, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework type selection so that it is based on the total refaults from all tiers of each type. Also allow a type to be selected until that type reaches MIN_NR_GENS, regardless of whether that type has a larger min_seq or not, to accommodate extreme swappiness. Since some tiers of a selected type can have higher refaults than the first tier of the other type, use a less larger gain factor 2:3 instead of 1:2, in order for those tiers in the selected type to be better protected. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: David Stevens Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/vmscan.c | 82 +++++++++++++++++------------------------------------ 1 file changed, 26 insertions(+), 56 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f767e3d34e73..a33221298fd0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3093,15 +3093,20 @@ struct ctrl_pos { static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, struct ctrl_pos *pos) { + int i; struct lru_gen_folio *lrugen = &lruvec->lrugen; int hist = lru_hist_from_seq(lrugen->min_seq[type]); - pos->refaulted = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + - lrugen->protected[hist][type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); pos->gain = gain; + pos->refaulted = pos->total = 0; + + for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) { + pos->refaulted += lrugen->avg_refaulted[type][i] + + atomic_long_read(&lrugen->refaulted[hist][type][i]); + pos->total += lrugen->avg_total[type][i] + + lrugen->protected[hist][type][i] + + atomic_long_read(&lrugen->evicted[hist][type][i]); + } } static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) @@ -4501,13 +4506,13 @@ static int get_tier_idx(struct lruvec *lruvec, int type) struct ctrl_pos sp, pv; /* - * To leave a margin for fluctuations, use a larger gain factor (1:2). + * To leave a margin for fluctuations, use a larger gain factor (2:3). * This value is chosen because any other tier would have at least twice * as many refaults as the first tier. */ - read_ctrl_pos(lruvec, type, 0, 1, &sp); + read_ctrl_pos(lruvec, type, 0, 2, &sp); for (tier = 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, 2, &pv); + read_ctrl_pos(lruvec, type, tier, 3, &pv); if (!positive_ctrl_err(&sp, &pv)) break; } @@ -4515,68 +4520,34 @@ static int get_tier_idx(struct lruvec *lruvec, int type) return tier - 1; } -static int get_type_to_scan(struct lruvec *lruvec, int swappiness, int *tier_idx) +static int get_type_to_scan(struct lruvec *lruvec, int swappiness) { - int type, tier; struct ctrl_pos sp, pv; - int gain[ANON_AND_FILE] = { swappiness, MAX_SWAPPINESS - swappiness }; + if (!swappiness) + return LRU_GEN_FILE; + + if (swappiness == MAX_SWAPPINESS) + return LRU_GEN_ANON; /* - * Compare the first tier of anon with that of file to determine which - * type to scan. Also need to compare other tiers of the selected type - * with the first tier of the other type to determine the last tier (of - * the selected type) to evict. + * Compare the sum of all tiers of anon with that of file to determine + * which type to scan. */ - read_ctrl_pos(lruvec, LRU_GEN_ANON, 0, gain[LRU_GEN_ANON], &sp); - read_ctrl_pos(lruvec, LRU_GEN_FILE, 0, gain[LRU_GEN_FILE], &pv); - type = positive_ctrl_err(&sp, &pv); + read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp); + read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv); - read_ctrl_pos(lruvec, !type, 0, gain[!type], &sp); - for (tier = 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, gain[type], &pv); - if (!positive_ctrl_err(&sp, &pv)) - break; - } - - *tier_idx = tier - 1; - - return type; + return positive_ctrl_err(&sp, &pv); } static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, int *type_scanned, struct list_head *list) { int i; - int type; - int tier = -1; - DEFINE_MIN_SEQ(lruvec); - - /* - * Try to make the obvious choice first, and if anon and file are both - * available from the same generation, - * 1. Interpret swappiness 1 as file first and MAX_SWAPPINESS as anon - * first. - * 2. If !__GFP_IO, file first since clean pagecache is more likely to - * exist than clean swapcache. - */ - if (!swappiness) - type = LRU_GEN_FILE; - else if (min_seq[LRU_GEN_ANON] < min_seq[LRU_GEN_FILE]) - type = LRU_GEN_ANON; - else if (swappiness == 1) - type = LRU_GEN_FILE; - else if (swappiness == MAX_SWAPPINESS) - type = LRU_GEN_ANON; - else if (!(sc->gfp_mask & __GFP_IO)) - type = LRU_GEN_FILE; - else - type = get_type_to_scan(lruvec, swappiness, &tier); + int type = get_type_to_scan(lruvec, swappiness); for_each_evictable_type(i, swappiness) { int scanned; - - if (tier < 0) - tier = get_tier_idx(lruvec, type); + int tier = get_tier_idx(lruvec, type); *type_scanned = type; @@ -4585,7 +4556,6 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; type = !type; - tier = -1; } return 0; From patchwork Tue Dec 31 04:35:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2539E77188 for ; Tue, 31 Dec 2024 04:36:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F9996B0095; Mon, 30 Dec 2024 23:36:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 284FD6B0096; Mon, 30 Dec 2024 23:36:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D4796B0098; Mon, 30 Dec 2024 23:36:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D34226B0095 for ; Mon, 30 Dec 2024 23:36:01 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 62CEB47B38 for ; Tue, 31 Dec 2024 04:36:01 +0000 (UTC) X-FDA: 82953990672.18.6BDD170 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf14.hostedemail.com (Postfix) with ESMTP id F1521100005 for ; Tue, 31 Dec 2024 04:35:06 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=N5wNiM41; spf=pass (imf14.hostedemail.com: domain of 3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619715; a=rsa-sha256; cv=none; b=J4PcB+EEcs9+dUqq5KX4SuciIdqTz9H4YF30SWoFHfXJ/dWv9PPaEsfXLz54Uq6a9926SR hh9TfzIYlEebKnBgal4Fo4FMPB06/IlrerijUA86/ISXKDQwsdmFXMpEraiaFFn4vXYT1Z FXO64Nuy7qjaSfPEQ8sjNug114emVCc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=N5wNiM41; spf=pass (imf14.hostedemail.com: domain of 3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o6XRtkbexDya+FE96V37S0c0XRTdkGfwnwLQlfXry/0=; b=AVEiqAHH44vRljO+QZbhPOWZ9mgoYcLdJz3Hr3mKto8jHF/ge4PnbODTVRIg93pvLcscns rQBX5HHkioIg6jvHp4vkXd2AWj3VxKkrkpVus/pGO5XbSNOodftoocQsg3do/s4fWw3Sc+ gtyPuKppH96CDAZ26bkKR/SwzFYpmt0= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21681a2c0d5so105657775ad.2 for ; Mon, 30 Dec 2024 20:35:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619758; x=1736224558; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o6XRtkbexDya+FE96V37S0c0XRTdkGfwnwLQlfXry/0=; b=N5wNiM41PdzNF6sQilN/Wupp5LXqdl9ypv81KIrF3qBTP6NBL6reJ7chdM+jXz3s7Z O8BdvUf2JES9BA0ApbJ4aN63RSGwSd0VVS/8AQpMYrtHYdCQIWXi+j8IxoupFl/vwYGt nOH79VaJoNvIhYCKN5UGbbYiXQ1A+4ciiPkrI/vQkJyuZhoZtt+OabeLgDWH36ltcZqB CidAJWzrjo3UTwlQe29TIWxEc1iSp0CPNrVN3/4RtkOD6bLrnX0luVmCajxjAf6eSsV2 QfVeyWttcmX6wyfF0+A2MxmGMTflzMjzfm9ig/O2OQqDhdi8pVppZDWXFZWfDcEHhHpu GG6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619758; x=1736224558; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o6XRtkbexDya+FE96V37S0c0XRTdkGfwnwLQlfXry/0=; b=n3cYMLI71exB+OvAx8vg5w+5AYeaiaHNb5OsQTUiKFMEvTr9DUKdXlAYLh8iSX4+N/ nQw1QTWzELgUvc77h/N7UjQu8gSlc4TVjqETSou2dkUiwZls1ducQHmeBzkWigMOiPys fjdVi4YN4HS0B3SvsGD5z4rBtEgZSsuKpMZJOFiP/yyKSqEsvAOY/7i3Fbjwx8S4CAvh VmDjo9yx+I5Zac+rQ22iGqkDhitidni5I/fsGvZG3lukzR+STUI6lmHzpw2IRkQTb3s0 PvB27dmU6LuL60R8+VjEcqicf3y+KxUX42STSg6sNetcM6gdD/0SzqsQVGbjO9l4/9MJ 236g== X-Gm-Message-State: AOJu0YxQvUV/Wn3TgD7JEg1vg/l1f3DdD7mwDYY0GC8y0ef6Sorg5bEk 8R9/ZWjDA1dzK273bk2vqc1jDFxnSs2SSjzxW1o9e6nNTel+HeEYlJsreUN0d9yv9a1upGfGo5r img== X-Google-Smtp-Source: AGHT+IFGxXIKapCxHqzF+SSkn6gmP8awPE8NmQhmaf+NKsYjxZAWuTACPo/hzKbtWVqcyzBxJYfvlIdIpEw= X-Received: from plmm9.prod.google.com ([2002:a17:902:c449:b0:215:6f59:ff5f]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d502:b0:216:4348:149d with SMTP id d9443c01a7336-219e6f25d60mr601318685ad.53.1735619758185; Mon, 30 Dec 2024 20:35:58 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:36 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-6-yuzhao@google.com> Subject: [PATCH mm-unstable v4 5/7] mm/mglru: rework refault detection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kairui Song , Kalesh Singh X-Rspamd-Queue-Id: F1521100005 X-Stat-Signature: 4zfj1neupxhnzx3nkkwrkeqi4he1r1e8 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735619706-433142 X-HE-Meta: U2FsdGVkX197kM3DCsWQgLvdnmSp9mXiirH5pQFgkil/xYxvKWmCG3KM5vs3/G01iQ602OH++/gQ54muyg1vdpxpjQYprfQ08hsOPYxzwTG3SuHpvl6dupPmahx0+pWhkf0Z6gNQTR9yB072xu+Rmxt99rvj7T3uXpPX99uimGke5bb/Cu/bTLw4Jud+1nxQk9xFxOs/YK9JPSLKeQwvebIQyw4AMBs9U54bo7SM3wrmjiLdgLQQtr2jD+9EyA0dYWYdV9vTEjTpJgomaoFklYc/B0vJX8BTLEW3GurKuxfyViF7Wup2sOZUAEsZY0I1I0uy6mJ9QPz2I9BudT24P2pl9jHNC94h9Rl4RiiW7QAJCrkufSiOhar90V78gZ2E5wzajgRNLQ1cEmPT7Vq1k/1zm5xpE9AObghd8KdSTzQR0KyyfLvfTcdfY8YytPWflTR9I+A6/QOfqWNVEMSxTIYD3+FeO2p5anlZcZbyx8OPOutPciqyZ8RxLBevlkjN24Eq0oymLwdiEzmV2ZDaQYnmEqVkDO7i8+rAZPJXU+ZmTiwaPxyA+01tv/D6H168QQr5Z4iOd5o7NDevYCa1BhTrbqP6YERvi4JwiMg6bV1S2HjaPd8edPcSW9ZUWYmDDiydznPNHTrlieRThzH7AJhUycRMqWLe7ztohGZqWD1ReZMdlHR2negnwPTWFCbnydqSctPqtyXHgxjOJr9yLl87PNxB9S2gS2ZoGcGH5dLcFZcyXru5fb8zCM8K4pK1aJVnEJsYI6oV2PzGYSa1LZ+QeIyDrdUAqG8Sq5impO9PRbVjKRQnYubqVfqd2mkxcZp5dHW8+lXNx5f+g9xvJvKcSskaeJUpFb49GdDLEunpVcPuK3pQXulfVB+IVpRyZS4EHRIK8p/Ls422FzAj9T/N5sYxN3l0DlqUWfmBla7QErErcmIAcXEDgGYSIfPAIuXIcyjh+3GjNBsSqb9 p9dbq9lD eTiHcwhbgQ0NPFTOZdKoDxbvxSAMfgEhuNd1NGCXf9bINBKWa4wWD/os/Aiw27hvYc/5NRViisb16H34H6SxdVIL20Fojg2n9eK3p3XvtyrKbDDZY/MBxT3mJrq7jdxYtpu7xB412qcZWKue01yD8h7w7lMIGvjDsrBTzFTcd8TKyxaRdtqvg2RxsnmBVqn1Q2XUkBeTJa1Jk45FVCtWq0c3sQiuG5vcPIz6nckU/FEs43d7bVawrc6+dLDW547NuFpWRNyMlBI2ClmkGxq17SBD8BTYhZuYrhNXZVmXdlBJymA151+WMdCw6rm8K+jpxg1QcHFGIrd4k+nsUlhws9l8flfiKbYay8l1CiIsXh6i7TS/ppS/iUjYiSsLHU0Qx5zven+gqzto5marYOa8E3548rilC8dGcMfVTMrbUNzqOyd0+njLLalK4SKeclIKAbaj5mGjomYB819qX92907vnvLsyz+cJA7RpR1fMtXFFH4lWMhPvwpU4YzuSkz/Y6SBdGfbQ5GmHctaxBlKZ21pD1IGwSLIh9wb5Yksi3CRKanrMpGRG7OglC5iFjomOlNzMZ7Ynr5LYK5q+xykWTsKsW20jQezel5Khegq8tjefM0Ao= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000674, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework workingset protection as well so that the comparison of refaults between anon and file is always on an equal footing. Specifically, make lru_gen_test_recent() return true for refaults happening within the distance of MAX_NR_GENS. For example, if min_seq of a type is max_seq-MIN_NR_GENS, refaults from min_seq-1, i.e., max_seq-MIN_NR_GENS-1, are also considered recent, since the distance max_seq-(max_seq-MIN_NR_GENS-1), i.e., MIN_NR_GENS+1 is less than MAX_NR_GENS. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: Kairui Song Closes: https://lore.kernel.org/CAOUHufahuWcKf5f1Sg3emnqX+cODuR=2TQo7T4Gr-QYLujn4RA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/workingset.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index ad181d1b8cf1..2c310c29f51e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -260,11 +260,11 @@ static void *lru_gen_eviction(struct folio *folio) * Tests if the shadow entry is for a folio that was recently evicted. * Fills in @lruvec, @token, @workingset with the values unpacked from shadow. */ -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { int memcg_id; - unsigned long min_seq; + unsigned long max_seq; struct mem_cgroup *memcg; struct pglist_data *pgdat; @@ -273,8 +273,10 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, memcg = mem_cgroup_from_id(memcg_id); *lruvec = mem_cgroup_lruvec(memcg, pgdat); - min_seq = READ_ONCE((*lruvec)->lrugen.min_seq[file]); - return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); + max_seq = READ_ONCE((*lruvec)->lrugen.max_seq); + max_seq &= EVICTION_MASK >> LRU_REFS_WIDTH; + + return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS; } static void lru_gen_refault(struct folio *folio, void *shadow) @@ -290,7 +292,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) rcu_read_lock(); - recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset); + recent = lru_gen_test_recent(shadow, &lruvec, &token, &workingset); if (lruvec != folio_lruvec(folio)) goto unlock; @@ -331,7 +333,7 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { return false; @@ -432,8 +434,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool recent; rcu_read_lock(); - recent = lru_gen_test_recent(shadow, file, &eviction_lruvec, - &eviction, workingset); + recent = lru_gen_test_recent(shadow, &eviction_lruvec, &eviction, workingset); rcu_read_unlock(); return recent; } From patchwork Tue Dec 31 04:35:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 358C9E77194 for ; Tue, 31 Dec 2024 04:36:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B84FB6B0098; Mon, 30 Dec 2024 23:36:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE6206B0099; Mon, 30 Dec 2024 23:36:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89CA76B009A; Mon, 30 Dec 2024 23:36:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5EEE86B0098 for ; Mon, 30 Dec 2024 23:36:04 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0C292C0D33 for ; Tue, 31 Dec 2024 04:36:04 +0000 (UTC) X-FDA: 82953990672.15.00D097C Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf20.hostedemail.com (Postfix) with ESMTP id 1FCD91C000A for ; Tue, 31 Dec 2024 04:35:12 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BvuR3Lc6; spf=pass (imf20.hostedemail.com: domain of 3sHRzZwYKCKkhdiQJXPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yuzhao.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3sHRzZwYKCKkhdiQJXPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619717; a=rsa-sha256; cv=none; b=FsV2/zGPcsdmLGXlnO05A+M7UHNfig6VEhOL9Y6gKr4mYj/55MXkQHefsUFh/OoO/uoAYf xr+baC4ZlIzXGZ4KjkyhxkSmP2z1KYlDhnBrE/IzDJYC8PZw2AvBmhze8T8VIuHnM2KclK u179DxN27n0bJZIvezcqTvhJeGUt7pM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BvuR3Lc6; spf=pass (imf20.hostedemail.com: domain of 3sHRzZwYKCKkhdiQJXPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yuzhao.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3sHRzZwYKCKkhdiQJXPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HKwjES3e0jLy2VuhFIOMlzL3YWtlXtEvY7LLG+FXeWU=; b=AVa1/VvCE9irx4Ct5JQ0RxcOpqG5QiQcg4GV6kywINpJxB5Dru0MeN8NKdiNMW1hR5/VFv xDfUYpujyydc0f1TwN6U8D3QCjsmkBzv/P4JuP0fWSVhlmmSSB2gouYAGKWtuUK3jVWnJn 3WUhwcv0u7KTJJvizOSu+DDveo3S5To= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2161d185f04so96103535ad.3 for ; Mon, 30 Dec 2024 20:36:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619761; x=1736224561; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HKwjES3e0jLy2VuhFIOMlzL3YWtlXtEvY7LLG+FXeWU=; b=BvuR3Lc6q6H6lw8HBkJeQDKPpl8Ew6BC1yqORxK0dOi1qezHLXUjujwYQo66NxfZnZ fmbLdP7fShzCYXCAx4x9sa5EiKABZyHADLEw+thO1doysMA4lcBXricY9HvNPAQxrEFi brpx02tJvUJ0kk58nm1ewcRHHkQAuAbPgOkmVpjK491jZPJ9vd1z331oxMrk0IB9R0R/ RhfRPFllkqKbFRsMuiIjWiAR59dw0Qgb7u2dgQ5FdQ1wuEg7E/N3wQ09QsTaYROm0G8O YiiWlXvgiB0dEJX5cZIkB71IwLpcBdU+Nn1H2dbAAQfoz6ddWAHj3cSvRNJ1RPsfiwk6 bL/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619761; x=1736224561; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HKwjES3e0jLy2VuhFIOMlzL3YWtlXtEvY7LLG+FXeWU=; b=CBT6hbwOMv4zScmG/4RpwHojEluN552Ba0kORlqHqmZqdXCJZ0vdyovpWp1lwwLnXJ izCkD76bVjxxop+EPuryzVN4YKOvxauwAInLlp5eT5ii4+fbozlZVkOAjjCGZX6yc1fr rb4LvH5az7s/+QV8XKQhuIj0Gv9O/xizNWVqBf1cDX1MCLN24tEnIZJk+ZVGqQ92u/xV z1vESd0eJwO5/fXSAeseqMXvTYHlhpaoopW6eh+Vocm3S0/jsF2p1LxMZSqLYArmF587 Em4X2/kpWKTPVGyGrbSLZxYq4aAPC0q9+Xp8O6p5c99nsuX6nasFvkhz58WABu1Sow0r Muag== X-Gm-Message-State: AOJu0YzpHoxx8DxJp5NR5Y6ZPo44EIPItyu+iuVtlUDm+eJRL5JW57F0 5DsYzpzuaJdMkADW2XL8yO/6VgvH6PDlYlyoyIidA899iIE5yo+EUyV2L9SzHgs/FVHgAfkWoEK 0Tg== X-Google-Smtp-Source: AGHT+IHWy99TyDT8NaxCGIoVaFrDEtF9zWOWP/0IUrjxMV7CP85GkRQziNTadZKOeAd6kPsRyTnxbmlSQC4= X-Received: from pgjs19.prod.google.com ([2002:a63:f053:0:b0:800:502a:791c]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:78a5:b0:1e1:a0b6:9882 with SMTP id adf61e73a8af0-1e5e046331fmr54638220637.17.1735619760939; Mon, 30 Dec 2024 20:36:00 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:37 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-7-yuzhao@google.com> Subject: [PATCH mm-unstable v4 6/7] mm/mglru: rework workingset protection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kairui Song , Kalesh Singh X-Rspamd-Queue-Id: 1FCD91C000A X-Stat-Signature: xmn85njn9qq3r1nfjdoqqxbaubabitss X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735619712-583886 X-HE-Meta: U2FsdGVkX1/X/IrEjNMxXRayE9e1e9JLVNSkMibWuVvrbOrHqolH2CFGqcs33m/3MF6F0WQIlnf7ThIvu1qhhARmvCsQB9jqefuqb3r//YqRF3hJw7Lpasze+jRUMw4+mCzlT0cjEwo8dh9spYS6TwjRaUHoLYQBY1WQWo861fXuQInqAlFgx/G7s04Fxh+zJxJC4Ga7N1w3vWRfZexwzO67rJL8cAw8KlvyGmK9kxKRedV057fmlZopqW+ptYr8ZJ6xJ0eEEjxSEoOn3hSMzJyWCCoATFVva/m7pZx5oQQjPo7sRTl3NNsC8V54c58LMais9jvlCXz8PWAHKEOVaeVbPPqX29lMGvd4lJzAwyOweR1vsd6vGjkIbTWfG/kKb2+2QfSXJFJGr8hVeS8ReomflaGUuRvVhkq1+h/cMWutsTF4eDvMgFx+h2QCEyCUNKuUdSCM2pTRCxVojQAC1YttNRFF7aCE3Ws9mzf/sKImVpz82O/Z2CPKVUsjfSsAHaIXu8nCLD7ySnsQjqjUtbIpOGqyE0FhQhr7wmQfcdqxEBe8vWlbm2sqLKmOysV8JEv2zX4SAFUX9yc7gDE8RHoNEBqa5XuwrQu6JBq9v8jgJInkOhbmiAEU5pbhJEwmb3TFOk8vvogfoSHtzihli5LR2BsYCBDBVDRXlerzM7RyHdC6GKVz+fxdnPr/NArnh4a2Mp/33W+f11ywU73C/HYLNnmjABwfZFoUAmWshG16Scn/aXy0RgPulXpQTv5ubec8+mIXwPLkjLCEWkwobeirO3VaWb1+qlUMUaRifs+t6vCo8yqjbhyE6PbWk2FiL+W+WX0aR1qPg3BgZEg6uNQTZuQ1NcTXErDs7TIf5FHUKxwtv7THjWR9ksFm0Hgb2SlMLptMl7rAADuO3ANbHcII6QeRfJZRvEzBTDRLSJQMR/VVpTHxvRDHQH9VQ8YvUZ4GKot1uRtrFfsYYFg DwcRuhxb AzXod/1grkDxruFXepc7DFHIISceg9yXn4BsteeK9EzijZl2irgKHxSCcDzqG3jSTCIKswfijct0wF2GFPgZwzHpWrRkRZ1wDOMZ53gH7yGC1T0Be7VFy18Hgwh8HxDNAmwc0cRCOk0FJ6Wt7Ed10vni7s0ftm5VPvWXbdgDU5bNdfYxjMsmT0RIFBH9a3dmuFhvvTkYeIuAZE2FbTvMA6BN55GYtybvgXRdkg76MEyZBMCUSo61Qwxsu406RPu5thkYkhbDDsv+jKSE00Bnvn8qu2d/K6RdHABfpwqWquIFF5Eqj0egVxYRgv+Hq2PDDbdjpOxrHJ9H2ALqGs95fEd/iyJuEVvbNUUQRDTIoVAFsdGUlbHIlrpVuJ9Lo9WEiXp2TVYNLv5AYEVPmALinrB9PkiprvJfLlN3zmOE2jrlh7qcDxwfFkvWKpZXzVTHeJV53qJzM1V7YWjFaaiFiU/I1TcB3ZXxbEdWUE7MIbuVlB7Lpm6km8Pr9sWXPxV0DLj7smvmiUB8B1QI/81TnOaz8QgYF0AU0rb6EddZQTG8eq65HWVwqlrIGR6hCIjbmppkhje6Qpi92kFGR+VUCOGBu60U+q4M7qgbtdCf4a/rtCjc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the aging feedback no longer considering the distribution of folios in each generation, rework workingset protection to better distribute folios across MAX_NR_GENS. This is achieved by reusing PG_workingset and PG_referenced/LRU_REFS_FLAGS in a slightly different way. For folios accessed multiple times through file descriptors, make lru_gen_inc_refs() set additional bits of LRU_REFS_WIDTH in folio->flags after PG_referenced, then PG_workingset after LRU_REFS_WIDTH. After all its bits are set, i.e., LRU_REFS_FLAGS|BIT(PG_workingset), a folio is lazily promoted into the second oldest generation in the eviction path. And when folio_inc_gen() does that, it clears LRU_REFS_FLAGS so that lru_gen_inc_refs() can start over. For this case, LRU_REFS_MASK is only valid when PG_referenced is set. For folios accessed multiple times through page tables, folio_update_gen() from a page table walk or lru_gen_set_refs() from a rmap walk sets PG_referenced after the accessed bit is cleared for the first time. Thereafter, those two paths set PG_workingset and promote folios to the youngest generation. Like folio_inc_gen(), when folio_update_gen() does that, it also clears PG_referenced. For this case, LRU_REFS_MASK is not used. For both of the cases, after PG_workingset is set on a folio, it remains until this folio is either reclaimed, or "deactivated" by lru_gen_clear_refs(). It can be set again if lru_gen_test_recent() returns true upon a refault. When adding folios to the LRU lists, lru_gen_folio_seq() distributes them as follows: +---------------------------------+---------------------------------+ | Accessed thru page tables | Accessed thru file descriptors | +---------------------------------+---------------------------------+ | PG_active (set while isolated) | | +----------------+----------------+----------------+----------------+ | PG_workingset | PG_referenced | PG_workingset | LRU_REFS_FLAGS | +---------------------------------+---------------------------------+ |<--------- MIN_NR_GENS --------->| | |<-------------------------- MAX_NR_GENS -------------------------->| After this patch, some typical client and server workloads showed improvements under heavy memory pressure. For example, Python TPC-C, which was used to benchmark a different approach [1] to better detect refault distances, showed a significant decrease in total refaults: Before After Change Time (seconds) 10801 10801 0% Executed (transactions) 41472 43663 +5% workingset_nodes 109070 120244 +10% workingset_refault_anon 5019627 7281831 +45% workingset_refault_file 1294678786 554855564 -57% workingset_refault_total 1299698413 562137395 -57% [1] https://lore.kernel.org/20230920190244.16839-1-ryncsn@gmail.com/ Reported-by: Kairui Song Closes: https://lore.kernel.org/CAOUHufahuWcKf5f1Sg3emnqX+cODuR=2TQo7T4Gr-QYLujn4RA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- include/linux/mm_inline.h | 88 +++++++++++------------ include/linux/mmzone.h | 82 +++++++++++++-------- mm/swap.c | 24 +++---- mm/vmscan.c | 147 ++++++++++++++++++++++---------------- mm/workingset.c | 29 ++++---- 5 files changed, 204 insertions(+), 166 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 34e5097182a0..f9157a0c42a5 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -133,31 +133,25 @@ static inline int lru_hist_from_seq(unsigned long seq) return seq % NR_HIST_GENS; } -static inline int lru_tier_from_refs(int refs) +static inline int lru_tier_from_refs(int refs, bool workingset) { VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH)); - /* see the comment in folio_lru_refs() */ - return order_base_2(refs + 1); + /* see the comment on MAX_NR_TIERS */ + return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs); } static inline int folio_lru_refs(struct folio *folio) { unsigned long flags = READ_ONCE(folio->flags); - bool workingset = flags & BIT(PG_workingset); + if (!(flags & BIT(PG_referenced))) + return 0; /* - * Return the number of accesses beyond PG_referenced, i.e., N-1 if the - * total number of accesses is N>1, since N=0,1 both map to the first - * tier. lru_tier_from_refs() will account for this off-by-one. Also see - * the comment on MAX_NR_TIERS. + * Return the total number of accesses including PG_referenced. Also see + * the comment on LRU_REFS_FLAGS. */ - return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + workingset; -} - -static inline void folio_clear_lru_refs(struct folio *folio) -{ - set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1; } static inline int folio_lru_gen(struct folio *folio) @@ -223,11 +217,43 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen)); } +static inline unsigned long lru_gen_folio_seq(struct lruvec *lruvec, struct folio *folio, + bool reclaiming) +{ + int gen; + int type = folio_is_file_lru(folio); + struct lru_gen_folio *lrugen = &lruvec->lrugen; + + /* + * +-----------------------------------+-----------------------------------+ + * | Accessed through page tables and | Accessed through file descriptors | + * | promoted by folio_update_gen() | and protected by folio_inc_gen() | + * +-----------------------------------+-----------------------------------+ + * | PG_active (set while isolated) | | + * +-----------------+-----------------+-----------------+-----------------+ + * | PG_workingset | PG_referenced | PG_workingset | LRU_REFS_FLAGS | + * +-----------------------------------+-----------------------------------+ + * |<---------- MIN_NR_GENS ---------->| | + * |<---------------------------- MAX_NR_GENS ---------------------------->| + */ + if (folio_test_active(folio)) + gen = MIN_NR_GENS - folio_test_workingset(folio); + else if (reclaiming) + gen = MAX_NR_GENS; + else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) || + (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio)))) + gen = MIN_NR_GENS; + else + gen = MAX_NR_GENS - folio_test_workingset(folio); + + return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type])); +} + static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) { unsigned long seq; unsigned long flags; - unsigned long mask; int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); @@ -237,40 +263,12 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, if (folio_test_unevictable(folio) || !lrugen->enabled) return false; - /* - * There are four common cases for this page: - * 1. If it's hot, i.e., freshly faulted in, add it to the youngest - * generation, and it's protected over the rest below. - * 2. If it can't be evicted immediately, i.e., a dirty page pending - * writeback, add it to the second youngest generation. - * 3. If it should be evicted first, e.g., cold and clean from - * folio_rotate_reclaimable(), add it to the oldest generation. - * 4. Everything else falls between 2 & 3 above and is added to the - * second oldest generation if it's considered inactive, or the - * oldest generation otherwise. See lru_gen_is_active(). - */ - if (folio_test_active(folio)) - seq = lrugen->max_seq; - else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) || - (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio)))) - seq = lrugen->max_seq - 1; - else if (reclaiming || lrugen->min_seq[type] + MIN_NR_GENS >= lrugen->max_seq) - seq = lrugen->min_seq[type]; - else - seq = lrugen->min_seq[type] + 1; + seq = lru_gen_folio_seq(lruvec, folio, reclaiming); gen = lru_gen_from_seq(seq); flags = (gen + 1UL) << LRU_GEN_PGOFF; /* see the comment on MIN_NR_GENS about PG_active */ - mask = LRU_GEN_MASK; - /* - * Don't clear PG_workingset here because it can affect PSI accounting - * if the activation is due to workingset refault. - */ - if (folio_test_active(folio)) - mask |= LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_active); - set_mask_bits(&folio->flags, mask, flags); + set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags); lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8245ecb0400b..9540b41894da 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -332,66 +332,88 @@ enum lruvec_flags { #endif /* !__GENERATING_BOUNDS_H */ /* - * Evictable pages are divided into multiple generations. The youngest and the + * Evictable folios are divided into multiple generations. The youngest and the * oldest generation numbers, max_seq and min_seq, are monotonically increasing. * They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An * offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the * corresponding generation. The gen counter in folio->flags stores gen+1 while - * a page is on one of lrugen->folios[]. Otherwise it stores 0. + * a folio is on one of lrugen->folios[]. Otherwise it stores 0. * - * A page is added to the youngest generation on faulting. The aging needs to - * check the accessed bit at least twice before handing this page over to the - * eviction. The first check takes care of the accessed bit set on the initial - * fault; the second check makes sure this page hasn't been used since then. - * This process, AKA second chance, requires a minimum of two generations, - * hence MIN_NR_GENS. And to maintain ABI compatibility with the active/inactive - * LRU, e.g., /proc/vmstat, these two generations are considered active; the - * rest of generations, if they exist, are considered inactive. See - * lru_gen_is_active(). + * After a folio is faulted in, the aging needs to check the accessed bit at + * least twice before handing this folio over to the eviction. The first check + * clears the accessed bit from the initial fault; the second check makes sure + * this folio hasn't been used since then. This process, AKA second chance, + * requires a minimum of two generations, hence MIN_NR_GENS. And to maintain ABI + * compatibility with the active/inactive LRU, e.g., /proc/vmstat, these two + * generations are considered active; the rest of generations, if they exist, + * are considered inactive. See lru_gen_is_active(). * - * PG_active is always cleared while a page is on one of lrugen->folios[] so - * that the aging needs not to worry about it. And it's set again when a page - * considered active is isolated for non-reclaiming purposes, e.g., migration. - * See lru_gen_add_folio() and lru_gen_del_folio(). + * PG_active is always cleared while a folio is on one of lrugen->folios[] so + * that the sliding window needs not to worry about it. And it's set again when + * a folio considered active is isolated for non-reclaiming purposes, e.g., + * migration. See lru_gen_add_folio() and lru_gen_del_folio(). * * MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the * number of categories of the active/inactive LRU when keeping track of * accesses through page tables. This requires order_base_2(MAX_NR_GENS+1) bits - * in folio->flags. + * in folio->flags, masked by LRU_GEN_MASK. */ #define MIN_NR_GENS 2U #define MAX_NR_GENS 4U /* - * Each generation is divided into multiple tiers. A page accessed N times - * through file descriptors is in tier order_base_2(N). A page in the first tier - * (N=0,1) is marked by PG_referenced unless it was faulted in through page - * tables or read ahead. A page in any other tier (N>1) is marked by - * PG_referenced and PG_workingset. This implies a minimum of two tiers is - * supported without using additional bits in folio->flags. + * Each generation is divided into multiple tiers. A folio accessed N times + * through file descriptors is in tier order_base_2(N). A folio in the first + * tier (N=0,1) is marked by PG_referenced unless it was faulted in through page + * tables or read ahead. A folio in the last tier (MAX_NR_TIERS-1) is marked by + * PG_workingset. A folio in any other tier (1flags. * * In contrast to moving across generations which requires the LRU lock, moving * across tiers only involves atomic operations on folio->flags and therefore * has a negligible cost in the buffered access path. In the eviction path, - * comparisons of refaulted/(evicted+protected) from the first tier and the - * rest infer whether pages accessed multiple times through file descriptors - * are statistically hot and thus worth protecting. + * comparisons of refaulted/(evicted+protected) from the first tier and the rest + * infer whether folios accessed multiple times through file descriptors are + * statistically hot and thus worth protecting. * * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the * number of categories of the active/inactive LRU when keeping track of * accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in - * folio->flags. + * folio->flags, masked by LRU_REFS_MASK. */ #define MAX_NR_TIERS 4U #ifndef __GENERATING_BOUNDS_H -struct lruvec; -struct page_vma_mapped_walk; - #define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) #define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) +/* + * For folios accessed multiple times through file descriptors, + * lru_gen_inc_refs() sets additional bits of LRU_REFS_WIDTH in folio->flags + * after PG_referenced, then PG_workingset after LRU_REFS_WIDTH. After all its + * bits are set, i.e., LRU_REFS_FLAGS|BIT(PG_workingset), a folio is lazily + * promoted into the second oldest generation in the eviction path. And when + * folio_inc_gen() does that, it clears LRU_REFS_FLAGS so that + * lru_gen_inc_refs() can start over. Note that for this case, LRU_REFS_MASK is + * only valid when PG_referenced is set. + * + * For folios accessed multiple times through page tables, folio_update_gen() + * from a page table walk or lru_gen_set_refs() from a rmap walk sets + * PG_referenced after the accessed bit is cleared for the first time. + * Thereafter, those two paths set PG_workingset and promote folios to the + * youngest generation. Like folio_inc_gen(), folio_update_gen() also clears + * PG_referenced. Note that for this case, LRU_REFS_MASK is not used. + * + * For both cases above, after PG_workingset is set on a folio, it remains until + * this folio is either reclaimed, or "deactivated" by lru_gen_clear_refs(). It + * can be set again if lru_gen_test_recent() returns true upon a refault. + */ +#define LRU_REFS_FLAGS (LRU_REFS_MASK | BIT(PG_referenced)) + +struct lruvec; +struct page_vma_mapped_walk; + #ifdef CONFIG_LRU_GEN enum { @@ -406,8 +428,6 @@ enum { NR_LRU_GEN_CAPS }; -#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) - #define MIN_LRU_BATCH BITS_PER_LONG #define MAX_LRU_BATCH (MIN_LRU_BATCH * 64) diff --git a/mm/swap.c b/mm/swap.c index 649ef7f2b74b..746a5ceba42c 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -387,24 +387,20 @@ static void lru_gen_inc_refs(struct folio *folio) if (folio_test_unevictable(folio)) return; + /* see the comment on LRU_REFS_FLAGS */ if (!folio_test_referenced(folio)) { - folio_set_referenced(folio); + set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced)); return; } - if (!folio_test_workingset(folio)) { - folio_set_workingset(folio); - return; - } - - /* see the comment on MAX_NR_TIERS */ do { - new_flags = old_flags & LRU_REFS_MASK; - if (new_flags == LRU_REFS_MASK) - break; + if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) { + if (!folio_test_workingset(folio)) + folio_set_workingset(folio); + return; + } - new_flags += BIT(LRU_REFS_PGOFF); - new_flags |= old_flags & ~LRU_REFS_MASK; + new_flags = old_flags + BIT(LRU_REFS_PGOFF); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } @@ -417,7 +413,7 @@ static bool lru_gen_clear_refs(struct folio *folio) if (gen < 0) return true; - set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + set_mask_bits(&folio->flags, LRU_REFS_FLAGS | BIT(PG_workingset), 0); lrugen = &folio_lruvec(folio)->lrugen; /* whether can do without shuffling under the LRU lock */ @@ -499,7 +495,7 @@ void folio_add_lru(struct folio *folio) folio_test_unevictable(folio), folio); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - /* see the comment in lru_gen_add_folio() */ + /* see the comment in lru_gen_folio_seq() */ if (lru_gen_enabled() && !folio_test_unevictable(folio) && lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) folio_set_active(folio); diff --git a/mm/vmscan.c b/mm/vmscan.c index a33221298fd0..74bc85fc7cdf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -862,6 +862,31 @@ enum folio_references { FOLIOREF_ACTIVATE, }; +#ifdef CONFIG_LRU_GEN +/* + * Only used on a mapped folio in the eviction (rmap walk) path, where promotion + * needs to be done by taking the folio off the LRU list and then adding it back + * with PG_active set. In contrast, the aging (page table walk) path uses + * folio_update_gen(). + */ +static bool lru_gen_set_refs(struct folio *folio) +{ + /* see the comment on LRU_REFS_FLAGS */ + if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) { + set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced)); + return false; + } + + set_mask_bits(&folio->flags, LRU_REFS_FLAGS, BIT(PG_workingset)); + return true; +} +#else +static bool lru_gen_set_refs(struct folio *folio) +{ + return false; +} +#endif /* CONFIG_LRU_GEN */ + static enum folio_references folio_check_references(struct folio *folio, struct scan_control *sc) { @@ -870,7 +895,6 @@ static enum folio_references folio_check_references(struct folio *folio, referenced_ptes = folio_referenced(folio, 1, sc->target_mem_cgroup, &vm_flags); - referenced_folio = folio_test_clear_referenced(folio); /* * The supposedly reclaimable folio was found to be in a VM_LOCKED vma. @@ -888,6 +912,15 @@ static enum folio_references folio_check_references(struct folio *folio, if (referenced_ptes == -1) return FOLIOREF_KEEP; + if (lru_gen_enabled()) { + if (!referenced_ptes) + return FOLIOREF_RECLAIM; + + return lru_gen_set_refs(folio) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP; + } + + referenced_folio = folio_test_clear_referenced(folio); + if (referenced_ptes) { /* * All mapped folios start out with page table @@ -1092,11 +1125,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!sc->may_unmap && folio_mapped(folio)) goto keep_locked; - /* folio_update_gen() tried to promote this page? */ - if (lru_gen_enabled() && !ignore_references && - folio_mapped(folio) && folio_test_referenced(folio)) - goto keep_locked; - /* * The number of dirty pages determines if a node is marked * reclaim_congested. kswapd will stall and start writing @@ -3167,16 +3195,19 @@ static int folio_update_gen(struct folio *folio, int gen) VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); + /* see the comment on LRU_REFS_FLAGS */ + if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) { + set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced)); + return -1; + } + do { /* lru_gen_del_folio() has isolated this page? */ - if (!(old_flags & LRU_GEN_MASK)) { - /* for shrink_folio_list() */ - new_flags = old_flags | BIT(PG_referenced); - continue; - } + if (!(old_flags & LRU_GEN_MASK)) + return -1; - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); - new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS); + new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; @@ -3200,7 +3231,7 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai new_gen = (old_gen + 1) % MAX_NR_GENS; - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS); new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF; /* for folio_end_writeback() */ if (reclaiming) @@ -3378,9 +3409,11 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, struct pglist_data *pgdat) { - struct folio *folio; + struct folio *folio = pfn_folio(pfn); + + if (folio_lru_gen(folio) < 0) + return NULL; - folio = pfn_folio(pfn); if (folio_nid(folio) != pgdat->node_id) return NULL; @@ -3757,8 +3790,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) while (!list_empty(head)) { struct folio *folio = lru_to_folio(head); int refs = folio_lru_refs(folio); - int tier = lru_tier_from_refs(refs); - int delta = folio_nr_pages(folio); + bool workingset = folio_test_workingset(folio); VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); @@ -3768,8 +3800,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) new_gen = folio_inc_gen(lruvec, folio, false); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + delta); + /* don't count the workingset being lazily promoted */ + if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) { + int tier = lru_tier_from_refs(refs, workingset); + int delta = folio_nr_pages(folio); + + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + delta); + } if (!--remaining) return false; @@ -4155,16 +4193,10 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) old_gen = folio_update_gen(folio, new_gen); if (old_gen >= 0 && old_gen != new_gen) update_batch_size(walk, folio, old_gen, new_gen); - - continue; - } - - old_gen = folio_lru_gen(folio); - if (old_gen < 0) - folio_set_referenced(folio); - else if (old_gen != new_gen) { - folio_clear_lru_refs(folio); - folio_activate(folio); + } else if (lru_gen_set_refs(folio)) { + old_gen = folio_lru_gen(folio); + if (old_gen >= 0 && old_gen != new_gen) + folio_activate(folio); } } @@ -4325,7 +4357,8 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c int zone = folio_zonenum(folio); int delta = folio_nr_pages(folio); int refs = folio_lru_refs(folio); - int tier = lru_tier_from_refs(refs); + bool workingset = folio_test_workingset(folio); + int tier = lru_tier_from_refs(refs, workingset); struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio); @@ -4347,14 +4380,17 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c } /* protected */ - if (tier > tier_idx || refs == BIT(LRU_REFS_WIDTH)) { - int hist = lru_hist_from_seq(lrugen->min_seq[type]); - + if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) { gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); - WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + delta); + /* don't count the workingset being lazily promoted */ + if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) { + int hist = lru_hist_from_seq(lrugen->min_seq[type]); + + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + delta); + } return true; } @@ -4374,8 +4410,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c } /* waiting for writeback */ - if (folio_test_locked(folio) || writeback || - (type == LRU_GEN_FILE && dirty)) { + if (writeback || (type == LRU_GEN_FILE && dirty)) { gen = folio_inc_gen(lruvec, folio, true); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; @@ -4404,13 +4439,12 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca return false; } - /* see the comment on MAX_NR_TIERS */ + /* see the comment on LRU_REFS_FLAGS */ if (!folio_test_referenced(folio)) - folio_clear_lru_refs(folio); + set_mask_bits(&folio->flags, LRU_REFS_MASK, 0); /* for shrink_folio_list() */ folio_clear_reclaim(folio); - folio_clear_referenced(folio); success = lru_gen_del_folio(lruvec, folio, true); VM_WARN_ON_ONCE_FOLIO(!success, folio); @@ -4600,31 +4634,24 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); list_for_each_entry_safe_reverse(folio, next, &list, lru) { + DEFINE_MIN_SEQ(lruvec); + if (!folio_evictable(folio)) { list_del(&folio->lru); folio_putback_lru(folio); continue; } - if (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio))) { - /* restore LRU_REFS_FLAGS cleared by isolate_folio() */ - if (folio_test_workingset(folio)) - folio_set_referenced(folio); - continue; - } - - if (skip_retry || folio_test_active(folio) || folio_test_referenced(folio) || - folio_mapped(folio) || folio_test_locked(folio) || - folio_test_dirty(folio) || folio_test_writeback(folio)) { - /* don't add rejected folios to the oldest generation */ - set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, - BIT(PG_active)); - continue; - } - /* retry folios that may have missed folio_rotate_reclaimable() */ - list_move(&folio->lru, &clean); + if (!skip_retry && !folio_test_active(folio) && !folio_mapped(folio) && + !folio_test_dirty(folio) && !folio_test_writeback(folio)) { + list_move(&folio->lru, &clean); + continue; + } + + /* don't add rejected folios to the oldest generation */ + if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) + set_mask_bits(&folio->flags, LRU_REFS_FLAGS, BIT(PG_active)); } spin_lock_irq(&lruvec->lru_lock); diff --git a/mm/workingset.c b/mm/workingset.c index 2c310c29f51e..4841ae8af411 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -239,7 +239,8 @@ static void *lru_gen_eviction(struct folio *folio) int type = folio_is_file_lru(folio); int delta = folio_nr_pages(folio); int refs = folio_lru_refs(folio); - int tier = lru_tier_from_refs(refs); + bool workingset = folio_test_workingset(folio); + int tier = lru_tier_from_refs(refs, workingset); struct mem_cgroup *memcg = folio_memcg(folio); struct pglist_data *pgdat = folio_pgdat(folio); @@ -253,7 +254,7 @@ static void *lru_gen_eviction(struct folio *folio) hist = lru_hist_from_seq(min_seq); atomic_long_add(delta, &lrugen->evicted[hist][type][tier]); - return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); + return pack_shadow(mem_cgroup_id(memcg), pgdat, token, workingset); } /* @@ -304,24 +305,20 @@ static void lru_gen_refault(struct folio *folio, void *shadow) lrugen = &lruvec->lrugen; hist = lru_hist_from_seq(READ_ONCE(lrugen->min_seq[type])); - /* see the comment in folio_lru_refs() */ - refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + workingset; - tier = lru_tier_from_refs(refs); + refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + 1; + tier = lru_tier_from_refs(refs, workingset); atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]); - mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta); - /* - * Count the following two cases as stalls: - * 1. For pages accessed through page tables, hotter pages pushed out - * hot pages which refaulted immediately. - * 2. For pages accessed multiple times through file descriptors, - * they would have been protected by sort_folio(). - */ - if (lru_gen_in_fault() || refs >= BIT(LRU_REFS_WIDTH) - 1) { - set_mask_bits(&folio->flags, 0, LRU_REFS_MASK | BIT(PG_workingset)); + /* see folio_add_lru() where folio_set_active() will be called */ + if (lru_gen_in_fault()) + mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta); + + if (workingset) { + folio_set_workingset(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta); - } + } else + set_mask_bits(&folio->flags, LRU_REFS_MASK, (refs - 1UL) << LRU_REFS_PGOFF); unlock: rcu_read_unlock(); } From patchwork Tue Dec 31 04:35:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E14A9E77188 for ; Tue, 31 Dec 2024 04:36:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73E466B009A; Mon, 30 Dec 2024 23:36:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C7406B009B; Mon, 30 Dec 2024 23:36:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F54C6B009C; Mon, 30 Dec 2024 23:36:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 281C16B009A for ; Mon, 30 Dec 2024 23:36:07 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D33CF47C26 for ; Tue, 31 Dec 2024 04:36:06 +0000 (UTC) X-FDA: 82953990546.24.ADDD200 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf04.hostedemail.com (Postfix) with ESMTP id D441140002 for ; Tue, 31 Dec 2024 04:35:15 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="rd9/zckB"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3s3RzZwYKCKwkglTMaSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--yuzhao.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3s3RzZwYKCKwkglTMaSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619720; a=rsa-sha256; cv=none; b=iJdGOopubFhKtlkyIh1WFH+8OxAVLRc9LmSMtXQSGxWxpdeiJ47LsODJ0r/RE5o1vis6Xr 5gl9YkmjWYPHC6+OhUu6jgpvOqnJXRXDPdmUQIskuAKcV7G6cwJH9dyNUV35765TOA+ukI YIgIIWe3QcbNEqgFHA/s5VHif9dMsl0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="rd9/zckB"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3s3RzZwYKCKwkglTMaSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--yuzhao.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3s3RzZwYKCKwkglTMaSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619720; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uSnhijC2SU9G2JpiGLLPy0X8NVAvldR4ppcAmuCmJd8=; b=oGqtCECPVBLVHJd5mSKX56NvxAg9nPqRj2J/tGimstrWicT579/V+H+ebfXhWH2JTR+hyE S/LwD4LYXZnvmoX0Tn18ZtKsphYOzGInaVxK0pxKchP4AxFolqEVkHQ+aRJXgByNW89rJF fteIsrDZlNYseyTKe2nFLx/Q9txnugk= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2163c2f32fdso210110075ad.2 for ; Mon, 30 Dec 2024 20:36:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619764; x=1736224564; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uSnhijC2SU9G2JpiGLLPy0X8NVAvldR4ppcAmuCmJd8=; b=rd9/zckBSSUR+v3drfFoKeoes3nY9E/5zFBR/FiZCeAQewCmMfmnBQaYbZbC0uU1mO j2eZs3wGNcuzBZLg/IHnFzrkYwtx3HQrsPaaP3R7JVdG512PlGOYrgsXFBmbjGP0x2vY ZouGAxpXt4hPzJdotyhQniFTeD0yKPmS900zCK0yoMUOzVpRoJGFVwHy0jsaoVKBf0Mt ioF2NYRcUS0fsCHUkJWv7wMjg6JNNU9OpbdacBUPZ1pGw0BTL9fQbUHIUW7XtwRpqn4Q qfQQkKZn1XM+bLzeX+jl4BoxdP0od9Xl4rsbV6Zs4GLXCPzph+6Sfd6yVaMlpBeLVCd9 7sUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619764; x=1736224564; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uSnhijC2SU9G2JpiGLLPy0X8NVAvldR4ppcAmuCmJd8=; b=jmswIpFu1qgBD5X9caH1d6qD8KEcVpF9ZDWzk1Tljcgp3Y5YZR8lYb6CEW0jMs6egG Evaog99MfFvMKeTBx7EMy/OlSX9MlzzBkRWXQIov1PKEk+BTlzVXcPUl//saQ92JqNeX 2wl83nrIWifQItKA/eBuL4WUlTJ9gmymlX6IkASdND4vzeCjgj4IHYiSIkYzc65bx6/u vWqQiuiZDDYrLJkMIazhGwQyvnjnGro+o+Sqp3AcDi2N92QCNkhIVQXb6CyylHumvwBZ LUNIwpfTcxK2YPaI1sY7GoNAposmg2pvFAzRfzj25wVdDkT6kuW44JroREyjjXN+EOGX q6yQ== X-Gm-Message-State: AOJu0YznCL+NweGe6BCoG3c3Giiu2riOA2H7VVYXXLRDCJe77J43VoQH MPJ+p9ihXDLucB8WhaAUnx7vmiHbbgcYNky4PujNpWLtDMCE+WLE0cMkCcDdKt5zbwm5asXSxg3 eGQ== X-Google-Smtp-Source: AGHT+IG+LWeqA7gxlbhJ4Aa6fTniLxLak1adrDfoZ2HZ8A15k6jPqKUZa911/S4AQd/cYmc6w8UN3NHudRE= X-Received: from pgbda9.prod.google.com ([2002:a05:6a02:2389:b0:7fd:577a:6d1e]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ef4e:b0:216:7761:cc36 with SMTP id d9443c01a7336-219e6f14bbamr619775545ad.43.1735619763742; Mon, 30 Dec 2024 20:36:03 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:38 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-8-yuzhao@google.com> Subject: [PATCH mm-unstable v4 7/7] mm/mglru: fix PTE-mapped large folios From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Barry Song , Kalesh Singh X-Stat-Signature: tqfuiaj7ifcnuaqqseinjeoaaqsqsft4 X-Rspam-User: X-Rspamd-Queue-Id: D441140002 X-Rspamd-Server: rspam08 X-HE-Tag: 1735619715-776609 X-HE-Meta: U2FsdGVkX19FahI0FQ0sQkMHKzINHEcP7PLczPn1+MiEe0WY9kegFjJ0Xllhi8j4UWpW9ggYs1MlbXDyNuzk+y5eGbdpmX08YPsYM/0oZMpDFPG03R7OhZ0zjD8r1rkjjVxzsNfpcNMV2XaGOwxpdzSYwoLgDjph+0b9M7gTifBNLQMbdfDzoBpPgMcQFZjQFOeOrLrU0A8nJLsQE+n2i3DoMCkRaqcWa1t+ibRwHHJLRrcPeg+vZEajXMtMF0kAq41Eun21xuY9y6eX8osbzCrSaEFma+6JBErRfSaF1hjbzyvku06ZNr/XSLwSCcFjCECbPu2T4ZQWFnhcejkClNs2iYTEzNjNDcDKs5kfNHSxgY2rzcpsT5buZMOv/uLj7I79UskghtCZfjtJeHm1qUAD9hU6O2tZvmILrtsTgrywUCs6jCGsBqoQQf7BPQ8YlTZZKflOHzzo9CjbiAg4Oz8ij/U87BD9lJGoxdIgVs/nc/IwLH1IYo1sunusWy2PW51D4iHe1OuySRd6CNX5/79Y/0hxcHD0yJEPj/BHfcwQckshdCKttRKDGOIknF174NghIJTF2OiNcIaFgEO+2NRQiJCY3ls7yoixYdLQCiOolXG9llQ0XtCglXqPGpAXbqSIUu08zDwa3fkAOpjhyjyXK5TY/aplNf+M4NVFBLq3UZscsePChEUTy+KQtke21ofwqx9iDDC73MkAv1u5BSw1csUPtKcoW+vAdjICYrt+jV4QdRnWBhgPwuJEko/QiRzN2gVnJwOeSanHPqiZAgifZEYhVUaE/L1b8Sn3g5PjHKvXLQDWtVnVBp4alibXBYpLmlBCFA46YqLEo57YsCsOExrL5I2RHK0t1kREdUvPaw+u6yTHYZzSKnrt4OoGv29yWDJMKOaOYek6zb4BQPqAg7ggkYW6mCQC/JOmnedrxlidupIRD8MChsp/WZ+XrtFJWPsyEMWCx57WK+R kZPuBV9M PY3OpGdhAlmmGZM17CAUpj+NOBpYuyxPL58DsNULJBXu3NGPFiR3nzQ2lKERe84b+ctbt3Gvu2a7VitkwykxfMsizx7vo+zD9vOOIEV5F2BXkmXUmyuibbV0k7hEOTos0qeDcilXpXM8V4obptIZYfw2gKbur97Mk+j8kGoHnQwk5/FnIthdHX2jdR0Wz0hPpWbK9OvyJAd9VZ/4366xQrqlZqjCOxyfNgE6QNzZTnh8BXwrY3bNg7uzVnkM3iinXPYTEpqxyNznQBH4qIbunBepWDTGqWI5Xjvpl11RKLtNfbJv6y56FvzR/CJtJgALaVhOfH3i8hCuhvWviMzk+hVnZVnTlr/muVaU6qAttCHUukwEMW67koWvCaoF6sq6IV+iWsVRXEB03AuOxs0mW0QgAvJ2NbfUviF/bg5/DDrbF3J+a3sP0MJdI8M+CBsVPskZUV//recs3MAjt9c57qZPtrHZV6Cxg4gpTDDuqzQonJdLZMBbdRcdC4PcoPtTwIz5EGdeM8RFr5m2MbEnmNjvF70AgvU69UdK4yLnaPMxfqdERsdzmoUO3Fw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Count the accessed bits from PTEs mapping the same large folio as one access rather than multiple accesses. The last patch changed how folios accessed through page tables are promoted: rather than getting promoted after the accessed bit is cleared for the first time, a folio only gets promoted thereafter. Counting the accessed bits from the same large folio as multiple accesses can cause that folio to be promoted prematurely, which in turn can cause overprotection of single-use large folios. This patch reduced the sys time of the kernel compilation by 95% CI [2, 5]% on Altra M128-30 with 3GB DRAM, 12GB zram, 16KB THPs and -j32. Reported-by: Barry Song Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/vmscan.c | 110 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 72 insertions(+), 38 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 74bc85fc7cdf..a099876fa029 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3431,29 +3431,55 @@ static bool suitable_to_scan(int total, int young) return young * n >= total; } +static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio, + int new_gen, bool dirty) +{ + int old_gen; + + if (!folio) + return; + + if (dirty && !folio_test_dirty(folio) && + !(folio_test_anon(folio) && folio_test_swapbacked(folio) && + !folio_test_swapcache(folio))) + folio_mark_dirty(folio); + + if (walk) { + old_gen = folio_update_gen(folio, new_gen); + if (old_gen >= 0 && old_gen != new_gen) + update_batch_size(walk, folio, old_gen, new_gen); + } else if (lru_gen_set_refs(folio)) { + old_gen = folio_lru_gen(folio); + if (old_gen >= 0 && old_gen != new_gen) + folio_activate(folio); + } +} + static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct mm_walk *args) { int i; + bool dirty; pte_t *pte; spinlock_t *ptl; unsigned long addr; int total = 0; int young = 0; + struct folio *last = NULL; struct lru_gen_mm_walk *walk = args->private; struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec); struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); - int old_gen, new_gen = lru_gen_from_seq(max_seq); + int gen = lru_gen_from_seq(max_seq); pmd_t pmdval; - pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, - &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, &ptl); if (!pte) return false; + if (!spin_trylock(ptl)) { pte_unmap(pte); - return false; + return true; } if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { @@ -3482,19 +3508,23 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (!ptep_clear_young_notify(args->vma, addr, pte + i)) continue; + if (last != folio) { + walk_update_folio(walk, last, gen, dirty); + + last = folio; + dirty = false; + } + + if (pte_dirty(ptent)) + dirty = true; + young++; walk->mm_stats[MM_LEAF_YOUNG]++; - - if (pte_dirty(ptent) && !folio_test_dirty(folio) && - !(folio_test_anon(folio) && folio_test_swapbacked(folio) && - !folio_test_swapcache(folio))) - folio_mark_dirty(folio); - - old_gen = folio_update_gen(folio, new_gen); - if (old_gen >= 0 && old_gen != new_gen) - update_batch_size(walk, folio, old_gen, new_gen); } + walk_update_folio(walk, last, gen, dirty); + last = NULL; + if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) goto restart; @@ -3508,13 +3538,15 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area struct mm_walk *args, unsigned long *bitmap, unsigned long *first) { int i; + bool dirty; pmd_t *pmd; spinlock_t *ptl; + struct folio *last = NULL; struct lru_gen_mm_walk *walk = args->private; struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec); struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); - int old_gen, new_gen = lru_gen_from_seq(max_seq); + int gen = lru_gen_from_seq(max_seq); VM_WARN_ON_ONCE(pud_leaf(*pud)); @@ -3567,20 +3599,23 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area if (!pmdp_clear_young_notify(vma, addr, pmd + i)) goto next; + if (last != folio) { + walk_update_folio(walk, last, gen, dirty); + + last = folio; + dirty = false; + } + + if (pmd_dirty(pmd[i])) + dirty = true; + walk->mm_stats[MM_LEAF_YOUNG]++; - - if (pmd_dirty(pmd[i]) && !folio_test_dirty(folio) && - !(folio_test_anon(folio) && folio_test_swapbacked(folio) && - !folio_test_swapcache(folio))) - folio_mark_dirty(folio); - - old_gen = folio_update_gen(folio, new_gen); - if (old_gen >= 0 && old_gen != new_gen) - update_batch_size(walk, folio, old_gen, new_gen); next: i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1; } while (i <= MIN_LRU_BATCH); + walk_update_folio(walk, last, gen, dirty); + arch_leave_lazy_mmu_mode(); spin_unlock(ptl); done: @@ -4115,9 +4150,11 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { int i; + bool dirty; unsigned long start; unsigned long end; struct lru_gen_mm_walk *walk; + struct folio *last = NULL; int young = 1; pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; @@ -4128,7 +4165,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); struct lru_gen_mm_state *mm_state = get_mm_state(lruvec); DEFINE_MAX_SEQ(lruvec); - int old_gen, new_gen = lru_gen_from_seq(max_seq); + int gen = lru_gen_from_seq(max_seq); lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); @@ -4182,24 +4219,21 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (!ptep_clear_young_notify(vma, addr, pte + i)) continue; - young++; + if (last != folio) { + walk_update_folio(walk, last, gen, dirty); - if (pte_dirty(ptent) && !folio_test_dirty(folio) && - !(folio_test_anon(folio) && folio_test_swapbacked(folio) && - !folio_test_swapcache(folio))) - folio_mark_dirty(folio); - - if (walk) { - old_gen = folio_update_gen(folio, new_gen); - if (old_gen >= 0 && old_gen != new_gen) - update_batch_size(walk, folio, old_gen, new_gen); - } else if (lru_gen_set_refs(folio)) { - old_gen = folio_lru_gen(folio); - if (old_gen >= 0 && old_gen != new_gen) - folio_activate(folio); + last = folio; + dirty = false; } + + if (pte_dirty(ptent)) + dirty = true; + + young++; } + walk_update_folio(walk, last, gen, dirty); + arch_leave_lazy_mmu_mode(); /* feedback from rmap walkers to page table walkers */