From patchwork Wed Aug 2 00:19:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 13337412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F40DBEB64DD for ; Wed, 2 Aug 2023 00:19:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAABF280108; Tue, 1 Aug 2023 20:19:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C5B0D2800C8; Tue, 1 Aug 2023 20:19:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2340280108; Tue, 1 Aug 2023 20:19:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A3D062800C8 for ; Tue, 1 Aug 2023 20:19:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 65A2AA0378 for ; Wed, 2 Aug 2023 00:19:50 +0000 (UTC) X-FDA: 81077256540.19.25D8462 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id AED9A180010 for ; Wed, 2 Aug 2023 00:19:48 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cm4Hai0R; spf=pass (imf16.hostedemail.com: domain of 3I6HJZAsKCLgiYjcqfqglefemmejc.amkjglsv-kkitYai.mpe@flex--kaleshsingh.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3I6HJZAsKCLgiYjcqfqglefemmejc.amkjglsv-kkitYai.mpe@flex--kaleshsingh.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690935588; a=rsa-sha256; cv=none; b=EGIBQldmsVfpeEIUBTnca1lKQl/g+XwNR+ZWcI4YyWkzdzKDZuJkn+w2qkk1b3e8kx5Oz+ EHpzsUe0ednxsVT7SDOtbUdxlqRyzY/fXFEAp5KyiwTBWW7iN/pz2JM+smK7HFMrDnC6lb IzqaJUIVCcthFFT5PKwuL0U+eeg2I10= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cm4Hai0R; spf=pass (imf16.hostedemail.com: domain of 3I6HJZAsKCLgiYjcqfqglefemmejc.amkjglsv-kkitYai.mpe@flex--kaleshsingh.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3I6HJZAsKCLgiYjcqfqglefemmejc.amkjglsv-kkitYai.mpe@flex--kaleshsingh.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690935588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=DXVsJSb9QIFOr/Cp0N+IT34pRw+slorysgAR9VBW/lM=; b=3c+ImGkBlecbuMzXhxHQ+xMhyN+LqRDC/AMRhYhICaLHGCr/ek+G30ZD7xmNhBkwg2WfAa 3u48AZ522bxwZ9JnVd2LUk595uN6NR1Zie1Z/GSe/4h4R/gwFhd7vXOFdA3Q7KN/0SX9K7 JZAca4+clsd3J6HtP6K8z+XnPdb5wHg= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-58473c4f629so65096107b3.1 for ; Tue, 01 Aug 2023 17:19:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690935588; x=1691540388; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=DXVsJSb9QIFOr/Cp0N+IT34pRw+slorysgAR9VBW/lM=; b=cm4Hai0RZYcjkOHwUMqYM8wK0SfBbq6tE/k3HPH9GmV1zBmvkpZFMcYL+r+O5/fJbr yVq3tNLgKEwKoly7oloPtCW9d6iQlb7bYEeqwEpyCEWI2sHa/uqdKA7oUbAlOw9+D+33 M61tOUgNo8fyyGNNXnxgduT41K+3pcW/oHczkSpQIGsd/5fEuI7YxntjfIFiICf3vyWk ViI9qV2uesdgd1VSF3s2Yl584bAMuwZjnbv8NIkoqTZnJcemCyasU28fzTMYmIe37CrO 2DtipcERam4z0HdG/CPTBtBJiKPAvtwqUVxtyD00zOgHXb6Rmfr2KoGteO8WsS1p37hY Y3eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690935588; x=1691540388; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=DXVsJSb9QIFOr/Cp0N+IT34pRw+slorysgAR9VBW/lM=; b=Jo7jdCDSXvmZFOs7S2m1a23g/do7Ht05QitxC4nZ5jOQMfcPmALs40lXecmATlbueV j6sMeBN0N5Jmf+3Abj5NZysSFld6MCAhh8pbi6Yong9sGvaQ4crdapky8IzF8k2ufFNg fJW1egaLC0/PMUgnUX6rZJsWDDIrq5g0h7F5hbJBrHXTOq0vU3hbyfjGpZRBRqRI3Jjm /JL9tX+6NhdMIDPmYBfOI6GuANAbacW2kCDX/TFk2Go95jh15yImDuSTzPQsKB9MWn+d 9NAzLKjQKLZDcmfhbyNW0yvc2yDikj1kVt3LXjBjRk0SFAobnrH/G+32lTqHnzEDd377 Vn6g== X-Gm-Message-State: ABy/qLY7DOk9qmWvrv/50mJirC+g8XUXZvDReoirl6Qkf+TFsshelYRG ZySCd56VRrb3dZdQqNpZfRX/b71pKXUFX+Q9Qw== X-Google-Smtp-Source: APBJJlFOYOnPjUqct44iRohOnc0mUP9ReqoYJeBrA5AVDrjW9wn0H6QdpajsQPU0LLHzoebktFORyvnYqlxBKUc5/Q== X-Received: from kalesh.mtv.corp.google.com ([2620:15c:211:201:aa5f:ff75:318:255]) (user=kaleshsingh job=sendgmr) by 2002:a25:ab86:0:b0:d0d:a7bc:4040 with SMTP id v6-20020a25ab86000000b00d0da7bc4040mr103482ybi.0.1690935587776; Tue, 01 Aug 2023 17:19:47 -0700 (PDT) Date: Tue, 1 Aug 2023 17:19:35 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230802001938.3913174-1-kaleshsingh@google.com> Subject: [PATCH 1/3] mm-unstable: Multi-gen LRU: Fix per-zone reclaim From: Kalesh Singh To: yuzhao@google.com, akpm@linux-foundation.org Cc: surenb@google.com, android-mm@google.com, kernel-team@android.com, Kalesh Singh , Charan Teja Kalla , Lecopzer Chen , Matthias Brugger , AngeloGioacchino Del Regno , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AED9A180010 X-Stat-Signature: n3hwu7s5n35ype5imndikj4x97r6zheb X-Rspam-User: X-HE-Tag: 1690935588-499924 X-HE-Meta: U2FsdGVkX19cf79r84xwspswdwNN+XCJCJuNccABBd1tGYnux4UZPXAEp4WTcmPW4HHTy4NLzDhgT+Kff+P+YW0qBE/ew+Ml/fNeHoahoVRkhwGq05gacI4DE4RBniEdMk27aGlk3Viu3xUnxWBJq/Zatj02Gl2L9xPhu9J+hFpMzSPvtEK30xbsbf9fB2gyKr4E3iBUJOWaoYhMfJCCE6iTqmLgNHYdAEwZwoBIyLY0zd8l5F6ukMpZOeZjD9y5yt04wfP70dkUW1nT+QvbC5obEc2mOgzu1v2ccJLldWmLdLRLGikLNqlnC4wNCY8tCdfb/COsXTZy8SnkIK2JATVOotbbs29hleWXQilP5G0Wtsuh+g6gRQ98teztHJwB368UtVj7Q+yYkzuO14rAZPEQKEWjPjla2gNr7ggjKNL6uIMBTE87QCl/ED5F3l+St+Q5WPBtfHZ9j6R57Rp7G4emwTItw0VJgBZv0LOaeDP3MSHm1+tq7SEILURCL1IjgC5oVF+Fu+prWWepakg15wWaGfIKs423N8QNKgaeZOMYkIn4w1qnXmPMYFCfZO5/t+LlzGf3JdVog/mTbqXxHr2lAC7JI90GT6iJy7+04oCljMXreutQNLkDutNFO0S2JWAtC7cuaAjlNLaoYOrbP6pVrqTd4xSaQsdszc7jfeWJlnlHYlkD6r41a6OQjcxlwZvyCtNvSUM7PeGzFCfutj9JbWwMuUdkI9DCf/ImndbvjkMMDGasN8D6k0J5NZ9zQs2WwPlU1OSXj6aBxqQH/l85R/oU0qls+DoRICYRZEmn8U1ppeiibc4DOpUeBO93NwQ/mkpBaCrKAJKt0gM65sId2tYJDX0UFzpztm71lGqZajXeAXScflBo3gwcmgYEy1MjfnbG/Rzn11Jj1G/u6JXpCAeLB3lOoiSh7JBivUvNpDu6xkgS7rF7Tp0jjRfSuzqzX4h1N2Tm1AseNA9 Y22s+nDl 6G8tz0yLIHYGWioBxZRjvV7GHJIA4Hf9T26Gcu3bsO12At8fkotnC1ewuMZq0YoMBDa+v/szsCs/PWjd0EKifa4lOM39+7SKLxa3o140vF/FUuf7UJGjM67y3NqTd5aHByWAFUXRiGo2pRxqMxkR4ZqJ2oXDIL5zuTLKWJAd7xeDdjR7mQ1mWURSzKJbYzPotHaVaLGRZljo5YwXUfg8nzJQKJFMtQTAl9nRTEmLhOUhiFUfIRpF8aapthmT9aLvGiC2XZkqG8JTM0/ErRV5u6n+R16ocsXI1bQP3vgcH18VEUmlpHrjoXTnAues9yEE8ZjK8tgcwkynHlREzrdgZ6WkUDHgU+N9BrouT3olfJkiIIwDVPf/EpHEWN7GDh6/nTE2+aEUuJZJ+jU2rqTU/siiUFvJvmNodMcBb6nxm6rgMJVdIod2d+6U0crHDpk8vc11vwybkh3ij+V9s8028itPu8VDlKPNVSU0Tb66S4eKgEymLEHCE4NVZL52yw9iGNUCoQPrgcKeD9fjIppW0VhGLOrg5DsQP27I1jg1mRjEHC2iMxmccTACtJF47LJaVWm+EafruJI+kJDOGydLk4fsaRhyW6vhy7hztt+BmzMvhiSosG9gR/EKcH86T8NS97+yZ+Hw43rhyqh/xBxK7O03KUd0XxDfvnlSZp2WbsRJenCHXMZEolTxyCkIFIH29ll2O9SkT+8h06ysJWVMMI4NcZrzw302qmqwdo8OvtFd3Up/5a6phkJBo1GosjGVZnwufnfjNKTJz3/ARGfdy8baIIlF4lOY5QgDppv9KEnJMhUPcdup3s2aXenneoOGUbh1s2t7+AOz00bfEUElJAGT10JzVyHDWFVIaZvCVkafKnJokTOF60GjBoHe0FkLRm7HSFBnZL1AMoZ4K5tj65fA4+DLAfkEiepvsjGB2U0n4WDA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.012603, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: MGLRU has a LRU list for each zone for each type (anon/file) in each generation: long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; The min_seq (oldest generation) can progress independently for each type but the max_seq (youngest generation) is shared for both anon and file. This is to maintain a common frame of reference. In order for eviction to advance the min_seq of a type, all the per-zone lists in the oldest generation of that type must be empty. The eviction logic only considers pages from eligible zones for eviction or promotion. scan_folios() { ... for (zone = sc->reclaim_idx; zone >= 0; zone--) { ... sort_folio(); // Promote ... isolate_folio(); // Evict } ... } Consider the system has the movable zone configured and default 4 generations. The current state of the system is as shown below (only illustrating one type for simplicity): Type: ANON Zone DMA32 Normal Movable Device Gen 0 0 0 4GB 0 Gen 1 0 1GB 1MB 0 Gen 2 1MB 4GB 1MB 0 Gen 3 1MB 1MB 1MB 0 Now consider there is a GFP_KERNEL allocation request (eligible zone index <= Normal), evict_folios() will return without doing any work since there are no pages to scan in the eligible zones of the oldest generation. Reclaim won't make progress until triggered from a ZONE_MOVABLE allocation request; which may not happen soon if there is a lot of free memory in the movable zone. This can lead to OOM kills, although there is 1GB pages in the Normal zone of Gen 1 that we have not yet tried to reclaim. This issue is not seen in the conventional active/inactive LRU since there are no per-zone lists. If there are no (not enough) folios to scan in the eligible zones, move folios from ineligible zone (zone_index > reclaim_index) to the next generation. This allows for the progression of min_seq and reclaiming from the next generation (Gen 1). Qualcomm, Mediatek and raspberrypi [1] discovered this issue independently. [1] https://github.com/raspberrypi/linux/issues/5395 Cc: Yu Zhao Cc: Andrew Morton Reported-by: Charan Teja Kalla Reported-by: Lecopzer Chen Signed-off-by: Kalesh Singh --- mm/vmscan.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 4039620d30fe..489a4fc7d9b1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4889,7 +4889,8 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) * the eviction ******************************************************************************/ -static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) +static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, + int tier_idx) { bool success; int gen = folio_lru_gen(folio); @@ -4939,6 +4940,13 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) return true; } + /* ineligible */ + if (zone > sc->reclaim_idx) { + gen = folio_inc_gen(lruvec, folio, false); + list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + return true; + } + /* waiting for writeback */ if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { @@ -4987,7 +4995,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, int type, int tier, struct list_head *list) { - int gen, zone; + int i; + int gen; enum vm_event_item item; int sorted = 0; int scanned = 0; @@ -5003,9 +5012,10 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, gen = lru_gen_from_seq(lrugen->min_seq[type]); - for (zone = sc->reclaim_idx; zone >= 0; zone--) { + for (i = MAX_NR_ZONES; i > 0; i--) { LIST_HEAD(moved); int skipped = 0; + int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head = &lrugen->folios[gen][type][zone]; while (!list_empty(head)) { @@ -5019,7 +5029,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, scanned += delta; - if (sort_folio(lruvec, folio, tier)) + if (sort_folio(lruvec, folio, sc, tier)) sorted += delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); From patchwork Wed Aug 2 00:19:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 13337413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34BDEEB64DD for ; Wed, 2 Aug 2023 00:19:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72C5D280109; Tue, 1 Aug 2023 20:19:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DC102800C8; Tue, 1 Aug 2023 20:19:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A3F4280109; Tue, 1 Aug 2023 20:19:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4A9032800C8 for ; Tue, 1 Aug 2023 20:19:58 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1582D140CEF for ; Wed, 2 Aug 2023 00:19:58 +0000 (UTC) X-FDA: 81077256876.20.637625F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf15.hostedemail.com (Postfix) with ESMTP id 50499A0004 for ; Wed, 2 Aug 2023 00:19:56 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=liEq2TRJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3K6HJZAsKCMAqgrkynyotmnmuumrk.iusrot03-ssq1giq.uxm@flex--kaleshsingh.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3K6HJZAsKCMAqgrkynyotmnmuumrk.iusrot03-ssq1giq.uxm@flex--kaleshsingh.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690935596; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oILqqqD951CZa6vFO5lwCGBJV16jZziGq+aTih9CvRM=; b=Rce/PhT4I+FbTArla+yQLW9BA9lOf+ZU+meFlGqYZZPjvjMUOVwXMNFPgUmPF9ozJiekaK 159p1d3UddRyiKPvOiN8Vr9U4NSPECSB8Z8en3blvGRQujyI1aFVir4J3kcx9N450uxupt 7L/GORmT2Gq6wN06mdcXFt1BRlqm/RI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=liEq2TRJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3K6HJZAsKCMAqgrkynyotmnmuumrk.iusrot03-ssq1giq.uxm@flex--kaleshsingh.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3K6HJZAsKCMAqgrkynyotmnmuumrk.iusrot03-ssq1giq.uxm@flex--kaleshsingh.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690935596; a=rsa-sha256; cv=none; b=m59EDZ89/Xx06Jr1x7p9r/q7e1W179QngztxO8H81WWlLQi6cndLt8CeOiO1QLpSk8cROp 1TxIQ5CNAAsNaAl7pk/M7OS3OCzKN51QmrL4tULNcTfjfduELoQZusu3H+KtIw7ZPFt8Rj OlGdFlFFZ2cs/fYAkaghwGhoUFrVlSg= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5734d919156so56862757b3.3 for ; Tue, 01 Aug 2023 17:19:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690935595; x=1691540395; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oILqqqD951CZa6vFO5lwCGBJV16jZziGq+aTih9CvRM=; b=liEq2TRJ6uphA1n1WRctdr12b14csuDNelgmOB3P8uUfovqp001PwMVQtOenzPkHJi whIzAspyWYUBqUAPEv0+bxGp/kCYHZiDZCj4aPoXHdFKRGOuA0OAPUxh2lZcKRB9DmY4 +sVvZ4qnTj42uetZgk1spG0aKDlJUYjhXZPRCghRJr5QNKGlAn8bEhwP4JK0KqRABLaT fhKrvbkr8HehDdLCVM2/wiJlQ2zpWgZOa9ODGSx5g36gQV2AuKOzP/RpJ1d7pqrnbT8W MmvhG/Y18dFWO4yMQQmXypjHnDfl+eqfjiVHy0xIyI41FCe/pW+Xv7cf+eIVYSkn4fkR 1ENQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690935595; x=1691540395; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oILqqqD951CZa6vFO5lwCGBJV16jZziGq+aTih9CvRM=; b=OBenagICB6gb76n014MbjZFnnA3TYfoM48y3TkXa/bw99UHgOA5NuaOfgkIJCw+XjZ bNKd3YGKJr6MxqPJQIf0jNPKNkdN5sR45A7kerWrKoM5GvJqiaBB9tmsCFMa3vswkpNS jM2cC8cWUlAlwcothlLwDdu1INMNG/2GLUlPbaEciR7Wogvn3NLPbJhYZ9grc/InrY80 pwf6M685/mJbVt3BmQ8YHidTcAhzyVnNheXtj5D4K94qllBDjSn7KoiQPrlIsDqqa7Ka ecRxYoh49VvXmbw5bT6Y2QQJTGvQdvY/TFQDm2a+szegk0JWPIzx+WYZtVtSe22K5wF+ X7Rw== X-Gm-Message-State: ABy/qLZIBhgyUe0/mH+i10uzsRBpnLa3kJ8RXaaFE088/OILqU/ZD8Lh cXkzSlgWqlU4M0vRIgieU1Q7bKmHOI7KJyV9pA== X-Google-Smtp-Source: APBJJlGVSXTcSf5HWtAwgVhVMNFG4pScdB5810WzB/3elwhK2y7pd9aNX9CzOkX3wIMvukUfcHZ5QeIzWXNdnmzMyw== X-Received: from kalesh.mtv.corp.google.com ([2620:15c:211:201:aa5f:ff75:318:255]) (user=kaleshsingh job=sendgmr) by 2002:a81:b301:0:b0:576:fdbe:76b2 with SMTP id r1-20020a81b301000000b00576fdbe76b2mr121491ywh.3.1690935595421; Tue, 01 Aug 2023 17:19:55 -0700 (PDT) Date: Tue, 1 Aug 2023 17:19:36 -0700 In-Reply-To: <20230802001938.3913174-1-kaleshsingh@google.com> Mime-Version: 1.0 References: <20230802001938.3913174-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230802001938.3913174-2-kaleshsingh@google.com> Subject: [PATCH 2/3] mm-unstable: Multi-gen LRU: Avoid race in inc_min_seq() From: Kalesh Singh To: yuzhao@google.com, akpm@linux-foundation.org Cc: surenb@google.com, android-mm@google.com, kernel-team@android.com, Kalesh Singh , Aneesh Kumar K V , Matthias Brugger , AngeloGioacchino Del Regno , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 50499A0004 X-Stat-Signature: 9n9cg518c3gapphrdpid5jfg9jwpbrrf X-Rspam-User: X-HE-Tag: 1690935596-383757 X-HE-Meta: U2FsdGVkX197AnyuCw5I6Bu9j+pZZnUpGBabuLaxiUV13E6uWkKeHQMsf320vdDuyWDXF64/KR+/yuUYyA6vb9b0YyTmTl6L/E5kV9On71nzobchzxmfwXmf5ML+tmNb4hcqyOomNJ/KWCKldmzqJpJjRhszvDahk1xjsYd8uvHSt3RYqtEfk1j71Mb+ZNyaLqrRUwCYOSo0t0OKyRgZN71m7xmO/JxmbDB6H/wCcMR8XnONYko6nLogmiUjHWx97N0n1+EaQAwRzFfTbOicC5sdIMdxRVQojLK+zpW6Escg+d+lGBJBXHk28PMj+flGm1AfGucOBmg+DULD848U1WLOy4JKSV8X/O20PzfReKw7H8VWxcJrD4jHyrkzB+8wEweYAT8aaBZChsxqBzK7d2fjPQDB1NAVNomv9M2ag9Ijs95H4ZZ/jD8g0W8A6GODHwdl/18ZssN/aIzpxJC3Wteplm+v7LhMAslNBblQ39p0qq/FwTwyPnPBR9eOgPGIisi2xLV9l+qVlLUWaX1VceEQwTZO+UHOFJROIeLszYrD98yOLAA+WLODiWVM/PjNsJxI2p8Mrnrp7u7FDkA40E5hp8FIhxUqfONkHYTLS/BOTOkmrcAgOf5SMRiLztNFxQFbxnQynir32LkJv/DsGaDkLQOflibAZHVLEpcA14PdA0cZpdErs4kbDnPYZoUXNKB/Mga+OMQtN7WS5C4JXKeOE3N3zNlmpUReuLqZwxELiBt/x1sVKbj8wxkOnP68N5M4RsfGm4onsuFcNsbv+T+LANTgmdfgooznyJ/DKuOgBtK/Ru6NY0Jv5r4vVEAzMHi0iic8/dvOEC++IpdUc7vL7fiWXklTfOzVI8Lx1XqbOtcR7lEusiB82W/jltBKfJOotetu8e8ELqulRkot2YdAcNdEYxDNb4oDjZIWUpl2/iUuxNDvz01N34hYR2vLsyUXkMBJUNPVXvZfOdl gHgRrShH 52Irh7eRYGuqrELUBjcS7Nsoxfd72Aufm+1eZlkf6/PlrY9TOxr9aSNRszOhC7fjE8/k4f2VAbsa28Nq0fTFCdVNx/b3ZPEMSYFIu/HBnn7txIhP0q0eGFmPQwyTDD9mFU8qdPwG+kxFPlNRe//8GuOvrufLg+GClRQ/TF+7/j6kN6pJgiAiUFxgjMvjJdgjFTPF+lJZLv+996D2Q1OJUHjzXDvOv3qJZn2G1LmgAotgcwgLVjHBLC3rqskszH0UdkxjtJ/QjzNIMxa+SI548edK/CMLkS3ULNh2ATQFCISwoaEcy/t3tMrpezB7wJOsHnElVI/6hPzHcf2CLJuI1xuz2CttrsLThp4tXee0rqfi5QqxLlepMecRGIXcjyXZL4fTlss/Z7miMnhbyKr0EH4ejCSdN2ZRAi8DbEyD26FqTD5Udv356ZzuZ7j8jtX0n1//vF+tVb2vG4r95tMQCssQDHuWmK1zbwK4qBjtZt0D+R3sE3eF+rEcE7VU+eHpu0RWoeg7K7CYS6FwH6U2pYCj2VPDj5nHMhfQsOc4jns6Wt1fFA7lGVcmW7Y1t0tRVOL5vyJwC9UJorVhH766cyMhSB1Gv/EZrv3icvWL9Y7m+p17tBM+R6KLOmxEYTIiRx7cXXlFxLbrUsdzu/d+M13i6p52R0LZZwHmtpgH+7StaClrEUyGQvj/GEhC2xk7N7m8Czur67/Uf+WtXHk/ukc+m+ZRl1f4qDUXB6FEOh4Cnv7ZCG6LIXYrPqgN5tOWa1gOsQ0vpp8oD7N633zlS+shwJX/MX/+6ZCEyH65h0kUSJC7PQGGgtPAxxoay8eXUkBbjXHeSzTOXCyE6as6OXQqnlH/9Cw+1SGBmHxJATS2UFVoaGKf4PUOykA6qUFAh7tEvj7KkUP3jx8ZUTBzZQSL7EN9PlUleDUrksqcbdjuMvf8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000017, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: inc_max_seq() will try to inc_min_seq() if nr_gens == MAX_NR_GENS. This is because the generations are reused (the last oldest now empty generation will become the next youngest generation). inc_min_seq() is retried until successful, dropping the lru_lock and yielding the CPU on each failure, and retaking the lock before trying again: while (!inc_min_seq(lruvec, type, can_swap)) { spin_unlock_irq(&lruvec->lru_lock); cond_resched(); spin_lock_irq(&lruvec->lru_lock); } However, the initial condition that required incrementing the min_seq (nr_gens == MAX_NR_GENS) is not retested. This can change by another call to inc_max_seq() from run_aging() with force_scan=true from the debugfs interface. Since the eviction stalls when the nr_gens == MIN_NR_GENS, avoid unnecessarily incrementing the min_seq by rechecking the number of generations before each attempt. This issue was uncovered in previous discussion on the list by Yu Zhao and Aneesh Kumar [1]. [1] https://lore.kernel.org/linux-mm/CAOUHufbO7CaVm=xjEb1avDhHVvnC8pJmGyKcFf2iY_dpf+zR3w@mail.gmail.com/ Cc: Yu Zhao Cc: Aneesh Kumar K V Cc: Andrew Morton Signed-off-by: Kalesh Singh --- mm/vmscan.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 489a4fc7d9b1..6eecd291756c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4439,7 +4439,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) int prev, next; int type, zone; struct lru_gen_folio *lrugen = &lruvec->lrugen; - +restart: spin_lock_irq(&lruvec->lru_lock); VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); @@ -4450,11 +4450,12 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) VM_WARN_ON_ONCE(!force_scan && (type == LRU_GEN_FILE || can_swap)); - while (!inc_min_seq(lruvec, type, can_swap)) { - spin_unlock_irq(&lruvec->lru_lock); - cond_resched(); - spin_lock_irq(&lruvec->lru_lock); - } + if (inc_min_seq(lruvec, type, can_swap)) + continue; + + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + goto restart; } /* From patchwork Wed Aug 2 00:19:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 13337414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 611EEC00528 for ; Wed, 2 Aug 2023 00:20:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D954028010A; Tue, 1 Aug 2023 20:20:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D44832800C8; Tue, 1 Aug 2023 20:20:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0C7A28010A; Tue, 1 Aug 2023 20:20:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B37D52800C8 for ; Tue, 1 Aug 2023 20:20:06 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 737DC803F6 for ; Wed, 2 Aug 2023 00:20:06 +0000 (UTC) X-FDA: 81077257212.16.4004E9C Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf17.hostedemail.com (Postfix) with ESMTP id ABBF74000B for ; Wed, 2 Aug 2023 00:20:04 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="sjS9Sd/9"; spf=pass (imf17.hostedemail.com: domain of 3M6HJZAsKCMgyozs6v6w1uvu22uzs.q20zw18B-00y9oqy.25u@flex--kaleshsingh.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3M6HJZAsKCMgyozs6v6w1uvu22uzs.q20zw18B-00y9oqy.25u@flex--kaleshsingh.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690935604; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sQqMGlFvgvQ3yvXEv1c3TG8Zu+eyY93FF0SWidKY8Dg=; b=R8JukKtojoiM7esi3HzEeBcfB9ICnYgktEM1ULPheb7sdsY1ZG0AK5KOPCLcgGW3lHNLaq S+y1AylJ55/tCoi4OxVMPLW9iMt572bdAlJLbFdKA/OJk+uf7eXwTvhZ4nP78CXeisHB5D t5nSSTtZD8EqtaHFFw/dOKUM5TXDV0o= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="sjS9Sd/9"; spf=pass (imf17.hostedemail.com: domain of 3M6HJZAsKCMgyozs6v6w1uvu22uzs.q20zw18B-00y9oqy.25u@flex--kaleshsingh.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3M6HJZAsKCMgyozs6v6w1uvu22uzs.q20zw18B-00y9oqy.25u@flex--kaleshsingh.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690935604; a=rsa-sha256; cv=none; b=PxUgXmQvYIxGqQ3nDVie4jfSaNHoQMzK39J5cNIW1NrivNG2BVBY2bCZvkmxn2LD34THcX dGpKbNc6MXXzDfzAdChzo8y/4Nk50Ecf64yLp8a6BZd/EXFnB0IuG+e+oDusD7ZiMN1u41 ceLCnTlaD2uI1pvuX9kB3rLlOQN1muU= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d052f58b7deso6401737276.2 for ; Tue, 01 Aug 2023 17:20:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690935604; x=1691540404; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sQqMGlFvgvQ3yvXEv1c3TG8Zu+eyY93FF0SWidKY8Dg=; b=sjS9Sd/96ChnfQYaR1cO5sk2N9oknVX/qJB2yXG052CqPpTjyq29z2og4PvyAUfuFb BbtTttNXygAAyEGj4VDc+gc5Tv/jJm9VVufAhbVnxD9d2fZv9SEDE3pJ6CmBfTmsJ7/M 30pcgTpZEa5b5WZTWX8yFJI4winvc57m0RSu8uswlhLrD88MgHDJlMnwkJUHOUbaDiEX XFB8o/mR1Hsb5CgE+Zf1jutf81sOk0xLQ2PlXcTmJRipNCPc6eQScIC9DvSTQlXma76v y6rPEG/+3EoY9yWeyWMpqj6nc4br7aYthBDJY27Nv4jLmE5k9Zm7VQi5fBGYNijEtZwE qVdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690935604; x=1691540404; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sQqMGlFvgvQ3yvXEv1c3TG8Zu+eyY93FF0SWidKY8Dg=; b=aueWoIsTe7b0T8jCvFJ/6+jsyEQKDpgHwc/nO46zf98q/yADeVSTPNT5FtWCuz69xz PbwLRifrnYtlW4SDDeFZV6xdLvpZTPNiS0VFP7dDCpYLdVMFmzkYh5WxjNE/eiAv+KCs wuF9L5/fXC20nQyUNPF51cQwTUYqXlEACp+oLCDBiarxqInQlKMmNec3z7rs+B8UK9qB a0KNsJvVb0ID+2n0peeCx1gtpaejM8hAXlRGxocUqO6jPhTzA7SZBNykvLd6Bo+NFD4X KJ0yBf59xO4WaF6dJc++h98MksillmuDijTGVP1uRA4CVM43z2XVr/t5/tAMuUHHoNiY vGgg== X-Gm-Message-State: ABy/qLajam35CNzvjsLgQVOz6B4ftvmc5JiSi3fRZFOcIe9n5sUHV2kT jCHofWaXiy+J5fTDA2pQLoKvHxFEFGUeDluF5Q== X-Google-Smtp-Source: APBJJlEf7/u2/Z616hvXFPIg4IjabnaZmfD7pavvMQxNa0+Yi7QX2OhFnd4BlpzLjlUWlOxu9nZ4P/MXGt70cu1ftA== X-Received: from kalesh.mtv.corp.google.com ([2620:15c:211:201:aa5f:ff75:318:255]) (user=kaleshsingh job=sendgmr) by 2002:a25:48a:0:b0:d37:353:b7eb with SMTP id 132-20020a25048a000000b00d370353b7ebmr33827ybe.11.1690935603866; Tue, 01 Aug 2023 17:20:03 -0700 (PDT) Date: Tue, 1 Aug 2023 17:19:37 -0700 In-Reply-To: <20230802001938.3913174-1-kaleshsingh@google.com> Mime-Version: 1.0 References: <20230802001938.3913174-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230802001938.3913174-3-kaleshsingh@google.com> Subject: [PATCH 3/3] mm-unstable: Multi-gen LRU: Fix can_swap in lru_gen_look_around() From: Kalesh Singh To: yuzhao@google.com, akpm@linux-foundation.org Cc: surenb@google.com, android-mm@google.com, kernel-team@android.com, Kalesh Singh , Matthias Brugger , AngeloGioacchino Del Regno , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org X-Rspamd-Queue-Id: ABBF74000B X-Rspam-User: X-Stat-Signature: r8j84sa4d5i4fyurktpq9a9utkwp5tp4 X-Rspamd-Server: rspam01 X-HE-Tag: 1690935604-811559 X-HE-Meta: U2FsdGVkX1+WHa/gr7iov9KwsM9YxV//UtnfwQyVmssuXOgJTRmGSvdcB9TszFVnGfvXdcAxfYmAOd53uW9IPqfvC/vT+qBRG/cPupY3v8m0nqY8pWSW60eO65PlY7UcV+ifi4IJRALbzNrBPp48rg8dBKfzvl6P3f+4xCVLxnwUnd432elgZFHvD8DFTCMwAHcVxk6wAWsYQtB+7BPj4vZYwsfhL1fKImJ5pPBkt1CS+XmlVK8hoPvoE2qcWfvrGCnmN6+pDgf9gT9n+KdoD0WGQS5/Knsy9f+FjZUjHU13ehAj4aZooXm4NbhuCNQ8XCdtc7ZyWl/JPU68e3Zy8JKTryh4ZGauesBEf1jBBF/Qyx1kdNnu5F2OPwdj9YPLoYZcazQUUopIJzkKC8tvacxHKyFBUibZC4HEmIHdsetMkitO4902j4bIHqK7enhHNbr9tCilaDuKWL26a485qNQ8UsyMWOVsTmGUo2VI9QAQvN7RPChV6j2HKYmn7Medw0hkuD/a89d7PTMuIZUZokZobhkdkBHvGpulfR2Sili/ivxaUSRCHXoatnGZpxd0AXeShTxNRHq3sfYis4I5ClodMMdOklBzzlR/tZquqYY706VW44/2RJwse5W90Oe5GL1exf3wfhaQd+OnIhwZT0XNL6z31nB0Z/ak0hn3u7Z3eOf5UgGqlWtuVpKjkNkxegkNmrE+fO+oAChOFvaELwCZyqTo5eeb7awyIv9ceVqcvlsVpEf/7NH2FD4DWQ767cirahx6Vse5fAjJqzkVvhKqFfR791ULYeIUz6rVY9TCDB/KhvOS1QiQkxGhsO+OXSgVJy4+eG5MUueVSAXsMqRH6S6i2aGo9q4LFLVpEvKWaNpIcPzbccskIvAEzK6o2Thg2KGcl4d7m0pmTY8yNyIH4XK/ZaGm2DG381occp1Ty9gZXbX2fkll7v2zJP3q6xmGRTEH51t6AXWXZXz 8/I+kRky n3z526spNZkTv9gourwRp9hmVdD6wIv+sQf46hfSdSts3kOwqqcmJ3jgX4Dbfr0fm6451jMOa+tvFCgsyPDMlvMrHlw1BF7vkX5iNebzTUF2UYRR6G8Fw8pHBUatDx2Be9+6+l50o9kTQdp3uYsaCI2iUpHOwlAla+Rp8MBjmWWsc+nttn8jjTImD8KUUMGMfroXOHi39wkVMu4TScdwcr2CFpEX8lQzWCcV7vvvKczckuYO8gljGEmDbVbeSP4hHuf4RIdD9pZp3tJebf7KEK0aUMXOM1m5VR4oafcKBB805YmApQLZXwE2j4/J/HeapzNaJ7CUA9rWfxWCLp0uCz75rvx2Pfz60/xgDxsWSNEAa6BZ805zwVQi6R6QJWr3DKaA1dv0Sk7Ay23dXlP6ltTSsTCwOpmVC8vVlcezBgW40CiZni5B2MbUruiZcdPHWj4YERnDcTry6rIEcdEOB2vrfk8xuoh8fGZ8hg+y+jR8DF0YS9NA+CESKwfFWM8xc5eTxWcjZocuLqdIleUjeWl8qz+4Uiu0g+oigT/3Ly2CJhj2bHt7r/ptjQkyowB2cKorYYlxoYNlO+hEyENimEz9XKEqWFzCx5EqGFFLhV9/s6ghBnE3zajF+UcUpeKoi8drYYyTpKpWu+346IWcbtZGt2Cb7jlz1M0MF08m+0Nf+yCiFZXVOwQ+QuPj7Pzj9Sl2z3nvk5hxWcD623KmuQ+qj/YYj8y/ulwNdk79GAz6Em1umzp7xTqEuWVeoOeCfj5pzUcA2D9Fw7tuQrL+RXVTqLigmJF8e/tUqTAJQY+BX0TSabucdgKLz2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000079, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: walk->can_swap might be invalid since it's not guaranteed to be initialized for the particular lruvec. Instead deduce it from the folio type (anon/file). Cc: Yu Zhao Cc: Andrew Morton Signed-off-by: Kalesh Singh --- mm/vmscan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6eecd291756c..b4329f93a682 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4656,6 +4656,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; struct folio *folio = pfn_folio(pvmw->pfn); + bool can_swap = !folio_is_file_lru(folio); struct mem_cgroup *memcg = folio_memcg(folio); struct pglist_data *pgdat = folio_pgdat(folio); struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); @@ -4704,7 +4705,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (!pte_young(ptent)) continue; - folio = get_pfn_folio(pfn, memcg, pgdat, !walk || walk->can_swap); + folio = get_pfn_folio(pfn, memcg, pgdat, can_swap); if (!folio) continue;