From patchwork Sun Jun 18 06:58:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13283773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E982EB64D7 for ; Sun, 18 Jun 2023 06:58:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02DB78E0001; Sun, 18 Jun 2023 02:58:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 003F56B007D; Sun, 18 Jun 2023 02:58:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0DDC8E0001; Sun, 18 Jun 2023 02:58:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CE1376B007B for ; Sun, 18 Jun 2023 02:58:23 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 97BA4403F6 for ; Sun, 18 Jun 2023 06:58:23 +0000 (UTC) X-FDA: 80914964886.30.41E77E2 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf16.hostedemail.com (Postfix) with ESMTP id D4343180014 for ; Sun, 18 Jun 2023 06:58:21 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="zOwV/CaH"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3DKuOZAoKCMcB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3DKuOZAoKCMcB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687071501; a=rsa-sha256; cv=none; b=tlEeGhPU2bzE+D76dYmxXEYZDtXvZFz5u6CpC3VxZWjWrRwqqfo2B09zZ2fWl2dYI0Eimg K2whzsIjAy+mAubcwlOTc7EQC30CbMKy3TbRqQa2aqWajjHr3/MJG9ilbVhjdUxF2RYS4G 9OGRD6qVG/xKRsWJCAOoONpoHk/N2+w= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="zOwV/CaH"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3DKuOZAoKCMcB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3DKuOZAoKCMcB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687071501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=YwXvQVSaypElesS3BE3qysx5jmIbVwSYC6z5eyt8hE4=; b=CDxqcB1Bl3ESK+rZxRXlYbHC1dhdTQfSNdop6iqn3wolp1NhJi+rTe5aXXhboSG7AUBWi9 sv1g1erbtA4mTRzCIgtFz8PQe26cXccDj5cgKY9Yby8hgzDo+SXHvzzzJ17FZ7kZYFiljV 9+HwbTTp2mmYaD2nawM/E3p4cGJz2UY= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-666e3b5d305so784473b3a.2 for ; Sat, 17 Jun 2023 23:58:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687071501; x=1689663501; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=YwXvQVSaypElesS3BE3qysx5jmIbVwSYC6z5eyt8hE4=; b=zOwV/CaHWAn0TdbeDv0YfetFHRqhjLbNtZofZk8gp1jhdDbApFYwHWaoGhfGloU8t/ ctAcxIf0+30pNej07lmQ6104qhnz655nV+5zO6qi9UaMQMboL93K3xQAOKbTh0qcXfl/ P09cWxqA2MK8VXEwo8LNT6A51QrppTROyxHRMUQFGJG6bHXe3KtwKdaPWg/jzyZxx6YD aCddrawGZdXnqKrbBY9qI0k68VFyjmbA4lyB+fCnMkH9IzjnjE9A9xiPEkbrWKosrxjF Fgyv7OpKDc1wOIhAVWrCmDBwHFHKObEurJqiCKO2a33jGe1yoE8KeFMDSuxwH8vwhDb9 RB6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687071501; x=1689663501; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YwXvQVSaypElesS3BE3qysx5jmIbVwSYC6z5eyt8hE4=; b=lSoPXPckpzRtAxnso9a5KVJCOR74SEhvVHhFXR7bvof3f0rbftSc1NwFTZRNWJFMPY xYdZuGis4UpRuXOvO4Il85PMp4mEL9vPMVwAM6bPW+0wJCGXT6HMutQE5iQdXxq0UulS Ke9WenuHZLKT5Qu178C1c6LoR8teZ0d8eQO6R96BVE+JcbPxN2D8hgcDB6xKwSha3AsH HQAUGuchm8NoEBwc8KnPlbIxcW7IGNrJB7LX9pdBWLyRmysHQodg2tQLRBkOvieb1kTq 6SizqdlSchyPvjpJdJyBVAeZr4p/SAg2QMvQ8+6xS2/F34EcMSeiRZ69IjNlx7dWayZE bhJw== X-Gm-Message-State: AC+VfDz2pew5OHaPbAgynD3JFKCYw/fDMyPx05U1uMaEhWsoMR8o8mWQ alL2iF2Cs939bzIeVSKQEE0jPv6Rf53xDY3I X-Google-Smtp-Source: ACHHUZ53HYo999vjmhrDC3s4hj57OygGWtz6Njl0zOWvVSrpTvDyQH1suL+U7VFBl2xQC7vNyDLGLfbC/tTModWT X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:aa7:8882:0:b0:652:b76e:43d2 with SMTP id z2-20020aa78882000000b00652b76e43d2mr1980763pfe.6.1687071500739; Sat, 17 Jun 2023 23:58:20 -0700 (PDT) Date: Sun, 18 Jun 2023 06:58:16 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230618065816.1365301-1-yosryahmed@google.com> Subject: [RFC PATCH 4/5] mm/vmscan: revive the unevictable LRU From: Yosry Ahmed To: Andrew Morton Cc: Yu Zhao , "Jan Alexander Steffens (heftig)" , Steven Barrett , Brian Geffon , "T.J. Alumbaugh" , Gaosheng Cui , Suren Baghdasaryan , "Matthew Wilcox (Oracle)" , "Liam R. Howlett" , David Hildenbrand , Jason Gunthorpe , Mathieu Desnoyers , David Howells , Hugh Dickins , Greg Thelen , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D4343180014 X-Stat-Signature: rcbi9e97oqf5jhhziy4mmewrjy99ase1 X-HE-Tag: 1687071501-799550 X-HE-Meta: U2FsdGVkX1/6CKmeISYl1ujsbZJNyo5bHcnM8o8Pus58dPOz9oJUOoq/2D3aMMdXQOrssWiuew6sO1DQRwtK5KyWduKK1nguKD0V06dOgg7097DTyET3mvayGqG4JdNdqCvKxiLuXAUJfSXUautHdlc+CdAW6LbSUw/Ekj0uZX5snbfCwuLURWRUrCzvvA7Ynxw/sHEXWJjZzQx5YiKSaDxED0eGNtj2JGAwKc52SNVGjOVZ8oxIX19TbvIh+pnlIjDLmjSAqs2G2Q/5E95qIjODZnTC+BhE5kpfOSiPbvi2dKcrO4DveL0yRgstK581NOtOlOx9VBTgacaoTQj+Cdl5+LYz7cRm8nubsOA+aT7wGZ9oIU5vazE8FhKxPzbHU9AVw0gJlD5VGs9IQ2tHKvSVFwHWMpLq8Dclmnn8cv8Tb1ie7KjOH4duQkTV7THouctUGLjSSRI0eT2jykcIUK0wtHTrkLBX9Z7PwnjrDSphTlH1sag/VSesn7sm4ceXF+ID0THRyrNWGR/WZlNWzfBcL2ufyxYXhojI/ANQ+Gnk1ympoIMpvhs/3WOEpCsWKgTcCJvzYOE3/uwtdne66Bj3KoSfVxv5umqFmY268NdV4Mf/XpXpH6ih4cqMBKBPj1NlmNiKZ3IgW+drkXZ3eQmwRQE+UTQUGakXvSAbAPDdDzSk+f0/5ZHQ3rOdBqBUowaRcuIllfo0FizSweXLBRj8yoTLmx4kF7sJKMr6HzHZbmoF38/JaoyIuAHssf6B1VFK3K1HbZyktIgfK9fq6GkgokBkZnAWDwdyACJRKGHzk3/2Lwj0soSjTSHS15NddGvjoN9iTzQoYVX9yMLDZbF94NJLTvyGmpNHuNhWukcL7UITkuD0gB/S5IMUcM7nS9oQGW/gv5CGbZmaZmCD0uvIRLe5IlF5DVvH7Akpgy33QuIzfMuEbPohqUEQR+YAswIbnp4+6YBtrSRDo1+ Dq7sgTWr MmmVDTAq7sS78zRt6Su8CbBAZRY/Ffk/gZWdjpJZJLVh0wG1mWoRm+pL/O7CVxYOi/w5aHDhIwzsIMHYsCxMlZQIpZlFwMPxqQhz5f9LnGOmF80NdABhAjbbm7yVy38i7vbOWLyRDHuD0288uxWXdfkzJ9m+DpPsKDBKakJ/VDMZ2GSMhS/abSs2Xf4cT7DiqgTbhF7myCgz7ucS5XKT1bTpOz5VPnSl6bsSXFkutCGjXw2aruSMW0iUIbbszgxwJxPLmcWAKb6OWNLA4wbx2YZ5QXVSEzOkuTBSoOxpCMtnW1itZQ4rQMpKzlOOfcgRL6t+tTJhyij5V+hvCPgvWBIx+Uy0IQBjif7wCbxYmDTMkLothvTn100t3RD6djdgAaFfl3C8bk3GD8DXJ0sPFfl+ES7DAnxTKskj8MMDcJBEMrIbXqRmhSkyDWLqRkTMKSMyjF1Xnuu8cUSbJZN1IfhT+6aHLZJE/disyN39d3VPTtKqRtO+fdpl7jihkM79fV5jApx/uedn1L2mxAO2SRZPr4QxveDZo5jbGeF7UY/6qxpTjkH6ZJpHKq9C7Cw/8Wy18/qjJanXPB1nMCPQ8oTl8+655UOzbqucDy+gPeVw/wEeo1YUvPuJzsbzgnPmoTL1wvAIl8ADJmzc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that mlock_count no longer overlays page->lru, revive the unevictable LRU. No need to special case it when adding/removing a folio to the LRUs. This also enables future work that will use the LRUs to find all user folios charged to a memcg, having the unevictable LRU makes sure we are not missing a significant chunk of those. Signed-off-by: Yosry Ahmed --- include/linux/mm_inline.h | 11 +++-------- mm/huge_memory.c | 3 +-- mm/mmzone.c | 8 -------- 3 files changed, 4 insertions(+), 18 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 0e1d239a882c..203b8db6b4a2 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -319,8 +319,7 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); - if (lru != LRU_UNEVICTABLE) - list_add(&folio->lru, &lruvec->lists[lru]); + list_add(&folio->lru, &lruvec->lists[lru]); } static __always_inline void add_page_to_lru_list(struct page *page, @@ -339,21 +338,17 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); - /* This is not expected to be used on LRU_UNEVICTABLE */ list_add_tail(&folio->lru, &lruvec->lists[lru]); } static __always_inline void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) { - enum lru_list lru = folio_lru_list(folio); - if (lru_gen_del_folio(lruvec, folio, false)) return; - if (lru != LRU_UNEVICTABLE) - list_del(&folio->lru); - update_lru_size(lruvec, lru, folio_zonenum(folio), + list_del(&folio->lru); + update_lru_size(lruvec, folio_lru_list(folio), folio_zonenum(folio), -folio_nr_pages(folio)); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0e5b58ca603f..4aa2f4ad8da7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2407,8 +2407,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail, } else { /* head is still on lru (and we have it frozen) */ VM_WARN_ON(!PageLRU(head)); - if (!PageUnevictable(tail)) - list_add_tail(&tail->lru, &head->lru); + list_add_tail(&tail->lru, &head->lru); SetPageLRU(tail); } } diff --git a/mm/mmzone.c b/mm/mmzone.c index 68e1511be12d..7678177bd639 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -81,14 +81,6 @@ void lruvec_init(struct lruvec *lruvec) for_each_lru(lru) INIT_LIST_HEAD(&lruvec->lists[lru]); - /* - * The "Unevictable LRU" is imaginary: though its size is maintained, - * it is never scanned, and unevictable pages are not threaded on it - * (so that their lru fields can be reused to hold mlock_count). - * Poison its list head, so that any operations on it would crash. - */ - list_del(&lruvec->lists[LRU_UNEVICTABLE]); - lru_gen_init_lruvec(lruvec); }