From patchwork Sun Sep 18 20:47:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7144ECAAD8 for ; Sun, 18 Sep 2022 20:48:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1D52940007; Sun, 18 Sep 2022 16:48:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECDD86B0075; Sun, 18 Sep 2022 16:48:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6CC8940007; Sun, 18 Sep 2022 16:48:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C93896B0074 for ; Sun, 18 Sep 2022 16:48:00 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A3ED480A52 for ; Sun, 18 Sep 2022 20:48:00 +0000 (UTC) X-FDA: 79926393120.01.9882C14 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf22.hostedemail.com (Postfix) with ESMTP id 325CBC0018 for ; Sun, 18 Sep 2022 20:47:59 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-345188a7247so238910707b3.22 for ; Sun, 18 Sep 2022 13:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=yukdN/NNiZkdpQOUkkw8VV3nsFMmM4H3QYKUUrnAd/U=; b=fS4bzBU3GIyv0KbuVZ2mzHkMN8Br35m0dNT58hAHVd1a0YnJpYGj5HG0ZGYFKes8ot kTGltQr120sEuMISpTlFs7MqNWzX9u2EwNrDY7SLDtb1+N1v2JVT4Hr2elH7Jw6UfVvB S/j2E69yjui8WW0v+wRrlMYWsT41SVqwXasx27guX1h7+aJKZGKHmwSvv8LkSMLDikBB oMd875pjB4vIEqeXkYW6YhbkPxHwdISa2lyktBTJGp+26aopdlpX215c6x4iidCRM0DQ tSO8+VcFeUJEKKGh0a4RqquQigezf7d/tg1xls1omXRtgbsFG0qDGlMHZTqHPa/f5iWb SU1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=yukdN/NNiZkdpQOUkkw8VV3nsFMmM4H3QYKUUrnAd/U=; b=d5jHL3NvkCoobWxoz126BnrK4vivboQWOKpFWmWRKPLr52jzs3j8V281ud7qDcIpSp l3gGMNpDHXUDk1FL+E0Nscr+uwXpCRzWxsTpplIsjPcz/urVtKUr9vlzYlKEY76sJHDN q1vj6GXBwM0x4IfLItlsxB1UMNob6g6am6rZYfESq5lweGyGrZA+mqP9eI8jgNnQZTqQ /iqhB0MzNY6CoXrP/Gtx9C7IcmM88D9obCFi/PXbVuW0gzTqvEa5GEOWvEDLelX+56TS f4iKA9/1YFqgGccEHE6iErh7rEEWXJgdcSTtpTYkP7hPj1lMzHW50Ty/y5Hv3ItixwxR lagQ== X-Gm-Message-State: ACrzQf2O/sWhmg2uPlhhOF2u52PpVksBbjdm6vminobxam1l+sqTGONj aPw8KGuTTfyi2+mhja0DcigAb7WZLr4= X-Google-Smtp-Source: AMsMyM5EIZEMGRedxI/d4mtyPvfC50YMjnZSbOfZxEdEsW0skxjj99QQdYzOfAOYUD9f2WwXIoIe+K9NgQE= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a25:4c03:0:b0:6b3:e5e4:cf29 with SMTP id z3-20020a254c03000000b006b3e5e4cf29mr2686638yba.553.1663534079319; Sun, 18 Sep 2022 13:47:59 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:45 -0600 In-Reply-To: Message-Id: <20220918204755.3135720-1-yuzhao@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 01/11] mm: multi-gen LRU: update admin guide From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao , Mike Rapoport ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534080; a=rsa-sha256; cv=none; b=sTusQDpQdLDWhgoJDqmKCiuIwBkSqsZTZd1YPA8i+SUOEA3IYGpoEiLVCIRlvBDGfmJHTr tZBt4JXfU7kiB60oD7UihrWVAdoPCePAeRByl9AeQmObYBn8SttarcBUchZIaS54nNpAI3 y0Lon7mK0Av3V1rWNVa4g0R/eLMvR0M= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fS4bzBU3; spf=pass (imf22.hostedemail.com: domain of 3_4MnYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3_4MnYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534080; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yukdN/NNiZkdpQOUkkw8VV3nsFMmM4H3QYKUUrnAd/U=; b=2YGShC4I9GOoxEwdRos2jGQz484ZgunxfXXrGf1Fvg+PBSLpeYhovL3WoE6vdu3O7LI6Ma pXj6WPjfckhbihi6E7lg/+Jz1heHKQccWJighc9PZ22vlukLNziU6tCtLkcNYGJhEWT1lg PUaY7ip0yoQBQg0LfQGIUd1wuFWb8ng= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fS4bzBU3; spf=pass (imf22.hostedemail.com: domain of 3_4MnYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3_4MnYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: 4enqrdkee3ok3pjxe877jxgindbm9i69 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 325CBC0018 X-HE-Tag: 1663534079-72220 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000034, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Link: https://lore.kernel.org/r/YvoOAJhJlvCr35Xc@kernel.org/ Suggested-by: Mike Rapoport Signed-off-by: Yu Zhao --- Documentation/admin-guide/mm/multigen_lru.rst | 50 +++++++++++-------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst index 6355f2b5019d..33e068830497 100644 --- a/Documentation/admin-guide/mm/multigen_lru.rst +++ b/Documentation/admin-guide/mm/multigen_lru.rst @@ -119,15 +119,18 @@ within ``age_in_ms``. E.g., ``min_gen_nr`` contains the coldest pages and ``max_gen_nr`` contains the hottest pages, since ``age_in_ms`` of the former is the largest and that of the latter is the smallest. -Users can write ``+ memcg_id node_id max_gen_nr -[can_swap [force_scan]]`` to ``lru_gen`` to create a new generation -``max_gen_nr+1``. ``can_swap`` defaults to the swap setting and, if it -is set to ``1``, it forces the scan of anon pages when swap is off, -and vice versa. ``force_scan`` defaults to ``1`` and, if it is set to -``0``, it employs heuristics to reduce the overhead, which is likely -to reduce the coverage as well. +Users can write the following command to ``lru_gen`` to create a new +generation ``max_gen_nr+1``: -A typical use case is that a job scheduler writes to ``lru_gen`` at a + ``+ memcg_id node_id max_gen_nr [can_swap [force_scan]]`` + +``can_swap`` defaults to the swap setting and, if it is set to ``1``, +it forces the scan of anon pages when swap is off, and vice versa. +``force_scan`` defaults to ``1`` and, if it is set to ``0``, it +employs heuristics to reduce the overhead, which is likely to reduce +the coverage as well. + +A typical use case is that a job scheduler runs this command at a certain time interval to create new generations, and it ranks the servers it manages based on the sizes of their cold pages defined by this time interval. @@ -137,20 +140,23 @@ Proactive reclaim Proactive reclaim induces page reclaim when there is no memory pressure. It usually targets cold pages only. E.g., when a new job comes in, the job scheduler wants to proactively reclaim cold pages on -the server it selected to improve the chance of successfully landing +the server it selected, to improve the chance of successfully landing this new job. -Users can write ``- memcg_id node_id min_gen_nr [swappiness -[nr_to_reclaim]]`` to ``lru_gen`` to evict generations less than or -equal to ``min_gen_nr``. Note that ``min_gen_nr`` should be less than -``max_gen_nr-1`` as ``max_gen_nr`` and ``max_gen_nr-1`` are not fully -aged and therefore cannot be evicted. ``swappiness`` overrides the -default value in ``/proc/sys/vm/swappiness``. ``nr_to_reclaim`` limits -the number of pages to evict. +Users can write the following command to ``lru_gen`` to evict +generations less than or equal to ``min_gen_nr``. -A typical use case is that a job scheduler writes to ``lru_gen`` -before it tries to land a new job on a server. If it fails to -materialize enough cold pages because of the overestimation, it -retries on the next server according to the ranking result obtained -from the working set estimation step. This less forceful approach -limits the impacts on the existing jobs. + ``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]`` + +``min_gen_nr`` should be less than ``max_gen_nr-1``, since +``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to +the active list) and therefore cannot be evicted. ``swappiness`` +overrides the default value in ``/proc/sys/vm/swappiness``. +``nr_to_reclaim`` limits the number of pages to evict. + +A typical use case is that a job scheduler runs this command before it +tries to land a new job on a server. If it fails to materialize enough +cold pages because of the overestimation, it retries on the next +server according to the ranking result obtained from the working set +estimation step. This less forceful approach limits the impacts on the +existing jobs. From patchwork Sun Sep 18 20:47:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED568C6FA82 for ; Sun, 18 Sep 2022 20:48:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BDF56B0074; Sun, 18 Sep 2022 16:48:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 46CB3940008; Sun, 18 Sep 2022 16:48:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30D8E6B0078; Sun, 18 Sep 2022 16:48:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 247306B0074 for ; Sun, 18 Sep 2022 16:48:02 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E2728AB47F for ; Sun, 18 Sep 2022 20:48:01 +0000 (UTC) X-FDA: 79926393162.12.4CAD2B3 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 948191A000A for ; Sun, 18 Sep 2022 20:48:01 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-349eecb5106so87472237b3.19 for ; Sun, 18 Sep 2022 13:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=j+OKHvKHG3fiMZDwJa80B6CugUV7TkNbiQbcStsRR/A=; b=hRuwBHeOlq07Z7oeNy0aK0jN/+XxMPz+xmH9wlezJnHOu18HbCg5bLy+gNTHzGzi4D 2yRbVUYAscIUe4jz1RMvL+CChY99mu55goMsLgfCC+GIfq7p2fCasGRSdtHt3WjjSnum DFe3Mb7I/YR0MDUWFG1Jo/oSE8B4oJXBLmhn8fyifBOZRtZ6db4ingqmvI6zdtxxSX4O Kg5Zv9TTkJPwLp/pqWZF32gKX1L6DBS3LDvr/taGLv8Rc+TUzopIpAeW6tQV/tG6NIMj tExAj35S6+fhL4VMP3kANHngr7oA+0/wKPCggtt5gMMZoiKWeodUgUU5DIWIYrIBuZ26 6HNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=j+OKHvKHG3fiMZDwJa80B6CugUV7TkNbiQbcStsRR/A=; b=w8ydKOz1BqYEvaLvLY0dWsRekqQITqrdYEle6/w/TOsUFwkeOT/SvqPayzYOButZYv FNRy4bd/hy3hBSffj0NArME7lpzUVyHlgyF4s3+sjOt3d1QS7EyVOmh3HdqWhpN/a5Sr 5TEdFUnA6aOvuqhQ9AW1BtuvfZqjLMeUzDR+wbN7F2FrtAA5yM3KCoCuGsTdeSW2E9Gf nuLgvpNV7ZefkRSyfsk3SJN8i38dibr4xg3extQba4LxsBAJJ4GU/v1Y3s7nBvJEdbx/ 7irlTMVv6aWRoJjOKQHAFpDeehcVreDvDcJz6prDlPZfR442TbRk23Bkrrg0rIORnzld dEBA== X-Gm-Message-State: ACrzQf2/sOsZt4gnoQkjr4dRADdxZaX0TCSegjHdAUL/BqkGHNXy+6nA o4v0/HXoFmUI51cYkFy0IlSAP93nc/Q= X-Google-Smtp-Source: AMsMyM538fxHaYlPJHI/OX6nai3ZPu7HSzj3U6KyRuCVm6+qJcbD2kyMoOeRVxU6InrzDbY4CJ76kInB16Y= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a0d:eb8b:0:b0:349:f236:cdd4 with SMTP id u133-20020a0deb8b000000b00349f236cdd4mr12550792ywe.198.1663534080921; Sun, 18 Sep 2022 13:48:00 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:46 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-2-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 02/11] mm: multi-gen LRU: add comment in lru_gen_use_mm() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534081; a=rsa-sha256; cv=none; b=Ylbk+F65MGGaGMpfFAn9dujhR5Owea12q4dCw0nQBW2Z03ASOboHaEfULckqrdpcpzyGyt /T2bJVBwpqonRqDKu+mmP4eI/uVpCHuWx19dJfQjwjGW07VcGjDz7wPL06/kc0Avs/v4Rp pSNC3bUdhlmuCPMSzwa8C69R9U+eD3Q= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hRuwBHeO; spf=pass (imf19.hostedemail.com: domain of 3AIQnYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3AIQnYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534081; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j+OKHvKHG3fiMZDwJa80B6CugUV7TkNbiQbcStsRR/A=; b=iz8lyLZ1M86hVi0KhUuN7l15U8Z2/06ljwHi1kmyo8M5CmAhSGKgwxWR8cf6cxpqTlVAt3 DWIjyHWzg4t+hHcrzfvVKGUhXVYwf78A5n+wTkfeVYyzcUfW928ZKzW5jSQGHmlLUs5BLj 159Vs5EdUIG3BXE50WeaOFm+2oTmiUI= Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hRuwBHeO; spf=pass (imf19.hostedemail.com: domain of 3AIQnYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3AIQnYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: 97xsre9fahg8fdcxhpithjaukp1s9ptk X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 948191A000A X-HE-Tag: 1663534081-25519 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Link: https://lore.kernel.org/r/CAOUHufY91Eju-g1+xbUsGkGZ-cwBm78v+S_Air7Cp8mAnYJVYA@mail.gmail.com/ Suggested-by: Andrew Morton Signed-off-by: Yu Zhao --- include/linux/mm_types.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 882c31e03cca..500e536796ca 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -18,7 +18,6 @@ #include #include #include -#include #include @@ -770,6 +769,11 @@ static inline void lru_gen_init_mm(struct mm_struct *mm) static inline void lru_gen_use_mm(struct mm_struct *mm) { + /* + * When the bitmap is set, page reclaim knows this mm_struct has been + * used since the last time it cleared the bitmap. So it might be worth + * walking the page tables of this mm_struct to clear the accessed bit. + */ WRITE_ONCE(mm->lru_gen.bitmap, -1); } From patchwork Sun Sep 18 20:47:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63254C6FA86 for ; Sun, 18 Sep 2022 20:48:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9C92940009; Sun, 18 Sep 2022 16:48:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2432940008; Sun, 18 Sep 2022 16:48:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C618940009; Sun, 18 Sep 2022 16:48:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 83C90940008 for ; Sun, 18 Sep 2022 16:48:03 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 56C79160302 for ; Sun, 18 Sep 2022 20:48:03 +0000 (UTC) X-FDA: 79926393246.17.71A4C37 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 135CC4000C for ; Sun, 18 Sep 2022 20:48:02 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id u12-20020a25094c000000b006a9ad6b2cebso23623206ybm.15 for ; Sun, 18 Sep 2022 13:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=oLphItSceKYTp0uPvFiu9gpbpwSB2H2p/Gmn36qHs5Q=; b=Gr2Dwa6beWc7RsEPq11s6/aXuhMiGVUf5rv5bcGpTulgEyMysSFBYN+ACqhwRxusnw BmwHxlF+9qPIFPwwHUCau/CQ5YI7KgA7+DQQZosSJusDEjd5gZnOFgH/2hUQQXcRk5yv qMTWCIaaqZkh33z3ZBqAvCvqVk2tYhjseyAQQE03Pm+XnmLf039IEVeiyuH/zb5vRsb/ 6yYBvOEMaHNGZPc4sX/XvwgYPXO+g+mPkeBAO9bJUB1J7rvIj6dQ0e2+MEZn36d0vmT6 fRfrwl15mOtf+5DJrhNgE+Ct/amOGR0GHwtckx73xKP0fgbKA8t0daE8tOJZSPS4d1X9 3S/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=oLphItSceKYTp0uPvFiu9gpbpwSB2H2p/Gmn36qHs5Q=; b=BwQGwbABxCbhvjJAIzk5TC0SmRtyUiqFPHuQzxC7Osdb6Wibd7R18OGxr+taFENJs6 p0HyOVmU8eiXdWWuo5NDjfI0lsfxreZxruvqT1LBDz5c7o1crlN/2LxRz/JcUQktN82C G/m2F2GElmTe2YTje035maxNLpHlOWtuU5Zr/UilBbOicgLr6JWD8WHfoucsB0GTGTjR 33rDbtEpK5IXkDEAHS3d1oS6nT4SKs/8EP1aB3+uwVbU5GPSitfdaPwMnNyMi58UgJvX WCDJB8Pit0FpAmk1lX9+U1a9sj41H/ZvkgK67ObtkX73xD1ofh4mxBMHOWgyK1Doyx00 6rdg== X-Gm-Message-State: ACrzQf3jI0LJcFidYT3pUxIE6rh0Jx85cjfDN3B9PZ7ch5B7QBFqdceA zKpQkZypqSmeMhqbWpJCkyLzkeeQj0Q= X-Google-Smtp-Source: AMsMyM6ofQh0byKCv5h0Eeua94mtAvgxlikfbLfXlDuMWYponQOZBI6ROWGKX6HvRMrEnKjIlLRf6Q1GCEk= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a25:6c3:0:b0:6b0:4336:9d81 with SMTP id 186-20020a2506c3000000b006b043369d81mr12177705ybg.119.1663534082383; Sun, 18 Sep 2022 13:48:02 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:47 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-3-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 03/11] mm: multi-gen LRU: warn on !ptep_test_and_clear_young() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao , Nadav Amit ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Gr2Dwa6b; spf=pass (imf04.hostedemail.com: domain of 3AoQnYwYKCMcB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3AoQnYwYKCMcB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534083; a=rsa-sha256; cv=none; b=Cnfrhw90n2MsECmZT/b8XPaVwDLgwLJbwWRf5r/u8hIAHJIbwYlAWy6ThLPcqI1r8zxq7F 289hxHV6O0vOPIwAeo3T5p2fFP7QO7FFeYepMMkDOfFEMHsco/cfUgh2ofqhpW0+8cKsd6 zr/HXNCtkj0tC38J+i46MeE8EUIHM8A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oLphItSceKYTp0uPvFiu9gpbpwSB2H2p/Gmn36qHs5Q=; b=CQrOUuq8iML+pWaJ/spdwo6UKiA354czpmlC8oQj9e0V93uNYav/2ANGWyGZFpq1L1i67B +pwVunzbb99nCMWWqMZOeAph9A1Yiowk9sKFpDkeR2ZVIUH0UK2jx7NARnrc1bQWEHqzJB x7SbGHlYg7qADOOc0l/kZS1q7E38tEY= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Gr2Dwa6b; spf=pass (imf04.hostedemail.com: domain of 3AoQnYwYKCMcB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3AoQnYwYKCMcB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: 696zhkqtb3nky4ipmhhhcyorayz4ab37 X-Rspamd-Queue-Id: 135CC4000C X-Rspamd-Server: rspam03 X-Rspam-User: X-HE-Tag: 1663534082-659046 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Link: https://lore.kernel.org/r/0F7CF2A7-F671-4196-B8FD-F35E9556391B@gmail.com/ Suggested-by: Nadav Amit Signed-off-by: Yu Zhao --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5c4d9d0867ec..629518813b59 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3904,7 +3904,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, continue; if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) - continue; + VM_WARN_ON_ONCE(true); young++; walk->mm_stats[MM_LEAF_YOUNG]++; @@ -4637,7 +4637,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) continue; if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) - continue; + VM_WARN_ON_ONCE(true); young++; From patchwork Sun Sep 18 20:47:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C5A6ECAAD8 for ; Sun, 18 Sep 2022 20:48:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C94494000A; Sun, 18 Sep 2022 16:48:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0756F940008; Sun, 18 Sep 2022 16:48:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBD0394000A; Sun, 18 Sep 2022 16:48:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB256940008 for ; Sun, 18 Sep 2022 16:48:04 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9B088A03C1 for ; Sun, 18 Sep 2022 20:48:04 +0000 (UTC) X-FDA: 79926393288.02.276DDCE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 41B7E180006 for ; Sun, 18 Sep 2022 20:48:04 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id n6-20020a5b0486000000b006aff8dc9865so12710178ybp.11 for ; Sun, 18 Sep 2022 13:48:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=Lw4scwFqzp1HreT7ffbgAOkeM6+LsydrTcmSI+YwUBg=; b=i+DsSbyQyJa9qdQMAHmq56D1M1P8DGQCwaK6s+/VBX6H8yAssf2yez96SlWRqiTTGe wQd/ywEIQEMsT5az0U0OZSmwRmIW8KQT1xMa4nTw8ddhOICbEoYsVNbTJGevtayyzbDx SW5ONhnfrXUJP1/QaIs2tJclO/T20QgE/BY4wp1OD3Hy00DixaxiVNHfDBPCRO6fXjtS ne1ZZ7y7oqP6tQ0I4h25wiQYeUDCYfvzACQ3Yycps18qPqYIpY/g8vy7/P9hga2JwTPx c2OOAN/+5CnQ7N7kyL8T5ncebdN+DA9ywGgUYdnj+Fy1wuY03MK+Q5nU7J5FJSuXYomw xP5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=Lw4scwFqzp1HreT7ffbgAOkeM6+LsydrTcmSI+YwUBg=; b=dZp7bLVHc9cB/PwjV7hhFYSu6KehmQo/G9xNL1Rm0ytzkN7ElWl1eZ4hvT+3tKBrCH HAFNCC5g9C9fxKqnmTD/1rc6g5vDF2uV9pLiTw8I6KLhs0IzfgPZQfB85c1zsV3SgRyo d4MLvsRKaw6ppMZJJxpWn/RBQnmnBRmiby7UbZypRboWqDVEFWj6Tu6+6M7IbVz7IxIB YhKROJfRGw44tXagp64yOLA+8lNEEikhGtPjjJ8Y0xHcchc/hWdeW30igEzj8j7/OBp6 DnTegKF8SkGoUzOBJEiuGPHRfLnI4hCYApJzMP0R9noX3JINKrx+llhVRSA5vo6Ce+L8 Mlag== X-Gm-Message-State: ACrzQf2Xxy3SULaCmei5yOgN2xLC1heVFQvniwDFWxhYV/GWaUOvEch/ iQr/6Ohi07h8QBlI0iQzvKqUSprZKz0= X-Google-Smtp-Source: AMsMyM41HFDvPrAKBbudcLn8GRnN0tvMl7hkoDh0FrTlA0nyM+Se6UA6dczZsxeYcseJbgADNU4yhD1FHRI= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a5b:cc4:0:b0:6ae:2a6c:59e6 with SMTP id e4-20020a5b0cc4000000b006ae2a6c59e6mr11920830ybr.59.1663534083673; Sun, 18 Sep 2022 13:48:03 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:48 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-4-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 04/11] mm: multi-gen LRU: fix warning from __rcu From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao , kernel test robot ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534084; a=rsa-sha256; cv=none; b=Eb5Yaujs9aj97rgJg/YnXGARuUSKursuW92drs8Wm//4HNHNh3PJfnDc3zCPJ54FgZO2NU uEuufQg3SIaqXAyjiEqTOBnh5QUZbk7/HcI+3YJu4Dp38e/lwsXZXL210gOPQ8qs7Bl+Tt KedrGB1Zq/rvM2TuZYfrE87i+B2i/zA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=i+DsSbyQ; spf=pass (imf24.hostedemail.com: domain of 3A4QnYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3A4QnYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534084; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Lw4scwFqzp1HreT7ffbgAOkeM6+LsydrTcmSI+YwUBg=; b=zcmHx9w+mghnmdsrp7JWhv/1rbulCx3PrnPPnywdui5DCcZjpDI37Jo/gEyAS3V9AUQEiV xiOQzuhPhh7PEjzOXO7JHrzyiPaoGoCajcHoi728MLOZiL6maLiosnBVOEjLI89rlAsJRa PvXYB2xo0CrcXgSTx/nhVBb/FZjFJjg= Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=i+DsSbyQ; spf=pass (imf24.hostedemail.com: domain of 3A4QnYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3A4QnYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: s46xqdig33rzsms9ojzo1bwo1knotzzg X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 41B7E180006 X-HE-Tag: 1663534084-964627 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Link: https://lore.kernel.org/r/CAOUHufY8+O_MwkrPzgfBKSFwLJ1v6LPDox1yGV0tjX4ozjh2Dw@mail.gmail.com/ Reported-by: kernel test robot Signed-off-by: Yu Zhao --- mm/memcontrol.c | 2 +- mm/vmscan.c | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8086edbdd39b..ac6440daf208 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6289,7 +6289,7 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) return; task_lock(task); - if (task->mm && task->mm->owner == task) + if (task->mm && READ_ONCE(task->mm->owner) == task) lru_gen_migrate_mm(task->mm); task_unlock(task); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 629518813b59..8291808ec535 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3270,15 +3270,17 @@ void lru_gen_del_mm(struct mm_struct *mm) void lru_gen_migrate_mm(struct mm_struct *mm) { struct mem_cgroup *memcg; + struct task_struct *task = rcu_dereference_protected(mm->owner, true); - lockdep_assert_held(&mm->owner->alloc_lock); + VM_WARN_ON_ONCE(task->mm != mm); + lockdep_assert_held(&task->alloc_lock); /* for mm_update_next_owner() */ if (mem_cgroup_disabled()) return; rcu_read_lock(); - memcg = mem_cgroup_from_task(rcu_dereference(mm->owner)); + memcg = mem_cgroup_from_task(task); rcu_read_unlock(); if (memcg == mm->lru_gen.memcg) return; From patchwork Sun Sep 18 20:47:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEE41C6FA86 for ; Sun, 18 Sep 2022 20:48:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46FCE94000B; Sun, 18 Sep 2022 16:48:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AA37940008; Sun, 18 Sep 2022 16:48:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24BE094000B; Sun, 18 Sep 2022 16:48:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 16DA8940008 for ; Sun, 18 Sep 2022 16:48:06 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E0D1081048 for ; Sun, 18 Sep 2022 20:48:05 +0000 (UTC) X-FDA: 79926393330.03.7D0D862 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 9594A100008 for ; Sun, 18 Sep 2022 20:48:05 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-349e6acbac9so89567637b3.2 for ; Sun, 18 Sep 2022 13:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=9fbu9/AcjqREk+tOcTZ0vFQGumfdYFLY0ujIQ9fmw6U=; b=DT/BXh3iJqzMhyJl9tq44ENviA2gqUFRt1e0POy5yIY6e6a1ZU2f2UbjtBSizGG3n2 ngXv5hn+GpRCNyo4i3umQ6g2rRyHUnW8BNz5GOFUB5cNzroNH7MzsKMOKY0qJaCv4SMP CB0uzpXNB9W2/rGxeC9q+gjBoFpXjhBob6nJ/hTflbgN+QglBbMC8PM771rV1jY4VKaU MWdrkx9Xwa3fsQfiAGixdtnUAIm+fZWdT68QClwccdrOoS8zHllPBgOruk6AZZWzrQfS UtmVPZZIBYQ+AkqVMkxxo6VBmvri7R29/kAqYPT2lSLPXaejWDdbtT1gOfrN25fpf8kT No7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=9fbu9/AcjqREk+tOcTZ0vFQGumfdYFLY0ujIQ9fmw6U=; b=sdD0CKyjJBoLt7l99WB4ebhU2d+2LeLYP71VbJzFnDkdFOTpSJxnnmetva2GmbYx7w gJPRHyy9kSJFixbvOGOkLbz2RrJ/lSPcI5NobzZA1O9ub3aXo85ilYexiUk/29wNBmpO MO/5Fm0TLg8Y3Xe8Hf14woD4/kUWseFTWprFhd0eLyFDI21GUBe5zajwOKnus1FzaK2p Bhpdp5Fmf2/epYjbmLVz5B6PmTg4oTGxkHJNQRb6WNdpB56opZ4X8kQKCO9AP9Z5nRnV tDaO7f29aAa4trA95sTHlq2w9IasC6TJqTllFo0OuUvHZVcLjreLSzRDqwAQu0OprEqP i3Sg== X-Gm-Message-State: ACrzQf29NpU0sbWQM1OCeSlKLX340Sl17FqelGnsToXNF1OsuFVaIW/4 G5B2YP1u6iPkJ1sp4l7ULCC1051Kz1c= X-Google-Smtp-Source: AMsMyM67plZ1qP8bFbUsVexyiFwS2xUKgwLHoQkzH5A5xCnZ9z4ssVrJrMYYXPUL1E81yD/3jrPT4HkL6N4= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a25:24c1:0:b0:6aa:11fe:9136 with SMTP id k184-20020a2524c1000000b006aa11fe9136mr11994402ybk.373.1663534084973; Sun, 18 Sep 2022 13:48:04 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:49 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-5-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 05/11] mm: multi-gen LRU: fix warning from seq_is_valid() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao , David Rientjes ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534085; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9fbu9/AcjqREk+tOcTZ0vFQGumfdYFLY0ujIQ9fmw6U=; b=mH7t60XoZxnKuxPJhZE5UybxPue53SrBKCIIKl6rZMXMUTytBU58F06aCB9zr7zff0Vw1w Sij+hzpK2mV/tvTzhN6k36aC0qUTGedfxJFxFqAwl3l8eLii9To7DQ2l/W0oPXCEgEv9q2 a71/AcSCp812pYxR7hkBh5quIG1FTjU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="DT/BXh3i"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3BIQnYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3BIQnYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534085; a=rsa-sha256; cv=none; b=jubonHeaZgI8OdZd52z06u3almDNoWMvGvs2IELcs8CBU65KzWt2iImuCR/IgfTZUQKC6Z X99dz0swowB0paPJtSDBfC9FX6zY9grSw6+4a1AJ7jYjAwHRlyJFwB4jOYSJEZepZUjhti lHYNNwgAP9OAjHQkvad5oYQP8QjbxW0= Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="DT/BXh3i"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3BIQnYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3BIQnYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com X-Rspam-User: X-Stat-Signature: jqttmyx1jkecsyqi7fwwkinkjidbqaim X-Rspamd-Queue-Id: 9594A100008 X-Rspamd-Server: rspam12 X-HE-Tag: 1663534085-461811 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: v5 (link below) calls inc_min_seq() in the correct order but rewrites in the later versions broke it. WARNING: CPU: 0 PID: 0 at ./include/linux/mm_types.h:613 context_switch+0x375/0x3e0 __schedule+0x443/0x610 schedule_idle+0x27/0x40 do_idle+0x322/0x360 cpu_startup_entry+0x65/0x70 rest_init+0x1e7/0x1f0 start_kernel+0x529/0x5a0 x86_64_start_reservations+0x13f/0x150 x86_64_start_kernel+0x1f2/0x200 secondary_startup_64+0xa6/0xb0 Link: https://lore.kernel.org/linux-mm/20211111041510.402534-7-yuzhao@google.com/ Reported-by: David Rientjes Signed-off-by: Yu Zhao --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8291808ec535..e509e896c019 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4307,7 +4307,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); - for (type = 0; type < ANON_AND_FILE; type++) { + for (type = ANON_AND_FILE - 1; type >= 0; type--) { if (get_nr_gens(lruvec, type) != MAX_NR_GENS) continue; From patchwork Sun Sep 18 20:47:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54FB5ECAAD8 for ; Sun, 18 Sep 2022 20:48:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0C5C94000C; Sun, 18 Sep 2022 16:48:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6DFB940008; Sun, 18 Sep 2022 16:48:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B259D94000C; Sun, 18 Sep 2022 16:48:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A4DE4940008 for ; Sun, 18 Sep 2022 16:48:07 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 78CA0C012A for ; Sun, 18 Sep 2022 20:48:07 +0000 (UTC) X-FDA: 79926393414.08.869B832 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 1FCCF40010 for ; Sun, 18 Sep 2022 20:48:06 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id u12-20020a25094c000000b006a9ad6b2cebso23623255ybm.15 for ; Sun, 18 Sep 2022 13:48:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=ADjZhLB2lH6XDF/bluSQyDYiqynm0JA4dN820og+pN0=; b=EKBvpt34rlhTK5I0GzuLGC8jyVXUC0KpgyukD4r5rAEYDacS720kmut1oA8mJjm+3n STilIqNq6htNbfcq+OvkYN5PdVhRFOhuGlYsjB7Dl8g94rjLv6q7VrRrjwzplvOUjxxd N7TEgHbk65sbS6caMP+kOfyBRkLwJk7PtxP8kQorx4XVG/B5j0ZF2EMRTEtxwznM4/Lv EYli7mpYWzi0gnYoAJIoY5oMbUDksCsq7Vvq2lZS8EJY+o40LeVA4Nw4sYqM8LM3Ijou acV/kG0ki44dDdfybZrU1RwDxjBb6SIeu8M4qZnjw7KKG1MXJcvtmH/j4ohFIkIn+PyH OL3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=ADjZhLB2lH6XDF/bluSQyDYiqynm0JA4dN820og+pN0=; b=Iul7ck08RksEENcQRWzVlqbZUOHEUrS4WRS3kZrbAV8d9EIJ13RxZXlj5DZ/oWoikC aGJkuRm72oEzTZu3DH/SmJENNEf4Q3kcAHI5QpyOlTEVZ1/JwWOOBrSMZQPvBjpnE1BU WlWv34HncjXMBXJSpbbSQEyviFHKf/NpEbVFjUsytXP8a1+9UmN+YA+NMVvKMNHZdB7F 0Gvle+J6sSFYtL8Jdd4KBNrpjSolVQFNze6hAovMLKjeAJbAngZbDT/2pRW4TJ8OHL2F +surHFt9F6N+sILzqMp+ZhdeLRQwX6/N5A+glUbo1U4kQ7v/3hFhbnOTZGzNHZFuRLqm //fA== X-Gm-Message-State: ACrzQf3/St41TynnCOJAQ9wEw97ByIvv0LUSLG6f+9KqbYSo23BbGyob RBw+ynU3O0SS3JzeiLnaZCB8s2SajuI= X-Google-Smtp-Source: AMsMyM5D6LMhndcaLHs/g0PMv4A6e/Blkb6flfK2s8k2DE/ZlN4ZC43SwHhh91v8K+zpiIgrBJ4WuL27ZaU= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a05:6902:1146:b0:6ae:72f2:62cb with SMTP id p6-20020a056902114600b006ae72f262cbmr13337927ybu.615.1663534086599; Sun, 18 Sep 2022 13:48:06 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:50 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-6-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 06/11] mm: multi-gen LRU: delete overcautious VM_WARN_ON_ONCE() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao , David Rientjes ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EKBvpt34; spf=pass (imf04.hostedemail.com: domain of 3BoQnYwYKCMsFBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BoQnYwYKCMsFBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534087; a=rsa-sha256; cv=none; b=1Qu906BCZeAE9bU/7QUIBFnzKEE5nmTrGVA1naaGWCaElrFxTiujcs7jWIgS1wGyanyEEf y1dUMfA1cQJxOYLmMByza2OY05BfRyk44xBAPAu6KmGNHLl0+4bl8/cQ06FDERryybkHdm urvxiwHP1qbtgSpes90XqfSzXdgX/BQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534087; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ADjZhLB2lH6XDF/bluSQyDYiqynm0JA4dN820og+pN0=; b=QE4TAPn/Pe/0KNnVJZMhx9RHkKJKh6rJFkp0BitXApfSA70zp5Q1t3HtOBahlakiMByAQ3 vMpCdI437G2E3rKVCjxknuRoHwtNrMQ9FO3SfNoKXpuXK/npjWALD2YRPwJEpxolR/B17g 98XgyT6NyRM6PCOp/mALzBIvFTMaRcU= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EKBvpt34; spf=pass (imf04.hostedemail.com: domain of 3BoQnYwYKCMsFBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BoQnYwYKCMsFBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: 6t4nk311fcox1w7cxduutoy5t6ruk4un X-Rspamd-Queue-Id: 1FCCF40010 X-Rspamd-Server: rspam03 X-Rspam-User: X-HE-Tag: 1663534086-945354 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As commented, it is "unlikely but not a bug when reset_batch_size() is pending". This one was never reported but another similar one in lru_gen_use_mm(), which is already deleted, indeed happened. kernel BUG at mm/vmscan.c:4592! Call Trace: ? get_nr_to_scan+0x89/0x2a0 lru_gen_shrink_lruvec+0x1aa/0x2b0 shrink_node_memcg+0x68/0x450 shrink_node+0x25c/0x620 shrink_zones+0xf4/0x170 do_try_to_free_pages+0xa5/0x280 try_to_free_mem_cgroup_pages+0x161/0x1f0 try_charge+0x2b1/0x940 __mem_cgroup_try_charge+0x102/0x3d0 do_anonymous_page+0x152/0x440 handle_mm_fault+0x9d2/0x1030 __do_page_fault+0x388/0x550 do_page_fault+0x2b/0x60 page_fault+0x45/0x50 Reported-by: David Rientjes Signed-off-by: Yu Zhao --- mm/vmscan.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e509e896c019..77d2d08950ba 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5171,9 +5171,6 @@ static bool __maybe_unused state_is_valid(struct lruvec *lruvec) for_each_gen_type_zone(gen, type, zone) { if (!list_empty(&lrugen->lists[gen][type][zone])) return false; - - /* unlikely but not a bug when reset_batch_size() is pending */ - VM_WARN_ON_ONCE(lrugen->nr_pages[gen][type][zone]); } } From patchwork Sun Sep 18 20:47:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC9A7C6FA86 for ; Sun, 18 Sep 2022 20:48:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3369C94000D; Sun, 18 Sep 2022 16:48:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E736940008; Sun, 18 Sep 2022 16:48:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1614894000D; Sun, 18 Sep 2022 16:48:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0667A940008 for ; Sun, 18 Sep 2022 16:48:09 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DCC301A021B for ; Sun, 18 Sep 2022 20:48:08 +0000 (UTC) X-FDA: 79926393456.14.4A564B4 Received: from mail-io1-f73.google.com (mail-io1-f73.google.com [209.85.166.73]) by imf14.hostedemail.com (Postfix) with ESMTP id 9692D100008 for ; Sun, 18 Sep 2022 20:48:08 +0000 (UTC) Received: by mail-io1-f73.google.com with SMTP id y1-20020a056602200100b006893cd97da9so13913174iod.1 for ; Sun, 18 Sep 2022 13:48:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=k+YWtvvaIypH46pOKiODFPRAWHrTFXoO8GcKmdHjqbk=; b=OlMG2YYSoaI4mZjuUmKIILmFBQzOnDlntGN26Om2dA9+EwzyWhX97ME2QjP4FF54Wq HcSym0iULyVcDBaU2cx2CdanghelI8OWweDwCogiLpqhjrPBg3H9piqlPZovPx9B0iCv 52bBOovYofIH7wCGf6K7eBN3tymCCD5V7lAvBFWCB7yzDJnTCoFdlukwqfX9fPdripCg P6NO83BFhBxtUiuahMHxOuRS55HLoV8lTlmOJVTQbpTDOYeJRhzlkaMyJOQlWHTzbDAS cOzq69Bjn+8Q8iNzV14aVV7vRz/WQ8Rhdu9uAzUk52ZlcRjJ2WE5IFCohHPs+DceuPOv f/8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=k+YWtvvaIypH46pOKiODFPRAWHrTFXoO8GcKmdHjqbk=; b=XwwVcYqD+ITa4Wkwmr9r8DSUV3bvKbvKvAtefIm1gxeWsxM+rKH/RN2Erccqs+APeo GJqHcKfR4VaSA3BRlOqeStbQLd4pPhwSuwpETC5TmO8PEwwOarkh++ACIvjzJx6JVy4e P9WzK5RXgHyiqDsHwUxXq4D65A7A99mOOfDjEERt2LXiTW4/mxp/swIQJ3jQGOGyj8vj Eo1FdfFqwoBt7889r+IFO1/buhBchI0sSSzjUEznHP3aOf1BmUVSkRUu7ES8P+7GznKC i8wZbFBU2+9R27mCzzBXJTsbcGDLicgnC/aNbWSi3pAcBUyrE6VtuJpme1CejWFn5a+p 0Eew== X-Gm-Message-State: ACrzQf3F7pif0/JJExKUmzR4LHCVjvc1WlBJw9ZU7LXXZZmf4Ll8l9kX +34XVA2197cozd/EULl6t/KV34CKXYM= X-Google-Smtp-Source: AMsMyM6S1PVX7RT1nMZd0WLX8Q7GYSpbx2WFMj/oICTcp1esFdFpsSG/YWX1wuL0PQMTWVzz34FUpFZvhtw= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a5d:8a02:0:b0:6a2:250e:a800 with SMTP id w2-20020a5d8a02000000b006a2250ea800mr5958799iod.153.1663534088059; Sun, 18 Sep 2022 13:48:08 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:51 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-7-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 07/11] mm: multi-gen LRU: dial down MAX_LRU_BATCH From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao , David Rientjes ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534088; a=rsa-sha256; cv=none; b=vvAFG6vNXJMx7lJB9Rsng/XUP3x5vzh18PKaIsFQcyZ9IC9J59mdHAW4Xg952KBrZiUb2k 5hKv9S6YNTH+c7jIjOJX8RcPNgKcsBnN7p+fwn8Ugy5uNEJdIP693/hxpNC/q5Ixgolcai 9UNGs+DIw2vXPpN1YoQz2N7jNklWXaM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OlMG2YYS; spf=pass (imf14.hostedemail.com: domain of 3CIQnYwYKCM0HDI0t7z77z4x.v75416DG-553Etv3.7Az@flex--yuzhao.bounces.google.com designates 209.85.166.73 as permitted sender) smtp.mailfrom=3CIQnYwYKCM0HDI0t7z77z4x.v75416DG-553Etv3.7Az@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k+YWtvvaIypH46pOKiODFPRAWHrTFXoO8GcKmdHjqbk=; b=s0Uovs0dKqdomnCI1MbhWSrydWMJt4Stbsoy5DePsnTeW8LLfPtv0Qn8apuctp+ONEBxBx FaWOc/7fVSfm2Lg1VZ+qk6nQMPJTkPBhgn+C8snbe7pE2JKKPmJ26bAhYW97ayNGy66aQN uXrgGt/ozKkD6IkmFgcMJeK9SQj+tPQ= X-Stat-Signature: 57uwyigb3tueiuq5pu646jb8r6egkg57 X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OlMG2YYS; spf=pass (imf14.hostedemail.com: domain of 3CIQnYwYKCM0HDI0t7z77z4x.v75416DG-553Etv3.7Az@flex--yuzhao.bounces.google.com designates 209.85.166.73 as permitted sender) smtp.mailfrom=3CIQnYwYKCM0HDI0t7z77z4x.v75416DG-553Etv3.7Az@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9692D100008 X-HE-Tag: 1663534088-623454 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A need_resched warning was reported and already fixed by adding a need_resched() check to walk_pud_range(). Dial down MAX_LRU_BATCH anyway in the interest of direct reclaim latency. WARNING: CPU: 22 PID: 2771 at kernel/sched/core.c:3637 scheduler_tick+0x339/0x410 Call Trace: update_process_times+0x7b/0x90 tick_sched_timer+0x82/0xd0 __run_hrtimer+0x81/0x200 hrtimer_interrupt+0x192/0x450 smp_apic_timer_interrupt+0xac/0x1d0 apic_timer_interrupt+0x88/0x90 RIP: 0010:walk_pte_range+0x1c2/0x6a0 walk_pmd_range+0x1ed/0x490 walk_pud_range+0xe6/0x2b0 __walk_page_range+0x111/0x690 walk_page_range+0x4e/0x150 walk_mm+0x110/0x200 try_to_inc_max_seq+0xdb/0xb10 lru_gen_run_cmd+0x153/0x1c0 lru_gen_run+0x150/0x210 stale_page_run+0x62/0x730 kthread+0x148/0x1b0 ret_from_fork+0x54/0x60 Reported-by: David Rientjes Signed-off-by: Yu Zhao --- include/linux/mmzone.h | 2 +- mm/vmscan.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 2b3f273faf68..4c8510f26b02 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -381,7 +381,7 @@ enum { }; #define MIN_LRU_BATCH BITS_PER_LONG -#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128) +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 64) /* whether to keep historical stats from evicted generations */ #ifdef CONFIG_LRU_GEN_STATS diff --git a/mm/vmscan.c b/mm/vmscan.c index 77d2d08950ba..2add99eecd0c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4400,7 +4400,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, } while (mm); done: if (!success) { - if (sc->priority < DEF_PRIORITY - 2) + if (sc->priority <= DEF_PRIORITY - 2) wait_event_killable(lruvec->mm_state.wait, max_seq < READ_ONCE(lrugen->max_seq)); From patchwork Sun Sep 18 20:47:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 460E2ECAAD8 for ; Sun, 18 Sep 2022 20:48:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D98A594000E; Sun, 18 Sep 2022 16:48:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D47AD940008; Sun, 18 Sep 2022 16:48:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0E7494000E; Sun, 18 Sep 2022 16:48:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B0FC3940008 for ; Sun, 18 Sep 2022 16:48:10 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8605D1207D7 for ; Sun, 18 Sep 2022 20:48:10 +0000 (UTC) X-FDA: 79926393540.10.89E3E24 Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf10.hostedemail.com (Postfix) with ESMTP id 4E144C001D for ; Sun, 18 Sep 2022 20:48:10 +0000 (UTC) Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-1278ff55da4so13434339fac.18 for ; Sun, 18 Sep 2022 13:48:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=9GLicrM8HZnHJRJdZ3RIuVbsRgADZtKmcAsJemCjhkc=; b=DyM3DX2OXRJARSUvzx4Yl7vLAglGMSIwOOJDlNmxNSkMmOkD/FkLhRSVn+2qH5ZOm3 QM6m4PszlF2otJ/D7d0b7DLUfbwv0BjOuuMkG5sOH010eWgLk1J6dJ/7S3vU3UJR/NJA doVzKKg9pm1URMMEcA7jrvZyhnQy+FpTRH66pkLYNlyXB/3rrIE2qIaFGFPw23+ysIsc t+wGqeMpvFNCvUL5QhE/BuDNDraZDQqrpdqaxc1qmTXcgMc0eQYRY8qrwUOVpsYC85Dy 0uhMEnixelD14cdYd7NIDPpZUa+v96Z1rJf/DiYO/RzWmNvg5f1Sdt7mQ8V11ed+t+f1 bz9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=9GLicrM8HZnHJRJdZ3RIuVbsRgADZtKmcAsJemCjhkc=; b=GICMFFLinz7Hm1QHodfbkUMffjESuPB9kjAXbbgFgIZdGy+nutqoMHPjhlKvciJwzh UGFdEqTMeUsI2YmIaEIJud573We5yNBW6hrSDRtjAL1q9MxzZWXT+XPDaK6mHh/+8myS IiIJiE4awAT8BqYY9jQiJWIHF2PJ5Aj0tZjjvvcjnD9p9Rw5PRkfCt3FD1vwYRWJOaKs 953ktBpGpsRM3gPyuUO1ikSiCRvvdXiKWoOxGujwjeRgSv7krnmYo18li+FsfDXoJaj7 8BgxzplV0ikltpR1CO+Od4MYPf50Tut5UkW/MdE4TOM3yyCKs/+hMcYFcTdMjp3f4Oix 1gng== X-Gm-Message-State: ACgBeo2pzFrBS4ZW26n+74tZ/UScq00g+3kQwduHpUie4qOnUx3WVJ2C sK/vI0jLwGU4JWafsDpDevYgAmdfdfU= X-Google-Smtp-Source: AA6agR7qXwx//DIVRPJC6EweKVG+aTNhUGvr456YoXdjoyIRZBMzrDHXOfZl740Sgvb+xyVNwp21Sm5sy/c= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a05:6870:568b:b0:127:9fdd:f31b with SMTP id p11-20020a056870568b00b001279fddf31bmr14262728oao.79.1663534089448; Sun, 18 Sep 2022 13:48:09 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:52 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-8-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 08/11] mm: multi-gen LRU: delete newline in kswapd_age_node() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534090; a=rsa-sha256; cv=none; b=OJKizy9iioI2jXtAnvL05rCda87ld/0JVvYTygdp5kUYb+1QqBxDXD6f0vA8vAkDtSjqLB tHdkVqoxzV2LSdkA8cbcl9MwK49AwLA4mrXr+k7de1x39mFmi2grKszP1Y6oy90gcilHXk VXSodaxFLOLB1fTwmuMbKWrgSYLnRQw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DyM3DX2O; spf=pass (imf10.hostedemail.com: domain of 3CYQnYwYKCM4IEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3CYQnYwYKCM4IEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9GLicrM8HZnHJRJdZ3RIuVbsRgADZtKmcAsJemCjhkc=; b=hkJNq9r5J5aCZhGK3+LlPzAbuqz2TOcs/68L/uXCaufFKvcfNoKSjCRziX+iQ2sBOVuPAg oPmr7pH1POW9hFyAAu1RmpeSWzdIaeWf+M3lBzxw5ytNeeUzIjWYc9RtQ5cjo+t51pSyr0 wfB5Dy4kzu7hGgPPKc6mgqejj7p+ubA= X-Stat-Signature: h319ayfrwdmxgq53p5tgiie6r8x5keph X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DyM3DX2O; spf=pass (imf10.hostedemail.com: domain of 3CYQnYwYKCM4IEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3CYQnYwYKCM4IEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4E144C001D X-HE-Tag: 1663534090-430323 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000010, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Yu Zhao --- mm/vmscan.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2add99eecd0c..1c5c357bee42 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6699,8 +6699,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, } #endif -static void kswapd_age_node(struct pglist_data *pgdat, - struct scan_control *sc) +static void kswapd_age_node(struct pglist_data *pgdat, struct scan_control *sc) { struct mem_cgroup *memcg; struct lruvec *lruvec; From patchwork Sun Sep 18 20:47:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEFE2C6FA82 for ; Sun, 18 Sep 2022 20:48:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 471D194000F; Sun, 18 Sep 2022 16:48:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 35D1C940008; Sun, 18 Sep 2022 16:48:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BA1D94000F; Sun, 18 Sep 2022 16:48:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 06C29940008 for ; Sun, 18 Sep 2022 16:48:12 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D8603A03C1 for ; Sun, 18 Sep 2022 20:48:11 +0000 (UTC) X-FDA: 79926393582.18.63DB261 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 8D1A640007 for ; Sun, 18 Sep 2022 20:48:11 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-34a03cb9679so66693717b3.21 for ; Sun, 18 Sep 2022 13:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=D5bqFcM02ok2Op9XffwMYvsMuGuxkAbvMVE2W4HraDE=; b=TuDIfV7XafYybhVc761tfvG9sZFGTpi7Z8r9tEqHNPVEXnmpLRIenytt+RetlqRrKs mf88Piy3neW7vqCl474V/f+fZ0orMw6pjtedVlOocOA5EKDXfDjHcUieCjP9WSBX/Yju I8GdCFCyNlr9GmLbqAG9lzOiRmCM00SZzhdFBQ3EWWSZSecmHrP8zHgQRRZmAzH/704m 6sJEt54coI+D0qeZ8ufdG8q9uEIjvPRXdVLlZmPx6AttrOqXcCGor8Gq07/LqLZ+ACsB o50zXl3bhSZ91zW9DIJODMOWLRlRn8Hru0YByEotUQVpIeQFp0E4AHaK2yVMQfebrDSw xshg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=D5bqFcM02ok2Op9XffwMYvsMuGuxkAbvMVE2W4HraDE=; b=Fz7lfbg1M3XOXgkuwYwNYUPTKP3ldfp6IuACNtRMII7U4MdJeensFuHXxAGCBtJWju 1RjpKrCyif+fUy1p4VqS/lGQAhGS8+QSLTT42Bs9h3yMlR2VJK+ouOCeq/jvOjWmSZt+ uy3s0d7v10oemYSUWU//IEdqeBSrksHOo6SNtkVAarxszmIZfsEOOTDCNAfc5Kyr1NKS i/BXQdC8m3CsGuUVVnJySEzTfoZjWYAEf5v0c2xBt4heOrLS8Srhvv3e5RmCOwnaOji4 dEnHbaJbHAQbRvktp7DqjV4VsGeCwTV7gUfWIchmtBhZEcWUFOADXm9WGhm/mdwFqS5d f2Tg== X-Gm-Message-State: ACrzQf0Z/3CV42zmtQ5dvEcveknQ8VnR2ovAbtddffjhLp4pxu8jYjL8 8FjXUEnkCmvFVV83fRyIUJWK38a9Vx4= X-Google-Smtp-Source: AMsMyM7VeW75cNZqfLRi0jskMtexUZEuaL+tNX+J26vrr0HuZQg0eIjJXeqHSwz5wdE1Ck7zGF8tY1AFf0M= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a25:7784:0:b0:6ab:4853:e6f9 with SMTP id s126-20020a257784000000b006ab4853e6f9mr12909970ybc.476.1663534090968; Sun, 18 Sep 2022 13:48:10 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:53 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-9-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 09/11] mm: multi-gen LRU: add comment in lru_gen_look_around() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D5bqFcM02ok2Op9XffwMYvsMuGuxkAbvMVE2W4HraDE=; b=d1A5+1zMZtFtP4cUkByHmY0pcHvNeCo45MYlRUc9ijQHciVLW6JbDz1uiPHRx1lfiV6kL3 UPMjQ089lblxp3iUkd5CGyh5UP5fF5T1ucWyPVTuiDcTuPpHA7EoIPd+wYUtBGOdnlegyo T1ZzxmTo/782xlyEeB//1tYYRf3vsB4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TuDIfV7X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3CoQnYwYKCM8JFK2v919916z.x97638FI-775Gvx5.9C1@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CoQnYwYKCM8JFK2v919916z.x97638FI-775Gvx5.9C1@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534091; a=rsa-sha256; cv=none; b=3TAq5AECX231zgfwuHrC0SN5k6nn2mhYziZLFdXomvfJhKASlRNDpo36TSapfOGRO8+WMQ N/nNEEC3GViOlVwOt14hDyQWVQ1HUbtUKUXoQeXE6nDPZhItnPk4nVOFd2lp2f+p8w9j08 rSrI0I/RzaIVz985f41zVsWN209leH4= X-Stat-Signature: eusdejqn1ap585jqfdg4qwwtpmunmhtw X-Rspamd-Queue-Id: 8D1A640007 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TuDIfV7X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3CoQnYwYKCM8JFK2v919916z.x97638FI-775Gvx5.9C1@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CoQnYwYKCM8JFK2v919916z.x97638FI-775Gvx5.9C1@flex--yuzhao.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1663534091-969012 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Yu Zhao --- mm/vmscan.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1c5c357bee42..f6eab73bdfb9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4604,6 +4604,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (spin_is_contended(pvmw->ptl)) return; + /* avoid taking the LRU lock under the PTL when possible */ + walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; + start = max(pvmw->address & PMD_MASK, pvmw->vma->vm_start); end = min(pvmw->address | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1; @@ -4619,7 +4622,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) } pte = pvmw->pte - (pvmw->address - start) / PAGE_SIZE; - walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; rcu_read_lock(); arch_enter_lazy_mmu_mode(); From patchwork Sun Sep 18 20:47:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53DACC6FA86 for ; Sun, 18 Sep 2022 20:48:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8CFD940010; Sun, 18 Sep 2022 16:48:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 950E5940008; Sun, 18 Sep 2022 16:48:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 755A2940010; Sun, 18 Sep 2022 16:48:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 65551940008 for ; Sun, 18 Sep 2022 16:48:13 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3ABB8C012A for ; Sun, 18 Sep 2022 20:48:13 +0000 (UTC) X-FDA: 79926393666.23.3574AC0 Received: from mail-il1-f201.google.com (mail-il1-f201.google.com [209.85.166.201]) by imf04.hostedemail.com (Postfix) with ESMTP id E912040007 for ; Sun, 18 Sep 2022 20:48:12 +0000 (UTC) Received: by mail-il1-f201.google.com with SMTP id o2-20020a056e0214c200b002eb8acbd27cso18515814ilk.22 for ; Sun, 18 Sep 2022 13:48:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=kHKKwMzpUGjXWL0kFP6rN6rEGcUyiSLgiCEgbGar96E=; b=Yj+AkXfSoZoe96HOU5B3jM2wroc05eNPTFeByQdkSPci8jc82UE/hkeJOA86FOTSw2 SOT1uCHPydR6W2ZR1i0jChC02Z/KBCWcRMgziQEBqrx0BIWQ35vtCxjE+q+RwuOi4+JX CwmgN0zCIiTAzy78nikwhCws2lIiHQNXJVHwIHjKyayxrtqIs5vty+hmIVHBOU+yCJzQ yWWDWrJVnfmUcW6zfTASlyNMfNmD+U92FDjdaLh81bjsMKoNrw92KGZ5574L3wiywFZL RAGCt5nwTQ6Kv/f25o3MPz8lOxtisXcR0HWROg8Tk6ACKMf+7h4QJVkeMYf+InAh75Au F2bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=kHKKwMzpUGjXWL0kFP6rN6rEGcUyiSLgiCEgbGar96E=; b=DCl987mqBW0ipZq6SIWjXZkzCgQeUm5uUFP9kF1U+heVbG8LdjTGl/kZJpjZxoksn0 LBp447kwPR8ZPv1buiQ4QV8CoQKTCpWAYlaoRnQ8HOzydRx/Xlpy98OXgxeiDIo4poUv ruC83Pp1hfen3mDIzB0AeNj9ULbxpV53+Pq2jmgWrHqmU5O3/HOhZGXXRvWN2TzD8Plh 1DKTeui2Klk32lRUWcx+EDUFKaTn5jp4ctqrc3OfiuEp4UWf7G1vSuP/Dl4VQ1EBDkLC ZgAtIBBefj9Mx2jg2klGZlY4eVhHONpY/Sxm07mTZvmcKnenL64amkGyL0X6JCwA9kHA 7rgQ== X-Gm-Message-State: ACrzQf0L08L+wYSVbSIapIYc+YsgQOPW/5bCanOJrrIoQmx1CPi7i1j2 SRT05CoQQTJdiG1QwWjOS0wvP38a3PU= X-Google-Smtp-Source: AMsMyM6baew0MYTta3xEIkNjyvoxjG5PDbmEYdcpgyifTX4ObSi0SaqQHHOuX6ApD8MLK4X4i31aUA44gnM= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a92:d843:0:b0:2f3:5f18:bbe7 with SMTP id h3-20020a92d843000000b002f35f18bbe7mr6221368ilq.108.1663534092309; Sun, 18 Sep 2022 13:48:12 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:54 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-10-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 10/11] mm: multi-gen LRU: fixed long-tailed direct reclaim latency From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534092; a=rsa-sha256; cv=none; b=N53MLWFUWpqeI92NhVfPdCKzoXzPyXY/pTD7BTQAwc8zgA2UgdXFOLzPLAAnu3TM64mqNf F03SiaQA2pg9MkcORkZmWJ8+bZQHVdXYIX45SO9sjFAEts+y3CiH74gAF1ZKq2BXuGqSCx ttsdxcqKurhYtMHxX13b+uxjSx+eHdc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Yj+AkXfS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3DIQnYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com designates 209.85.166.201 as permitted sender) smtp.mailfrom=3DIQnYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534092; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kHKKwMzpUGjXWL0kFP6rN6rEGcUyiSLgiCEgbGar96E=; b=7vmGf7Ij/BGTzW5+0tta2diwrgT49CsikZBAn+K3ZLpMoZsvLyi5baCZKqpD4Q4pzbjc1D 5MqaFaUYcd/Qoo1yxTP7Q20d3lx/vGS9/BbP95KuRSTQ/fnU5ctznMOm25TgdYQyLkSKij JXlb9SSMredgG72Ej4vQaNNdanzzeBE= X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E912040007 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Yj+AkXfS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3DIQnYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com designates 209.85.166.201 as permitted sender) smtp.mailfrom=3DIQnYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com X-Stat-Signature: pti8uacujf4rowr5fya6i53agk4kb3uy X-HE-Tag: 1663534092-580964 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Long-tailed direct reclaim latency was seen on high-memory (TBs) machines: MGLRU is better at the 99th percentile but worse at the 99.9th. It turned out the old direct reclaim backoff, which tries to enforce a minimum fairness among all eligible memcgs, over-swapped by about (total_mem>>DEF_PRIORITY)-nr_to_reclaim: /* adjust priority if memcg is offline or the target is met */ if (!mem_cgroup_online(memcg)) priority = 0; else if (sc->nr_reclaimed - reclaimed >= sc->nr_to_reclaim) priority = DEF_PRIORITY; else priority = sc->priority; The new backoff, which pulls the plug on swapping once the target is met, trades some fairness for curtailed latency. Specifically, in should_abort_scan(): /* over-swapping can increase allocation latency */ if (sc->nr_reclaimed >= sc->nr_to_reclaim && need_swapping) return true; The fundamental problem is that the backoff requires a sophisticated model and the previous one was oversimplified. The new one may still be, but at least it can handle a couple more corner cases on top of the above: /* age each memcg once to ensure fairness */ if (max_seq - seq > 1) return true; The NR_FREE_PAGES check at the bottom of should_abort_scan(). Signed-off-by: Yu Zhao --- mm/vmscan.c | 105 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 70 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f6eab73bdfb9..50764b2d462f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -135,10 +135,9 @@ struct scan_control { unsigned int no_demotion:1; #ifdef CONFIG_LRU_GEN - /* help make better choices when multiple memcgs are available */ + /* help kswapd make better choices among multiple memcgs */ unsigned int memcgs_need_aging:1; - unsigned int memcgs_need_swapping:1; - unsigned int memcgs_avoid_swapping:1; + unsigned long last_reclaimed; #endif /* Allocation order */ @@ -4524,22 +4523,19 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) VM_WARN_ON_ONCE(!current_is_kswapd()); + sc->last_reclaimed = sc->nr_reclaimed; + /* - * To reduce the chance of going into the aging path or swapping, which - * can be costly, optimistically skip them unless their corresponding - * flags were cleared in the eviction path. This improves the overall - * performance when multiple memcgs are available. + * To reduce the chance of going into the aging path, which can be + * costly, optimistically skip it if the flag below was cleared in the + * eviction path. This improves the overall performance when multiple + * memcgs are available. */ if (!sc->memcgs_need_aging) { sc->memcgs_need_aging = true; - sc->memcgs_avoid_swapping = !sc->memcgs_need_swapping; - sc->memcgs_need_swapping = true; return; } - sc->memcgs_need_swapping = true; - sc->memcgs_avoid_swapping = true; - set_mm_walk(pgdat); memcg = mem_cgroup_iter(NULL, NULL, NULL); @@ -5035,7 +5031,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap sc->nr_reclaimed += reclaimed; - if (type == LRU_GEN_ANON && need_swapping) + if (need_swapping && type == LRU_GEN_ANON) *need_swapping = true; return scanned; @@ -5047,19 +5043,13 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap * reclaim. */ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, - bool can_swap, unsigned long reclaimed, bool *need_aging) + bool can_swap, bool *need_aging) { - int priority; unsigned long nr_to_scan; struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); - if (fatal_signal_pending(current)) { - sc->nr_reclaimed += MIN_LRU_BATCH; - return 0; - } - if (mem_cgroup_below_min(memcg) || (mem_cgroup_below_low(memcg) && !sc->memcg_low_reclaim)) return 0; @@ -5068,15 +5058,7 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * if (!nr_to_scan) return 0; - /* adjust priority if memcg is offline or the target is met */ - if (!mem_cgroup_online(memcg)) - priority = 0; - else if (sc->nr_reclaimed - reclaimed >= sc->nr_to_reclaim) - priority = DEF_PRIORITY; - else - priority = sc->priority; - - nr_to_scan >>= priority; + nr_to_scan >>= mem_cgroup_online(memcg) ? sc->priority : 0; if (!nr_to_scan) return 0; @@ -5084,7 +5066,7 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * return nr_to_scan; /* skip the aging path at the default priority */ - if (priority == DEF_PRIORITY) + if (sc->priority == DEF_PRIORITY) goto done; /* leave the work to lru_gen_age_node() */ @@ -5097,6 +5079,60 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * return min_seq[!can_swap] + MIN_NR_GENS <= max_seq ? nr_to_scan : 0; } +static bool should_abort_scan(struct lruvec *lruvec, unsigned long seq, + struct scan_control *sc, bool need_swapping) +{ + int i; + DEFINE_MAX_SEQ(lruvec); + + if (!current_is_kswapd()) { + /* age each memcg once to ensure fairness */ + if (max_seq - seq > 1) + return true; + + /* over-swapping can increase allocation latency */ + if (sc->nr_reclaimed >= sc->nr_to_reclaim && need_swapping) + return true; + + /* give this thread a chance to exit and free its memory */ + if (fatal_signal_pending(current)) { + sc->nr_reclaimed += MIN_LRU_BATCH; + return true; + } + + if (cgroup_reclaim(sc)) + return false; + } else if (sc->nr_reclaimed - sc->last_reclaimed < sc->nr_to_reclaim) + return false; + + /* keep scanning at low priorities to ensure fairness */ + if (sc->priority > DEF_PRIORITY - 2) + return false; + + /* + * A minimum amount of work was done under global memory pressure. For + * kswapd, it may be overshooting. For direct reclaim, the target isn't + * met, and yet the allocation may still succeed, since kswapd may have + * caught up. In either case, it's better to stop now, and restart if + * necessary. + */ + for (i = 0; i <= sc->reclaim_idx; i++) { + unsigned long wmark; + struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; + + if (!managed_zone(zone)) + continue; + + wmark = current_is_kswapd() ? high_wmark_pages(zone) : low_wmark_pages(zone); + if (wmark > zone_page_state(zone, NR_FREE_PAGES)) + return false; + } + + sc->nr_reclaimed += MIN_LRU_BATCH; + + return true; +} + static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { struct blk_plug plug; @@ -5104,6 +5140,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc bool need_swapping = false; unsigned long scanned = 0; unsigned long reclaimed = sc->nr_reclaimed; + DEFINE_MAX_SEQ(lruvec); lru_add_drain(); @@ -5123,7 +5160,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc else swappiness = 0; - nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness, reclaimed, &need_aging); + nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness, &need_aging); if (!nr_to_scan) goto done; @@ -5135,17 +5172,15 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (scanned >= nr_to_scan) break; - if (sc->memcgs_avoid_swapping && swappiness < 200 && need_swapping) + if (should_abort_scan(lruvec, max_seq, sc, need_swapping)) break; cond_resched(); } /* see the comment in lru_gen_age_node() */ - if (!need_aging) + if (sc->nr_reclaimed - reclaimed >= MIN_LRU_BATCH && !need_aging) sc->memcgs_need_aging = false; - if (!need_swapping) - sc->memcgs_need_swapping = false; done: clear_mm_walk(); From patchwork Sun Sep 18 20:47:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EBC6ECAAD8 for ; Sun, 18 Sep 2022 20:48:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B003940011; Sun, 18 Sep 2022 16:48:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 137DF940008; Sun, 18 Sep 2022 16:48:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7F42940011; Sun, 18 Sep 2022 16:48:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D89E3940008 for ; Sun, 18 Sep 2022 16:48:14 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B64B0A0353 for ; Sun, 18 Sep 2022 20:48:14 +0000 (UTC) X-FDA: 79926393708.19.A1DC0A5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 75EA0C0004 for ; Sun, 18 Sep 2022 20:48:14 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-349f88710b2so80747247b3.20 for ; Sun, 18 Sep 2022 13:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=4dMrFmMaEOJAkK/RVbE/otUUczJ0DfDH7kAUZcwg9Yo=; b=mic9okYpjd7t6Jy5SOgxblMABXWu8QZ9mNcw9JTed6mYg+nmY7RzlqvVB2ErsX6N4a yAB4ietgY9OuD9fXPDEDPy0eToAZXtqKs/HCTWY2T8YtgmimfMEjFHyKlZ3J1c3Bh+/F yDHNs3ETSKaBkxzO3Vnq0/rKmhydea7Fa4MjlC61kjBSNXAIjgpRMvQmTc5OeBgB6v1O 8C5jJrAFdrqUM8VKYasNHPNHq2UVOMeSsx0mLqNPrcbNniW+xgU7kyUtHqO++Bnc/Tmd nzC4EG5g7KoIuQDITJteNalaKHst/GzH45goMiHGNz53zLsTlp/9NkinSzaTjk7xT25Z qpeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=4dMrFmMaEOJAkK/RVbE/otUUczJ0DfDH7kAUZcwg9Yo=; b=6uNu6OzDLOnfeWXQJ0Dm2H9YVXUF+JtjzHoZw7pjPRss90bQr0ePrBsyfUlE995mvO Fph+T3XhVn6Fqglyhw+Q3kgFdu563299T3VRhgVO3ypvKCmSrB5gXqm8fqjQxM7ZViCw lPbUZ6C3r3slIFfzxPnW0RVYkW/9HwvbCjvYDdWk3M2FizZJ9ugh844vmhmhveV+H4lw v2mJGp8MbGJW6r0WPLsI/Vsgxq9I84f4s1edJiv5RALvGOI+QvaXLNEDM7/9Zjb6WcCN bjkAuAjSIvpKXuDMtv/wDL7pRdX6bWUvjQIxP6wlucpge2v6q6eDo/kWigr27yX3w+vu JpDw== X-Gm-Message-State: ACrzQf3GUhpCsX0DO/+0tvaf5De2uGb/HDr30aWykkLFa8ZWmuD/sSKI nvYUGHaVVzxjhWeYUp5KoCOpl4t8CoM= X-Google-Smtp-Source: AMsMyM69sdTfA4Q/ZiGuBDxMN+9Z3VE4aZlnyp0qY7hf8grZciFnwRGs3pmk9Nqfr2T6UdRBc3cvPBYSRRA= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a25:3491:0:b0:6ae:e4dc:93ec with SMTP id b139-20020a253491000000b006aee4dc93ecmr11873221yba.26.1663534093878; Sun, 18 Sep 2022 13:48:13 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:55 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-11-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 11/11] mm: multi-gen LRU: refactor get_nr_evictable() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534094; a=rsa-sha256; cv=none; b=muAeCD+q7FzSjhX+zcCHA6+fGhVLQZM+VGGAdVba8lHzL6iCjih5w+wMShIST6cw/jTTbw lRhqMpt2Y2yneGdIbfg849KpunaHGJ2DuJN2pKp/k0C7SbsAqZZ6Zz5x6YAzQBjyF7ICFu h/LbERvDucXHiHEdsOLHe8641GfGrL4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mic9okYp; spf=pass (imf28.hostedemail.com: domain of 3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4dMrFmMaEOJAkK/RVbE/otUUczJ0DfDH7kAUZcwg9Yo=; b=PmFz7yCHJtu5kIjwILxdVFpsmDdhUquUUEr69fHK2A91YbsJFgedkFzu8vhVG7nRde6bXU fbt1SdZbQPe4BJ+bykixq0iXsWTXMWxduxc0JxpAA4xkgJ9x7SJN2j/jXY8U8a+sksG8I1 IbPDUy9mapXyzhe3Mqb294TBCY7tYLM= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mic9okYp; spf=pass (imf28.hostedemail.com: domain of 3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: kdz4jantwsuriickbj74z8o8rgajmipt X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 75EA0C0004 X-HE-Tag: 1663534094-673552 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous commit made this refactor possible: 28 insertions and 36 deletions. Signed-off-by: Yu Zhao --- mm/vmscan.c | 64 +++++++++++++++++++++++------------------------------ 1 file changed, 28 insertions(+), 36 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 50764b2d462f..a8fd6300fa7e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4418,14 +4418,15 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static unsigned long get_nr_evictable(struct lruvec *lruvec, unsigned long max_seq, - unsigned long *min_seq, bool can_swap, bool *need_aging) +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsigned long *min_seq, + struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) { int gen, type, zone; unsigned long old = 0; unsigned long young = 0; unsigned long total = 0; struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); for (type = !can_swap; type < ANON_AND_FILE; type++) { unsigned long seq; @@ -4441,35 +4442,37 @@ static unsigned long get_nr_evictable(struct lruvec *lruvec, unsigned long max_s total += size; if (seq == max_seq) young += size; - if (seq + MIN_NR_GENS == max_seq) + else if (seq + MIN_NR_GENS == max_seq) old += size; } } + /* try to scrape all its memory if this memcg was deleted */ + *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; + /* - * The aging tries to be lazy to reduce the overhead. On the other hand, - * the eviction stalls when the number of generations reaches - * MIN_NR_GENS. So ideally, there should be MIN_NR_GENS+1 generations, - * hence the first two if's. - * - * Also it's ideal to spread pages out evenly, meaning 1/(MIN_NR_GENS+1) + * The aging tries to be lazy to reduce the overhead, while the eviction + * stalls when the number of generations reaches MIN_NR_GENS. Hence, the + * ideal number of generations is MIN_NR_GENS+1. + */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) + return true; + if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + return false; + + /* + * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) * of the total number of pages for each generation. A reasonable range * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * eviction cares about the lower bound of cold pages, whereas the aging - * cares about the upper bound of hot pages. + * aging cares about the upper bound of hot pages, while the eviction + * cares about the lower bound of cold pages. */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) - *need_aging = true; - else if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - *need_aging = false; - else if (young * MIN_NR_GENS > total) - *need_aging = true; - else if (old * (MIN_NR_GENS + 2) < total) - *need_aging = true; - else - *need_aging = false; + if (young * MIN_NR_GENS > total) + return true; + if (old * (MIN_NR_GENS + 2) < total) + return true; - return total; + return false; } static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned long min_ttl) @@ -4488,11 +4491,7 @@ static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned if (mem_cgroup_below_min(memcg)) return false; - nr_to_scan = get_nr_evictable(lruvec, max_seq, min_seq, swappiness, &need_aging); - if (!nr_to_scan) - return false; - - nr_to_scan >>= mem_cgroup_online(memcg) ? sc->priority : 0; + need_aging = should_run_aging(lruvec, max_seq, min_seq, sc, swappiness, &nr_to_scan); if (min_ttl) { int gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); @@ -4506,7 +4505,7 @@ static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned return false; } - if (nr_to_scan && need_aging) + if (need_aging) try_to_inc_max_seq(lruvec, max_seq, sc, swappiness, false); return true; @@ -5054,14 +5053,7 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * (mem_cgroup_below_low(memcg) && !sc->memcg_low_reclaim)) return 0; - nr_to_scan = get_nr_evictable(lruvec, max_seq, min_seq, can_swap, need_aging); - if (!nr_to_scan) - return 0; - - nr_to_scan >>= mem_cgroup_online(memcg) ? sc->priority : 0; - if (!nr_to_scan) - return 0; - + *need_aging = should_run_aging(lruvec, max_seq, min_seq, sc, can_swap, &nr_to_scan); if (!*need_aging) return nr_to_scan;