From patchwork Tue Feb 28 08:50:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13154534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BE36C7EE31 for ; Tue, 28 Feb 2023 08:50:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 321FB6B0075; Tue, 28 Feb 2023 03:50:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AD556B0078; Tue, 28 Feb 2023 03:50:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 124C36B007B; Tue, 28 Feb 2023 03:50:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F1A516B0075 for ; Tue, 28 Feb 2023 03:50:10 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BF2A6121371 for ; Tue, 28 Feb 2023 08:50:10 +0000 (UTC) X-FDA: 80516078580.24.A490E97 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf30.hostedemail.com (Postfix) with ESMTP id 0952580011 for ; Tue, 28 Feb 2023 08:50:08 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QpsaKClB; spf=pass (imf30.hostedemail.com: domain of 3P8D9YwoKCNwWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3P8D9YwoKCNwWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677574209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tgJC52QM6QqWzETYUfTKUV+TRKWCtnTb99Tc/z4xtcc=; b=NgEbCQ0ETeJkoWjKnwXYCmckPQtaJxOOnEQt4k+NTzmyf5wbS9ZxsdZK9MFge2y6xSId02 CrSLaSGsCIr+2cERMaY9OuDg+LXewa5eUCp29I+3OWrC2Gqlr/UxqfpYMJzTFbQLtcEokK ZRM6onML8J89vRm1z8uM3zWrLPcqpXg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QpsaKClB; spf=pass (imf30.hostedemail.com: domain of 3P8D9YwoKCNwWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3P8D9YwoKCNwWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677574209; a=rsa-sha256; cv=none; b=mcQer18pAij4KCISMYl4KaD1Ub8bdX0+n3kndXuxd+nb82WsfFqM30LTcN1ftL7zDKBvBi 3AT6OfBXQSwBRjGMSRqBMR48I7Q70jgR9GBzUFy7keG1LrQOBJpimj1s0VXqD9kPhYNNRY qFazNvQeU/uoO7u6GxX+cYTERcbOPfo= Received: by mail-pf1-f202.google.com with SMTP id a10-20020a056a000c8a00b005fc6b117942so1839304pfv.2 for ; Tue, 28 Feb 2023 00:50:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1677574208; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tgJC52QM6QqWzETYUfTKUV+TRKWCtnTb99Tc/z4xtcc=; b=QpsaKClBM6T/l157IH+7qI9Rdj6857P2of63Qy25b1ULux6HWBa7tPwczUKR822sQm Jtu4ES6VL+1NRykXmz5aiv4K55zgEU1gyffkFKfifRDKyIBnA0uJ1jjdMSq3yLorLuby A7J9nOJUGetgqw5d1Alpv1sY4oIyc7bWZD02dBipyIYqxTGJWAjh1Aoxhy0vJ4ZuLYiU ZcI+vNG4fis3fO4Q1DKAztlEJeJR69f7liSjn2WRt9qFaSe5iB8T7zMWLuiVof/9K0MV SOrTQToIej9jdyCvxdFzm7JPtyyAS6uRvP9Ms6dFChBa7ZvjPPagDLqcGbXyvNXqmUa6 KqIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677574208; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tgJC52QM6QqWzETYUfTKUV+TRKWCtnTb99Tc/z4xtcc=; b=E5dVp7YheiZ8SWugXKQgt0LE+KtviaWCSZKaDzoOLjMOj2OQV4LQMpd5YYIeZ7zXFp CSz8ynZxkYi5hrRaMqqdUpB/5W3a8A5Rit2wkWylNJ9NTThcApAqpitlvVLpFuId+Bam S3l5haxl7DHGkOqpOUja+8LHECu2CpiOYrEisV0IRBlnSTaYfOdaKP+UNckxEpHwQZMn Gf1jr5JAK1QjpMViizj507YCfmb6XzAsAWapdLbc5xv2CQQOP1WoYOnF2gyivVO2QXe5 8oV5PJFINJ8t8rIwjq7dZHPYRIpWiFFLKTQVV28X4+C2At6WFSwzFYgO6QSKVYkMmno2 Rmkg== X-Gm-Message-State: AO0yUKW6QdS7OylQL58keq3bVWxyuqs+mKCafsQ87NNAdRsp5tB1GPi0 aXd7va0PFsKpJ6dBc7wEGYg3zSe0XM+bBo6l X-Google-Smtp-Source: AK7set8t0cooG+i7Em12R7p9gzkWi0x9flCtehwMdDtbP+QKpSIaBR21aL7xJpXfLxb2tbubY3iK1iSuvetMpUhV X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:b21b:0:b0:502:f4c6:3992 with SMTP id x27-20020a63b21b000000b00502f4c63992mr521403pge.4.1677574207767; Tue, 28 Feb 2023 00:50:07 -0800 (PST) Date: Tue, 28 Feb 2023 08:50:01 +0000 In-Reply-To: <20230228085002.2592473-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230228085002.2592473-1-yosryahmed@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230228085002.2592473-2-yosryahmed@google.com> Subject: [PATCH v1 1/2] mm: vmscan: refactor updating reclaimed pages in reclaim_state From: Yosry Ahmed To: Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0952580011 X-Stat-Signature: n3x3ie17om48nxjdahdzc8hnnahzq3wn X-HE-Tag: 1677574208-872245 X-HE-Meta: U2FsdGVkX18/TEguOGpeFLTUgCtlzF9s1svyg0hE1dvSQZvaxhiYYtQjpWddyC9Ouse4AFOnqh5Xf7lgmT47yJnOfHyS3gaLI1YSeTd10V4xcZ5bf2ntk8Ut34Ozx3FuovnnrvI+zpVNaHqDb4t4yix3Ei9xjQLfYGyJoc2dFZ738+wIZZZDrJC5X72gyXqHywODFY08Sd3Sv7+NxOLJgNohuxW2K+TTM0PMfE++cZiF4fCv/vhNv/WGjcK9IRBQ16yj5Eb5yR4gWqAbQf2GSElmeJ/0NJm1MJF9T0L3XfVk1+tX5l02bEbLiSNL+B3VM12fyeX7zvIsuO9wZyakfWnf4wTBieQVb7sfDPifZ6YfPUm4WkAKs0ptFj96YOnM+QIZjEexwowrwewOqHA+ElWUdm4N/d7B5Svq1r84QjHF01yoN43e3A0ojmO/P0YBWVxxL8g/pa0pJM2qXLeqOw20xYelB2Tlu6h7vOPwxCL9RXamEWyXQapWqToy48pK6yn1m7Aif00pWnS4umhxMnSMskTZp6PFSCQbq9WkXl6K1uy88AOX6XRbHcAPgU+bShISMngqipo2Jd2nA//y2UA8Jg/ety0DsoKB5BdxzjYSAAlseclxqtNF80ff4Ny56cNNR1TYk2riHzbd59bxWLCjF86Ej5Oci5NkhZ0quqdRqfiR+IfXaP5pZje78hpMji/vvpTyhqdQX9tK23s51f/Xnc36hm/DMzMWW+pQJNwInbbb5imuL/a/fBc1QY9qKv2Z/PBKBl8xgS3eT1/bQLGAAbCCEx9cWb326ZYH4c4eVur/tqUBc8iVcm3F3wZ24X/pkv3jvqvHvcEL6ul3jwzQZOEXkeCFT526cg5arzWGHSjb33sepu7P0e1NBrNLPMlLC4fAlPytSrDV51XV48ogGkhYdpy04XBL252Y5c1w+V+RCHE8vys+B4/JjLWn0vzdRTsnLS1dGCHrV8p Fe9YJSjq /EyPcrDKFh1SboDoLDdVSX8L6eTKMFjlYIMlkGA3WssM2YSSufOx16IaAs11OfmYPQVUi+Wjqz57j1nvfZG89sCOVxSTsfpD9RB1v9hPcRAjMnCZdJ7yLN/DygYXzm6Y9JObGIXk1yeA0o6ZBVJ3eYs+cu5i5Ar7kccbIJ4PYycDE769fiIqmn6U8xrQiAhKF7DZ+SIzGEB1uO3Ai3/dyqDHSAQwlnlc8fIN2NzEFxrNe+JHq4N5zngwSj6HZqgz07McBEk57Tmvt3jv5E5YOJdKh+84NCFciIGUHt4t+qAdmmKdMCXd5vurhFapl20UZ68jedFAXuHtuK9KdfGWdU0ZUSXoD2+OvbQ4YbV/O9Tsml/CeGZQbbb8+yQ0EQJd3OmDQqj2uKks5GPsUPCbd7+o67741w2YJr9yNEc7lPK8tFxvvK1EtgIksR7MiaFZCdEqqlFNckQvCHyImdKzMCJEUNNgdvsaAe6g3/cFpwjyF+j7gD6tgKK1TGKKJ9JyDZI1K/oVLjDCqPDFOPn0Kt2FggfFgwi21fZuLvU1NkhSZtD7kRCf8LuZ+VqTo2Y572SWygoDT7ucbYobfZm1k9hATVgiUm9+zVWnxIqa743c3gyt7AsEgHd5/QYb0Ev+GDCFa4roz6/LpgtHyNInlxbONsV0l3NSyKqMpE3CIwSLDZNHPffb5LAL0fVTUkveDMcgNrL1v9vw3Ebwe84x24O/Dv61tEWVRcEQwl8Q6nG6Bb3jxYMNWveONL2UjKpOmVXmUV3VEp75CQBtce1imgVfKPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During reclaim, we keep track of pages reclaimed from other means than LRU-based reclaim through scan_control->reclaim_state->reclaimed_slab, which we stash a pointer to in current task_struct. However, we keep track of more than just reclaimed slab pages through this. We also use it for clean file pages dropped through pruned inodes, and xfs buffer pages freed. Rename reclaimed_slab to reclaimed, and add a helper function that wraps updating it through current. Signed-off-by: Yosry Ahmed --- fs/inode.c | 3 +-- fs/xfs/xfs_buf.c | 3 +-- include/linux/swap.h | 5 ++++- mm/slab.c | 3 +-- mm/slob.c | 6 ++---- mm/slub.c | 5 ++--- mm/vmscan.c | 31 +++++++++++++++++++++++++------ 7 files changed, 36 insertions(+), 20 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 4558dc2f1355..1022d8ac7205 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -864,8 +864,7 @@ static enum lru_status inode_lru_isolate(struct list_head *item, __count_vm_events(KSWAPD_INODESTEAL, reap); else __count_vm_events(PGINODESTEAL, reap); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += reap; + report_freed_pages(reap); } iput(inode); spin_lock(lru_lock); diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 54c774af6e1c..060079f1e966 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -286,8 +286,7 @@ xfs_buf_free_pages( if (bp->b_pages[i]) __free_page(bp->b_pages[i]); } - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += bp->b_page_count; + report_freed_pages(bp->b_page_count); if (bp->b_pages != bp->b_page_array) kmem_free(bp->b_pages); diff --git a/include/linux/swap.h b/include/linux/swap.h index 209a425739a9..525f0ae442f9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -153,13 +153,16 @@ union swap_header { * memory reclaim */ struct reclaim_state { - unsigned long reclaimed_slab; + /* pages reclaimed outside of LRU-based reclaim */ + unsigned long reclaimed; #ifdef CONFIG_LRU_GEN /* per-thread mm walk data */ struct lru_gen_mm_walk *mm_walk; #endif }; +void report_freed_pages(unsigned long pages); + #ifdef __KERNEL__ struct address_space; diff --git a/mm/slab.c b/mm/slab.c index dabc2a671fc6..325634416aab 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1392,8 +1392,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab) smp_wmb(); __folio_clear_slab(folio); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += 1 << order; + report_freed_pages(1 << order); unaccount_slab(slab, order, cachep); __free_pages(&folio->page, order); } diff --git a/mm/slob.c b/mm/slob.c index fe567fcfa3a3..71ee00e9dd46 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -61,7 +61,7 @@ #include #include -#include /* struct reclaim_state */ +#include /* report_freed_pages() */ #include #include #include @@ -211,9 +211,7 @@ static void slob_free_pages(void *b, int order) { struct page *sp = virt_to_page(b); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += 1 << order; - + report_freed_pages(1 << order); mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order)); __free_pages(sp, order); diff --git a/mm/slub.c b/mm/slub.c index 39327e98fce3..165319bf11f1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -11,7 +11,7 @@ */ #include -#include /* struct reclaim_state */ +#include /* report_freed_pages() */ #include #include #include @@ -2063,8 +2063,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) /* Make the mapping reset visible before clearing the flag */ smp_wmb(); __folio_clear_slab(folio); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += pages; + report_freed_pages(pages); unaccount_slab(slab, order, s); __free_pages(&folio->page, order); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..8846531e85a4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -200,6 +200,29 @@ static void set_task_reclaim_state(struct task_struct *task, task->reclaim_state = rs; } +/* + * reclaim_report_freed_pages: report pages freed outside of LRU-based reclaim + * @pages: number of pages freed + * + * If the current process is undergoing a reclaim operation, + * increment the number of reclaimed pages by @pages. + */ +void report_freed_pages(unsigned long pages) +{ + if (current->reclaim_state) + current->reclaim_state->reclaimed += pages; +} +EXPORT_SYMBOL(report_freed_pages); + +static void add_non_vmscan_reclaimed(struct scan_control *sc, + struct reclaim_state *rs) +{ + if (rs) { + sc->nr_reclaimed += rs->reclaimed; + rs->reclaimed = 0; + } +} + LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); @@ -5346,8 +5369,7 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) vmpressure(sc->gfp_mask, memcg, false, sc->nr_scanned - scanned, sc->nr_reclaimed - reclaimed); - sc->nr_reclaimed += current->reclaim_state->reclaimed_slab; - current->reclaim_state->reclaimed_slab = 0; + add_non_vmscan_reclaimed(sc, current->reclaim_state); return success ? MEMCG_LRU_YOUNG : 0; } @@ -6472,10 +6494,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) shrink_node_memcgs(pgdat, sc); - if (reclaim_state) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab = 0; - } + add_non_vmscan_reclaimed(sc, reclaim_state); /* Record the subtree's reclaim efficiency */ if (!sc->proactive) From patchwork Tue Feb 28 08:50:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13154535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AC84C64EC7 for ; Tue, 28 Feb 2023 08:50:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FF9B6B007B; Tue, 28 Feb 2023 03:50:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 988916B007D; Tue, 28 Feb 2023 03:50:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 828DA6B007E; Tue, 28 Feb 2023 03:50:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 721996B007B for ; Tue, 28 Feb 2023 03:50:12 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 48B79ABCE7 for ; Tue, 28 Feb 2023 08:50:12 +0000 (UTC) X-FDA: 80516078664.11.9FB8110 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf27.hostedemail.com (Postfix) with ESMTP id 8BD5740008 for ; Tue, 28 Feb 2023 08:50:10 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Qx1HMjnO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3QcD9YwoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QcD9YwoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677574210; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=th38ntbA4Oc2oiVVTEFN7kVCZXMfBHW5bqzoJX/NKxM=; b=6PYAxpPEL4AGZlCyB96siDuRNya7FmUGNLCgAo0adNpe26wOR2VmD5T7v/2RFfgS7YWMzu U47EuHhVJH4b1BW1Dgx5leHm2byX5bBDIx3wXiasnycJ9dxD/FqwxDqO7yr9kHWj39M0Bo Cd7jPWsWeMTJi1GiIFTe+4GWBm9wZCs= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Qx1HMjnO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3QcD9YwoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QcD9YwoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677574210; a=rsa-sha256; cv=none; b=onbYtLXOfbK7U6a5ETMYLcmDv2w2Yoizf4MQX5e7gGRRtVUqhmQ2U5QKJ086Y3Gv/a58zE s7/qBha8XcBC+fSvYr/c3u3rBoU6lvsDgHVmCZ3K+yyxoLzheaAFjkGYSg5quldEV6Jclq bRCDl8FadGquNrpkAJ8eDY/5h1/mkoE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-536cad819c7so197084357b3.6 for ; Tue, 28 Feb 2023 00:50:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1677574209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=th38ntbA4Oc2oiVVTEFN7kVCZXMfBHW5bqzoJX/NKxM=; b=Qx1HMjnOH9OsA3ycuQn28p/IaZ+Hp3kNuLp2QUi9h0Ycfnoty+yQNpg+WRxNuY/vYp jOY7XFohXtJylm8Zhy5PNGpA0U7IS0oAfCING0t+nz1L/hwrLlAjdn6QB4BQ7g4vN6HH OAiYlZHTTmYK4E/ibEh6T98sf4VZmgYow+qhvCLJm9G0PRCymGehRtI1hbaEgguuPSwP c7t1cNU/lIriIMXf5vdSZTLjo71hA+1rjgeHrdym+NEDcQOTi8Gy7EZ3GPG7NMQg7zho MG8uusZE7ELVwRFmlhIaXFZ9jTXMsB870R8rLfTvIx/zMtY4Y5kaKuhA8atMfdGBrjnt RIMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677574209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=th38ntbA4Oc2oiVVTEFN7kVCZXMfBHW5bqzoJX/NKxM=; b=McsPlu7tp39cu1tAFx9jjq3evi5X5hmDBQjKwpaUaf2pVXRnSSqbher7d6FjrFocA0 od96DNfcZs15CXGT/+bXB3m/1lOTPDhgbeGPSCS+DHcQJ7RNn523BOk2DqdFd2x9vs/+ O39Wo1DMYGF0ZvZkJ/oetwbV87KLdcErdpgZygv3Qxi8Cvg+L4rbvH/Ssqd53PDc09si UJyRxUZjY5O3KOIr43rURVqdV7Yi/gkd5+VjLNWEbY/yS1ie3XhQ20STF0+0V9XtsDTw VDCVrMa97PFci9Z6plrCfl3DaZK08KGLkr7mFCxcevbrIw97OBewM6qQSrtwplkthltO oOUg== X-Gm-Message-State: AO0yUKVvk++zphAMVDD3JC0iNzhQSfRlRSR8SA9cZvlSOigZHD4IGi/z lffVnpvsmg1p9NIwcTOBSiIiwzUVwrSxR7gv X-Google-Smtp-Source: AK7set+DvIV3bw19wB6Hwqeb1fxI1Ob9E6fp4FkYQsDq5MINr8X9kGIXdnH2rMDNI8ge0fm/8/tJGYKLqoW41WxG X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:ad67:0:b0:525:2005:7dc2 with SMTP id l39-20020a81ad67000000b0052520057dc2mr1184169ywk.1.1677574209698; Tue, 28 Feb 2023 00:50:09 -0800 (PST) Date: Tue, 28 Feb 2023 08:50:02 +0000 In-Reply-To: <20230228085002.2592473-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230228085002.2592473-1-yosryahmed@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230228085002.2592473-3-yosryahmed@google.com> Subject: [PATCH v1 2/2] mm: vmscan: ignore non-LRU-based reclaim in memcg reclaim From: Yosry Ahmed To: Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8BD5740008 X-Stat-Signature: xpm7jsp53oz97y7gju5qym6g7cyeum7x X-HE-Tag: 1677574210-988675 X-HE-Meta: U2FsdGVkX1+GIOLfMICXcsHSaCioI08rz8YKcSSTOc/hq+Qoj2ak4MqnmF/ZjaVyXQnztegubwvzhuG2pJL7hjjYZhTTiWeQ1gNE39sjBJHgEjtsTksiYUJi8jtgFcDIiiZaptM7938UoTxsdcvR3EHOsKR8AgghD6QXJ+5XCqPlGPdpCrkJm3HhGA5KLHScNkQybaW8RcTxOq/3Mv64tCspENgi734Spc7Fe7aexESBB5dRCBWXIFbneccYacLKDsWq1M60yVyOTprMD1Y17tdr1U/4lQnVyLRo6M60zdMdGNetq1SMjqc2U1uVFvnK1+neXJ5GcBKBr4sRQoL8UhWCpEUE0RhIEyKCZx/00OV/rznPmVAT6fHRgTCFV/lZDYgMmbBQQjX5BkfUjgRrFHYMYCuuQsHYmTdB+JpRM8FD1uI+jlIBuAlTEXYr8wZDsJPYE83VJgTp8Se4HzxETP/Q8LdJPAqX8gfAwRFKewNmi0DwYpeJKt0ZifzQRySLRWjst0189LQ/rUfI+wUukmUfWAQwOwbvjQbExPyEtjPMFr/n61iTxXlRIMARSvqpnw368KdzM4Ckj1bQlXxQT8vSFxBSE6dUDQGVr1YpFTa01piPy4BofhJfB5dV00FWicB0SoDKSQU9yY6VJiokZIPQDxJAR1Lv+Z4D454wACndSYrxpKimFTDsqp2fFILfBUiCTzihMzkAJm52rOIOnqlqEpOvtnqWjAAa8XmOZcfMGfn6KySiD8EX74IeDmfTvIg3eQ403ApFbZDu+unnGSCprl6OlIH9fgHYXeEWYDaO68Cy5baMbZA0pWuXoHB53paURSmVe8UhkC0raecIIg6IXxloOFZA99m4rAPm9qmaEC+koybKMHbJW17UpP9sT+s4I3kzTNIljN6I/WQLMseNYbDFGBOGGQM99k46VzlYEe/SopBk6SKCjvfvBgMpkN7FLp6CZ3mep9FoE2f nk5jWpFL TN3cZUomgVMaObgMMnwdfl6iYqU0/Sp/r3fyHAs8OOrP+AmUzJXCPGBMib1WSXcK7ktTBbvAWV+wMzMUxGZAqIU8pz5BNE38yiU9ueedUfYX4CQ6o+nvPuudT8ww45zELtGQ7rffKml+7PFNJxx8bFvCoxWUPvQgVG3jODFHF9mGekWIWQwor3+wLmUE0WnLg+ptNfkeQEVf7NlMF5f1oBJ56KZeDI+gqXajzUzV5mUgs1ES2Gb0RAbNYG/84krXcAWv2zBSE3IxZShL064w6L8jJtzNZGwzMS/6Sa5xhzn+rAZGlmb02O0d4HIkpacsgoGRUBzzjxgjFGazdpJ5dltrGAkTmSzyyi2/YkZPjV4RO+Uzkewi3GTzO+mhvVX8JGYmTrhr9Oof/IgWwv2mSY1Uj+Vj7I6H3Nfkf4OkNoistiK2M8phfsYfLtuMaHq06yT7NH56vRRyARd966Jx/fete1PhoRYV3mq+eolwqHRe5y/Cgeyv9+CSp8ITXQdVkwpbfNvMXZ0wZAF2ZkSnOHNmgWmh8EB+P+5/EHQ5PWBIP4ZTFUGT0YweQeiQkpMqht5ELBqKMZqrhMjSRIRFw+TzuanR6ZPhd5dBXrZZZSzOBE/h2cs+Z7A7vuI58kvUmS8rJpk4gMkzjf7PuInZ9N2auf3V3tL+XFZaJWXJl1T+Sy6MIWAL9tdBT8jQOhiE0jW1qquTEZCZ+wt4jrZbGKh//ztfNsabEhT6a X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We keep track of different types of reclaimed pages through reclaim_state->reclaimed, and we add them to the reported number of reclaimed pages. For non-memcg reclaim, this makes sense. For memcg reclaim, we have no clue if those pages are charged to the memcg under reclaim. Slab pages are shared by different memcgs, so a freed slab page may have only been partially charged to the memcg under reclaim. The same goes for clean file pages from pruned inodes (on highmem systems) or xfs buffer pages, there is no way to link them to the memcg under reclaim. Stop reporting those freed pages as reclaimed pages during memcg reclaim. This should make the return value of writing to memory.reclaim, and may help reduce unnecessary reclaim retries during memcg charging. Generally, this should make the return value of try_to_free_mem_cgroup_pages() more accurate. In some limited cases (e.g. freed a slab page that was mostly charged to the memcg under reclaim), the return value of try_to_free_mem_cgroup_pages() can be underestimated, but this should be fine. The freed pages will be uncharged anyway, and we can charge the memcg the next time around as we usually do memcg reclaim in a retry loop. Signed-off-by: Yosry Ahmed --- mm/vmscan.c | 50 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8846531e85a4..c53659221965 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -188,6 +188,16 @@ struct scan_control { */ int vm_swappiness = 60; +static bool cgroup_reclaim(struct scan_control *sc) +{ + return sc->target_mem_cgroup; +} + +static bool global_reclaim(struct scan_control *sc) +{ + return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); +} + static void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs) { @@ -217,7 +227,35 @@ EXPORT_SYMBOL(report_freed_pages); static void add_non_vmscan_reclaimed(struct scan_control *sc, struct reclaim_state *rs) { - if (rs) { + /* + * Currently, reclaim_state->reclaimed includes three types of pages + * freed outside of vmscan: + * (1) Slab pages. + * (2) Clean file pages from pruned inodes. + * (3) XFS freed buffer pages. + * + * For all of these cases, we have no way of finding out whether these + * pages were related to the memcg under reclaim. For example, a freed + * slab page could have had only a single object charged to the memcg + * under reclaim. Also, populated inodes are not on shrinker LRUs + * anymore except on highmem systems. + * + * Instead of over-reporting the reclaimed pages in a memcg reclaim, + * only count such pages in system-wide reclaim. This prevents + * unnecessary retries during memcg charging and false positive from + * proactive reclaim (memory.reclaim). + * + * For uncommon cases were the freed pages were actually significantly + * charged to the memcg under reclaim, and we end up under-reporting, it + * should be fine. The freed pages will be uncharged anyway, even if + * they are not reported properly, and we will be able to make forward + * progress in charging (which is usually in a retry loop). + * + * We can go one step further, and report the uncharged objcg pages in + * memcg reclaim, to make reporting more accurate and reduce + * under-reporting, but it's probably not worth the complexity for now. + */ + if (rs && !cgroup_reclaim(sc)) { sc->nr_reclaimed += rs->reclaimed; rs->reclaimed = 0; } @@ -463,16 +501,6 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) up_read(&shrinker_rwsem); } -static bool cgroup_reclaim(struct scan_control *sc) -{ - return sc->target_mem_cgroup; -} - -static bool global_reclaim(struct scan_control *sc) -{ - return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); -} - /** * writeback_throttling_sane - is the usual dirty throttling mechanism available? * @sc: scan_control in question