From patchwork Tue Feb 27 17:42:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40822C54798 for ; Tue, 27 Feb 2024 17:43:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B3A56B0101; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C5B4940013; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 276806B00FA; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E16286B00F7 for ; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 48F99140962 for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) X-FDA: 81838304730.26.F14E2A6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id DC3F64000F for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hNus+nOY; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8ClF8cOCV2qWtW1lr6Cu4rAKz4XJquX7C1JC7chVGGM=; b=VDbJE9culqkt3CfU31gzQdF2rc4Lz4pVIBPiK+N5KfDzVcq6PGD9dKCqCu3+ZnvFUCOlbQ AuJ/vYnp2artpIBFaNC89CONou64M9qBBp9hy1hLRS+NWpvM3vsd2u/MuGWl/woFt3omkP ZFPkEzzMKkehLf23kZy+NJMM/E2SkhQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hNus+nOY; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055784; a=rsa-sha256; cv=none; b=U2VsNJWAekL4ObnaJHz+ehtXc1Yw4wDYH8pHsjj+o7JNMo09zh4x63x8irBBV6Viq9hS+J O7tNrxgkub1kiJN+Fa+OE7FtxDLCZlfqsA8yNqnDTbOQctcCCR4x7s036gkD0GuV+rSuq6 TQ8lB6K/CFYQCuvNxjUCoijXRCqQGoA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8ClF8cOCV2qWtW1lr6Cu4rAKz4XJquX7C1JC7chVGGM=; b=hNus+nOYLvgBoVAtnbFN9m/+eD c+YOj98yR+0TfCdEeJzbmwsJ8APrBrPuu1utJAJ8+ScYCqrQFEseONSWvwe1n12qK3QxvaDIjukDR JxTk1BW+P1yhps/2vzt1RuHp25NtdLuR1F0D/yYFTFPQYWcWrFSPt6qhti5G3xWsTpIrIcblO0xGz mGBAti+s+YwqBJkIU7Jxfabav0q7hW6dXO70prH/xHtVbLxnWmH5Jkt5Jf+KdPug8l7sNx7q3PW16 aDvuNdgUl/Z6MHsMqA9EQqj578cB06M+Epw9V9cJxO7YkZWJtKFxeCOhATa5WJ8gu92q97Rze66zC c3sAwmxg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002yrm-0j14; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 12/18] mm: Free folios directly in move_folios_to_lru() Date: Tue, 27 Feb 2024 17:42:46 +0000 Message-ID: <20240227174254.710559-13-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: DC3F64000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: b6jhamr8ykaexo6xo38jeqf11w8auwxu X-HE-Tag: 1709055783-519772 X-HE-Meta: U2FsdGVkX19tU4Z7UcmJ1iNYQ+lB6FugB4XgCfUWUXfeiuXRrhE5zF1DzyhchXznqgFGtYeJZ6C2e0+0NIyC1znZYXq6R4QDGuO+btPzpfyCWr4c+CME3v8UButiUiwmbcJdmCvMEAIi+jwjrDWYr99f/AWU9BiIpN84YsrCwg2ZwQph1MpF+i48GyU2+Znd18nNhOdTlvTDyVoIk4B8xjBdAOyRnxcxpjzo2NlnV27UVflOOcLvPdlhgudp++gYtq8/BLOxZm+wFcf6sy+jM+94cd4L/isWUl8zvXQayDmwugx72uf62OaB+mIuawq6RReCwOK11MtkjkW/AfT/tME1A48KOV2Pqjp9YW8EDFpqBm/TuK15+u55wKIDeZQTwgwYGlXveeRjkWz+BqcZ0h8CCBjh9MghP4H58ojF0dC73b2bt/UdlChjiCAJ91wiDiLEonf5PEjyeuhUNiO5KyplzTnmXM+2EZORG1sw75HNV4VJQZtCvjnG2yulxvbay1z6uKs8+RpWJLklKs+vWFPZ1xzICqDBatkI4lUGKvlwAZFQaaMx+Ld+mxC0dN7kdEvbl3kfxw+zcHx8rooSoEboFxVhi164A8JkH99zEbJV5RHgi8JJKXKoeNu5cHpTrtKEOIU1BtvHEDqjVmCEBeJ9k8Xfg+AJdfooEBHL0m2ywKLmuF4ayNffEDWgcZX+BIA+mgNvoAerrYuMMiL1la09UCqim9UuytZs8oUNDp/f3ouIydeuo2539wsT0jj9eSH2K3FLnOSndsA66j70gkjMqtugktEFCwYrPGcmuDuq7LHW+eQdqbDgLBGzuRQOF0nFiyxvVAU7D0pcxOY7ccsU0EiirXFIyF7rQOU0Ef9cWLTmF13zmjxI7HE0NHdVO/CfdH4AFkFIFSKeW2C0WRBx130mQ/0H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The few folios which can't be moved to the LRU list (because their refcount dropped to zero) used to be returned to the caller to dispose of. Make this simpler to call by freeing the folios directly through free_unref_folios(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0c88cb23cc40..c86c4694bcb1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1801,7 +1801,6 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file, /* * move_folios_to_lru() moves folios from private @list to appropriate LRU list. - * On return, @list is reused as a list of folios to be freed by the caller. * * Returns the number of pages moved to the given lruvec. */ @@ -1809,8 +1808,9 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, struct list_head *list) { int nr_pages, nr_moved = 0; - LIST_HEAD(folios_to_free); + struct folio_batch free_folios; + folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); @@ -1839,12 +1839,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, if (unlikely(folio_put_testzero(folio))) { __folio_clear_lru_flags(folio); - if (unlikely(folio_test_large(folio))) { + if (folio_batch_add(&free_folios, folio) == 0) { spin_unlock_irq(&lruvec->lru_lock); - destroy_large_folio(folio); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); spin_lock_irq(&lruvec->lru_lock); - } else - list_add(&folio->lru, &folios_to_free); + } continue; } @@ -1861,10 +1861,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&folios_to_free, list); + if (free_folios.nr) { + spin_unlock_irq(&lruvec->lru_lock); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; } @@ -1943,8 +1945,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - mem_cgroup_uncharge_list(&folio_list); - free_unref_page_list(&folio_list); /* * If dirty folios are scanned that are not queued for IO, it @@ -2085,8 +2085,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate = move_folios_to_lru(lruvec, &l_active); nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); - /* Keep all free folios in l_active list */ - list_splice(&l_inactive, &l_active); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); @@ -2096,8 +2094,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); - mem_cgroup_uncharge_list(&l_active); - free_unref_page_list(&l_active); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -4601,10 +4597,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap spin_unlock_irq(&lruvec->lru_lock); - mem_cgroup_uncharge_list(&list); - free_unref_page_list(&list); - - INIT_LIST_HEAD(&list); list_splice_init(&clean, &list); if (!list_empty(&list)) {