From patchwork Sat Feb 17 02:25:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB7D4C48260 for ; Sat, 17 Feb 2024 02:26:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 275B06B00A5; Fri, 16 Feb 2024 21:25:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D21F6B00A7; Fri, 16 Feb 2024 21:25:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEF686B00A6; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ACBBB6B00A2 for ; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 776C9120148 for ; Sat, 17 Feb 2024 02:25:55 +0000 (UTC) X-FDA: 81799705470.08.632DEF0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id C441E4000A for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Gge7ZHIB; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136753; a=rsa-sha256; cv=none; b=TNHdiYyzVA7hkZOW5OA16fUn8jtw2C4eRdVlIgZkBCFt9A/vvrwnhFOO0Rjn6zVseU12Cx tqb4caZngAos5Rv9cYZ3zAIVCGcOIeV+40fPBN00fGuHaep1SOEHh2RhxOHmqNDgzlXozE inyEUEw9OSohfisdCVkTy7rSZaRLSMI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Gge7ZHIB; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xLNBtKmJzlmAoHK/ONEef9lzcMsCVs4HBTSM/I8zSNQ=; b=eMr70a4F1YRNXv8vCV2KuFeS1HhUcZgEwApTC+5V6zqbvJo/5Iwbdg6tqJ67i6o8G8oflm JMqNM5sX1vUzxu5zqVrRSXH2rDZjFDqZm6nTykTfvqEOLOpJMOU0I/ijzMgbSMyO1UBO9C 89OrgpOKY2gw/oankcqWrz+Ua5l5aLU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xLNBtKmJzlmAoHK/ONEef9lzcMsCVs4HBTSM/I8zSNQ=; b=Gge7ZHIBCsdoopoK8itgybcL9X G43A/PoZIaMIDTcFkMOk0/C5fynUIdwcvg6wfhdzCZve2P4Q2bJDxnQf9eHxaYJWDyhQeZxoOHV+0 E7w0rQgVcFTux1Lejv8497vGNvmDbna2F2eTRAG3SXGU7pxfb3cboBsDWOe5GTVv/INiJeVpZK2OX eTFP4JGp4Naq3aI0PgRszR9CK7c02QjTfLLnSmGZf9wD+J7Vwo1IV+s4r8KEjNYxL2M4SPNoP4MqZ wu3UMS33Ku8odhv47qpB4QjTM8ycRMRBj7GijW134oKmKZGlkjVVOcNW854ix9XP6sNvG/wCrvrkd /eYx+fRA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HDa-1Hmp; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 12/18] mm: Free folios directly in move_folios_to_lru() Date: Sat, 17 Feb 2024 02:25:38 +0000 Message-ID: <20240217022546.1496101-13-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C441E4000A X-Stat-Signature: on3gxz9du4oh7bzknsjw8habi34acmfs X-Rspam-User: X-HE-Tag: 1708136753-378124 X-HE-Meta: U2FsdGVkX1+cCQjVH6w3ouK0y91/qVOb2sH5lyKPlUWsgfCMCMWdUGOYm34coB0s0A6TI6/2UyPVUTGJEdtGCShd5b0Whm0ZdAcK9wzKiuusIj/PyEjo0JWdg7fI47d9DfKGFvWVffhx7vXrRV38wImzWHoOciZrnNzNcdGGd7jj4Gnz5/OpXm69PsSIPy1nrAQspNcxptleXWzoo/FnIThvG0WLdsiQOk1znz7tfaEUZWutvktuVHBefJ1H2dt7i0m0b3dcPkkS8rIXytYiESdOgO9ruYRLXE4uOp/2Ha/eCXLB5qLREy4shkfezw8rsRiggAMBIdGa2yISy+X6Qb2JnGnRlTtkp92ptG0GRmFkaoffW5hoFGNbV02idnHYVGK46dJDlPJ2qDbH02ztT7qLwwgFkgEjAkVKf1dpFzobDHn97PbrKdtbc8sRtBxei53EoqOUSvHbBeYLz5WKR5RyViJLJQt2H+rxk+CAZx/C99uRj0WsfKKmNQP7BC3DWS3lul/xfyp7FQiU8DNRfZ9lLNB+bVcBl9AjEZLkQGxGT+trJbCMaBBTanCXBTBYa2JFK6/1/YwGzPn4L72KSwx8AXxEAPVBp5GGsRkGffC7JfQyx9mHQxg4cu+onhmr2TIT1BHkCteocMNSmMWmKWQOAjSe0S4s2ZXxZ3VXdjlpFUP8BtK5JBBhC0bn3up9ZGmKZTMhCKYr+t4GfJsTzdZbBkGCtBIFDUZqFCbVVSB2k4a4s6P3weVLOsW8zJKbnWOBJrrqDwDMdYQB5zi1xiKGRdpuPfypI2YJYogMgtWW678luQ/oxfTpuYZd50Z/ftI1mNzAG6GvZyH7BucZ3zLe6OGXaLHztn0twcaYLwEtTJV7outcxf0fzkZeX2rP6rpOxTwKe/fWZorSv3PlcmYnsZJrDzVHMixb1UwfM9DPhFMpiO2xg5P5gdLaDILmXLB42YRZQddXaWCEASe 21YLIMn+ /9pXvHRaE8rkhbV3FU/7G/Rj4o5XioP6PvN2/Xa/ndEs6TeFUDXfhngLNfDVn9B+F2Z8vzQW5QIRvMzcpUgfax8p+TUQaN0Q19jeKYBCVVkRxls5ytnhvAl776jKFz//C4FW4IybydZaNUEBa/C5qYdCBcNQICva1knT2bTWNH9Rj8YLwwOnGPpaYeP44hGaRB476WKCwGuoLiWUzPr9LeTWqotgAN3fjP/td X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The few folios which can't be moved to the LRU list (because their refcount dropped to zero) used to be returned to the caller to dispose of. Make this simpler to call by freeing the folios directly through free_unref_folios(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e2292855f58e..84f838330c3b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1798,7 +1798,6 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file, /* * move_folios_to_lru() moves folios from private @list to appropriate LRU list. - * On return, @list is reused as a list of folios to be freed by the caller. * * Returns the number of pages moved to the given lruvec. */ @@ -1806,8 +1805,9 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, struct list_head *list) { int nr_pages, nr_moved = 0; - LIST_HEAD(folios_to_free); + struct folio_batch free_folios; + folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); @@ -1836,12 +1836,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, if (unlikely(folio_put_testzero(folio))) { __folio_clear_lru_flags(folio); - if (unlikely(folio_test_large(folio))) { + if (folio_batch_add(&free_folios, folio) == 0) { spin_unlock_irq(&lruvec->lru_lock); - destroy_large_folio(folio); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); spin_lock_irq(&lruvec->lru_lock); - } else - list_add(&folio->lru, &folios_to_free); + } continue; } @@ -1858,10 +1858,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&folios_to_free, list); + if (free_folios.nr) { + spin_unlock_irq(&lruvec->lru_lock); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; } @@ -1940,8 +1942,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - mem_cgroup_uncharge_list(&folio_list); - free_unref_page_list(&folio_list); /* * If dirty folios are scanned that are not queued for IO, it @@ -2082,8 +2082,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate = move_folios_to_lru(lruvec, &l_active); nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); - /* Keep all free folios in l_active list */ - list_splice(&l_inactive, &l_active); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); @@ -2093,8 +2091,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); - mem_cgroup_uncharge_list(&l_active); - free_unref_page_list(&l_active); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -4594,10 +4590,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap spin_unlock_irq(&lruvec->lru_lock); - mem_cgroup_uncharge_list(&list); - free_unref_page_list(&list); - - INIT_LIST_HEAD(&list); list_splice_init(&clean, &list); if (!list_empty(&list)) {