From patchwork Fri Aug 25 13:59:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BA2EC3DA6F for ; Fri, 25 Aug 2023 13:59:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78B8A8E0018; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 691F12800C9; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 385372800C8; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DB1E12800C9 for ; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B44C0140651 for ; Fri, 25 Aug 2023 13:59:28 +0000 (UTC) X-FDA: 81162784416.17.58F2749 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id BDA104001E for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dgiWRr1o; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971967; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=92K/vCjYxmyDWMu+BIreMC71v+Kez53bhjphz+8AFr0=; b=Y/gM0JxuHYylgR0oQVfmNPEVzvIvTSA4ktfHk2EJjBrI2m+1GBYfGUNF0DGFBSD3AyuJ0i yEqtn88+WVNTc8ZGzyzsU+XxsvEcgHyYRROI4IlaicjneYR/443MKxqnK9vVVv5MPvHMOn tA+tXKCxJOrar0R2sdpXQxGhal0eDH8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971967; a=rsa-sha256; cv=none; b=0/WG/oc6+vlp/XGGucNiPox5aStuC/EdSb1Xc1KkL0E/0ubKik2mXeYyV1OlwzfgEWI6yB QEL7nm06l/WcJ8tN15AwwVM4kXYHwgdbdJsEJ8AEUKP1fLxLMU0IgoTf929AL+KHMYxl+U xtiBi+Fj7IoLj6iAIV5HN+957e7gUl4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dgiWRr1o; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=92K/vCjYxmyDWMu+BIreMC71v+Kez53bhjphz+8AFr0=; b=dgiWRr1of/ichTG1+kGip0HMsi 8qAZKCcIk4ZXHWzX9YABfjoVDZrnaVerKkZbXidkuDfInHD82l0W59JaFyr4drmQyAo2GSIPX7ebx tVfdRKiXRWSjuNLMF8E+EWS7FZi0wNeeG8amEDbyCpy1LQKlAWjW0xEu3WiIVY1Bfg5lYH+5NsMxk ygEwCxFaIH7R3lC42lemfA2wFNKQl5Qq3JsrFIsVFGX21yZMK5qJMxo9p4vKYoB+oKBoyqGZCLIki +9lCD5i7ks/p88uENxhkHZErPORN47vxYyzB0vx0slxBfpHtUT3d/GjhjTC/h1kyJj3d0t5mx2O9L f8472d0Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLO-00HTRN-Ae; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 12/14] mm: Free folios directly in move_folios_to_lru() Date: Fri, 25 Aug 2023 14:59:16 +0100 Message-Id: <20230825135918.4164671-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BDA104001E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: eq3dobjtcnnmt5w7znhqiefw9c8qy1wu X-HE-Tag: 1692971966-426 X-HE-Meta: U2FsdGVkX194tjh3hHoACEQ/PSfLdm8Y7XlwuTOmi6ZZ3eU5W5nmecaIouGyq7SZHHnh/NsXFHbyBpr0Xks5Q35XbAIwlXjpPhOJuAyLY4abodtIsNxuTzGnhonhITjluVP5hUyKcIpWpax2wO0deq/8wt02oQI4PkYLYzuT1tKKgf5yFuPEvmXBKUs8+fwx/bwqkrTfVE/t4xdqv2vZgCUGVX+LW9NDgrl6J+lAmdLbahC+TqjFDDzCDBOW1wzPlQYt3S4+o5sWYGGk4nP7bTCtB0RnYXAjtkBvA710tjnP5LavxbTQgXIIuxcubh6NqUvPaA9ZmbNjE32s6udNL2jTXGSoERjvJ1cTC8MYw0ZfypZ1HNKuQtRwxreiUmwqhOyGmcbFquDUAbYPMoG+6PCwsa3dO5S76j0AI3COx76/WHtzsybT6CfP2VO2xb+eLnsQT/xpqVtvXwWfHwWxpg6eLvwA222ZzC7+vo9r32mkL1eqEgPBWbuYGjaGoHRKiDa8rwGlFl/QGmNgpy4bm4xH8ENep0u0a12EeJ+e3I7X9Tf7cVoi4EFL+aYvzJQsagbsTCixKOWbBVviPpFOfVrwyDz/gIWtXBfj8vhn5ngMpAWom9iSHecDMVtTWifDT0pqUiFYpN+UHNvIHAopNUwcL5nXZmz63G+YIgsmm3gGq+RI9TEQc58zHTJI6oMa/8+sbhBgx+6uy6UZFdDBqLRC7bv5IT2QKbp5NdS7QSvZW+eITz4PKJqhkdUcM4Lnb3FyZItlwWP8LaYc4tSqjVA2Jqsbk2ZB95pYROH3By+39d1ji9zVu8u1/nUbpeVQlnNXWQD1zEb1fzvLNDprPZaCtyO8KJVUlRvJ5rdDszbcqz69mYkNNybHGuFHdpCzm+lXhyBWb5dOs3Fa4gvIwW8BQwDlq4SO+hPqr0VXYxG93Tf7yigeEuhV11MRz59bcZWnfwBJiWSkjr1wcU3 VWKnBgH5 yVzS2e9/6SU9GnSmV45qwawnHblAYDhJOYarVIwZTOlWIDjI27OHU2j5fYEpnk9OeWhGB/zsoOFwESJKAZSXuWrLIGWRFzYFJt4w4amO4orFVWIv+GppFB/tHRw1mEK+NNqoJO8TNAaQtctLtDn8OoQscVrxPgrthXKCLHlH2oCoUqleHnaqWY8EameYd2ZTEa4OOlWXLeVGARVLZEdMP52sPoyc61wRMCtXt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The few folios which can't be moved to the LRU list (because their refcount dropped to zero) used to be returned to the caller to dispose of. Make this simpler to call by freeing the folios directly through free_unref_folios(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 965c429847fd..d5080510608e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2489,8 +2489,9 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, struct list_head *list) { int nr_pages, nr_moved = 0; - LIST_HEAD(folios_to_free); + struct folio_batch free_folios; + folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); @@ -2519,12 +2520,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, if (unlikely(folio_put_testzero(folio))) { __folio_clear_lru_flags(folio); - if (unlikely(folio_test_large(folio))) { + if (folio_batch_add(&free_folios, folio) == 0) { spin_unlock_irq(&lruvec->lru_lock); - destroy_large_folio(folio); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); spin_lock_irq(&lruvec->lru_lock); - } else - list_add(&folio->lru, &folios_to_free); + } continue; } @@ -2541,10 +2542,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&folios_to_free, list); + if (free_folios.nr) { + spin_unlock_irq(&lruvec->lru_lock); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; } @@ -2623,8 +2626,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - mem_cgroup_uncharge_list(&folio_list); - free_unref_page_list(&folio_list); /* * If dirty folios are scanned that are not queued for IO, it @@ -2765,8 +2766,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate = move_folios_to_lru(lruvec, &l_active); nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); - /* Keep all free folios in l_active list */ - list_splice(&l_inactive, &l_active); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); @@ -2776,8 +2775,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); - mem_cgroup_uncharge_list(&l_active); - free_unref_page_list(&l_active); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -5238,10 +5235,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap spin_unlock_irq(&lruvec->lru_lock); - mem_cgroup_uncharge_list(&list); - free_unref_page_list(&list); - - INIT_LIST_HEAD(&list); list_splice_init(&clean, &list); if (!list_empty(&list)) {