From patchwork Sat Feb 17 02:25:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D35C48BEF for ; Sat, 17 Feb 2024 02:26:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD4616B00AA; Fri, 16 Feb 2024 21:26:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A85B66B00AC; Fri, 16 Feb 2024 21:26:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 867B96B00AE; Fri, 16 Feb 2024 21:26:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5EAEA6B00AC for ; Fri, 16 Feb 2024 21:26:01 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3DF95A0037 for ; Sat, 17 Feb 2024 02:26:01 +0000 (UTC) X-FDA: 81799705722.11.8F3569C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 7BD5DA0012 for ; Sat, 17 Feb 2024 02:25:59 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=d5HskI6H; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136759; a=rsa-sha256; cv=none; b=uVT2YkSV9O1fclJ+UQ5zswzod0lmtIs+al1C6Y6qNyIVr+85o0HxffR8UeaJSK5R3oMviC wDVxHsrPb721DFUwRJMY92ZG9QpocSQxcEMcNy2COVrajaXv4Hp9LzndxRLzKKg9H4QS9Q pjU1AevjJxkiYPUJMaJ21ankaLQR1g0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=d5HskI6H; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T+EzjsqxKT4x+MbA+ssZUn05QbMd3khRsC4WSdCchLM=; b=AB+xvHXYRga4rF21GGfFhiGsKPWc6jWPrfdu0PI6kphr3IBeh4I/rfBDI3dZSi6IOhAL44 d9wg+LI2dzR7l6srDh5kAP5K4ysye2RIhYYDZ0KA6GxdEq1Y12dMOthfkmkAJ2MCGSIMDe mHTpLd0ZfddSzUdZqhcxdNtZ0IS5JQk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=T+EzjsqxKT4x+MbA+ssZUn05QbMd3khRsC4WSdCchLM=; b=d5HskI6HQw1DaiS4wY9GYLeiwv NIem8+/ZEbCvTU5v9MYOhh/6fLfjkZnGS3hHX/gd5bwFYoiQkkldfyFrePPwnkbe3jCEbliI4dbAM OlzK5EAfQ2ZuUfmxxq9vYGL4YS/7QAZacBmZLFGYCMgfrS4MmvwrZnQrWzar7Mwly3M2rXS/vvZ5D /Y0u0Sa7rwvdpN8gcgsynLf2CIDh5xvagtMXVNBXvnq6cbWPbPUIUH0wHSHhOMewyB1cAN2Y0G76G bMoWoUFXm5Ukt8adxbe+ezRplDv13oWwJuuuDlJasnnGCpeEkJQCDSdLiKAzetNCrvmPgWQTG4toV vT4vJPVg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HD7-1LGF; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 01/18] mm: Make folios_put() the basis of release_pages() Date: Sat, 17 Feb 2024 02:25:27 +0000 Message-ID: <20240217022546.1496101-2-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7BD5DA0012 X-Stat-Signature: pnem49453efa4iwzg8thqbno57wd9g9y X-HE-Tag: 1708136759-654469 X-HE-Meta: U2FsdGVkX18bCcC6ctRr4v/yg3WcQ8ShYoRDWT00wDFG5OBPuPe8elBBlxxNbUl8TaVEa9g90GX45p7uwCVQNnXgIVuyl2o98P5tGjvj30/85YE07EVLDrvKpLEGghQ7bbYEGyun+kjoKjo4nbYLO1TBELzdLUWGJFmkzBDay9ELJ8ofoCg7X16O736AOlxAUxLsnohnNW8UibplP+gesEdrY9ELo0wUql/74a1UbNW71ZQwciIxi5Ij8yDx4TMTYiYExl2XUlrRzeNs+SUbTuYgB1MpjtuvHcoPtteoNeFlgGJUVszGfUTVE2xUN1YbEzCxIEL0rFTXJhcDxa6uV4T0QEsRztMn+sL5vaDKsqhz/Nf603w4xqpbex9ZwWK+jofN1rheWf+RlijEfsehIm0IbMXVQV4Gy8Ep8X75y5r7LfU8q+lgojoA498uePgCj9XufTbWeAV4vqmbipdZBBy2JN1fN8hlQ6YSVes+NcPKZmI522Qdr0EXFWP8/uvQl1x82vTsDl+vAiP288oXkVxBeD+zvasYTzM21G9mHyRO/7XN6kYjrAFhgksMSzwZPpXfIprJXhhcAnVHSmS0k9/S+966XrcbpvL0amw3oW7YpFZFLrE29jakQNUxdLjwwdSCW86IB6mrHne8IkeGLdwTk5+dtOiG9l8fZ7jbVLN2rftVWE+aok7xc2jMxevgYhsc7uo3Xk6/9ddn7l71hOFgG3UVinWwav079ofJgzLx2S9TJ0H0gN0yAO6pcgE41f4sdh5H6msun+0EcLAzPNWf5f7/QlTLbVTrhZX86mT4g+lN0iwaa1dENLqKZdjvGRK4XgzludDpVz2ZqtJg6UPwNwGEt2KdUJSFA3CEoHHnYKGCGCZLmiSpMAcilYqqosCI6btepxrGBQCa0I8lyoAO493MkQXPOeNZWTps8HatuUuw9Zy7Y2Hyc2/q9k8r7FHNUCdrQ8rkrpeXJLL OU/8tzp9 ShubFYQ7FcDsLEtsxMJLVAo/7VUf09UFumpBys6uIS6sqOve9oyW6kb8PD1kaiXz5HYQBYl+a8LhvU5i5cbzUN37pKthemXaVoQSLcVefo+XAe8B1Vl2ZsOdOppMUziqha1SeL+73+CxyH35efCEbz05xXD9fvlGlnIf+ywD9fCl/koPwOmX7KF10YJJcSm0lMgdHkVAP7gupFP6Ebb42dnkgx41xduy9JL2Q X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: By making release_pages() call folios_put(), we can get rid of the calls to compound_head() for the callers that already know they have folios. We can also get rid of the lock_batch tracking as we know the size of the batch is limited by folio_batch. This does reduce the maximum number of pages for which the lruvec lock is held, from SWAP_CLUSTER_MAX (32) to PAGEVEC_SIZE (15). I do not expect this to make a significant difference, but if it does, we can increase PAGEVEC_SIZE to 31. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 19 ++--------- mm/mlock.c | 3 +- mm/swap.c | 84 +++++++++++++++++++++++++++------------------- 3 files changed, 52 insertions(+), 54 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6095c86aa040..2a1ebda5fb79 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -36,6 +36,7 @@ struct anon_vma; struct anon_vma_chain; struct user_struct; struct pt_regs; +struct folio_batch; extern int sysctl_page_lock_unfairness; @@ -1532,23 +1533,7 @@ typedef union { } release_pages_arg __attribute__ ((__transparent_union__)); void release_pages(release_pages_arg, int nr); - -/** - * folios_put - Decrement the reference count on an array of folios. - * @folios: The folios. - * @nr: How many folios there are. - * - * Like folio_put(), but for an array of folios. This is more efficient - * than writing the loop yourself as it will optimise the locks which - * need to be taken if the folios are freed. - * - * Context: May be called in process or interrupt context, but not in NMI - * context. May be called while holding a spinlock. - */ -static inline void folios_put(struct folio **folios, unsigned int nr) -{ - release_pages(folios, nr); -} +void folios_put(struct folio_batch *folios); static inline void put_page(struct page *page) { diff --git a/mm/mlock.c b/mm/mlock.c index 086546ac5766..1ed2f2ab37cd 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -206,8 +206,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch) if (lruvec) unlock_page_lruvec_irq(lruvec); - folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } void mlock_drain_local(void) diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..7bdc63b56859 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -89,7 +89,7 @@ static void __page_cache_release(struct folio *folio) __folio_clear_lru_flags(folio); unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in release_pages() */ + /* See comment on folio_test_mlocked in folios_put() */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -175,7 +175,7 @@ static void lru_add_fn(struct lruvec *lruvec, struct folio *folio) * while the LRU lock is held. * * (That is not true of __page_cache_release(), and not necessarily - * true of release_pages(): but those only clear the mlocked flag after + * true of folios_put(): but those only clear the mlocked flag after * folio_put_testzero() has excluded any other users of the folio.) */ if (folio_evictable(folio)) { @@ -221,8 +221,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); - folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } static void folio_batch_add_and_move(struct folio_batch *fbatch, @@ -946,41 +945,27 @@ void lru_cache_disable(void) } /** - * release_pages - batched put_page() - * @arg: array of pages to release - * @nr: number of pages + * folios_put - Decrement the reference count on a batch of folios. + * @folios: The folios. * - * Decrement the reference count on all the pages in @arg. If it - * fell to zero, remove the page from the LRU and free it. + * Like folio_put(), but for a batch of folios. This is more efficient + * than writing the loop yourself as it will optimise the locks which need + * to be taken if the folios are freed. The folios batch is returned + * empty and ready to be reused for another batch; there is no need to + * reinitialise it. * - * Note that the argument can be an array of pages, encoded pages, - * or folio pointers. We ignore any encoded bits, and turn any of - * them into just a folio that gets free'd. + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. */ -void release_pages(release_pages_arg arg, int nr) +void folios_put(struct folio_batch *folios) { int i; - struct encoded_page **encoded = arg.encoded_pages; LIST_HEAD(pages_to_free); struct lruvec *lruvec = NULL; unsigned long flags = 0; - unsigned int lock_batch; - for (i = 0; i < nr; i++) { - struct folio *folio; - - /* Turn any of the argument types into a folio */ - folio = page_folio(encoded_page_ptr(encoded[i])); - - /* - * Make sure the IRQ-safe lock-holding time does not get - * excessive with a continuous string of pages from the - * same lruvec. The lock is held only if lruvec != NULL. - */ - if (lruvec && ++lock_batch == SWAP_CLUSTER_MAX) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; if (is_huge_zero_page(&folio->page)) continue; @@ -1010,13 +995,8 @@ void release_pages(release_pages_arg arg, int nr) } if (folio_test_lru(folio)) { - struct lruvec *prev_lruvec = lruvec; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); - if (prev_lruvec != lruvec) - lock_batch = 0; - lruvec_del_folio(lruvec, folio); __folio_clear_lru_flags(folio); } @@ -1040,6 +1020,40 @@ void release_pages(release_pages_arg arg, int nr) mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); + folios->nr = 0; +} +EXPORT_SYMBOL(folios_put); + +/** + * release_pages - batched put_page() + * @arg: array of pages to release + * @nr: number of pages + * + * Decrement the reference count on all the pages in @arg. If it + * fell to zero, remove the page from the LRU and free it. + * + * Note that the argument can be an array of pages, encoded pages, + * or folio pointers. We ignore any encoded bits, and turn any of + * them into just a folio that gets free'd. + */ +void release_pages(release_pages_arg arg, int nr) +{ + struct folio_batch fbatch; + struct encoded_page **encoded = arg.encoded_pages; + int i; + + folio_batch_init(&fbatch); + for (i = 0; i < nr; i++) { + /* Turn any of the argument types into a folio */ + struct folio *folio = page_folio(encoded_page_ptr(encoded[i])); + + if (folio_batch_add(&fbatch, folio) > 0) + continue; + folios_put(&fbatch); + } + + if (fbatch.nr) + folios_put(&fbatch); } EXPORT_SYMBOL(release_pages); From patchwork Sat Feb 17 02:25:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56E95C48260 for ; Sat, 17 Feb 2024 02:26:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA78F6B00A0; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CABD6B00A5; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E0A06B00A1; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1BD626B00A5 for ; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EAE9414012F for ; Sat, 17 Feb 2024 02:25:54 +0000 (UTC) X-FDA: 81799705428.09.BE4878C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 7F9A8180021 for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=lPd5VdaU; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mlFgA+f7ZWhuxah7PZlx2X3QOGE0MGG8XgVHVNnzo5M=; b=ZzBo+SFKOg4pGWCQ8/24EAk/AsdQrIhDn4Bp35mdIMbtXAunSEfPGqvC6j/uZ9KuA4xomw b60+jOHijOnMVi40SRowZCYxhriC0fO0JhlXWbRG/Op7JxYggtrlgWcRKBbtLLi7r+Q8og TXmdRVbldhS2nNAhuzjUTUvMQDPBZig= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136753; a=rsa-sha256; cv=none; b=e74milBUQ54clwpnvzpNCA11hhu/Lh9a+Me5SI10K4ADXrlhXBZDyXyk9BIhpjZb7OTrFc dWRVZljbM0lR5Lbgk8X4aUhEAjRzql0i3qgdA2Y9BTTiX+THneYLlOjHgQ0ipGt4Ss6Qk/ tJGyMcNg5iEBJVubi41EJxx30slK/7I= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=lPd5VdaU; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=mlFgA+f7ZWhuxah7PZlx2X3QOGE0MGG8XgVHVNnzo5M=; b=lPd5VdaUaYfPQgDi+6DMFBDc+0 PmhjdIEG/r1n8vOV5rfv7vP//GosH9K/WIjysAHl8ydjEU5MKAt1vxCkqqwU3KY5HpPM7IKDe210D 9KIgHIjE8TSciUAxmYGNy8BUzN4ACbHLgeL5MoKTbLSzh3Bd16Lg98KeEcBknTUkoEkuwPXCA07/A rpuRNsOJNLQDbauT1v0NDCGky1tRbIkH8QNY2Vl6O9Ek/gVmvHRU7bilwdke+R2waXYBG0iAH61kl vDmjgKAd2QOuxJKmxpbYN0Avfy/fjemJwxXOpvFnicEMoMbg+7fiOLHibTO5yyg3bBgvmTmbhv4qE YcIz7JYQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HD9-1cgP; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 02/18] mm: Convert free_unref_page_list() to use folios Date: Sat, 17 Feb 2024 02:25:28 +0000 Message-ID: <20240217022546.1496101-3-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7F9A8180021 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: be3kk4913hb4t66y7fdgtdety8nji4eh X-HE-Tag: 1708136753-85791 X-HE-Meta: U2FsdGVkX1/vsaHgUjleIje/IL/rk7Ora6yx4YWbQjnaWwEhGwYCtElKcQq9Zd8SQMkm6XhyvkCb9fucWqTLEmOZ4r8kn6+ZJGR30kGJuGdt+whsjl63cmHPoqXAERD0H96ycjTy7WUXGQr3jttWDgMJU6/dRioVR2+krboHulrmQNukqBXdDrSmbI8fogQwyvlLpsSKa6bdk1YgBXrn/F4MOrB6PJkT+GzcfG4ZGucYiyNnhnOow6wnef38Fgd0qvtJwHVI++sTR7KEeN8I0aAu1OnCeTiuHxjJmjSgdDq3uDsLa8GVLznvL4F5utFjGvaa/MnreJcopB4/mrrl+jAWImSJfK4JUM1w+iauYvzK8gLQnlfeEAWe5cXNe1GLvuPm8XvyMBIMD+jPQ8IeNR0nRC61MmXjNpbgpPvqkeBPP27plCcu8jjeyKVdq7leLDCpzPVPWfGqFBMfawtQmz3MiiGMGQVPPOMBTG2Qbswqw3o74BEsq8drXrqddDQULwlB6BukebI6hQyhSRkxr8j02ornEfOEhidOJBc6UH6EAKDlZC+k0S5lDVedN4OZRoEDsWclf40RSXofMFWNxIbm8bvISwTDZdBTR9os1NKhhyGKW/zn3DLYdlnxbAuovZX0LSBoyI3eh6W9CG1MNTKuaH8IfT/ipxOD+xMPpeSho6xNjPwcucS1npgxmlHj33Eis7rgi//oURc7yH4wFYoVDxhLjCJKlh+EImH96iZRE0Cd3KdWl9LKZ1BDJ+y7FBpdPRB3mfHM/Zn+NCwj1milLuWVBcumiPa7isFCRcA827hGsvbGs4DkqDX8ofMvuiS2E8tfcTzVGqWIJkW1KPXXU9k9maLHGy726w51l4hfMGPmhY58TR56nPjOnPLovqnVxMjB4c0YqZbXxgXx0rxnGDRac8cV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Most of its callees are not yet ready to accept a folio, but we know all of the pages passed in are actually folios because they're linked through ->lru. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/page_alloc.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7ae4b74c9e5c..a8292c7a0391 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2556,17 +2556,17 @@ void free_unref_page(struct page *page, unsigned int order) void free_unref_page_list(struct list_head *list) { unsigned long __maybe_unused UP_flags; - struct page *page, *next; + struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; int batch_count = 0; int migratetype; /* Prepare pages for freeing */ - list_for_each_entry_safe(page, next, list, lru) { - unsigned long pfn = page_to_pfn(page); - if (!free_unref_page_prepare(page, pfn, 0)) { - list_del(&page->lru); + list_for_each_entry_safe(folio, next, list, lru) { + unsigned long pfn = folio_pfn(folio); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) { + list_del(&folio->lru); continue; } @@ -2574,24 +2574,25 @@ void free_unref_page_list(struct list_head *list) * Free isolated pages directly to the allocator, see * comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(page); + migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&page->lru); - free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE); + list_del(&folio->lru); + free_one_page(folio_zone(folio), &folio->page, pfn, + 0, migratetype, FPI_NONE); continue; } } - list_for_each_entry_safe(page, next, list, lru) { - struct zone *zone = page_zone(page); + list_for_each_entry_safe(folio, next, list, lru) { + struct zone *zone = folio_zone(folio); - list_del(&page->lru); - migratetype = get_pcppage_migratetype(page); + list_del(&folio->lru); + migratetype = get_pcppage_migratetype(&folio->page); /* * Either different zone requiring a different pcp lock or * excessive lock hold times when freeing a large list of - * pages. + * folios. */ if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { if (pcp) { @@ -2602,15 +2603,16 @@ void free_unref_page_list(struct list_head *list) batch_count = 0; /* - * trylock is necessary as pages may be getting freed + * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. */ pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, page, page_to_pfn(page), - 0, migratetype, FPI_NONE); + free_one_page(zone, &folio->page, + folio_pfn(folio), 0, + migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2624,8 +2626,8 @@ void free_unref_page_list(struct list_head *list) if (unlikely(migratetype >= MIGRATE_PCPTYPES)) migratetype = MIGRATE_MOVABLE; - trace_mm_page_free_batched(page); - free_unref_page_commit(zone, pcp, page, migratetype, 0); + trace_mm_page_free_batched(&folio->page); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); batch_count++; } From patchwork Sat Feb 17 02:25:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3090FC48260 for ; Sat, 17 Feb 2024 02:27:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5E5C6B00BE; Fri, 16 Feb 2024 21:27:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A0E466B00BF; Fri, 16 Feb 2024 21:27:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9245F6B00C0; Fri, 16 Feb 2024 21:27:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 802D66B00BE for ; Fri, 16 Feb 2024 21:27:24 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5B895A012D for ; Sat, 17 Feb 2024 02:27:24 +0000 (UTC) X-FDA: 81799709208.09.D031153 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id AEE30100008 for ; Sat, 17 Feb 2024 02:27:22 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MrcX6eiD; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136842; a=rsa-sha256; cv=none; b=pjFaUEilpVQZGZHM49XsMaWpju0hAhEPsKb9t4g75FeSB1N/DGygEN0bbQZJBByar69OZC U57/2i6m7MoTzgZb9zt2z6NhB4/Cakazm3pDy7pgAnsFHwM6BIq13SoP/HXQR9SVPW6Pxn Mjshh/d3DguLKy1niQP4U9j14wufAas= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MrcX6eiD; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136842; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ywr7en3i1eBptvHXurHGwggnE1YYYXLr2O4imJ1FezY=; b=d21V1jdjL4qXaXo3hdJ+LzNE6xLBlN5B7njDYwBT0/XQILdrEfLyIrU2p4yctG/f/z+Hy2 dYJEDDK1u6rDB+tmwhFyZ66E+FsGK2ajluQGpXwRLaYFKJu2eQSKQwi9VEX+2LAi0IgvDa 28HuZtagYKFqGLgDvkffsrHR/L7zSHg= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ywr7en3i1eBptvHXurHGwggnE1YYYXLr2O4imJ1FezY=; b=MrcX6eiD5VHikyl+maCAPGDwKW /zxx6HiCTYVda/CWo0uwueJstkMJ631CLbxs+LorXySVVclDhjo40K1990kYDhKv0J9Tz/AEPZDe/ n+BAONSDaT9GzPNx1IGPIfveWIWmaY3NOILvWAOoc8h2mvWM+r3h449s/xjzrm9FuCa+qluVnCRxw K23bSDPz8844eS3HZquqtZJzE7jgkvyorCcQhbURnlPOwH3fS/4+JydXKWTDjRfwN9EdMSsOl/QnV s2r482NsECD52/iGUL3FF7o0VmnL+sF1srzQuJXCj+GBRWQlersNsUdTNSN1KH8jrLcy68mzj7vBy MeaLV7kQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDB-1z2T; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 03/18] mm: Add free_unref_folios() Date: Sat, 17 Feb 2024 02:25:29 +0000 Message-ID: <20240217022546.1496101-4-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: AEE30100008 X-Stat-Signature: 7afxbou7pziibsbbuxb8barskjz94mj6 X-HE-Tag: 1708136842-646939 X-HE-Meta: U2FsdGVkX18FqnA6AkpMc7KG3UxTvLVTdGUtOJGqbOqUzWSdcXUohDsZAFbasvFoYWXbCx4Ih8JsI7S9LR18C8gdwY3lBni4pyLwUgiaZ/EOI6yzfGjNkLuIFPJu2FieuKY2eVtYCmRjcCJrX2/xr3ofeEtpZLvh5o9F+FKjz/HCG6QKgSeFoyE8LuoQ9VKaEd+G3ylZX6l0eQXg7ShUAZEvhoInd9FFY8Abw7MDG4Gtg5ieVlDpSMf7mceBcKGjUQOS4zRvonRPW6qNZmMMGXIAnk80Y69gdANZT76rHsL8bID1ppHpLn//1H9J7VmBJ1lTPo3yNdBtqs9WzaCqC02itq/BbGNHPoasA5GMKMNkkPKSwXSanYlP3CFA9t2l4/SHWzQOtVd3vbDfn+yyGk0n/nrl1T8m4ELlZJs86BJKNPMAlI3GOqlXxs6ZO/ns7HQvRvo43DkqyyM6iLwPnTIE3sxJqUctXbnxwfjTitQGshD/RF27xpPoKncQX2ElvhjzXQXMXmNy/77kpFAbE/KJ74AjlRv9VbGty9qTwwg/ezRuEXL7GdttYTjpfTF3IS1q6KxHlGQZ93eX4yntiMZ3Myrm39Nd93JqzeuceAtYmvhgIRORJPWXcny09TXkq+X2Sy48VbR0vHYouiSFy3ORslOHZ00BjFNSFuT9SfHm2H5EGkysjuEQWQempD6QgvAr1IAfyM99gzKDPH3FB51IrJ85KP6kMZAhH+G7Kco0R3eXZHl3LwfcvUACM6a+gvKOPP/pE3prnlIZR6JjU8SnsXtcMkeoUoPVLunVeGuMt4CWVCK5zltsT0O0xkSMAVUxB7ilfO7sBG5ew+Vby2Jgw6tKTdVZ43M7NI4ITtOCfof4taRnbtVS9FRXGNVkZ+8NcWAUOwAEH9u4256/+NfJow6BrIQGLmJNmBICeqsEaCfNShRHVbjKJEGiR9o2ujFjOA7d9V+t9UNVIcq +fwDRBIU +YS1KFVHELlDUcOzDlSbfEa+hYx9PnepmM+7E5h6Eto0RKQ3o4aOaeNPPwl9ZO0bKpo8OmQoBtPdlbhs1GSa/FoMFxKexMgUV0ZAFTOeXMItzBdvOgLD++mJTONthN3PMUVMOEWu5VGKE0/myb1Q6rL22xNJoOjwdP6HBloLaKoMk5vREGG9+t73tDvjPjaAjeHrhTSXpTuv5PRKng6K6o5hDgJXlxfxE0UVCPAc1Up3jNxcM4mwLHpqKuW6W9aQovnT8heryJMXIpHQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Iterate over a folio_batch rather than a linked list. This is easier for the CPU to prefetch and has a batch count naturally built in so we don't need to track it. Again, this lowers the maximum lock hold time from 32 folios to 15, but I do not expect this to have a significant effect. Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 5 +++-- mm/page_alloc.c | 59 ++++++++++++++++++++++++++++++------------------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index d6044c684e93..4d45b351e0fd 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -449,8 +449,9 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern int user_min_free_kbytes; -extern void free_unref_page(struct page *page, unsigned int order); -extern void free_unref_page_list(struct list_head *list); +void free_unref_page(struct page *page, unsigned int order); +void free_unref_folios(struct folio_batch *fbatch); +void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a8292c7a0391..8ef1c5c86472 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2551,57 +2552,51 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a list of 0-order pages + * Free a batch of 0-order pages */ -void free_unref_page_list(struct list_head *list) +void free_unref_folios(struct folio_batch *folios) { unsigned long __maybe_unused UP_flags; - struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int batch_count = 0; - int migratetype; + int i, j, migratetype; - /* Prepare pages for freeing */ - list_for_each_entry_safe(folio, next, list, lru) { + /* Prepare folios for freeing */ + for (i = 0, j = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) { - list_del(&folio->lru); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) continue; - } /* - * Free isolated pages directly to the allocator, see + * Free isolated folios directly to the allocator, see * comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&folio->lru); free_one_page(folio_zone(folio), &folio->page, pfn, 0, migratetype, FPI_NONE); continue; } + if (j != i) + folios->folios[j] = folio; + j++; } + folios->nr = j; - list_for_each_entry_safe(folio, next, list, lru) { + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); - list_del(&folio->lru); migratetype = get_pcppage_migratetype(&folio->page); - /* - * Either different zone requiring a different pcp lock or - * excessive lock hold times when freeing a large list of - * folios. - */ - if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { + /* Different zone requires a different pcp lock */ + if (zone != locked_zone) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } - batch_count = 0; - /* * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. @@ -2628,13 +2623,31 @@ void free_unref_page_list(struct list_head *list) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); - batch_count++; } if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } + folios->nr = 0; +} + +void free_unref_page_list(struct list_head *list) +{ + struct folio_batch fbatch; + + folio_batch_init(&fbatch); + while (!list_empty(list)) { + struct folio *folio = list_first_entry(list, struct folio, lru); + + list_del(&folio->lru); + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); + } + + if (fbatch.nr) + free_unref_folios(&fbatch); } /* From patchwork Sat Feb 17 02:25:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D29BCC48260 for ; Sat, 17 Feb 2024 02:26:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77B0A6B00AD; Fri, 16 Feb 2024 21:26:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 728666B00AE; Fri, 16 Feb 2024 21:26:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A23D6B00AF; Fri, 16 Feb 2024 21:26:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 466C86B00AD for ; Fri, 16 Feb 2024 21:26:05 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 221E61C005D for ; Sat, 17 Feb 2024 02:26:05 +0000 (UTC) X-FDA: 81799705890.04.DA07B4C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 878F9C0012 for ; Sat, 17 Feb 2024 02:26:02 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EWWh+9Q2; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G5MiSMIQieXIDtrNd7LuHEe7+/2xFOKZC61A4SySwyY=; b=D+mXh8E1tfqI5QLaFYX4jeeFfOz4vz49VhJJglsW6SFaoM2gyNTSmHpRscqScHjgAUekYf YLJdTv+X+F3WU0/xUlvs/dvzxKi9LdAuQR9Hrlc5KOmMtdMgLQtq1THi86p5teZuKV2cxA kiI3k1PKy5cRaqMooOJN5k56075VH2M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136762; a=rsa-sha256; cv=none; b=ubeV0c+bUKJC+sPc/djAcrpc7rB53IXXdD6wC7Q6/P2N5L9LcAekv6XOgSzcpa7j584pEF Pq1ONKbxB8SFJc9Q48kwI39elcmczNunVtH2PZ+fmzpirFv/h4Iud4MznMuP8BpmIUsxM1 H5pT8W50g814wl4blA2nNxHY12HgdJw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EWWh+9Q2; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=G5MiSMIQieXIDtrNd7LuHEe7+/2xFOKZC61A4SySwyY=; b=EWWh+9Q2NaP/tdzptWLwwS3e+l TbK5tXDMT84kMnJixuSxvd+VWASxbalExrQDegNcliaCEioL/LP8KQ53Anp9WTzOfqsJKSeG6WLas nbEJ761WPMTImxKnKdS4V/wlOUgaZk7Gat+9V+MAayObdtap97MQoeW6fQEqq34FTa0Pr2CbRJGWC /AjLLiPxFWbGTA/6cR7EYUilDB9xfQmH0xYzxyJoSifGYoTZokVPZF9SKJezlRPwybYbwlOncKI6r k1h0XZXkzroWGB9jIcmVvs/vIsw2HnvP9ahZ+G7KIqmBSY44Rkj8PD3Cmn5HhW/NU4hUkhY7+qK5/ xVliHhpg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDD-2KW6; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 04/18] mm: Use folios_put() in __folio_batch_release() Date: Sat, 17 Feb 2024 02:25:30 +0000 Message-ID: <20240217022546.1496101-5-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 878F9C0012 X-Rspam-User: X-Stat-Signature: b4hnzds4e55533zsm98a7a38fyqmg5je X-Rspamd-Server: rspam03 X-HE-Tag: 1708136762-439620 X-HE-Meta: U2FsdGVkX1/t+W9htuMRy0nLHl1u15v2QxqmodutSFWUW3yqFxI96FlFmdnpiO5ck6xNtMTrkwfPzTzwln9sw5orPTwKNqvmxmNZsLdz8kYzVZpgpBFiJSU+pNdsxzRi5E/DYjzvMFWRBh/Byt5bUSntIGtE/QBej+ezqBhGoG++Pxund/hWfZtM26KflDhDe4gFt9dlIzrMHuOp4XRPvML0Mxffpv3Z3Ft4v7h3qnRAGev5mJc9YBwtuihxdZ5QJZQ2kwX0A1NfkiMM86ljmfv+o7o4kemaxd+yul8LKkeTTzZs3AatM58Gexm1j++WREvnkdn+gVHd0ldsuXOr0sGHXlgKeUhXY+hLXYPVYn64Aucrz2wcs4Px73cTKY1mg3RZI54s1hZ4XqCQ6Xw7JX+yerNwuwl7wBScdDVVKRNa2XForXNERcvqpdaOLxtnMmapGk+P8jqBFPQPZr3YZq0hSUac3wpBSgdLrZllT54cnX0vdATfjqutRAk8UqN37gWWAI5Y3eX108N3V1AQbVGad7zq9eeTJGYl4A8VOKo68jvDLjJxEimcjg+mfGKVrlooiPXYU3MiE+VAHC8vmN6IN6wjK7HxSdoKe8YXssFaJupxNlLcY4IK8aZCYXt9r9wTz3Ee8Ji3bJZCqheZzW02KgHtCT9L7akHEuIy7FYjxW6veSVI8s2LYbbQYscjvCNZ5aLusU/R9hmYHCzP3ibo2gcp9395SUVH9mtFgZKf8ajfZ9y3wgdsH88y5kqB+Rmcu160sqzAxcq+gu/ZrBusnvo57soDhSWQL2hXA+SR7qSP8l2sWA79t3PMabpfhb0AeGT8Rzvu8vZCJgrbgOGdtwPah1v12XsSXvVEElCb9U9YPNRrSdGUkcOMZOKTgK6pabax1tjGtpQoDzPb7EBsJcrtDo+DWpnWsap0szYaYx/+YtdWaE1QOibRezMOlx4XXxaBE9WxMSUrQaW m0l4oOf2 nvvaC74TmKTDx1Yfq40d/AFJGh9W8W0P4Bi3FHS2T/dVAJUX48wlFg7oMlV76Mgnp0nKKb2JH7obCpU+CFG2oXTYa9EbtwzwTsBgEeZRyfm0c3A36yjGwJGlIapRVBRjxIsvUN2BJdPGz0jFzt2uwYGCrqwNtz1fwwIwurGkeFb7hyl8mGGoIB6x4rjJpCH0C+krJjZtf8dnuT0SWNfCWxQ/S3Sgm3+8K5gazQh73QCG4+Bvae77u6yqLV5Isap4+7X7oC004IqWBER1Rcm5tIogXsu0Lg9/Gx52j X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There's no need to indirect through release_pages() and iterate over this batch of folios an extra time; we can just use the batch that we have. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 7bdc63b56859..c5ea0c6669e7 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1073,8 +1073,7 @@ void __folio_batch_release(struct folio_batch *fbatch) lru_add_drain(); fbatch->percpu_pvec_drained = true; } - release_pages(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } EXPORT_SYMBOL(__folio_batch_release); From patchwork Sat Feb 17 02:25:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDBC1C48260 for ; Sat, 17 Feb 2024 02:26:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A2BA6B00A6; Fri, 16 Feb 2024 21:25:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 74FA46B00A7; Fri, 16 Feb 2024 21:25:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A3B06B00A9; Fri, 16 Feb 2024 21:25:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 468ED6B00A6 for ; Fri, 16 Feb 2024 21:25:58 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2103A1A0149 for ; Sat, 17 Feb 2024 02:25:58 +0000 (UTC) X-FDA: 81799705596.22.69E2829 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id A2F88A000C for ; Sat, 17 Feb 2024 02:25:56 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=fI7dBJoP; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136756; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nC0n0NXQQI6XzJ4rhAAADl5IL4OyomYweYGR8HN01FM=; b=VXkN5eEqHwi8NJLVeXQql/zUBhH9PC88e4v1sA+xSJk4h/q4+qQuYQcwRyp+/ZamIlFHUt 6j8NTkHQ5EpRbjp97aK1KLcN9UZ3E1zQh5RTooZhkKzKXIwcSNLKkY2xF0IMDzJkKHCpIi Jd9zRMZOgqPxQnsZAAxEJssJ8trWxQo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=fI7dBJoP; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136756; a=rsa-sha256; cv=none; b=fYqsHi5cA36uyCoKTluEXwoi+trblRjKhT89Jgl2cweWQQgKMGfhlZri7x55wCsmEv/YQ2 jVQ74zARLvBtrTi4L0nT6zp3auN/4cgDYQjX4jbfzPvuR2acQr5ZBR2+7rH+rIt1tQYzP6 2dCRg74nY2uHQ5aX8lZFTr5x+v0Sl2I= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=nC0n0NXQQI6XzJ4rhAAADl5IL4OyomYweYGR8HN01FM=; b=fI7dBJoPXDMAZIBrlVUZHCw2DQ x0nqj0o/1ZPZcGMtIHFg5f7rNBmOg13IX+eq8ptJTpSPuaaYGvvinOwKdsDISX1X9zDIma/rh990G F5rTI8ujHVhH4TiQFDA/21uHAfj90QSZ4iRg4nf3I/NMDBBMbKVxrmCa6iQhKe7mG9oD0jUqUZJ1B BfUxPo6p2FmArY/tWs37JVehAvQoo7t0a0/++JxbNTEuRbXqJsSgsvNr42v4QVdS7dBeX4AC5Jjew QtIFKz+b3y5DiwPoX+jeaNek80MWzL9hZzKnELK3zETwVbj1D6vlvSOMN3AXPc5+A3WfwSNEEY0DQ VB0cCnLA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDF-2ccd; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 05/18] memcg: Add mem_cgroup_uncharge_folios() Date: Sat, 17 Feb 2024 02:25:31 +0000 Message-ID: <20240217022546.1496101-6-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: A2F88A000C X-Stat-Signature: 6xew5ag5e9dqd5np7p4677bnro4j65jq X-HE-Tag: 1708136756-394914 X-HE-Meta: U2FsdGVkX1/R28d2XPrR+8zJywky6fwTJd3jkDGVul4MOxWvAv0dBHaN7gzJ4Vk9XNILqpUrpuFfK4kxHpJhdTPCcFgsV0mRAY4OA/14aShiKjmhzzEvF02oHMvshttN7hJM99pfMiUU0iI/60ZxHhEo1Wu5VPvJVYeo+jUluyx9NvUCsD4nH/H7cha0sCEEShorTemkroJVesNEsZTFtdBICJlfqFm6cVIcHc7wLkRNiijrMnkRW/L7fce6tRocYfYxZ0A9xjVpG/qgp5ZBstbSqISi5ijh523w3wwRMLD7MoawSOISG+1DPukQ3lGVO8OYgRafpqRs1HHOqbQhK9MwL7OkXId96p63k5LYSBpkeJNbPh2b96ZNOx0QHcMhUHUwc7C3wq4uX4h3UF20KtYSeYlFBk+VNiTsjPqH1EOocSd9r1WnNUy47bgitRLhOAVYq4UrNLvYCgLCdq0TqfVGwNqB8lI/Xi8702jBc6P0ka8uyINj+iTkv248xE6CUMIHtOpn9t9hWVQ3I++uayDYbYZ9a4Kdk6W0Af7GxJtzuZCUouerEkre2t8FHAYJyTiF1xEU28xUZEDVVJ9h40AIMn/K7HsKuWX90SeXBfmSbQUoEJayMdxT+LBXvpnRbcpNfDSpU0W4htmvv9oRYn+uS1rWoy44cG3xBD6djSeWHldSqy4rUBN1KltSn1XxPANiX3griCo6sMCFDRxu1nUVrqcNtvKj2N3UmlNJ0gunQS8Polwhqw7/LkrTqwORSsjk/6EQUfOsVkCicsGp5uRBpnQIzrmW2c1q3o3fUvRzCsV7TeVkZxdgNi0oDLM/DzTYVuNYmXCD0hGFlVeqSTPgZMtuQe+pZJMeQmWEkNrHyXzln06GdZxLm97HVyVYE5VAIPHBpef5SITzvp0+mczU+WoLl62enuIogkqS9bY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Almost identical to mem_cgroup_uncharge_list(), except it takes a folio_batch instead of a list_head. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/memcontrol.h | 14 ++++++++++++-- mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4e4caeaea404..46d9abb20761 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -721,10 +721,16 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) __mem_cgroup_uncharge_list(page_list); } -void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); +void __mem_cgroup_uncharge_folios(struct folio_batch *folios); +static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ + if (mem_cgroup_disabled()) + return; + __mem_cgroup_uncharge_folios(folios); +} +void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); void mem_cgroup_replace_folio(struct folio *old, struct folio *new); - void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1299,6 +1305,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) { } +static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ +} + static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 94d1b278c458..0499d7838224 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -7564,6 +7565,18 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) uncharge_batch(&ug); } +void __mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ + struct uncharge_gather ug; + unsigned int i; + + uncharge_gather_clear(&ug); + for (i = 0; i < folios->nr; i++) + uncharge_folio(folios->folios[i], &ug); + if (ug.memcg) + uncharge_batch(&ug); +} + /** * mem_cgroup_replace_folio - Charge a folio's replacement. * @old: Currently circulating folio. From patchwork Sat Feb 17 02:25:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FCD3C48260 for ; Sat, 17 Feb 2024 02:26:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8277A6B009C; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E53776B00A2; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB23B6B00A5; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 90B7F6B009C for ; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6AD3240133 for ; Sat, 17 Feb 2024 02:25:54 +0000 (UTC) X-FDA: 81799705428.06.F981686 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id CB428A0002 for ; Sat, 17 Feb 2024 02:25:52 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UowlmIBK; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lqE1h+TV7PuhvcThUGfshDf8IYmuBERhJxDPw5eXVAI=; b=CTVN5kyyyKRjZpNJ8+3lQxup61DVJ9tHdBWKH1Fd/NNGfCRqcokEf4CKNESP4pBODH1jVk g30+cZ/NN2y0NSKU9V5UnHMFNyH8REvRpwyyKlaIopQZw6Kv5hkTZ8ee5jOrQKGH6GY5rM jManJ5/Vibu2RWot9Ts6+4PuSGW+AtQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136752; a=rsa-sha256; cv=none; b=Eh9EoHuQrHgizTkqeSCvncMJFiuiDtdPyIKCkr+F/vVZkdt5qnhaQOkpRhncQ9Yf0rM52T 0f1WQDQO1zOesUP6emgBVY/Y8O82F/gs87XwN8RhbFvzHsm+8P+lwdRW5YqUJkbaBWdr51 uvLkeC7IWe/QICOFICJ1Z1a9ZCNYk/E= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UowlmIBK; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=lqE1h+TV7PuhvcThUGfshDf8IYmuBERhJxDPw5eXVAI=; b=UowlmIBKxTgGIyCseqXthtDi5C dAxQBiyGIH3YaJ+O2a1zd9SY+skK60xKEbaski+YRuZe+GoAgy7UtGOAdadWrxdgkClx76mX3Z89g KdumithpSggzZIL7TYoCLVIcGWhGoM4yptw4ij4N1A92Yay5PGqE8ly48kSI0kd/G8fSWbln6rrFg XxCMDsbqx5NGOpf9bw0/u3df9oyxxQ6jqff6PyZJu1VQqOtx97M/PdSZZeLzF77FR94pLTm62d4T+ M3cZ+8VFsBlnBEw8eeSDiNvmuW4Zd6YrtznxOBxQ52muDZUqdoSb5y6xXZXihCFg6VK5KszJT+ax4 m8QDPLOQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDH-2zqR; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 06/18] mm: Remove use of folio list from folios_put() Date: Sat, 17 Feb 2024 02:25:32 +0000 Message-ID: <20240217022546.1496101-7-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: CB428A0002 X-Rspam-User: X-Stat-Signature: jz39r6qbp81jw6ahpjuknypr4sd9jufe X-Rspamd-Server: rspam03 X-HE-Tag: 1708136752-137298 X-HE-Meta: U2FsdGVkX1899u7iz3btx/vUSYmHl0PsMfij+rOsWjlfx7uDbGby+sWJH5dv/TNcpjSVwCQ3BE4PfioyqS/+FZRSjEmlyZakp+3dTEiONdTgKYJriBQRYUewVD/TOOc/4EOkPEyuR9Kp+PQfmc3jLRCkFVe+L6xAdiolbv7QL0IkGhcE9WbRYGMfOyWOaz0cRi3hHA86zW6a0WjRLUW1kymkF6H185y63GIVTn89ODJ0ul2qUaBlTCrCa1m3BdpvMZJ5Vhmu4xXVFbDasRsRNX0mAoDwDhFf88cpDPjuVOlfTOmRhB0COG9heC3N6pkh6/EpdF1UkPOpJ/tQCQSChJfyn2isybzVjVe8jZ8QHuiTDuYwgBn6t60yo5LTIQ3F81vbcWy1IRI34A/ZJh+VpyD3ttS0YXv4U+MLQd0r7zq0v1ebNOEzGhBnPYKSY/BnoA0CfUZZTF1JxVOlHu6f0T6RJw7361dnYcRQvJb7JkyGdRhfwtXVa1R1cZjAYHyNJmv6ZG748Iy2uKqDD27cpfVk23YXg5l3564CFLsLfQXQfD3qo1CoaPldiiYC/XMHrSosQNnQ2UCAxUI6eec4iN5fTqCMY7qPWns1b4m3vUdhHRrxdVJ4Ka5Hy6LJSTnbRa3roCABrbT3r8s9vFaF6TjG7TC668pxLLYMolCvxZ2t2+hB/EboNt0sLsEBaHTQEK7Md7sjRQkX1GGhD2wnrfMq6C40IYQcLIjosZgMyEgNYH5ukf9RxHtvVMSM9kmx0WlTnb98fsrbCPmEvRx5i434AlNXgwwH8Th7sp9OkVDp1z5IsX9mdxIGDNLrvcXycVgiVQUg9mmTwLbpgpPN/dzrOthC2AZQS25x6amkA8nOJsZFoaSCZe4a+ZjavXFkD9aR+12+zZMSX7iX7ViQdaGIgLiUK4uwvAVx1VLWKfjvsvb8VSYg3eIqY/8bFsrhscMyu3YZGauBYX4ewEq vITtige3 eOtI0FaXEbKj9t4x0v8H0aiuUu8Xw6rEH6gKxdqYitC5X9GAvcbRpNJr1IUlp91xvTMPaQVWf2JHXJ6xaWRteCBsEdJ4gnzfBxvwcy95NmLtUaFSDcETaxsOYDKA6cY9O8kUdIqkRVQ4N7fLBLhTFr4SYq0eg9s1c2Tn1Gcad7Xq3n4TqtuyNqpnCgvEP5q7NI/Ti6PBokNBXDqonJP+KjeaJwfWhxpS0hAHYZYFqZfdgKs413hSfpnUDh2SuysmE37OQ5glUTSQBhjBx4N6qtk5nw4fYSNSmLiEI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of putting the interesting folios on a list, delete the uninteresting one from the folio_batch. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index c5ea0c6669e7..6b2f3f77450c 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -959,12 +959,11 @@ void lru_cache_disable(void) */ void folios_put(struct folio_batch *folios) { - int i; - LIST_HEAD(pages_to_free); + int i, j; struct lruvec *lruvec = NULL; unsigned long flags = 0; - for (i = 0; i < folios->nr; i++) { + for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; if (is_huge_zero_page(&folio->page)) @@ -1013,14 +1012,18 @@ void folios_put(struct folio_batch *folios) count_vm_event(UNEVICTABLE_PGCLEARED); } - list_add(&folio->lru, &pages_to_free); + if (j != i) + folios->folios[j] = folio; + j++; } if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); + folios->nr = j; + if (!j) + return; - mem_cgroup_uncharge_list(&pages_to_free); - free_unref_page_list(&pages_to_free); - folios->nr = 0; + mem_cgroup_uncharge_folios(folios); + free_unref_folios(folios); } EXPORT_SYMBOL(folios_put); From patchwork Sat Feb 17 02:25:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A16ADC48260 for ; Sat, 17 Feb 2024 02:26:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9CBA06B009F; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DF126B00A0; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15C196B0096; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D19EF6B0093 for ; Fri, 16 Feb 2024 21:25:53 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9E0EA16012A for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) X-FDA: 81799705386.15.F8ADCEC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id E479820005 for ; Sat, 17 Feb 2024 02:25:50 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nhsBxa5G; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zxPjxJ3sR8/tAmwn7dVHy6hIoko7fcejXVznXxtZqEE=; b=3BU2eC3nQNWHzT1mhE0ysl3JS9F+dzSaN/qCK79uQgVWRF5HGJCTaC/vG/ygytTQ3cYXVH kI7dTj0x7uKvRBWx8JEz00eGG1CHsGeGeUBlavq+rndoCrHu58d9bjj4C4ZpeJG+uim006 i58L5HPcMCsRKkVKilYukxTvKb8J/CE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nhsBxa5G; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136751; a=rsa-sha256; cv=none; b=QX2TlgyDCSirzDIMgxwQnwHHb5/Ua6id1alVlAsapeZSqotTN/s9I2g6Mn8zhhKFozHwcb zvOY/H4FXTHlZpPiUumjAx/p14bZWiT20/7oxlmlxNN/Jp/J2mFoE5lr35LLVIqYTlsEdh Fr1ddoUS3WnCSgzc1XBKp3Odo5bLPco= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zxPjxJ3sR8/tAmwn7dVHy6hIoko7fcejXVznXxtZqEE=; b=nhsBxa5GmjasHs7y9XXuz4lSOC qh55W/ZLqTWACzE0UNAAHrkiayh8BfTgdZBogGomTb6dh+ds1WK2bdxtXBIm5RZCd2gKF4khwEPDi xVnC+8PxTgzD7l5i1BO6WZ9n8c/NIRdX1VayJwS5lWBvP0QLCovNBONgMisfxDgwKRkzVqxVR9ufr qQJx2ehWxicFR1Qi0h/7vqFM1hRpfcR887LlqZ5JANGJwDzrJCBxQ3zLtdVQHZwlnnlKz8+4Stwga bJIrRwwXrv6iLW621cI9/fly/cfenuu/9GFNT+yVVfoEtg9SyUVBH3CpnsNk6Tw1SkVCUtcGjWU3S oafepyxw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDJ-3M6b; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 07/18] mm: Use free_unref_folios() in put_pages_list() Date: Sat, 17 Feb 2024 02:25:33 +0000 Message-ID: <20240217022546.1496101-8-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E479820005 X-Rspam-User: X-Stat-Signature: fb8b9qsz4t6j34wm6fb43g7sbndc64o8 X-Rspamd-Server: rspam01 X-HE-Tag: 1708136750-578330 X-HE-Meta: U2FsdGVkX19q1zzTTNeMilIVZ1NvuVNnBcBl3BzhZGH5a8shZg5Y3kbuXcgDrbiEuZX2M4qXcuIwbG14ILC7rfRBsFzkU17hDbQKZpgvZerpjUNVTLngZ1F3KlNHmjuSq0po+xLzaIaD4y+FvsBUvJpDC31OQv7khQH1MgG8Uba/mkm7Ce4YcKv995Csg+EkQE8AxXBQ4aZdNOg1JbJiTPlnu+XtrdmcAvSJwPjcXT7LnaKcPNN0PHHNJldYFoveiqq66ltiTywvIOwSoiFpM9wc62jOhxwl2nztmF8akhxY2eskL8LcuG9EJ3rQh+0RXcflrIiQWOKRX/QUWOlBD+y1WUPS7dbhL/9IZ20DXpxj52CsM8HoOVkKuJxkpl3jJ63P/muVn/GOz3CmxuRF3pO3xFEkqAHgn9VGeh7XGW3Loem3Pq0N9JLQU/Yc5+59uYGtDx1aRCeD7f3rPCVwirSkdEJRQp5BnQzFWsT4AAW+KT91KhBTdt8crVF8Zr2QE5x0VYf2kWzjr/0MukjWvp0f4p4irfE754eqi73Z2KJlL/scTv9x+BUuYmZDZZPGKzP/Ee+rKA8LtSYb80AOxn9VHekJR1LFO4NrdpMbmycmrR6Lt0RDxkabZ65GftLVblm5rn5yKjoeehZm0sJZ6LsCMbElJl28yar1rOX/Cb1jNbtL2kcgXIIENfgL+ArHH87HY/N2HpM078Zq8v2wY9XdcN8BhW/8Cr3HrwZE1AAE4tsgL78Av4l9XX7OWW0AO5tSw1EeZrK9XP0vBXUYksr37zv/RP/HHXXwGAImsMOkUjggBy32+G0xch/rmnrdDhtdFdllnHGuc2ecyhV3tYi1tE9boUNCHwSFD545ZtaVKOSbiS8+KGmNB39A9qIVXWoG9v9bp1K+8xjD3sja+VF6TolHvtZNBIA0Tc5O299ys0iuwtExkca2edBCX15rMJU8FLYM3a6aptgthGz MZ+NviEa 6Ca8MdXEvqYA4OEZmdFcQCG9kXGZfktinb0WDqDkuXJ+fv7Db3Bj+qJQ3x5qSdTDmb6tOgeXd95OiG4srj3eOy7UgqnTZ3v7u5G/KtG8BX4eInZ6Mcvzv3cDxTQIoAPZbsEwowK/U3tqK+47MOoLn8biImI+BAooaAhSeZRSfiFhQpgth9/nzQce4aeCwLdULZ+M3mXHeUG8OUAIc619JnW7lz3cyFC19FegD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Break up the list of folios into batches here so that the folios are more likely to be cache hot when doing the rest of the processing. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 6b2f3f77450c..fffdb48cfbf9 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -138,22 +138,25 @@ EXPORT_SYMBOL(__folio_put); */ void put_pages_list(struct list_head *pages) { - struct folio *folio, *next; + struct folio_batch fbatch; + struct folio *folio; - list_for_each_entry_safe(folio, next, pages, lru) { - if (!folio_put_testzero(folio)) { - list_del(&folio->lru); + folio_batch_init(&fbatch); + list_for_each_entry(folio, pages, lru) { + if (!folio_put_testzero(folio)) continue; - } if (folio_test_large(folio)) { - list_del(&folio->lru); __folio_put_large(folio); continue; } /* LRU flag must be clear because it's passed using the lru */ + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); } - free_unref_page_list(pages); + if (fbatch.nr) + free_unref_folios(&fbatch); INIT_LIST_HEAD(pages); } EXPORT_SYMBOL(put_pages_list); From patchwork Sat Feb 17 02:25:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D95CC48260 for ; Sat, 17 Feb 2024 02:26:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5E226B00AE; Fri, 16 Feb 2024 21:26:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A0EA66B00AF; Fri, 16 Feb 2024 21:26:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 889F36B00B0; Fri, 16 Feb 2024 21:26:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 76CD06B00AE for ; Fri, 16 Feb 2024 21:26:08 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2737B1A014C for ; Sat, 17 Feb 2024 02:26:08 +0000 (UTC) X-FDA: 81799706016.14.11C3867 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 8A03E140012 for ; Sat, 17 Feb 2024 02:26:06 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QmlP8Yff; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=21JMIPgwOOuKR2eeBq0L1sNsizlI1+DP/HMzZ2lCV+A=; b=VxLmf8utm5XuuYbszPY8FxJ35Dj+lsvpdpwUvxqPawm+XW77H+kQOLtPG8NsHMqzGrbtzr YO56r6OrlQlGTRhot5oHyqbMYxmGPb1u/K0eaMvezyJ0M328wUEQjjFhylBTaA4dXkJbVt 5/tzTT0E0W7CNlWj9549whxqGaOghZE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QmlP8Yff; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136766; a=rsa-sha256; cv=none; b=JdrCbcSRAtlsJHcsa1B/XdHkLZzpJb/6xwEyrlCgkraoSYX+OsFj6WvRKxHLbnXM22QiEO FLRabPDw0fvs3/Yv+uEXlLeGTViatJNsj4Y5exLGh5eaOZr3llctTI6ELeZGmArp38plTQ yDNfKpiaon4LBBdOiqR10BYw9iIq7xw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=21JMIPgwOOuKR2eeBq0L1sNsizlI1+DP/HMzZ2lCV+A=; b=QmlP8Yff9ZiFb4bAExFaWdYrmc vRivX9/H+y7nNEG5APYPNHaUDvu+SGtbMKSoK0b2uw91dYrUGaBioiCVmc/mWzSYI7ZSr53kqVURr DGgIm+DRXvSi2TjlMze82pPIRSOlnzOXpxG+wBgH+OuuC3o4G5OILC2oVgR1x7tgS+UWN63xt/nUJ MVzsygl/uIyLwMHCvMhzSCxyMPnIrkByIWYzCJKIOKZDjBjLwnD2QaEHbGsmS0IRAQfADduSWJTBm g5dr8pdz1Du0WFOU9lNHzGmTtklrq68ACahpSXvtAno4GdFdNeFAuv3Yhybk6TJSV8+a8ddHmvD3b iNaNMqDA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDL-3jz6; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 08/18] mm: use __page_cache_release() in folios_put() Date: Sat, 17 Feb 2024 02:25:34 +0000 Message-ID: <20240217022546.1496101-9-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8A03E140012 X-Rspam-User: X-Stat-Signature: 3rpy8m8zy6ksqfxucsy7sx7zeps9f1c7 X-Rspamd-Server: rspam01 X-HE-Tag: 1708136766-187192 X-HE-Meta: U2FsdGVkX18Tmh36Y3pk65pTM208wdgHPTyVTePpUNC59GW2Z15QkWmxkF/LzrXjfdTDI4B0uqNhULPl0q6vUl9OoCOjAlpyjqUKe8uuJT0zHUNy7G5uOdTUFnSkXoY8uQudjd9PoE7Gz8nJyH5Fgl+RnD5U3EiuyDLpXLaPEkxy4ftTo5aUBcSw2EUKPhxr+rXL+uOy/q6DCHFl5xyfcmDGP4rFQ1Rn838dcpoc1DZSfuuVqZozBZpnh4shXuMiGYyXKqzfabWMXRJ3jCRSt8xR6pdlQF1M97sN9tCknYI9Lq9SSoS5mB2C2WIZtV79hxuBQ4EC70HJ8g20qYniis88kZKTmSs2lPsydyZhWfXwUfGnHRkSyQ2DCAX3m9iLXx6TCjjpZWCdzpgwVAOGXLscPeKwof2hb4KHTV9qoCQH4VkZNImDBcAcs0QCLs9A6kvXX1XTBWYKyCwvZX/MP5xBTlaCv6OG/WpI5g9QLK5CqTSQlreseBOzY7h49e1/j+alQTvZB6Jj0Dw9RwOH/2RcDLaUJzsejGWXkpUioXiM4KkV/t/vbiY4FXFcR3wPNh4X4UDFDfPd+ALJjgBFzdaElFUqGhSm/6ly4Tyi0+mZlFtL3cQwv1HZWWD/go0qbUzjeXD0q/lyY7OMEol2YgH2HSd9uE1vvwyINE6vzeVorneCEH+vHXenOu0m02kGGDeTGBGNqfcLIiE9jbIGhMrqit+NKdHwXz+tEnRXc4frZUFVDYgYL3myMvtgH78MyD1a5j4Z2EvNR+tz0IdyLOpaRqjhzfzlT3sPXEnH0jgkOH0B5UO2hKQ3jfhFLwF2vCIFh9YvV3LlgLUsTm2RPYHvn+2wOONj5r8pTjJ7ukmBORqNuKKychH1vwdf6R4IkQvHB3+5Fjl5vyyGfHB9RjHdmVI9YjZHcalts/3nhA0KcwUbYZr+zLpKl0+5fBysEdiKaKzUl++7v3oNfLf IcfOX1Mj d44OgtdOehAzoLdPwdhNuTvf73cTmWPmEPGWywxZ3xKKNl5r+uPV7+EUNho3FkQNYvU5RzpZOQzTl6v9owin6zqrMh2BDuMLejLsxd7CY/kAsVvv268SakY5j2NRkNxPz0lX4eCJxAwz2uw/WFbbSKkbgUY2UDzkLy+XCl0gRIXcrDcX5CBlx+7XpIsP+Vw+OspYjsdJ0fBbbGacKyMWNvVm5p+aHpJHhU04aT2lCrPg89z7TfN1W+HvPLtLVuXgydxF569mvueSRyZzjfxMUa1ciPA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pass a pointer to the lruvec so we can take advantage of the folio_lruvec_relock_irqsave(). Adjust the calling convention of folio_lruvec_relock_irqsave() to suit and add a page_cache_release() wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 16 +++++----- mm/swap.c | 62 ++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 46d9abb20761..8a0e8972a3d3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1705,18 +1705,18 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, return folio_lruvec_lock_irq(folio); } -/* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, - struct lruvec *locked_lruvec, unsigned long *flags) +/* Don't lock again iff folio's lruvec locked */ +static inline void folio_lruvec_relock_irqsave(struct folio *folio, + struct lruvec **lruvecp, unsigned long *flags) { - if (locked_lruvec) { - if (folio_matches_lruvec(folio, locked_lruvec)) - return locked_lruvec; + if (*lruvecp) { + if (folio_matches_lruvec(folio, *lruvecp)) + return; - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + unlock_page_lruvec_irqrestore(*lruvecp, *flags); } - return folio_lruvec_lock_irqsave(folio, flags); + *lruvecp = folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/swap.c b/mm/swap.c index fffdb48cfbf9..21c2df0f7928 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -74,22 +74,21 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), }; -/* - * This path almost never happens for VM activity - pages are normally freed - * in batches. But it gets used by networking - and for compound pages. - */ -static void __page_cache_release(struct folio *folio) +static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, + unsigned long *flagsp) { if (folio_test_lru(folio)) { - struct lruvec *lruvec; - unsigned long flags; - - lruvec = folio_lruvec_lock_irqsave(folio, &flags); - lruvec_del_folio(lruvec, folio); + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in folios_put() */ + + /* + * In rare cases, when truncation or holepunching raced with + * munlock after VM_LOCKED was cleared, Mlocked may still be + * found set here. This does not indicate a problem, unless + * "unevictable_pgs_cleared" appears worryingly large. + */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -99,9 +98,23 @@ static void __page_cache_release(struct folio *folio) } } +/* + * This path almost never happens for VM activity - pages are normally freed + * in batches. But it gets used by networking - and for compound pages. + */ +static void page_cache_release(struct folio *folio) +{ + struct lruvec *lruvec = NULL; + unsigned long flags; + + __page_cache_release(folio, &lruvec, &flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); +} + static void __folio_put_small(struct folio *folio) { - __page_cache_release(folio); + page_cache_release(folio); mem_cgroup_uncharge(folio); free_unref_page(&folio->page, 0); } @@ -115,7 +128,7 @@ static void __folio_put_large(struct folio *folio) * be called for hugetlb (it has a separate hugetlb_cgroup.) */ if (!folio_test_hugetlb(folio)) - __page_cache_release(folio); + page_cache_release(folio); destroy_large_folio(folio); } @@ -216,7 +229,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) continue; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); folio_set_lru(folio); @@ -996,24 +1009,7 @@ void folios_put(struct folio_batch *folios) continue; } - if (folio_test_lru(folio)) { - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - lruvec_del_folio(lruvec, folio); - __folio_clear_lru_flags(folio); - } - - /* - * In rare cases, when truncation or holepunching raced with - * munlock after VM_LOCKED was cleared, Mlocked may still be - * found set here. This does not indicate a problem, unless - * "unevictable_pgs_cleared" appears worryingly large. - */ - if (unlikely(folio_test_mlocked(folio))) { - __folio_clear_mlocked(folio); - zone_stat_sub_folio(folio, NR_MLOCK); - count_vm_event(UNEVICTABLE_PGCLEARED); - } + __page_cache_release(folio, &lruvec, &flags); if (j != i) folios->folios[j] = folio; From patchwork Sat Feb 17 02:25:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78530C48BEF for ; Sat, 17 Feb 2024 02:26:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 009426B00A2; Fri, 16 Feb 2024 21:25:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E82BE6B00AA; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0BA26B00A9; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 990AB6B00A0 for ; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 63810120145 for ; Sat, 17 Feb 2024 02:25:55 +0000 (UTC) X-FDA: 81799705470.04.4B4BFF4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id C047120005 for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FejDrfEI; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+aNucfAnCh67XPHQO85RwlKe2XqWfNSUlxuOSTQj5iw=; b=5RmDgcOO+iNGevog94NVYp9hNBWBJC0rmOoaZZFW4x10oQt7zCsmDQez2Zw202Kjbr/6R2 koBIWFYYCgbgf3KPO5NF3J1KmZeQS4vvJyUHw8SeI5UXYRVELZ4Wanwu1aiT27txeupYYG J9WA30G7zqtT9/ZOs+kv1PwPF3tZ01A= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FejDrfEI; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136753; a=rsa-sha256; cv=none; b=lybm11dMFhqOT/KiciIn6VtIGDJpYkXl/aGf6aRIAW2V6BlKP/yB6vJrSGB2xeh2PApd0B LAfa7AdRgcHhS5WD5NT8lCm7nY9knFSdRzIclmRzgseEQx4aTuodmKyDv5dl3IV38teVsZ 8rqiVZ1p6AULYs74UOWjfLYfIByuUOc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+aNucfAnCh67XPHQO85RwlKe2XqWfNSUlxuOSTQj5iw=; b=FejDrfEIOWy7DYFg01udT8aGcx x24y4hdVgL8Kzm+LkOo+9ym9bOqm7jpDL9S1ptDMCFQ5Z78IlkodphVk3N6SI+MyoJjRDM9aglnMl m2AmpRiTdSRqEXYfsqWuIpGE1B62lNJSYtApcNalyA8YGwGIHsNfPUUafAuCYj7PnLthQ5C25IBF0 vSMbasW8KnxGYsyF6WzciNgoDzBoYxx/je0yTG0zUXAKWYUS1EUpi2YFc/Dd82BrlB1jBtf74yhqP slVNA3tWsOAB+CJs0/drv0VfBefj5sJTYQXgCbij810OKd++D9pi4eNiX0C386xAhBIOLMBSiX+tu csFh6tsw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDN-46fA; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 09/18] mm: Handle large folios in free_unref_folios() Date: Sat, 17 Feb 2024 02:25:35 +0000 Message-ID: <20240217022546.1496101-10-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C047120005 X-Rspam-User: X-Stat-Signature: bhg4acxsj9jbff337ihsncs54s9az1xo X-Rspamd-Server: rspam01 X-HE-Tag: 1708136753-828762 X-HE-Meta: U2FsdGVkX19fjbuFw0M90iAQqvesI8wTg4GCbIpipUWjUYok8qKVtVEswJsqB+eWXOoVbaubXp55J6ZsUJHmS7YywS5kbWNEnIjAcfCol2/6bqMeAYKq+++Ca8ISTiBZE7+8NnM1VERgrhgUBoT1ovQVcGoR2mmYqo2eewStjc+BGQ1ZyJLQvOmhylkMK481jb4yN8Xx+amioUA8+/CS1VMF3f6AlpkBNCwaRa5t6nnLdEI61GDBihBjA/a2/jBRjeNaeba4o2HHzB9w/ooxcGvFvowNdY4DOifwf1Pu0jn7z2kDN5LOOgBuS43/bd5gEUwEfEdknCHkzRg1XbvmModHq4NZnB4JEUIxp82QwWq9+U3dHVfOwRtbZlOgDgE5MmFf9OlJkZyYuK48ADXl56Dz8xeXSSYYSi3ezf9cLj1+7Zh1zEIXrP5lUJO+h3hmN1Pt+Cuu8Hqr/VjnBEppIftSmKbPC+vpcSqJKSBgazIuiU49Y2L1UKTBN/LNAs9xUG/Oq0jlgzORVpl2jlBu2Qiv5ZR8zvqG+megRkef7+uxY+nTTdWq+pHynvjRhBWISHh9S7vFjctjKLkCm0Aqqeo1fRhQZ+y0YnUKokIXRy/44Uby5vt0TCc3kTM003QLYCUNykXVs6ocTx0cGhCIAsbhjgi/lM9VfpUEoxJD9ZlcnFIOr1puKy4a+Rd2VhbMJoyKb///VIbk6KO/c4QtL2nMrpny/kM64yijY3R78aNKOy6ThIOioawsDof4rLeDfKGbxzXPfyOjnPAgyxEL7RK9qWee93wHmJhvfrI4tAO344aLoLsxn5WcVrwh5adOf86pHNtcSqxdIMFDrnLTMVM/vNIaKUIvPQPsS8bdkAXWSyb+eh+Y9Y9MHSXs3WDFqI2xFCWGG+52YOPur7Vba8VsgB5XKIXF5AWX9LNElKvt/qLJ7b6OiXLopSabXV21W8MbKIx2fy2865U06ry 97j+qM+X IAZ5PFnAfSKTPy00wCb8CM8vhqn4CgIjXnG3tRmrhPqXwPpAnLd4R8NtOnCkZCcvOv6r/gIkwzqWbTwz4ExNOKYkiDorTAuh1t+1NkMchV4lslywdcHazUhx7JCVD38I3Xya2Yw34wB7orMc3gHj8f2rL0np87XUm5keIp8LKZKRLfd14QaMDuvc3bz+Ap43ESh8Yu+0BkPL/zDsqVMc/1OyRvx7FY0O89kGw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Call folio_undo_large_rmappable() if needed. free_unref_page_prepare() destroys the ability to call folio_order(), so stash the order in folio->private for the benefit of the second loop. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8ef1c5c86472..eca5b153f732 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2552,7 +2552,7 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a batch of 0-order pages + * Free a batch of folios */ void free_unref_folios(struct folio_batch *folios) { @@ -2565,19 +2565,25 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) + unsigned int order = folio_order(folio); + + if (order > 0 && folio_test_large_rmappable(folio)) + folio_undo_large_rmappable(folio); + if (!free_unref_page_prepare(&folio->page, pfn, order)) continue; /* - * Free isolated folios directly to the allocator, see - * comment in free_unref_page. + * Free isolated folios and orders not handled on the PCP + * directly to the allocator, see comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); - if (unlikely(is_migrate_isolate(migratetype))) { + if (!pcp_allowed_order(order) || + is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, - 0, migratetype, FPI_NONE); + order, migratetype, FPI_NONE); continue; } + folio->private = (void *)(unsigned long)order; if (j != i) folios->folios[j] = folio; j++; @@ -2587,7 +2593,9 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned int order = (unsigned long)folio->private; + folio->private = NULL; migratetype = get_pcppage_migratetype(&folio->page); /* Different zone requires a different pcp lock */ @@ -2606,7 +2614,7 @@ void free_unref_folios(struct folio_batch *folios) if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, - folio_pfn(folio), 0, + folio_pfn(folio), order, migratetype, FPI_NONE); locked_zone = NULL; continue; @@ -2622,7 +2630,8 @@ void free_unref_folios(struct folio_batch *folios) migratetype = MIGRATE_MOVABLE; trace_mm_page_free_batched(&folio->page); - free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, + order); } if (pcp) { From patchwork Sat Feb 17 02:25:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC7D4C48BEF for ; Sat, 17 Feb 2024 02:26:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6195C6B00A9; Fri, 16 Feb 2024 21:26:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C8726B00AA; Fri, 16 Feb 2024 21:26:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41AFA6B00AC; Fri, 16 Feb 2024 21:26:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2F4FD6B00A9 for ; Fri, 16 Feb 2024 21:26:01 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0E17514012F for ; Sat, 17 Feb 2024 02:26:01 +0000 (UTC) X-FDA: 81799705722.17.1A0613C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 90D98C0015 for ; Sat, 17 Feb 2024 02:25:59 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EUdRxko7; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136759; a=rsa-sha256; cv=none; b=cOKRxffJq+Do8R5GxOOzDL73MCP6r4qEfNdovq/27Gy25P9ysr/u2cITNKl3SMIxo41jTl V2gMaBdZtuGScWXBsA0+k8AYEqycg3s8yYDrjXWP2PxcRYC18CaFDDyrhZ7aEpLMekX42R KMP8dfTd4kR4Ph7sAYXzYZxKpVDNBZ0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EUdRxko7; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PU6ugP91krgUdfd/+R+e4wrMYZesh+3xPHxHX0SsH9A=; b=n12Zf6Vb7qp8vXxY8nhD2yOuWmHFqAaDgCU5g7Ocnt1gQiBzy6hCV/+c0mBLwguuxkaZZv /kJeIwgh2KwRYWtXN4rz5QTUiHOOcaAnGjucubpicuAVI9T0i/mBVmB2W9zTHK+Dusk4Ki 9JVidpE1SkxHgnc28efmAQuLqejQA1U= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PU6ugP91krgUdfd/+R+e4wrMYZesh+3xPHxHX0SsH9A=; b=EUdRxko7M7qehAKii7MvZGshdP 7eMxyBsqY1HCAFZcmYVh1FWNeuTD+M9exLD5nM2RuFHWF1XMRz7bbijE8jIg+YZRY0cTBB5aa3dTw 39jJpyIzzcq89P7Syh4qEYbd/mjIk/1aaQNdi99c6NT4EhRI4JQTL+9RYOY0rCNW0XTrYYw4tX9ye 7QIV4dp2W9N60wFQ2Je+t4Ecgtt5rsMEULdL7FmZlW4B/qhQ/ouctBd+IvYskPj0K8AhTUlQ8B10B 0aI8MPJI8+285GAo0Uzy38dXj/hFMRGspY59SeRLPh6tavcg4OWdC3AmKzy9Ryo1rcn+FwKWuV7tK LqkiorPw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HDP-0ILD; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 10/18] mm: Allow non-hugetlb large folios to be batch processed Date: Sat, 17 Feb 2024 02:25:36 +0000 Message-ID: <20240217022546.1496101-11-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 90D98C0015 X-Stat-Signature: rycc6s3cpqk16ogbid98563xsrcxf4ii X-HE-Tag: 1708136759-315450 X-HE-Meta: U2FsdGVkX1+4nylLf+0JOX41MBS65fCNbMO0aBkOM/YmGN0HDtHK7JxPMzzzrY7KOtaTPxOj1qFoKgR7mj3wZ0ZeAeGizmpZ6q5o9ooBQXrWmPZ2q8Ab6uetdaxs6X6IxA4oHZAxbelX1sw5ucORHO69pAI+PITaPlClMC8lxoh+5RsOykxTqAnr/Kub6zqOuoStTzzTK66VXk8RdAYXFN2s20I77oIQNE64Js/P+tHIjvpyjKYDROKkKUmIaRa+Agu/0MldmwvnVN+zFshc3WcVbtiYnNQ8UPCDKkzV09/HUT/2/J8ixMsv+SD8eFZAqWnKmAl6Wwdv86MUlUj7s8ryG7+NQrsWC0zduiYn8Aa3v2ds0x3S16tAz/KBbHXv05Z4vNt6KYa4quZZjIw64xlRojEP25m9z6EGk6AW1aKRYwsfxobDZrJu+O5yZP3xg0ulItvaHIWmgDzUHI5YXPXSWa/MslVJIo43pv3vzuAB8lPi/lDTUeHsawWaz5i6sSIXHSioxYXg2ZBeGTV/x62RUnO2Rk8GBahBIN4gHVKoKA6Rczkf5KuRgTy4Rr/zJaZYBczCHs7F7+PuAu66/PdqaPswU3mHqTsqw1zhIwIbYUsFdXvkEij03Hhu7dsk5Xlv4wy23bSzg4Jwi8VvgYxs7nTVU2PJJUfno1pnU8SEyaTAMJiwvva4Jfc7f33Br/FXsIBcIwskcEHCCvSeH6/FKpcvfoVgXxB2+LjXIABTcJrbhn3deZh3L3t4icwJir1vuwZQjDjncg0Eqg3onSpnShi1+HYBsLRrdHTbmAlLXaPDucWOKE3tZqHbX664NsMk5YeCTAzP098ErEFiLk2RXIInYctqAVBqN7pZ0tlmsmSiyIKHOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hugetlb folios still get special treatment, but normal large folios can now be freed by free_unref_folios(). This should have a reasonable performance impact, TBD. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 21c2df0f7928..8bd15402cd8f 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1000,12 +1000,13 @@ void folios_put(struct folio_batch *folios) if (!folio_put_testzero(folio)) continue; - if (folio_test_large(folio)) { + /* hugetlb has its own memcg */ + if (folio_test_hugetlb(folio)) { if (lruvec) { unlock_page_lruvec_irqrestore(lruvec, flags); lruvec = NULL; } - __folio_put_large(folio); + free_huge_folio(folio); continue; } From patchwork Sat Feb 17 02:25:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19898C48BEF for ; Sat, 17 Feb 2024 02:26:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BBC96B0098; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E99C6B009B; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64DD66B00A1; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 239F16B009E for ; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F1FF916012A for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) X-FDA: 81799705386.11.F9E996C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 58CC720003 for ; Sat, 17 Feb 2024 02:25:52 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wUK393+U; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t5PILiViggT1JshWVG5xib00pMcFIIu0Z3GmM+yVzOI=; b=G1ZsU1ugK+Av4pHjLg5c5vopxbRgePqxPp14Ujks/g1gl7ctLGIN8aM2t+KLD2hFvr5zf/ O32lfqspgDm/QoXFY1IV35QEFNKoNxJg/YJ/ZAPPPyX+wcVSwEBWb+PRppzM1p2jtk5Bof 21H+s0YkWbeP0BXjCjfO1/jm0KbE7lM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wUK393+U; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136752; a=rsa-sha256; cv=none; b=NEFzlBl3B03wp9saXFHaKZ5Yc39RBD30ZLz4mkxqa4JndZUVvbDyiO0GsF0DFRWYAnDgGC bGwVysOdQQLOs7u9/PpNOVHYIrv7qsvEljyhHMyhlXScqdgRwU1X47MhuQhwoenCznV6qO /LR+AtvA3oY0Yqc1NJyDjmWh6hf5xyo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=t5PILiViggT1JshWVG5xib00pMcFIIu0Z3GmM+yVzOI=; b=wUK393+UoQ042NCda5+YFCHJp2 WIcR9OesVEaYPk0ax6Kxi7OeEuPDPbgXWaK5F1hNiemAQv05kYdQvJAelkES3IXXWhGN234V4CoC6 U7y3KB/sbt7wG/GqcUoLhXrqX7jyfYdsgxK1dUJ6mWLTelRnumX7kql/U4ZYlAbeSqz1WEwgxlkzD EzCcGgzfFJyolNQ69RFZ5fa5aPwcwG+xvkTsGBV84ZPTRUi4zuPlvVzAogRbymMpkWkmzIckktFAg 2OZtd9uRjwFgjKax7YOsGQ7KPTtqvmlVc/YUQGf5TneecGGpQqZ/2MH9feJ1g1JDkQ9qbdzTHwJPm YO4dYCzA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HDR-0ii1; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Mel Gorman Subject: [PATCH v2 11/18] mm: Free folios in a batch in shrink_folio_list() Date: Sat, 17 Feb 2024 02:25:37 +0000 Message-ID: <20240217022546.1496101-12-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 58CC720003 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 4aa9rx1mco6et5h793tymxraatsbufoj X-HE-Tag: 1708136752-179602 X-HE-Meta: U2FsdGVkX19tEsV/ImS9me+9eCLXpHUjnSnLbFwgpBGxsGXTTlMRLNaWPtxuGKxrC/wyOFh1oXDDEWWR39Sd4maEaNmaRTxmGS/9IT29APfrKIaf7zNTCnGqtGECtnMWtILjgWFqCUn93I2Ann6OLkroDjCzXh5Ql/C8O6n8NqEqGTVfq0TNJTu5kr6nQM08zEBv19PTUqVF317BAqz8iVPgQ84aSy3Ykl2t7ZL9rVd5i0D9ZvxJob1U2SOTWEtPXczdzD5oEnjKpB2sNDJ0qPukxCeXzpjPzek9bcuV0PnTtwzt16JD4SUAIHemjmOUcTh74zYfYZTpHHa/gA0oNvTACp56vq4+vgKBHXu5mPXQNs/f6CRhRH8WthCLVIP3FfzS4F9Svyr2Q5Q9gRx03c6NFpVdUPHNWSRADNOD5kvnOTUgCXGqxu9Mf+msiX//mM0b1Pg57K9RKCmoWqneOEcM0B+vsrImspj6RTrYK2c1IYHjs01AfGKpmUa68USJdtL3qM85h6eJVNaKc/wQNCfBAgp4XlegQtK+B1+RwVC+Ls/1mXUGAx9PGCW6SK2jtNHkLhSwq7jsi8n/IDuqTU9hO6XbA+yYw5FDwLA49zpR6zZLA4csDugFBlDKgyDPfdW/p8lQlK6mWic6hQyhHxLVioMOfGxqwQRYYRIJKvl7NuLw381VOYtg2AJC/xrvWSImciuwHSgSbryKXesT+IO6xNCdXaQCq7yV5vFbC9tOUsuGPCaNNKlu07THYw+cFbm2UfsC6wvbg2uqLhclRr0UHs/2AYTrStcjm8Sypcu4wJpm7NTasWyrucPnWolgYOkNRPNK8rRFKpM6lw0Yp7OjoLCgBWWvwA3VjA+HbqYBoNuNkDeoLYV5gRFtQDVQbGdDJTFJWU6sqA+8kXk6VkWmWEKZ8XxoYlQAFM1+0nmghapa98aom7ErfTxk5CMZwSGILgO/u/TNdGVKoQQ uuPpEhJi Cnj5sXPZWpKuEYSpo6dF6lqK5hflhjeH29H/t3IOptss3vbL3xgwTvADTNr7UfSFjf/J3IgYsiABT1Sm1KIOSGFaXLYkR7Og95M5ObXbo0fIqxaRJvfGGHWYSXD9gaql0QqiqZUleMeCoWqBgePu+RtAKYLktiJzkI0dRxWhKsAsSlAupqE7q2gzRhcPGjBGITOKExPFZiLyhGcF1OhxIG6V8HGnrfOOG1Pep X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use free_unref_page_batch() to free the folios. This may increase the number of IPIs from calling try_to_unmap_flush() more often, but that's going to be very workload-dependent. It may even reduce the number of IPIs as we now batch-free large folios instead of freeing them one at a time. Signed-off-by: Matthew Wilcox (Oracle) Cc: Mel Gorman --- mm/vmscan.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 87df3a48bdd7..e2292855f58e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1023,14 +1023,15 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct pglist_data *pgdat, struct scan_control *sc, struct reclaim_stat *stat, bool ignore_references) { + struct folio_batch free_folios; LIST_HEAD(ret_folios); - LIST_HEAD(free_folios); LIST_HEAD(demote_folios); unsigned int nr_reclaimed = 0; unsigned int pgactivate = 0; bool do_demote_pass; struct swap_iocb *plug = NULL; + folio_batch_init(&free_folios); memset(stat, 0, sizeof(*stat)); cond_resched(); do_demote_pass = can_demote(pgdat->node_id, sc); @@ -1429,14 +1430,11 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, */ nr_reclaimed += nr_pages; - /* - * Is there need to periodically free_folio_list? It would - * appear not as the counts should be low - */ - if (unlikely(folio_test_large(folio))) - destroy_large_folio(folio); - else - list_add(&folio->lru, &free_folios); + if (folio_batch_add(&free_folios, folio) == 0) { + mem_cgroup_uncharge_folios(&free_folios); + try_to_unmap_flush(); + free_unref_folios(&free_folios); + } continue; activate_locked_split: @@ -1500,9 +1498,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; - mem_cgroup_uncharge_list(&free_folios); + mem_cgroup_uncharge_folios(&free_folios); try_to_unmap_flush(); - free_unref_page_list(&free_folios); + free_unref_folios(&free_folios); list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate); From patchwork Sat Feb 17 02:25:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB7D4C48260 for ; Sat, 17 Feb 2024 02:26:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 275B06B00A5; Fri, 16 Feb 2024 21:25:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D21F6B00A7; Fri, 16 Feb 2024 21:25:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEF686B00A6; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ACBBB6B00A2 for ; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 776C9120148 for ; Sat, 17 Feb 2024 02:25:55 +0000 (UTC) X-FDA: 81799705470.08.632DEF0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id C441E4000A for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Gge7ZHIB; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136753; a=rsa-sha256; cv=none; b=TNHdiYyzVA7hkZOW5OA16fUn8jtw2C4eRdVlIgZkBCFt9A/vvrwnhFOO0Rjn6zVseU12Cx tqb4caZngAos5Rv9cYZ3zAIVCGcOIeV+40fPBN00fGuHaep1SOEHh2RhxOHmqNDgzlXozE inyEUEw9OSohfisdCVkTy7rSZaRLSMI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Gge7ZHIB; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xLNBtKmJzlmAoHK/ONEef9lzcMsCVs4HBTSM/I8zSNQ=; b=eMr70a4F1YRNXv8vCV2KuFeS1HhUcZgEwApTC+5V6zqbvJo/5Iwbdg6tqJ67i6o8G8oflm JMqNM5sX1vUzxu5zqVrRSXH2rDZjFDqZm6nTykTfvqEOLOpJMOU0I/ijzMgbSMyO1UBO9C 89OrgpOKY2gw/oankcqWrz+Ua5l5aLU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xLNBtKmJzlmAoHK/ONEef9lzcMsCVs4HBTSM/I8zSNQ=; b=Gge7ZHIBCsdoopoK8itgybcL9X G43A/PoZIaMIDTcFkMOk0/C5fynUIdwcvg6wfhdzCZve2P4Q2bJDxnQf9eHxaYJWDyhQeZxoOHV+0 E7w0rQgVcFTux1Lejv8497vGNvmDbna2F2eTRAG3SXGU7pxfb3cboBsDWOe5GTVv/INiJeVpZK2OX eTFP4JGp4Naq3aI0PgRszR9CK7c02QjTfLLnSmGZf9wD+J7Vwo1IV+s4r8KEjNYxL2M4SPNoP4MqZ wu3UMS33Ku8odhv47qpB4QjTM8ycRMRBj7GijW134oKmKZGlkjVVOcNW854ix9XP6sNvG/wCrvrkd /eYx+fRA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HDa-1Hmp; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 12/18] mm: Free folios directly in move_folios_to_lru() Date: Sat, 17 Feb 2024 02:25:38 +0000 Message-ID: <20240217022546.1496101-13-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C441E4000A X-Stat-Signature: on3gxz9du4oh7bzknsjw8habi34acmfs X-Rspam-User: X-HE-Tag: 1708136753-378124 X-HE-Meta: U2FsdGVkX1+cCQjVH6w3ouK0y91/qVOb2sH5lyKPlUWsgfCMCMWdUGOYm34coB0s0A6TI6/2UyPVUTGJEdtGCShd5b0Whm0ZdAcK9wzKiuusIj/PyEjo0JWdg7fI47d9DfKGFvWVffhx7vXrRV38wImzWHoOciZrnNzNcdGGd7jj4Gnz5/OpXm69PsSIPy1nrAQspNcxptleXWzoo/FnIThvG0WLdsiQOk1znz7tfaEUZWutvktuVHBefJ1H2dt7i0m0b3dcPkkS8rIXytYiESdOgO9ruYRLXE4uOp/2Ha/eCXLB5qLREy4shkfezw8rsRiggAMBIdGa2yISy+X6Qb2JnGnRlTtkp92ptG0GRmFkaoffW5hoFGNbV02idnHYVGK46dJDlPJ2qDbH02ztT7qLwwgFkgEjAkVKf1dpFzobDHn97PbrKdtbc8sRtBxei53EoqOUSvHbBeYLz5WKR5RyViJLJQt2H+rxk+CAZx/C99uRj0WsfKKmNQP7BC3DWS3lul/xfyp7FQiU8DNRfZ9lLNB+bVcBl9AjEZLkQGxGT+trJbCMaBBTanCXBTBYa2JFK6/1/YwGzPn4L72KSwx8AXxEAPVBp5GGsRkGffC7JfQyx9mHQxg4cu+onhmr2TIT1BHkCteocMNSmMWmKWQOAjSe0S4s2ZXxZ3VXdjlpFUP8BtK5JBBhC0bn3up9ZGmKZTMhCKYr+t4GfJsTzdZbBkGCtBIFDUZqFCbVVSB2k4a4s6P3weVLOsW8zJKbnWOBJrrqDwDMdYQB5zi1xiKGRdpuPfypI2YJYogMgtWW678luQ/oxfTpuYZd50Z/ftI1mNzAG6GvZyH7BucZ3zLe6OGXaLHztn0twcaYLwEtTJV7outcxf0fzkZeX2rP6rpOxTwKe/fWZorSv3PlcmYnsZJrDzVHMixb1UwfM9DPhFMpiO2xg5P5gdLaDILmXLB42YRZQddXaWCEASe 21YLIMn+ /9pXvHRaE8rkhbV3FU/7G/Rj4o5XioP6PvN2/Xa/ndEs6TeFUDXfhngLNfDVn9B+F2Z8vzQW5QIRvMzcpUgfax8p+TUQaN0Q19jeKYBCVVkRxls5ytnhvAl776jKFz//C4FW4IybydZaNUEBa/C5qYdCBcNQICva1knT2bTWNH9Rj8YLwwOnGPpaYeP44hGaRB476WKCwGuoLiWUzPr9LeTWqotgAN3fjP/td X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The few folios which can't be moved to the LRU list (because their refcount dropped to zero) used to be returned to the caller to dispose of. Make this simpler to call by freeing the folios directly through free_unref_folios(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e2292855f58e..84f838330c3b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1798,7 +1798,6 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file, /* * move_folios_to_lru() moves folios from private @list to appropriate LRU list. - * On return, @list is reused as a list of folios to be freed by the caller. * * Returns the number of pages moved to the given lruvec. */ @@ -1806,8 +1805,9 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, struct list_head *list) { int nr_pages, nr_moved = 0; - LIST_HEAD(folios_to_free); + struct folio_batch free_folios; + folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); @@ -1836,12 +1836,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, if (unlikely(folio_put_testzero(folio))) { __folio_clear_lru_flags(folio); - if (unlikely(folio_test_large(folio))) { + if (folio_batch_add(&free_folios, folio) == 0) { spin_unlock_irq(&lruvec->lru_lock); - destroy_large_folio(folio); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); spin_lock_irq(&lruvec->lru_lock); - } else - list_add(&folio->lru, &folios_to_free); + } continue; } @@ -1858,10 +1858,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&folios_to_free, list); + if (free_folios.nr) { + spin_unlock_irq(&lruvec->lru_lock); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; } @@ -1940,8 +1942,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - mem_cgroup_uncharge_list(&folio_list); - free_unref_page_list(&folio_list); /* * If dirty folios are scanned that are not queued for IO, it @@ -2082,8 +2082,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate = move_folios_to_lru(lruvec, &l_active); nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); - /* Keep all free folios in l_active list */ - list_splice(&l_inactive, &l_active); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); @@ -2093,8 +2091,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); - mem_cgroup_uncharge_list(&l_active); - free_unref_page_list(&l_active); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -4594,10 +4590,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap spin_unlock_irq(&lruvec->lru_lock); - mem_cgroup_uncharge_list(&list); - free_unref_page_list(&list); - - INIT_LIST_HEAD(&list); list_splice_init(&clean, &list); if (!list_empty(&list)) { From patchwork Sat Feb 17 02:25:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B44FC48260 for ; Sat, 17 Feb 2024 02:25:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 164B76B0093; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E6E56B0098; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA4676B009C; Fri, 16 Feb 2024 21:25:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D26AA6B0096 for ; Fri, 16 Feb 2024 21:25:53 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 97670140122 for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) X-FDA: 81799705386.11.07CE7E4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 02FC31C0006 for ; Sat, 17 Feb 2024 02:25:50 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=CLqvUfSV; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3y483NAn04LDI+CHl8S2xioP/KCmHcMseUB+d705X/I=; b=7iKdPfKYl9ArSdiG77AzjmTZ80tYFRXtMvHhEm0oDst45Lmdfi7QL53UJnJ3dNqpo/PyQY YgfvzaL9LgDnS7EtfzZts6o15aqTuOePOlE3XJrmlhcqIkg2xciEPoO2lpYq4+NUuEBmp5 32IQq07W3sl80MGRfiaGQrDrQ/1TRa8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=CLqvUfSV; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136752; a=rsa-sha256; cv=none; b=oLNMroW6T1GGvVc/qJhR4jpuYNYQncaehDJVTVEv9CwLqmHj8tJctFYBe/w7qFpyoiabou 87uZKJTFle447A1B04Tj2zzyJLauBmIJrmxJtQb24PxSW9ItjxPi1Gaq5AkghM/G9ON1WR 1x0tXtD1NgUWpJnXEVCEkVuvOUMX6SE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3y483NAn04LDI+CHl8S2xioP/KCmHcMseUB+d705X/I=; b=CLqvUfSVkzJ4Ut0h/YIba8YDeg 1XiVh/YyeslGmLtHC5uaUAYDwWT8C+3F1LdbFx2+tl1fL9oBL2g/JLL8rccWsiJGAzUzvjdot5o9h 7XjdKGWvNLRGtyk+EiRkyg9KzyW/6L76fAhmQ5Z+qkQ5cY9iVdZIvvWIm6/IAbwgeIrwJiXPnLJpw u6Nivo1xnB439PwCSi9A1uJYpCMoHvRJ9SxiB4fyXnAqzIqxOOLaEyLGIpktPSrApXuqw5GvLFJfN KS9lxz8vd7qSrkRUx0UFryWdSuiCVh7OThGE/lgUftpNokH2wBYUdnfCbNB7uAKRjaXFCXt+yrIio hhiqHK9w==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HDq-26Gf; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 13/18] memcg: Remove mem_cgroup_uncharge_list() Date: Sat, 17 Feb 2024 02:25:39 +0000 Message-ID: <20240217022546.1496101-14-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 02FC31C0006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: jzyh9iqsf718a3pnz1psbussm5apmfrc X-HE-Tag: 1708136750-6983 X-HE-Meta: U2FsdGVkX19aeaKSHWKiMGRPMonL6Ut+zGbq8qbRBBBf1hL5nU5wDx+JnTJyVJIQQQeEYcduJyP43PfRzdvYCWOKV/zCRbtlVKVhywjGbhKrfIkGVqQ5FRx6nTOGEAH45NvOI6gexfKFJ0MTaYIBdtkYjioaDJKBt88ymaYQi2Yyx8pQi8Jb8PxLBlDku3aNsrMpa59VJUvX3aOanUyNl//klKBHYEOlsHRel4Eu3tF1iSkkoaXAEK8vgDQTf9CZPcPybSV14RKMFziL4OcDje2FfJbZm/k4LfMF0VQ6ItDq2+/0YPfE9w0+p5mBVejSXFEzr+n4hSnDRwj8wDyagRzxyR4oMoudN2NiEdy5uX4LcOGQO+/Knyd0hxdKaN7uQpAayWNjpcimuSIYLHrEatYC5MFp5B8U8g6N+iEAxmqR/e62Fo/5nNcZ1LvTPBpzbr37nquh2rUi7ESYQ+hhdSOnLA5wNIxIZKJ5VGkLP8gOQ6LgsDwrHNuQ4oIYHVI7Jc4GuEHQBNGeuF6v+paEMCQyW6frn2FHZIGMYavThlpFh5PDAkay9ceni80KM9IYxo31TtHdsVqNqI1cgEoQWJ0NVClGV05Ibs697vGxQvmvVwcEx7nDRMzUwEEmBZHYawwgeridoLDTCcl8z3u4us46rm0PknS4tYEx7Wy3G4hirbv7PiIN4Wec8Unl7EXzkYV+3NsZWGgo1A6BptRBfKfEyV0tnPMzGavbm65fRpp1zwYe1Hp43Uzn8jAsk83sbe2TtL2iiX7OQk4Ro9TLY9fNVZ4yEs5cE8/UMslZYA9cBmvvhi5eumqFLBwe6Tle8YiqRlTMhrzXnAa8yPQtEwHsVdqAalXbsEsxdpmsbKY6sZYLWhIJ2gQT48OI/Cv9xnWFcYoied1Qb1veXLK1UWebp77pCva9Z/wtgGv/M4Muz2CzGlIgAElhRqPZFbPl8Syqb3IzDuwUmnsAG7K 19bFEtSI 3bqXXhJXZGYAxfd3Q/n2iWzeqhXGh0ICImMMUOV9BnKFx2lvaNgHVHeISvZBOr40BrMhRF1HvLB/PC8h2tNnUiKL324nyOL+Zbm0v4YGJRksG1lG9jrcZahWBs6CBnGSsFK+45GtIoWfkmtPKktlWnuz7r4/oz0vRa9/I/UHTh0STKSQhdyRKZ0kxm0U6Bm9kHUwKSiNMsDRgiDQrNwEmraD27lhxzjyezcFQfX6onI4tzn0I8gdDiuQCUuj0CyoC6F9kH1YbxZXdrYMVLXrgahCXIrdOwdr4dc+c X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users have been converted to mem_cgroup_uncharge_folios() so we can remove this API. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/memcontrol.h | 12 ------------ mm/memcontrol.c | 19 ------------------- 2 files changed, 31 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8a0e8972a3d3..6ed0c54a3773 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -713,14 +713,6 @@ static inline void mem_cgroup_uncharge(struct folio *folio) __mem_cgroup_uncharge(folio); } -void __mem_cgroup_uncharge_list(struct list_head *page_list); -static inline void mem_cgroup_uncharge_list(struct list_head *page_list) -{ - if (mem_cgroup_disabled()) - return; - __mem_cgroup_uncharge_list(page_list); -} - void __mem_cgroup_uncharge_folios(struct folio_batch *folios); static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) { @@ -1301,10 +1293,6 @@ static inline void mem_cgroup_uncharge(struct folio *folio) { } -static inline void mem_cgroup_uncharge_list(struct list_head *page_list) -{ -} - static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0499d7838224..d45b9f322a92 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7546,25 +7546,6 @@ void __mem_cgroup_uncharge(struct folio *folio) uncharge_batch(&ug); } -/** - * __mem_cgroup_uncharge_list - uncharge a list of page - * @page_list: list of pages to uncharge - * - * Uncharge a list of pages previously charged with - * __mem_cgroup_charge(). - */ -void __mem_cgroup_uncharge_list(struct list_head *page_list) -{ - struct uncharge_gather ug; - struct folio *folio; - - uncharge_gather_clear(&ug); - list_for_each_entry(folio, page_list, lru) - uncharge_folio(folio, &ug); - if (ug.memcg) - uncharge_batch(&ug); -} - void __mem_cgroup_uncharge_folios(struct folio_batch *folios) { struct uncharge_gather ug; From patchwork Sat Feb 17 02:25:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C519C48260 for ; Sat, 17 Feb 2024 02:26:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C30346B009D; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B27B6B0098; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4480E6B00A2; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 173236B009D for ; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CB4DA40133 for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) X-FDA: 81799705386.10.7694202 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 0ED7E180009 for ; Sat, 17 Feb 2024 02:25:50 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=pAFEJ5ZX; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N0E9uz7bGt4jDkC/SlTskrxduZu41qZA+MkWZ6jP464=; b=GsC4kMcNLBAFQ1Vhc7w/vNJSw3lDHNBD2rY79VNRclwA4glPNoDGyGmmVn7zyKC4U3qLbI 3pRIvfsYmNkI0mjq5/Z1mSCWhlRbMNxsLOsk6wM+A/yOBByWef11iL3u3V0oLqtmb+Si8B TOb30XtcWX1PryUMAqdYe1S0h1MR3k0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=pAFEJ5ZX; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136752; a=rsa-sha256; cv=none; b=55zvHFAZcuTrw30Nz8IeKzyrOx5WH4FUSGZyKk3gyygXcdZ15oIZr2vYhQ6L7OlnEcnYuA iQZ0rD/l/MJ7uxoohmav0EfjWFuhscVK98Eaygbe7J1hPF5kVnAs3Do/ab/KTSS5C1NP8y 3KHKamaCiP5TnOIeLvfJL7we6iTgDUs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=N0E9uz7bGt4jDkC/SlTskrxduZu41qZA+MkWZ6jP464=; b=pAFEJ5ZX3JLkNV5iscOBpMUqh2 9+jssXkC/noCrYPp8FQdAngEaHGcdPOSZRluE8GlPxlprCpKil2iQvm0nGLqfzjF2HApqo2LBfDPo GmhR9sKVKkKztzDLGzHsr+D2GiOwCw0xdfiwotuxGAmAvqgWklsqN8OdSVu65DMMMkDPiLnjCUKeB zBOm2oIL0bK5ZwvEQyMX5YpjzOpUVtjjS1Pgqfo5iJO0LW3gMQd+r1ULJHU8arMW4aUFJJCoh7DfP d87ZQU5N6/4bzWgri1p3Qm7Inb12N8MZ1+0K2wCR8AW5hpYwGKuCOjzVyeCkA4rJJFacnEWpWlymX CZBglKXQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HDz-2ZDw; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 14/18] mm: Remove free_unref_page_list() Date: Sat, 17 Feb 2024 02:25:40 +0000 Message-ID: <20240217022546.1496101-15-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0ED7E180009 X-Rspam-User: X-Stat-Signature: 8a4cdcx1k1d9ir5zxno17ij5qhgu7sng X-Rspamd-Server: rspam01 X-HE-Tag: 1708136750-656048 X-HE-Meta: U2FsdGVkX1+X/MdWPnMPTMcbuTRYiNJixWyGxraSrFHO6jw60MmVtbw516NXRoy3hpicstq2H9s3oINpsaz10yUhcSttTRHEtEx4K3b8gWv+7eLfS4qPUv2lXBN69NADS7JMB3BeO69eo7tvIny84T+b5proxVI3vcPgrq2lk7UVzpupE6pbXTXYRsmJr4P9xzyiJQiT8rg3bo5n8rprf3EcKVy6E0KTl41PONhxsM6egxZgZeoc5Wgk0Mp+TGX4wJ8ZAXRTV6H871nTfSWIX7tZIn1941Eo/tCkZMfpjY8aAgh5yPvXyfR60VIF5D5FFvhFI4fc3rFCty6pQy6bkx6AFRuym51SYsxKSxv3fCj2U4OYZOVROYdOQ/jdPtTSclY8HfTqHsYl9aafG/CQ7qpI908ikERkxqBsjfZtS3cvUuKpQOm9/uVKj36QhWxXjBs86KoujyycG1fIT1FEgELs5V/LnVsM+CDxr3pYfkpbZ+jhl9hZ8vmJI48cAteEjoIR7zj+LAJBlh4apVfVw7GqgGXhmkeL3FJzz0O5hhHQ4Kn9HntI8DLyDEnSqrYM2iwEfsqRlBq4mNZx6L4AgPOz/CVLRK0HKhjV7U4XQYuyJAUEb+btsaIOw/AKSw7NLRtrCP+OdIYCwqZZM+H5LB5n4YAzudXu65PMvIkn7GrxpAnZrZq4cMsUa6Jg64tH7bTuG7mfwWuwbRyNb+W3DGGT9HVsrlGU7v5dD6jaFdPPpidJu2xhuUt9CP8Soi/psz3ZAbT4H5PTU5SXLgXFnY98g9OJ13Ks1jOr4gWNXVV0Tr8zKWe02Rmn0N0OFNFXSyoHfZKoX2IaQqXQ24tq+3Rj3p0/zwRAePHV10O6lYIzvpkXYPAwDxCGe9GGq3M0ct4B3jixU1QjalElMJpzaIXCbhnQiSckZlqnJagN12PO7xa+ZjX/1nxL3+txAaSx3z0ZqhDWQb+MI+H5+QC RsEM1IxY GOeApI+zUzTSx3piz4VTKtQMmnoKpXaaaUdI8g7/orIyyZsQ2OUVKYIfb36uiaMO2U5lqWzdVYMpteHC5f8OEC5fFoKofZr77aQk7SsEEgBZjkMWge+OpxenCyQ8F3t2SfmpkVhsLIanMHHnUNSVneyqx52xgKaQX8fNiJ9GbmjxPXhsiXJ/vxYjD3yi4l1VsrZLbMXeRqqrE1XQBd3yl2KzmS+0wnbIYwaHPQoEZynThFYBuNIImqwhDpnZj01dcHJ+IhVUT7AJz8QfJK7xpMA9XYMXoM/MSXupv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers now use free_unref_folios() so we can delete this function. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/internal.h | 1 - mm/page_alloc.c | 18 ------------------ 2 files changed, 19 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 4d45b351e0fd..3e2b478c610f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,7 +451,6 @@ extern int user_min_free_kbytes; void free_unref_page(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); -void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index eca5b153f732..7600344b997e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2641,24 +2641,6 @@ void free_unref_folios(struct folio_batch *folios) folios->nr = 0; } -void free_unref_page_list(struct list_head *list) -{ - struct folio_batch fbatch; - - folio_batch_init(&fbatch); - while (!list_empty(list)) { - struct folio *folio = list_first_entry(list, struct folio, lru); - - list_del(&folio->lru); - if (folio_batch_add(&fbatch, folio) > 0) - continue; - free_unref_folios(&fbatch); - } - - if (fbatch.nr) - free_unref_folios(&fbatch); -} - /* * split_page takes a non-compound higher-order page, and splits it into * n (1< X-Patchwork-Id: 13561210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1093C48260 for ; Sat, 17 Feb 2024 02:26:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8BEB6B00A1; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B95E6B00A6; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 022606B00A1; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D6EC16B009E for ; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A88A24014F for ; Sat, 17 Feb 2024 02:25:54 +0000 (UTC) X-FDA: 81799705428.09.50F01D0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 2A160160004 for ; Sat, 17 Feb 2024 02:25:52 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gLoUYUrY; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S8nD4vbdK7S1c96UnUq9hUR/PZMJuUTm+SmBC6+SzBM=; b=rdb7R6LZB4QDe4m0e9aATLrjTikw1zygyp11MgHyEzrvZSzW9ejU7kcbhGccrmXpBl9T8C vt7/Yg9c6+ewbdCzk0KO5INwnAHuSVB6t6kcoAhEibulYbr9VepUMroA4wqR1PAYsfNwfd Pti6fO30IG4ZuDB9ABMcj+f+I4wcnGk= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gLoUYUrY; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136753; a=rsa-sha256; cv=none; b=1DhV38URsozYb5FdK2I0vF79DBegN7hFIyY3ditPHtvc73BoHP8Xko2ZvQd90v1Ab5NZIK aBqGwOZxknML/y4DlNKVMz2dO9V6BDPTzAZ/GmXpigBOCG4oOSu/Xr8OMZujDWO+KQo1j4 1sXxd1ifa0YChfjDe65jqSCJCpHztEc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=S8nD4vbdK7S1c96UnUq9hUR/PZMJuUTm+SmBC6+SzBM=; b=gLoUYUrYCCSZg497kNzr66g9hG w+eLsONxqtwglu5Fk2tnY22nRiT7zELD3HSpxJ9i3NPk50K7omVEE1SnevBqscUzHAZsHkv9t0W+9 iRRp6lnH7xwZq3ugXxRy8MSkYahWUZg9FRkDVfuCIYKdKzDltJbBFWikH1LCesN2SIwBxQ5Z7t7oe H7wf5rQa2vMxRKuS9SVUJEbWLqtHKhdxj3l14A3tkGqs23/V6orWUR53to2iqYqizpijyaXIat9tO 3gEGxBVT4xmVvgndEmO0ellCPlRZ98sh363pDUzT9cYDBG5Uid2jtUNVZTN5JP4bsejB7awD0VWWq 2Rj8sttw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HE6-30Iz; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 15/18] mm: Remove lru_to_page() Date: Sat, 17 Feb 2024 02:25:41 +0000 Message-ID: <20240217022546.1496101-16-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hkmec5dhtg8m7ta81kswz9r8ifot9b9s X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2A160160004 X-HE-Tag: 1708136752-406356 X-HE-Meta: U2FsdGVkX1/FjADaDvi4C/LXT1hlv7JNRFVBE/W8tWFJeeeekMmfOQB4fnMf3jRRwP6MADFHsFN2eO86YS5Wdh85LLXZVwNrw8h4PQOpiz1s31a4VIpJLSfKW4Mqz6byKB/wIC5Ou3U+GIoLOkV0dI5Ezmr5JqsiqsBIHtLdpU87ubj06NbalbD7NQHL+/1BLhwzYZ/jfDgZ1Rk9MELUuvY4xAX3EIL6lEFFZ6+4tZ20V+IZWE/F7rYepkaGThSu3VdpNUFVYXLh35GFltmItxFViJW+1H9jbEYU6m1zoWJ7cGE2WcQ8RA0zWrJt+lbAEH3QdtE2rI1R7ifbgK9NC6oKN/mz9Ja8MzqxaWeKKDN3RElOjaz0/fLhGoHwnFmEgYQChxdZQg4IINSkrsJaI9aIDcMZPsQKdHnZPEtsoWgrdJdFxx/hq2U5qwo0keA8LJa1TaRePvNAAZYYKHeXdTkMeew032CpBw+fCObsGF1wU2p11AV5/vN31U9TNDE/qyIvY50By8SNt941rkCcjdXlxrHD9U/xBFPgBmdTmTJzIZNx9nJUg+5LSlA3I6plueK+68OSYdcOGYEPqwBGMRmA+5WNzDcHMorQ3jRs+PhOGeJjHtig17T3xY0BOtfMGmW+JN2M3nwWEHSbiu/MVt0gWlGZ++q5wq4QF9n/Raw7Lyb7N6/G4DcDH7sdVkQ1e/YJSOsyw+aq0FHhoFh8elHZ40RGNUUfAxnxt86VZawqEhz8yH6YTk1ljw+vcNO2KIzYQyqoGaehThYxVq1tDrY5IL880r82kSOtYiD3xi0QkAEkSVYiXXWNtgecD+tmB0AouBx4+Ooxy0p+qgxV+5eaOYrVyGiVXohtvJmkH4vgpO51D/ECqqxhRk8PB7qVtJI/HWsXmWj0W2Zh6GI9HkWTLYfNhiPh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The last user was removed over a year ago; remove the definition. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2a1ebda5fb79..bb0ea42c2990 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -227,7 +227,6 @@ int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *, /* test whether an address (unsigned long or pointer) is aligned to PAGE_SIZE */ #define PAGE_ALIGNED(addr) IS_ALIGNED((unsigned long)(addr), PAGE_SIZE) -#define lru_to_page(head) (list_entry((head)->prev, struct page, lru)) static inline struct folio *lru_to_folio(struct list_head *head) { return list_entry((head)->prev, struct folio, lru); From patchwork Sat Feb 17 02:25:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1911FC48260 for ; Sat, 17 Feb 2024 02:26:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F8C76B009E; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D6A8C6B009C; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D4DC6B00A1; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 38B2B6B009F for ; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 13BFD140122 for ; Sat, 17 Feb 2024 02:25:54 +0000 (UTC) X-FDA: 81799705428.08.8E506FF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 5F43A80002 for ; Sat, 17 Feb 2024 02:25:51 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LWH6+duI; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+dEHqQtt2ZjC/Vyorv8zVIQOVxePOZ190NejO48fo3g=; b=SUPMe0nz1Du9tnIMToLLCQligzTh8J8IQ/snUByDM1HZdPFeXfcebcoK9X6Mdrz4hcwIdf CtyIBpsLYNcBC/Acajsu5WsD+3iBdaQAu+QBFFxQ3vCrwzCLYkU8wUWFLjiIlofM/biLef rbka/bEsrgo8OuKWXm20Ct6a07kPlJI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LWH6+duI; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136751; a=rsa-sha256; cv=none; b=fYEQvGzU6kwBSFXSfeBaekpXAdnZUBIikM+fhmRUlBcRQXxNk3CKILc6tPvv3BuQQItrHU fT65G0088UEWYb2Ld41z04UWpQmuXS+nIKp1+66ZWsACm8lcRu3mwDl/0D7FYcbbReFP5L QQ10Yl5aIxwOSIbL0M3WUBVqhfDVN2A= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+dEHqQtt2ZjC/Vyorv8zVIQOVxePOZ190NejO48fo3g=; b=LWH6+duIw2iuCstEZBr1Dm2J6E eto74roIyDFIruVYGMtJPUypuQ3br7LpQR31yt5oB943I21GlOausaMPz1uCrFyYkmCzeLfrdw3CL frOZ6r2Lsfy2CigWJtybTiwUR1tFpilxhEVKFxU9nDdm8NFFkjlqZKAS+fodmRufywq2anfBZvlMZ yhxZykZLoQhbbUvmEQz2lTevaaMLEG3zGwMQiVkr7515xWbIog8SgJEZnbcpXaaZZMHhhshzcuxjD S0tyVy21RZBue2M4lk6XgnNcYccJwgLEvmeA0XACSU+aHhGOvazRl5iTXDCL42W7aF/boHRLqpIyt GG3q2O2g==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HED-3bcA; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 16/18] mm: Convert free_pages_and_swap_cache() to use folios_put() Date: Sat, 17 Feb 2024 02:25:42 +0000 Message-ID: <20240217022546.1496101-17-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5F43A80002 X-Stat-Signature: 3s7oyz8p8p4tar7g8yn44ihb1nsfcfy3 X-Rspam-User: X-HE-Tag: 1708136751-39274 X-HE-Meta: U2FsdGVkX1+Tb5y18wvxSTut1oJVOGASzoXkxYcRxbevhqZacjtsOj98qStlHO7I6khf6GJuWmkCB7/+vDBWNBMUn6w3rCLf5c4shVAyACNomvrIdp3vAFHr/eXuhggpK1OfSgPzN1gWZqKmFmzmgdWcu/DjvZhd6YPQkdfC6T8gDOPpoLkyPfLBNfhhsYVkUAq5T1gHGBjMK7b8RtCukQusj/wWgds0Kp5rRtPudXkWx1AATaqMWojS5CTl92zV3Io9dWRpg2zJe74wusq8ufxcTUUatkilELFbeWKVRJv1pFFx5kQVRG573nzi/8wbr7EOV4bF7Wcr8yKCTP6kW+rrEPzKttbKxRxnBY8AeCVmwVR3+gmfnvdMMOnnxHYCJcY6fON7IVkxshDnWeKyBvDfn3EB+eIYj9QWxNPuLS2/rZGo0XkYnoB1hX7F13DkXKUPu9p1Ic5I0kyNql9tboKCH2bmIidvMBj5zFUXhuKuqfi1X5ohUJrtX4bhvNfasm16G2dN1BjkIqycPF1Aw7lsNpt6vQ6JczYd/wVX86OkqrMubFGDoQipmMNRpSHlPO6qeu7CCAu3jxMpjaoFkjRO/qQxkB86kZEo14BZgyV+bp4GcaAOUnbvovsihDtKS1e/tk0IBUnt/oNXTErV5v9IoEUbPPpR4Hi5jPOaoilsQmA+4U54Cvqxh9lS5+DvBVoZ8yhyRBHxwEipXQVU46Aj4B2uYn1tqCYnoDf9lN+6BdW3q25JHFq6Gu0B9QesaEB4ZWbIKavlXg9ub2zADLh3evbbU4UMg54bkQcIuoXJ2M5ihi4Gl9PJJpMmwgNOUWM3EGYByhyIpSLZaEuVT2IABMosd0Zalj0Tp3AHy0DtfdNLp0oev+wA+/ZceEjFvylhd7xSFW3UaPBy2W2Ri5TVdvw7M2EEsr8V0fvb+YMfq4MnpXHRzpU9zH8TaQj3qKs8T/osoH/huHFelKX dfGOr1F+ opFlcBZM0Yp4jM7ou0I2XbHD/Q2sijkaojAvZ+EgUI6A3nE9DiedK1N3kCxY3dfLG2GhI17fXFQUmDwufXITKxrXFtoHsMcp898HndFcFEKvyuE8T0ntQCDlbA9ENjfWi31Jtcv9dkHYTne+rnwQvm3lyWPcNgsOivkyn/zTcy+LjR5Uh+63dIQ40IIwdRctJRdD1zJlbJEgnDWbW937M34Boeg9M9oJKBrc9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Process the pages in batch-sized quantities instead of all-at-once. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 7255c01a1e4e..f2e07022d763 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -310,10 +311,18 @@ void free_page_and_swap_cache(struct page *page) */ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) { + struct folio_batch folios; + lru_add_drain(); - for (int i = 0; i < nr; i++) - free_swap_cache(encoded_page_ptr(pages[i])); - release_pages(pages, nr); + folio_batch_init(&folios); + for (int i = 0; i < nr; i++) { + struct folio *folio = page_folio(encoded_page_ptr(pages[i])); + free_swap_cache(&folio->page); + if (folio_batch_add(&folios, folio) == 0) + folios_put(&folios); + } + if (folios.nr) + folios_put(&folios); } static inline bool swap_use_vma_readahead(void) From patchwork Sat Feb 17 02:25:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A201C48BEF for ; Sat, 17 Feb 2024 02:26:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 33D006B009B; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B6B566B00A0; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7007C6B009D; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 359E26B009B for ; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0E8334014A for ; Sat, 17 Feb 2024 02:25:54 +0000 (UTC) X-FDA: 81799705428.11.1C5BA15 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 6551FC0005 for ; Sat, 17 Feb 2024 02:25:51 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=VJ0aIzZp; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o3omPoxMLsk94BWO3XT1KRKpFrR6WgQB/H8HRcd+FzA=; b=KeO1SKTj+HtzZi3SroKDlTwEo3IPm2gXOpqUme1UYRjb0fJLYwbPl2CECsnnKbq3Wl85dA HfOIIQbg0em+GsVDKAu3Fv0CIV4FzdToYBKF43Jv8acFaeb8iSuWKrWXijeJPB61ouoYdi grpxtRKDWR9z1+g8pKxMR1sxkFRmIAM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136751; a=rsa-sha256; cv=none; b=28bLqYpEvcnVmCbibZ1X3EUjIh7W9AndFSpC76h5yLyXnRO2DkoiZApvMn7/F+0A0xI3xg 3xRnXTzAFCdx7LjxsPj+ZFKgfzcXn3vKSgxq0ODYYl0D0ETfkQ7We5zME+xB0/EIrNXXyj DRlbV+19+tHxPiLMZdpoCbmlk/CdnmY= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=VJ0aIzZp; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=o3omPoxMLsk94BWO3XT1KRKpFrR6WgQB/H8HRcd+FzA=; b=VJ0aIzZpZYScSdiK1VA91ch2kK qgM6xBppoz7JwAnhOg7N2GpvgrnlKGIy+31R/uX9swaexH57A3L5qlKyO4iXpbFuys0kstjBI3IjP tB8CsONFpWCktgvql9jFRRENltvdUr3LDeUgZi/XZ8wUhbDIUR0q6xjnMxyWRTWbxleU3uRArfIit 21reC5P56+j91tkX1HWaCXACt/kXeLqPw6YWe3qnI2q7KmPsn8KBBKB2EbjN3Jmh/l/JA/UDtjAjf B2rITmRlLXT56B2QhBKwCKMpAcmXGY54+hdjsJiDu488KqsgcSu4ycguUGBQ5An4imZY2LaA6G9iA gUblaoEQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOj-00000006HEJ-4ATV; Sat, 17 Feb 2024 02:25:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 17/18] mm: Use a folio in __collapse_huge_page_copy_succeeded() Date: Sat, 17 Feb 2024 02:25:43 +0000 Message-ID: <20240217022546.1496101-18-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: jcu15b46u3ebr38dfp5cm8kuze5fg8p9 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6551FC0005 X-Rspam-User: X-HE-Tag: 1708136751-146579 X-HE-Meta: U2FsdGVkX19shWxub8GqZj16bf0VZa3fbNaT5o/ldd3UOanuvSDEeXq4vFStj62oNXEcIXaoH6e7dY4EiBlZc8YIwYmhbcwUl4S7Tb53d9l7LErlLpTVKK8EEGAOMjC3glHoDefEgPCumXoGoZbIX/jZ0GixQwF9sqBvRgUSRlsAc6srTKbnEN2aAKwhMWnM39YrLKQoeOATIPGrA7IX/k+zOkM7LfLs5CU+pekPXiJlqW/WgicT8EO8gd2Vra4ipqOgPmyx73WVhIHyWsSCftnGURFaLtoo0MA4T2VpKXxIwUgbszjPMt1dC4MxlHflGVOL1Z26sRxCvAIeZiVUe+IlGkuC3Pqs7Oy5ZrOx5iNSqczwmkOQ5fXvRM05x8FuyNd2J+2ieSJ0q/RgofRMZJJHP1SkEQpmKdgyI56meSY40dRh0S8ni7SRB0sjOG8JlW/6u33VRY7VGQXrZPet7ApwM96OWtbF5JHqAAQiL6qORxSeUzCla6IHaDwo/K0d+EMv/9W36MWtEjjH1S2zZCQRa7tqLqWfNFJBVN5qnOkzDlXESOETXF7Dm2X+g88yooDoF9MdF/LNsyNuGk452576iq7ETLcgAxxGTLNl3AQWbxIkB7ZutAAzqN0X7ruelOa9C9s+rNl3CjzsDLcZ5CxjYWJhXCu5c1xYVuaY0uzx4Zw/Y99GGpmGxhBIMfWYCecC9PCu8JxT8So/409d5yBZFGlWldE/Vib6mfiEDdxpqdN18B/HetJkclhKJKFMSnSSC99RaKTZtuvRTXwfw4sz8wKkGlJQ9Fc9Z2NTJpPvexp/RBsx39TqEvSMqpqjfvX0hamtcB5zLj808ppSi7HZR/wM9uKVxzXqPztN4s0aFd/RSR4vziXMot1K/TmsKmM9JkoM7HA3C7B2MAVxm5rdkke+Q1Ad1jAqienBnC3jRmHkOlANdkwsEJNG6VC/2PuGaECSUcKKek0qRlw nn84wOOw eBqk9EHBgHXxAHIHWGFZhiZY1xGGx7zdE5pVm7Squmm+DkpDbP+AzVoypYPXN9WXyngeUCm/3R70guVOqN0mSs3PPHi8ZE1XVh9Rmhr9soohxAQ1f9hW3DVo6DN1MjGDH7OrwAsn5cQhcSOQL1UY76H/sEW1TsMnJv+4soYpZG+F0Xku+EhmQ8GaA+0Po0tnzXEPd/pwim9b6f47kdbXgxfOfCTOu3HIHnmOB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These pages are all chained together through the lru list, so we know they're folios. Use the folio APIs to save three hidden calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2771fc043b3b..5cc39c3f3847 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -689,9 +689,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { - struct folio *src_folio; - struct page *src_page; - struct page *tmp; + struct folio *src, *tmp; pte_t *_pte; pte_t pteval; @@ -710,10 +708,11 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, ksm_might_unmap_zero_page(vma->vm_mm, pteval); } } else { - src_page = pte_page(pteval); - src_folio = page_folio(src_page); - if (!folio_test_large(src_folio)) - release_pte_folio(src_folio); + struct page *src_page = pte_page(pteval); + + src = page_folio(src_page); + if (!folio_test_large(src)) + release_pte_folio(src); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats @@ -721,20 +720,19 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src_folio, src_page, vma); + folio_remove_rmap_pte(src, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } } - list_for_each_entry_safe(src_page, tmp, compound_pagelist, lru) { - list_del(&src_page->lru); - mod_node_page_state(page_pgdat(src_page), - NR_ISOLATED_ANON + page_is_file_lru(src_page), - -compound_nr(src_page)); - unlock_page(src_page); - free_swap_cache(src_page); - putback_lru_page(src_page); + list_for_each_entry_safe(src, tmp, compound_pagelist, lru) { + list_del(&src->lru); + node_stat_sub_folio(src, NR_ISOLATED_ANON + + folio_is_file_lru(src)); + folio_unlock(src); + free_swap_cache(&src->page); + folio_putback_lru(src); } } From patchwork Sat Feb 17 02:25:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82184C48BEF for ; Sat, 17 Feb 2024 02:25:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5506B6B0096; Fri, 16 Feb 2024 21:25:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BB966B009C; Fri, 16 Feb 2024 21:25:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 022EE6B009B; Fri, 16 Feb 2024 21:25:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D32526B0098 for ; Fri, 16 Feb 2024 21:25:53 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1D30D80139 for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) X-FDA: 81799705386.12.58C43BD Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 9BF024000E for ; Sat, 17 Feb 2024 02:25:51 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hkPTBK85; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AeeOyRTuCkQBvJfx3AB0GRUPqnlTrrPrtcTW/4ajnA8=; b=R2k01zjinIsk/jiIFZpkbBOWSpdj+cq0w29ydZNjit7FdkmknOX3xEO2OXdNn6+HytxlIP mwr4roM7SXhQcZdRdPPpNsIWGm+iXqsFrDTfKeUGT4KUwNEYTtPfysHr1VC6uSegyOJtrO c75Qg74d1kzUTKBtLfpE9vibD7TUrMg= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hkPTBK85; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136751; a=rsa-sha256; cv=none; b=nEyEZvroHiYd+jwqBOK5Ne0lQHirtdWz4AJ60xAU+jpWpbk4g87DA5teaaSNQihbxbcqVh xip1IpRs/cnr7Y4qG2RFmbcMHRXGhs2pqU0wQ78sOiL97076pZGbEzmIns3hUDiGNBn5K0 clad8sE+cOe6iejj0EopxC/07x50Dfs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AeeOyRTuCkQBvJfx3AB0GRUPqnlTrrPrtcTW/4ajnA8=; b=hkPTBK85XC/Z0XtdzHiO4CQvuH rSYDAY2pBa/Hg+ZMTygcrKERy/WOGbTsqSPCmRiE+QRqduGSsbbsW3PJOt7vHzbD7952iRZ5E4L0c 7MRQPdguDVqYxyZqMnjU4Xw+j2jSwWeaF/pQR5nBlxTnPXRjV9234Lj0eBNd86NypLRS5WSosqLMn qR3PFH7jYOUPxhE8norcB0Rz8K8cgmIIV8jEFveLXdsrrGPVZ8x4taYg8ww7gEKgTXk12gkgz89Dg wMQyoYJjVQ8cj0O6t+m8QxGLf5HpEy0TpC6oLa3f5OCcMz61H4apUlOx271UuwA67uiNNq9R0T18M M+ngpCJA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOk-00000006HEQ-0bdW; Sat, 17 Feb 2024 02:25:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v2 18/18] mm: Convert free_swap_cache() to take a folio Date: Sat, 17 Feb 2024 02:25:44 +0000 Message-ID: <20240217022546.1496101-19-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 65cdngdd8qftdgqf7y8e1un5m8g8wsdc X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9BF024000E X-HE-Tag: 1708136751-656372 X-HE-Meta: U2FsdGVkX18TKCurtZqOVYvsv5YjnJCIJEnQwh2xixgC0NVvKSXcm0IT2t94EWSE+YxE6YWNvW58VHky0X7nT1Zf87ogcOnAbKs4jkHiGrmMYFh2t35LvsylfyRo592cJHC4/EuUv/Osf3NUsQQpkPi/5iYWvbKFCGMm2AZD7SLzI3KG7JiVJkaEp+p0T6bQAmUu28n99PiYZfJdNjH5sjWtvycJ6GdGkpdlO3e35WVJ4MWieThxITiC5PRnFw/i48W1Ol01PJ4dz9k12JNRRImwhiHFAF0Z/Bgh1BVTM3BvrgvU7QsWeg/Z3W3lFVt6b86JGlkI0ly/cYEz9fgmtU+DKq3mTDqxHB/jfIcDjsk4wEDqLfJEZSkFGOTjb0M1iS+oGnekRMpfWQ7mwsoKyrnvCHVbQu49Pgnp47v2iXSxl1p6R2uHjVS2lfa2tYtA1ujM8UKZ2iWnAzpNJq2G+n421o5rKU+2VJrVnILH06/doNtd3L2hX7ICPCIRmk32ZSZmh80jc4PpWnpq+JHyfiEbQOWdzGcKFsBTKOP1MLvEtGUGu9/TIQNastg6wuQguV9osNgurT0TUZWGSZNWChcr1paq/PoOtWU7xJPXKq/0DIbnSkEhckVzVsx0uDCiDcs84Mdq7moeUqA6u67IFKo4zhmmA7r0TRzs5MbTsrFgTA2L/BbDugdBXXTCltoXMZoybfctFrfP3zV0fwkOmXVXGXSkTjei5hCTz0xviQEgAAhpj4XysVyWFtjgX2CMA5AvjOpFK23aKsIZmrVG5VkW0689n9elXRcaA4R8CWvPJdhU/1NXvWTmrYkRgKycGT+WfMuRC70OW/kmlMFoaGA2qixXEwzENxJhFKxifuPigYBS0WAnFYKLd7sqv5Tuyih+RtSm6FCRlcNyQFsYaT3CK+jXoN1UpIJUN0ui0yU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All but one caller already has a folio, so convert free_page_and_swap_cache() to have a folio and remove the call to page_folio(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts Reviewed-by: David Hildenbrand --- include/linux/swap.h | 8 ++++---- mm/khugepaged.c | 2 +- mm/memory.c | 2 +- mm/swap_state.c | 12 ++++++------ 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 3e2b038852bb..a211a0383425 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -440,9 +440,9 @@ static inline unsigned long total_swapcache_pages(void) return global_node_page_state(NR_SWAPCACHE); } -extern void free_swap_cache(struct page *page); -extern void free_page_and_swap_cache(struct page *); -extern void free_pages_and_swap_cache(struct encoded_page **, int); +void free_swap_cache(struct folio *folio); +void free_page_and_swap_cache(struct page *); +void free_pages_and_swap_cache(struct encoded_page **, int); /* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; extern long total_swap_pages; @@ -524,7 +524,7 @@ static inline void put_swap_device(struct swap_info_struct *si) /* used to sanity check ptes in zap_pte_range when CONFIG_SWAP=0 */ #define free_swap_and_cache(e) is_pfn_swap_entry(e) -static inline void free_swap_cache(struct page *page) +static inline void free_swap_cache(struct folio *folio) { } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5cc39c3f3847..d19fba3355a7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -731,7 +731,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, node_stat_sub_folio(src, NR_ISOLATED_ANON + folio_is_file_lru(src)); folio_unlock(src); - free_swap_cache(&src->page); + free_swap_cache(src); folio_putback_lru(src); } } diff --git a/mm/memory.c b/mm/memory.c index e3e32c5b4be1..815312b2dc48 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3376,7 +3376,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(new_folio); if (old_folio) { if (page_copied) - free_swap_cache(&old_folio->page); + free_swap_cache(old_folio); folio_put(old_folio); } diff --git a/mm/swap_state.c b/mm/swap_state.c index f2e07022d763..3f58d6fd5b44 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -283,10 +283,8 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, * folio_free_swap() _with_ the lock. * - Marcelo */ -void free_swap_cache(struct page *page) +void free_swap_cache(struct folio *folio) { - struct folio *folio = page_folio(page); - if (folio_test_swapcache(folio) && !folio_mapped(folio) && folio_trylock(folio)) { folio_free_swap(folio); @@ -300,9 +298,11 @@ void free_swap_cache(struct page *page) */ void free_page_and_swap_cache(struct page *page) { - free_swap_cache(page); + struct folio *folio = page_folio(page); + + free_swap_cache(folio); if (!is_huge_zero_page(page)) - put_page(page); + folio_put(folio); } /* @@ -317,7 +317,7 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) folio_batch_init(&folios); for (int i = 0; i < nr; i++) { struct folio *folio = page_folio(encoded_page_ptr(pages[i])); - free_swap_cache(&folio->page); + free_swap_cache(folio); if (folio_batch_add(&folios, folio) == 0) folios_put(&folios); }