From patchwork Fri Aug 25 13:59:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DFCBC88CB2 for ; Fri, 25 Aug 2023 13:59:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC3FC2800C1; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D23A2800C8; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8E7A2800C7; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BD4708E0017 for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8CF22A06E9 for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.09.639BA30 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 0548040006 for ; Fri, 25 Aug 2023 13:59:22 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="kmX/bgzv"; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Uw/NH5iJzGksgzQyF3Y5iZkOcZVEYA15Wf8gjLzbAnU=; b=qPd7qVaR9nyjgshr16LhprEKKyItXH20Rsk5SNpDoIk79m6DGfhyfpDKb0O3W7rzN9zciv IDu/ZQEN0Va6zXZsYBVjuVPqC1sueab1B50Bk3fmmrAcS10hY1ngux7uTsbd3jHw9IPOu0 cXLRu8PzokFV9nQyaJuVftXj1bJxbO4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="kmX/bgzv"; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=KOIpzYC+6PjhHUyKL2f+lKGrDlEIMOdTWaaLE+CSS6qhsHbJFsx6GVmW8z4fJWJi37pOSv rP9tiAWSE5XVEbJ0/5KpjTedDBvpjLgLVbbLWNJRLmk3I7glcXHroZYThndSwzifImxRbq /pXZ3yTrLPIFHUurrQ+c+FKq8cepBIA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Uw/NH5iJzGksgzQyF3Y5iZkOcZVEYA15Wf8gjLzbAnU=; b=kmX/bgzv5G50PlMppO4cSG9SGO joQOgQD/tPRvRpB5jZMRiOFCXp4qoheyElp9kFKRsPVRbIAWCdD/rRG7D894OmnvQMsw1WWJhn0UR BWWGkDRb8aow3qkkLhGd2N9iPsrjMQ3jj0zz6QH6Aa67tnJE5DmL3g9w/WH4rQFCzRMc1ElCcq2b2 iUZRvY0ctQOT/hQSYBnTPrR4htJXF67uc4ATbPKKE9g172E2IxJcBPMc9fOSctAnjY9ClkncAaK5j bDTPuYDWeKPlX9eM/S6w5kT+WIO8RijTKJ/MTFAmA17AJUWtepK504fUo76ADji4uZKgt44kLeOVh VWkrWoFA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLM-00HTQT-VH; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 01/14] mm: Make folios_put() the basis of release_pages() Date: Fri, 25 Aug 2023 14:59:05 +0100 Message-Id: <20230825135918.4164671-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0548040006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: x4wty4u3ktypgw9wi9mpqxmb7ztzf3ri X-HE-Tag: 1692971962-223671 X-HE-Meta: U2FsdGVkX1/UeJ6xcMCVGBu3L6d3uu6wbanj8dc15LjV8aNUUgz39gEc6gTWGNOmdXr4mvZnHJvDjUo7ulhxVH6oDatrWK8O1Pfhyk77mXahW7lG3jXqtMNGSMAE2dNRgdI8ClPAUyvAw53Gw3gY0zsjQkZAAeHTMXF0F7Q6YZHQg3qyrKxVEVa8fIKeDWh1lOl+XfEAIkjDiPbDx654VL1wFAEi6CX9kqt6uACoCOswj9BxtItFqKNA6Fb0OL87xDYMM7ob4TZYZhDrj8s1klNN2ycLzlnuSWAm9dL1AIatlfEozthL9AgvsHJNO47s0ZBDiahZuUSJLUbzq1rZ6jFNYBXd1Wjx0+wXRQfG5gk5bGGm1Uvx8GKM4f/iYmQqk03ZZlUY54VJ+qWMVBrETJW3tgZ5ve4M8OwRwZNW8OIWsRQ+OdOHR1UAGHvLq0PqkLWNPH1W2nbiCisyVmVJ+bYU0flDyy9FJNS1Xe0P43nPmsSh6eWVd4Qy4N42cWYG+j8qCJRcy4hMYLjhlJuQJt1jvwNOQ5b+WfvziFiiPoFke+TxbDFToJx5uq4dKvAAd2wO841Oh41NuXrREFcJTy4Sed5bIE40493mz/Y+mCBXKX7zLUZMJT6TEtR0nhTFBXifY60qiwZP9g6o6oHBAS78aXhoRdgrw6B1FDEaigpRpCJ9zZeLH0ZDUNV0YOJncH3Q74Yrc3g3y/5/YK9iocEhX7yMafs+yvv+wF2Is/29vLhrcLG5qj1E7f7m95Sd8mUjGsM/64XnLt8HnvBFU6WeBUTb7L7ZZovxutWGCC1qgO9sa7RHuiGKxfmlOPoCmXAhzYyTe7bOgiaabwMHTp/C5T5F93qXT9V04/fsZs45fs6AoOpFESUs/rXeWb8kxtRbYracBKe5rQ+x2ORXdXf3mIe8/MX5N6Bj0sZh4DDCwo2aWMuFcSQWcOrMgFmSp07rbpq55/1L8t+CT80 hcSCLw/G EJ3g6l28+xHhCpbHBJKLzGpjEJiQdl375WvZXOuBmnbBqK5gaEnF9mhBowas7k6fd6LWskkwite8YJnJQ7cJZfDAmkvWujH79dw4grURleFEFPaFM4+OoIMvgMzq7i0X7lu5JJRg/WRBEeRY0khMeaqqYJdAyujoo03BGAgAwbDdBHfwDFbfXTP5WsIzBv43MqtP/hFzXagnVHRb//vFiQ+ux2yJyWfiYm9b5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: By making release_pages() call folios_put(), we can get rid of the calls to compound_head() for the callers that already know they have folios. We can also get rid of the lock_batch tracking as we know the size of the batch is limited by folio_batch. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 19 ++--------- mm/mlock.c | 3 +- mm/swap.c | 84 +++++++++++++++++++++++++++------------------- 3 files changed, 52 insertions(+), 54 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2c6b54b5506a..7d1d96b75d11 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -36,6 +36,7 @@ struct anon_vma; struct anon_vma_chain; struct user_struct; struct pt_regs; +struct folio_batch; extern int sysctl_page_lock_unfairness; @@ -1521,23 +1522,7 @@ typedef union { } release_pages_arg __attribute__ ((__transparent_union__)); void release_pages(release_pages_arg, int nr); - -/** - * folios_put - Decrement the reference count on an array of folios. - * @folios: The folios. - * @nr: How many folios there are. - * - * Like folio_put(), but for an array of folios. This is more efficient - * than writing the loop yourself as it will optimise the locks which - * need to be taken if the folios are freed. - * - * Context: May be called in process or interrupt context, but not in NMI - * context. May be called while holding a spinlock. - */ -static inline void folios_put(struct folio **folios, unsigned int nr) -{ - release_pages(folios, nr); -} +void folios_put(struct folio_batch *folios); static inline void put_page(struct page *page) { diff --git a/mm/mlock.c b/mm/mlock.c index 06bdfab83b58..67bd74a6268a 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -206,8 +206,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch) if (lruvec) unlock_page_lruvec_irq(lruvec); - folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } void mlock_drain_local(void) diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..7bdc63b56859 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -89,7 +89,7 @@ static void __page_cache_release(struct folio *folio) __folio_clear_lru_flags(folio); unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in release_pages() */ + /* See comment on folio_test_mlocked in folios_put() */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -175,7 +175,7 @@ static void lru_add_fn(struct lruvec *lruvec, struct folio *folio) * while the LRU lock is held. * * (That is not true of __page_cache_release(), and not necessarily - * true of release_pages(): but those only clear the mlocked flag after + * true of folios_put(): but those only clear the mlocked flag after * folio_put_testzero() has excluded any other users of the folio.) */ if (folio_evictable(folio)) { @@ -221,8 +221,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); - folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } static void folio_batch_add_and_move(struct folio_batch *fbatch, @@ -946,41 +945,27 @@ void lru_cache_disable(void) } /** - * release_pages - batched put_page() - * @arg: array of pages to release - * @nr: number of pages + * folios_put - Decrement the reference count on a batch of folios. + * @folios: The folios. * - * Decrement the reference count on all the pages in @arg. If it - * fell to zero, remove the page from the LRU and free it. + * Like folio_put(), but for a batch of folios. This is more efficient + * than writing the loop yourself as it will optimise the locks which need + * to be taken if the folios are freed. The folios batch is returned + * empty and ready to be reused for another batch; there is no need to + * reinitialise it. * - * Note that the argument can be an array of pages, encoded pages, - * or folio pointers. We ignore any encoded bits, and turn any of - * them into just a folio that gets free'd. + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. */ -void release_pages(release_pages_arg arg, int nr) +void folios_put(struct folio_batch *folios) { int i; - struct encoded_page **encoded = arg.encoded_pages; LIST_HEAD(pages_to_free); struct lruvec *lruvec = NULL; unsigned long flags = 0; - unsigned int lock_batch; - for (i = 0; i < nr; i++) { - struct folio *folio; - - /* Turn any of the argument types into a folio */ - folio = page_folio(encoded_page_ptr(encoded[i])); - - /* - * Make sure the IRQ-safe lock-holding time does not get - * excessive with a continuous string of pages from the - * same lruvec. The lock is held only if lruvec != NULL. - */ - if (lruvec && ++lock_batch == SWAP_CLUSTER_MAX) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; if (is_huge_zero_page(&folio->page)) continue; @@ -1010,13 +995,8 @@ void release_pages(release_pages_arg arg, int nr) } if (folio_test_lru(folio)) { - struct lruvec *prev_lruvec = lruvec; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); - if (prev_lruvec != lruvec) - lock_batch = 0; - lruvec_del_folio(lruvec, folio); __folio_clear_lru_flags(folio); } @@ -1040,6 +1020,40 @@ void release_pages(release_pages_arg arg, int nr) mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); + folios->nr = 0; +} +EXPORT_SYMBOL(folios_put); + +/** + * release_pages - batched put_page() + * @arg: array of pages to release + * @nr: number of pages + * + * Decrement the reference count on all the pages in @arg. If it + * fell to zero, remove the page from the LRU and free it. + * + * Note that the argument can be an array of pages, encoded pages, + * or folio pointers. We ignore any encoded bits, and turn any of + * them into just a folio that gets free'd. + */ +void release_pages(release_pages_arg arg, int nr) +{ + struct folio_batch fbatch; + struct encoded_page **encoded = arg.encoded_pages; + int i; + + folio_batch_init(&fbatch); + for (i = 0; i < nr; i++) { + /* Turn any of the argument types into a folio */ + struct folio *folio = page_folio(encoded_page_ptr(encoded[i])); + + if (folio_batch_add(&fbatch, folio) > 0) + continue; + folios_put(&fbatch); + } + + if (fbatch.nr) + folios_put(&fbatch); } EXPORT_SYMBOL(release_pages); From patchwork Fri Aug 25 13:59:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FE3DC3DA6F for ; Fri, 25 Aug 2023 13:59:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0C388E0016; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CD0B2800C1; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFD082800BB; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3B1A72800C0 for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F3EEFB267C for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) X-FDA: 81162784332.11.A69E6B7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id B04C24000A for ; Fri, 25 Aug 2023 13:59:23 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cC8hYhXO; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IbN4ox6XAesra77WmyaM3MdMG44S5bDbvXnZH45zTdw=; b=22obXB0Vu2E11xbfmt00Jd/RZ05iMa84JU9NDY3sI79/8MAA6IDxdH+rEYn3eB48H2yEvY kFd3bj9bpaEUOoPfS10elb1LEr0ppdWWX6PLT92LKx2w2B6ZJA5hIx911caTV2dxm0fDXB ogPh83oJ4mCnbi7//GKhMfZMWJKYCsI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cC8hYhXO; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=8bZ0Xq/OtktyKQI/ibnHggLRCp+jm1AtMLY6RVzdd8hfyTdJNqUEpupNUnmvkKV193HJsa H/iQ5r9yjd5NOy+KakMMA1z5iOaRzry1QJz9ayEsU5Nrtkfu5etnkahox64b9o08BMPAR8 VXtFPGUrG3PoJK5+KpYwzpoj97Fw3mM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=IbN4ox6XAesra77WmyaM3MdMG44S5bDbvXnZH45zTdw=; b=cC8hYhXOiS++B1m4VY0rS6TpUc wArYFQ/eD8PM10XdEwiK2b/xS7z0z5v3UsfDUn0tU6WKoB6XArj78sEDEfF8h9FNDkrPAGzOEpzwZ NUaxtJCX2TFg1ZISHMdrAhDdRKjibM3dm8NyV1zKlXIdpS+Fk5Z2OWIhAiI3p5xwMLSstncQe7104 9eh2+KPtKTxpIgRAPLvX5MrOHtR5G3s/iyFC7Vn0yzw2cc4CmXELXXVUBkcWQoOUU0X91pNw4jN4K 7zdLMBtmnHxth7r6iy+eUKkhAYJqfIg4LpbDWPtdyxjmm1RvOMzNZ6UNAkV+HEUw8OrV+vDn1m5FS gKfTF+Kg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTQY-3M; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 02/14] mm: Convert free_unref_page_list() to use folios Date: Fri, 25 Aug 2023 14:59:06 +0100 Message-Id: <20230825135918.4164671-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B04C24000A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 76piinf8yt6jf6y99isbqs6bctwxriba X-HE-Tag: 1692971963-575802 X-HE-Meta: U2FsdGVkX1+WEVF4FDERwpS8iXuZ7iEOyqbf2IPH57wGVJfjes+aZYcwAMDv68Z/fC0L9SSzwE85hDLV9HGUtI0lkXynfBAQjbEetz02WygKYmi7P7XeAutn5N7taoDjpekUr4s7x19Z/EF/EUl7SBbKv8z5EdrI+8W6qjI5f9HoDaUu8ntDf5BxvWK9n7vBuFVH8IitVVuQFRd7i3f8AYErOLFEkEAKibdjz/vjdcU8Z2BJqJaniax8ZJduSDNuuatzTmoq/J7oCWOLb5GZ62B8/tLEYhMnrjsdOui+LR+ZQXpI8GqNxMV2OHqAUaUgGdb8Qu4vLcGmyQrMD7n0NAX9fPZ/c8HLuSycFGvna8vm05/5WjYt9uVivDZCxkWEDnlhham9elZ0OqoqNnDfWtzSlT0KStoMGlADHUfKUP+mgI8fh4MkBd8EYuGwGemYSaXxITloHBttdWlzdgXqSfl8mldPQcaaSVAGf790R9Fu6ET6JzH3YhzWRvy4idVkgcHuIgIpJUi8vIY2ZRO+fhCMhfQWuc+ihtUlaJaJTMvU87+NRqsVgMca1lXRq8O06KGn09nkTBKAI+5Btvm30YYsI84nadaZcnN9DqtH8uizF25CWvFNrHe6xXQ5DSJbe9Uh44FiUfceUeSrKdLTmFhtXnQ1YbgnQ/TIsKk/CtJlbFH5FUwZ7+qJLeU6nMOpDy8Oy1svH5zjdp1OBQJa27cy92kjHXH4mtMdN8NXB+WILfS4DyBoDssz9WwtraVkAmRRbSyZGcmooIcJGsh/iuNJKfhQcUCHB/5gHb84FtdXdV3shhF3m5PVqDveAp5E7fQFioFDY5MvLcysiogTJZaHA/dM++YIA5vGYc4JPxnJEdNOoFSens210ZEaQ1ehpJx3n9PbNdIYIDd1Kq1S45ELKORqooQPYYZp+y/zBDw3jC+LQ0RiuKNkerka+VDNrb+o03xQl27yyEda7B+ 2U0DhRRL tMNY4+c/RoVuKizrl/volhsnAoI1MNsfPR3VQaBFAV9QXhDIKTlxteDKUpXU+MQtMgrPuYsUgwPLQ4EoUbfxUeupj5PDXe6kaDdZimqCjsI+8lV71otSLRDlxydWC3Re6YjoUwAF9cbp6XFGV1B3XG/7cIdn8Bv2jbXEr6jO4On2JG3swTtyZBkC7yMxLDKfFvO8wvOtMkwNf2CT9oSaH0Jffi7Il+4yV8TtO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most of its callees are not yet ready to accept a folio, but we know all of the pages passed in are actually folios because they're linked through ->lru. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/page_alloc.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 442c1b3480aa..f1ee96fd9bef 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2469,17 +2469,17 @@ void free_unref_page(struct page *page, unsigned int order) void free_unref_page_list(struct list_head *list) { unsigned long __maybe_unused UP_flags; - struct page *page, *next; + struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; int batch_count = 0; int migratetype; /* Prepare pages for freeing */ - list_for_each_entry_safe(page, next, list, lru) { - unsigned long pfn = page_to_pfn(page); - if (!free_unref_page_prepare(page, pfn, 0)) { - list_del(&page->lru); + list_for_each_entry_safe(folio, next, list, lru) { + unsigned long pfn = folio_pfn(folio); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) { + list_del(&folio->lru); continue; } @@ -2487,24 +2487,25 @@ void free_unref_page_list(struct list_head *list) * Free isolated pages directly to the allocator, see * comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(page); + migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&page->lru); - free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE); + list_del(&folio->lru); + free_one_page(folio_zone(folio), &folio->page, pfn, + 0, migratetype, FPI_NONE); continue; } } - list_for_each_entry_safe(page, next, list, lru) { - struct zone *zone = page_zone(page); + list_for_each_entry_safe(folio, next, list, lru) { + struct zone *zone = folio_zone(folio); - list_del(&page->lru); - migratetype = get_pcppage_migratetype(page); + list_del(&folio->lru); + migratetype = get_pcppage_migratetype(&folio->page); /* * Either different zone requiring a different pcp lock or * excessive lock hold times when freeing a large list of - * pages. + * folios. */ if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { if (pcp) { @@ -2515,15 +2516,16 @@ void free_unref_page_list(struct list_head *list) batch_count = 0; /* - * trylock is necessary as pages may be getting freed + * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. */ pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, page, page_to_pfn(page), - 0, migratetype, FPI_NONE); + free_one_page(zone, &folio->page, + folio_pfn(folio), 0, + migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2537,8 +2539,8 @@ void free_unref_page_list(struct list_head *list) if (unlikely(migratetype >= MIGRATE_PCPTYPES)) migratetype = MIGRATE_MOVABLE; - trace_mm_page_free_batched(page); - free_unref_page_commit(zone, pcp, page, migratetype, 0); + trace_mm_page_free_batched(&folio->page); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); batch_count++; } From patchwork Fri Aug 25 13:59:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90E2DEE49B7 for ; Fri, 25 Aug 2023 13:59:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7799C2800C7; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E7E412800C3; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68FCE2800BB; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 743AF2800BF for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 54079C06B3 for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.09.0CDBF6A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 8979980013 for ; Fri, 25 Aug 2023 13:59:24 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UN5WzuQe; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zCdiy3Qag8b4OHr2D3vhJJ/icmkUhqWPaFiKm7GSTKs=; b=OsKOJpuxaIWn/T43OIp3n0bDOCwwe2NISejhHWDBjowwoLeZBWZnlGLUc8CauSHx6ROY/t oGCD/m19WmUzQSx7vqFDTP3UHASaGpbW0hsMIuknTFF++ollNosCtFGQHHgXn21fjmN2us C8mLiXJYudw1i6b+IZjr3xvhCc5KJaE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UN5WzuQe; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=R3dNKuC/KJ39O4OIgDV3saB9wql8vnUVKnwnn5iBT1FespzfLgiVsCr+35l0dSZFRt3dAj F74Vx0LO9S0AiMRyK9rS7dhDeTPj4YI6wcSC3i6PStNkdcQQ8Vhvv4i06ClYD9iauZ/Be4 Kdnwy8MA+zXTF64J7/ONtE/LNFduY1A= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zCdiy3Qag8b4OHr2D3vhJJ/icmkUhqWPaFiKm7GSTKs=; b=UN5WzuQeGq8BlLoxmbwgj9IobM 4hgUtnvAJmQtyKiDuBTJGDknMAkv0Z/yWG3Mf+Fv1hL615v1el8iP9GmlZSyEULrqMhi2X970GZZE iWiJD/GoUJ+wER7fGpJRsX2xXWQLl22OOs5a/nICRSRMoFupcyAYQgW8RSRl455dIoYUAhTDkjeH8 ICWb9fAtiVCItY0P70yqqhREja01yoTKxtwHKHfNXai+U1RbKQ5uxVPUQ5rwYx2uiUkJN7+sc7TMu IPVeUxCeMqUzclUBrJrrC/MSITe/TXCxZI5c5lFz7L60of1TafNQ62d6LWUnvJ4SM5Y5cLfEc3Dn9 2LphzJug==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTQd-78; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 03/14] mm: Add free_unref_folios() Date: Fri, 25 Aug 2023 14:59:07 +0100 Message-Id: <20230825135918.4164671-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8979980013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3xtsrxpih8q5t8qobbqoq7atfkdwqo1u X-HE-Tag: 1692971964-611751 X-HE-Meta: U2FsdGVkX1/UM6QNqjdr85oG67V8U+ZglqKudfrxFtbVmKkRQ4QfqwBkMx0axluwb6Ro2SXvGAkeTeKlvOhZSR1O+laj0nUJlPlnSjy+gw6C6SPJtbCgxcnrzvk676l4GAzpiqo6Cl+r2DJ/QqLoCGO47AChAWUTtm32CK3F/196eGPtuA6BRElJYrJiKSOEnOVduK8RyxvfIaamaB6Zbecrs3hOm/88lauVicq5QC/dXTU6iGALuFyP076+tcycNQbpz+6yN9oLvemOdMODtxyNV8YpDcnvFldrSnCA6PJgYpU1oOzeePIG6XlDgUtgEx124EUzRl4D/93agu0ZlDDa2mHfIxVxL3iscDPZFv9Js0Nc2HL5hUuePsD8ZHammaarEYTQeOUxwBHaPB1vIZSFipDVBElkdm1BzWCTIZFCFPbz0nW5B8MdC5b5uBcgznM7p3+bEP4BWFNpFEcU9zQNkcSoyEMTRUIjr+3VH5DXX6iMnIOGVHH5RIPUffpFP+Dtbz/S9F8hopgNiNsyJd1FvlOjEfgVppDfkl0Sy52NdUee2WOlEpATGOWdc0JtfTAk21Lo0PWIkjpdWskwTNUGSAnD9YDfpoakASKzKPL6mW9Jpf4nEiyIOzTAXDB4LUNg2vXNTgOiN4GA6ERyNElwVcgl+AC3Rm16RvkorHsnVrTT30rozZhHVhRpmMUxAXj2QJxoQFoBsr1yb22swght+u7GDrCMYDgMNLqDN5WGZhYZszTz31WjjnRzt2KMO27rBpmk3BYPUfe5VYq5OLyOPP1VIqR7zEcSiwZE1B3BixaGirl7q79wBypnAy3Icq6/7mS8Wu5TFuaeTM/V/nYzjj6JqQCGkMf3MAmNBE1IH/Qvfu2rf3GKikaZXEEu6qF8L6rh2oRSn5Lvqxe7JoOhEA/MKGsu/BtVJChYlk0h5HVoXcpJCYJostok55JDFqMeIOEWMiClp5wmhnP A9aH7prF dMImJoNLSW9KSChzcBFeePmLlrxEjKFRQPePAj9tXbyqPhHjOweeRtLc5ot91z2q9LVBnXqAupYgPG2igwzYk44dt/bZNYVZbeP6rkVdN/b7Fe+1/2jc1AtIAQFxyt0HW3RGUS88CRlB675CbtrodinnJqvE8QVcWywh3zWLf2uOPZZJzH+/pj4q9tpyPK7UDI9qg77DM39Z2FCBYkDCVVqLk/PgKcsfIfiIsasydYF8Hul1s4d7Y5YG2cnfKV6FeXxUR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Iterate over a folio_batch rather than a linked list. This is easier for the CPU to prefetch and has a batch count naturally built in so we don't need to track it. Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 5 +++-- mm/page_alloc.c | 59 ++++++++++++++++++++++++++++++------------------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 7499b5ea1cf6..5c6a53371aeb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -441,8 +441,9 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern int user_min_free_kbytes; -extern void free_unref_page(struct page *page, unsigned int order); -extern void free_unref_page_list(struct list_head *list); +void free_unref_page(struct page *page, unsigned int order); +void free_unref_folios(struct folio_batch *fbatch); +void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f1ee96fd9bef..bca5c70b5576 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2464,57 +2465,51 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a list of 0-order pages + * Free a batch of 0-order pages */ -void free_unref_page_list(struct list_head *list) +void free_unref_folios(struct folio_batch *folios) { unsigned long __maybe_unused UP_flags; - struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int batch_count = 0; - int migratetype; + int i, j, migratetype; - /* Prepare pages for freeing */ - list_for_each_entry_safe(folio, next, list, lru) { + /* Prepare folios for freeing */ + for (i = 0, j = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) { - list_del(&folio->lru); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) continue; - } /* - * Free isolated pages directly to the allocator, see + * Free isolated folios directly to the allocator, see * comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&folio->lru); free_one_page(folio_zone(folio), &folio->page, pfn, 0, migratetype, FPI_NONE); continue; } + if (j != i) + folios->folios[j] = folio; + j++; } + folios->nr = j; - list_for_each_entry_safe(folio, next, list, lru) { + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); - list_del(&folio->lru); migratetype = get_pcppage_migratetype(&folio->page); - /* - * Either different zone requiring a different pcp lock or - * excessive lock hold times when freeing a large list of - * folios. - */ - if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { + /* Different zone requires a different pcp lock */ + if (zone != locked_zone) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } - batch_count = 0; - /* * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. @@ -2541,13 +2536,31 @@ void free_unref_page_list(struct list_head *list) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); - batch_count++; } if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } + folios->nr = 0; +} + +void free_unref_page_list(struct list_head *list) +{ + struct folio_batch fbatch; + + folio_batch_init(&fbatch); + while (!list_empty(list)) { + struct folio *folio = list_first_entry(list, struct folio, lru); + + list_del(&folio->lru); + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); + } + + if (fbatch.nr) + free_unref_folios(&fbatch); } /* From patchwork Fri Aug 25 13:59:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0342C71133 for ; Fri, 25 Aug 2023 13:59:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D0082800C0; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B720C2800C9; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3618D2800C3; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4D2DC2800C2 for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2B354406C2 for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.12.2CB0F4A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 6E61BA0024 for ; Fri, 25 Aug 2023 13:59:24 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=u5AMMwaL; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XrsnVnR8KiSf3aJSJOgE2uWljs21ZHQ5GjL1P6Fx1JY=; b=nJymcDaP4H74cHvI5Vm2D8ehIJKp3JD4bLU2CdEOaS6wt2GJLfxBczRfFe5xK+c53KEFHH oY73+7B0DeTQYrcJhCRITbyWCXLLldlvKnhULAjIouXi+ciROQp/U+gRfptBfxnowtcl11 7WIgcpqwuuY9JSho67mshE8wpyC6g2c= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=u5AMMwaL; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=dsgE74srBAwkWEvd4SCFeWYxz3GmYICq9YVeafVT+DeGqlAhs9qrFGmrwyrTHBdMJNvlDe XiJSVJLcDH9U4/paADCvkfUs05BMiTtuO5yngLz3uaxVe0Eiv7+frt43XyZDPyCsxt6+rY sKfXsSnGihLDUl2Hzdv8CtyZtttBlY8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=XrsnVnR8KiSf3aJSJOgE2uWljs21ZHQ5GjL1P6Fx1JY=; b=u5AMMwaLi83v019OUH8tZ+fDhH aoRucrIX4va3o3SCykt+Kqs47qLg8oprESE+kAmocXNYgSXjwnJ1QoaOUNqad00A5HNEfjyjwVOKL BJHyCVV/9cZn8we6QXHJoK6NODOkQ2qL6Qtv9y8AUV9ihLcYOl4V/kViXmaqSstRW8UYJEvUjrZe6 7tZctd4htN2ingVQrE9ID9xKWy78LjeMMvEdycwodbG7zIaAg3jCnTr+6uWt7DkjjOiB6JY48z/kH Ui5jmN1AG+nsBAkEE1c5fzAbH5myoV/4gP2ecRNYhfEhYD8gDf/Hnr+U04tKGay+zrKMNFSryedKM syYN30Gw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTQj-BC; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 04/14] mm: Use folios_put() in __folio_batch_release() Date: Fri, 25 Aug 2023 14:59:08 +0100 Message-Id: <20230825135918.4164671-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6E61BA0024 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 3ph7gp17ntdkqgynmpf45dnhtwtu1jzf X-HE-Tag: 1692971964-58703 X-HE-Meta: U2FsdGVkX19Ocu96P1wnaPPLg12+iuslFgelB8KP4l2W1o5VLXLz89sIQVRC9BLSIgxUXr4KttdjeyBuWXF3xbNVUiqsS8+RSqljQjVV5Le8bXDGB3jLRhK1C/GyO3DM4OtsWwWCMswN85jNryYHSOXuff7CGB8IiYZCTs0unlLAIWmCMPsy8cWLIJThRWn5kasuLreUFdc9N3UK58ccGhGBADoXmvnSRBAk5Pyz2a9pkt37mMjPgVkkspkPPowFU8PBla5Yxf74ytgthXtRNlGJjcuYthSgKcipdqgdXkK2AK04PHOUiNxs71oC5mVwby49YPBAX/sJXa7tWtvKv1AtnNwQS1TYrfYkGfze7iArqzGj1z7zTBGakei3Rn1JZVTNG0G1lWdRk9ZESRh1loeUoQNoJZ0nwlqkFvsoeqxy9XJob2wuanW318TTtmN6QDaWbEQS1vAoPahvlGwyVgbj9o0Fg9rgHFBRzjyzFXGW6BeqxQANoRzIcxnnw0BHHT93FA8E9q4NyANKPwDd6t40cLediI5LIZdLZafhwtZvPnoRmReuXGls7YumxEhq2WYccco8gv7fO+UjW5kL2S5tpgDQ2xpTzmQWA3d28xlyqR3mbm36k6ZY72oifJXpqLfQUF9jeghXo9v3MXVmCUJFH6dpapmd5wh2xkbIUrqUrgGebytR6JLmy3eiFkW5uwK2agKX69sjJpD4VSt/ThAvPjpRzYFORfMa7kLMik+GutOUONNWPbkWKVdv92C7D68dV+1sPF7vdnYtSPr6yKj/enbQ8yxrZeJOJdi/sIs9XtWET4PbunCqkKfLSk6GpcGA/k3MQs4/hez/zUYmIbsj1Kr3/tuHfoNPeQGIPbjKUcBHACy00wGCMcaeCnSaQ5yuQwIr2UqUmd97DRhFb4XaLdbL0VZlfy82Fst8QZo4ZMTpBrMCmNGojsW/uVIv/M7CHBglbSbWaVE9Uuk 3geTUGW/ w9g1loAtUSs2f/egUU9AbpMJWdUb8yxV6rp+L6pTv2Ae5469rb/hsevuvI7GjATqTGlAWEldmYqcvOUDoC1WGaN8nT1Frih/hOPM47z45NvVF5a/Xz7GnnkYvAeMl/bvwou/bQ9PbDntjn9pooif3/Pr+bWR3/LjZtHPk3yag7LSmGbd0VP8vRz+NcUJfxqw6cDyD9MQy4XXTilFJ7jXV9od3DifhDNW+snsr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There's no need to indirect through release_pages() and iterate over this batch of folios an extra time; we can just use the batch that we have. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 7bdc63b56859..c5ea0c6669e7 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1073,8 +1073,7 @@ void __folio_batch_release(struct folio_batch *fbatch) lru_add_drain(); fbatch->percpu_pvec_drained = true; } - release_pages(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } EXPORT_SYMBOL(__folio_batch_release); From patchwork Fri Aug 25 13:59:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8815DC7EE2C for ; Fri, 25 Aug 2023 13:59:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 891332800C5; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 83BB82800BB; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FC2B2800BB; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EDD112800BB for ; Fri, 25 Aug 2023 09:59:25 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C6B461A06BA for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) X-FDA: 81162784290.13.53B579B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 0A7E840009 for ; Fri, 25 Aug 2023 13:59:23 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=U63qpPXM; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CdqQz781axet48KQidYLD3qY3rX/OHB467VTCpi9mOE=; b=qtN1muwg3eOP5zv7ntu9/Ee+oTaqy3xiiGSYJEDHi2L4Wx61Kxm9GLEqwYZu2Ozs4LDEt2 FJ5iCKwR5YZaE/kxqWQe6j5v1JtalydWjCnwDr+WpHrWwBYtdMM0ZZoXbsfFmuqVMFNYAk yCdG39hiS6NLKQU693Cc0bjjP8gOucQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=b/rLda72g2lQhktN1BXFf7y2tfAWPVeKPpsmKt1lgYiRnnCiY4OFI5d3h8zYx4pZyJ9z4g vSem0FWiO9Yu5sGv89uDq+EPpajCfFZTW1LWAIktI2vgoY1f+6H+hSjzaRHrgK/zBUyQjQ VdX2t8s+ZNjjD5dIEXs5TJC9XW80kAU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=U63qpPXM; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=CdqQz781axet48KQidYLD3qY3rX/OHB467VTCpi9mOE=; b=U63qpPXMBk+wldMqManAPBRBNl BcGPVUTy0C5LAxj2n685rIO47vanLcxy1FQUxuEKEDeJUWwnPhMj0bdZL/JSKn1M31Kp99cTW6eNZ v1yIJJF2BlMgKgxopiqo1S0T07PglTDku/2EelJur11i02pVUP13ab8fMYuK0oYQIOENpUg5MttBY JD+SmozORdQwbdAc2fndBGOSxP2yRLgKwLFspaVSOLjsOZZ3IxTPw/upRmOXa3tMJTtJR82h09n55 NV1RUKdxqCOJ0Ir6xg3HwLiL55h6cP2fDQf/+Cy1bXSiBRHqrNhLycruLJWeBNZ9X+ekMa/pYqCLn +GV+yP+A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTQo-F0; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 05/14] memcg: Add mem_cgroup_uncharge_folios() Date: Fri, 25 Aug 2023 14:59:09 +0100 Message-Id: <20230825135918.4164671-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0A7E840009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: gmo6e7naoyeouoa4ocnkcurgeo9dn5pn X-HE-Tag: 1692971963-918165 X-HE-Meta: U2FsdGVkX18eaU4jWUkFmrE1BgdS7X2oIzG942R3BllOflPQESwEh01p5pzGTBRMPJ5xiubR1eEfv2311CyP7yLhVFZk5oZy2HAzOF0sAEURSq9kYaLcuShVyIziSPlmvNd8Fthhe2sp8lOGSI97Mqqd0O2gWunv1DcUN6em8zowe1qQQQMAz+DZAzqgGQlz4RfUEwO+qpFPUaU0lqHomUq7q4P75UNaJbH9yrIGnUC1FnY6A5XbSOSVgeJOyVCjjHB33Ks1q2QbWaBcGtrDlSGIjzmrZn0vPRN186gwJLto2XIeh5X+SUDqBHAahHEgcNoRcFnFnSNU4ACyvFBslUFTTaFPTOs4Wgow5sFBSweUj6uPSwAj1GHNBybJgRiEpwb1fRjXeTY8584WznoG49v4YMViqLCgITXXvYjoA7lmcuX/aa7DnFPTazcLU6fcC06bmJyu8pDlpWQtC4QBEsfo8YrMchHKYtEW8XQ6Jt02WVKZam9XNamjbql0Y7za6R3VVKAJTtwSuxIzy9ncoN0NEaQaXFWfpjaBkkYTPsOa39HG1cEU0z8f/Qhtxeq482LYLjHrBEMVlE1Xm8ebskT+G2NBO7Nusj6/uF40UbHDTurXWBRxC02Ub4jQx+9atOo3ggnhsEd5UARzhrADYjwuSmmLV+5LKk4LQ7F38iQVhPhsEcHVQ/S1pvkR0nH+YAIu07LRyWBsBMvP5tBPohNIQHm8BhJy4NQOXPbd3xM2zsqEnuZ1ZMs5Stuti7/i3aNOxjDjTob+GlrxZuTN/M1zjwx67kNrSSp4YG4rX/UzTdzghBhVVkZ+ZjCs3o+gV0kYzh6uk3fhEHgJ+LBgNj1RYkKI13WILU/w2NjH/rTHBV8g2XBoGrad4KMcu9ibi/aN5z/4JMxImRwJWwv4SRzRYCQfXpX5DJ02Fc3aJvl+U8zMN5LsZpgrxCrh2T1zH6WoYoOJeInCNOCb54g fA9XNR/M lEuFHwlXs9svRMInbNqlEysIK3aln6RFA7PhuYJdLxTG6KzK97z+sadgZcxNsxCdstl/s+fbe8LOPqhEzR3lys43rHEq40BCe3TbFSJAbAM2Hccwkvc1Fx8zTa1v4VozTACmgm6lHMjzZnrX3xPhWCINd/CE+Ro6r8mhUbjjHhbV2gm+kBelIArkC8zuxQ2pQi8OzpPomG4NuIZceY8AhhuTgY7WbNAtRLNey8msy0lz2PHTxOCstGbgqtg9WBqHuA953 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Almost identical to mem_cgroup_uncharge_list(), except it takes a folio_batch instead of a list_head. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/memcontrol.h | 12 ++++++++++++ mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..d0caeb1c1136 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -713,6 +713,14 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) __mem_cgroup_uncharge_list(page_list); } +void __mem_cgroup_uncharge_folios(struct folio_batch *folios); +static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ + if (mem_cgroup_disabled()) + return; + __mem_cgroup_uncharge_folios(folios); +} + void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1273,6 +1281,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) { } +static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ +} + static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 26ba9d1a66d1..a0fb008afa23 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -7190,6 +7191,18 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) uncharge_batch(&ug); } +void __mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ + struct uncharge_gather ug; + unsigned int i; + + uncharge_gather_clear(&ug); + for (i = 0; i < folios->nr; i++) + uncharge_folio(folios->folios[i], &ug); + if (ug.memcg) + uncharge_batch(&ug); +} + /** * mem_cgroup_migrate - Charge a folio's replacement. * @old: Currently circulating folio. From patchwork Fri Aug 25 13:59:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BD10C71133 for ; Fri, 25 Aug 2023 13:59:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 816722800C6; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AA792800C4; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88ADF2800BE; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 29F062800BC for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F144AB2671 for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) X-FDA: 81162784290.15.2C4DC7B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 3ACEC1A0009 for ; Fri, 25 Aug 2023 13:59:23 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ioYJ9ZBa; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5c7x1kFuKVswGl86DNxqX0EgO6SdlR0zTwx5NPZJoVM=; b=Lt6pIHOHf8TKISBnx/OrOSkTQTXq7h5FbG6gz1xO2kHm+uyGP9dIUoV4E3+5cVSrM3F67q rfghStLCEvNA6N/+TRfmiT38NMJyeJulsjdXSWOxwjWXTMgJKoxq9d7GDWQO3ksldMAawm 510PD4n2qsBhmsAx6C7RhjUAlhpc/Sc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ioYJ9ZBa; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=wARNHYagudVmjbKU8d6+Wo0vdUFy9HNDKezTQgCQxA/K0UcI2mqqr0aAvfFy8PvcJWprZS rwZRVpHwjnQYoIo4+j4UN6Qp/soPrGITZcGyAMjwgC1be9YwHpmNRZGr8fLK1SUuX4Ejp6 sfJapczplNY5wShbZ+m1km5uQah0Zio= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5c7x1kFuKVswGl86DNxqX0EgO6SdlR0zTwx5NPZJoVM=; b=ioYJ9ZBaG5iSJwtN2Q1RH/Ym/T 5XaS24qb1p2fVO9kC4OIAUVffmsOfQIFApsa0EUS9E3nqZfvnCmJXGKH/gXOlSbPZueY1mG8wDtHr yCbX7A8PHLvpUW2WyW+3xRYHOtNJAo2eke5P0vzW9KS18tahjNOc693mRbOLlXzAK0zovAZXskLMs Xzmza1eYOkta2Jy9vA5yXXtO5b6EzeQmgn4BjaO2l5x6uAlEsMHXxwhkDBSK7qGTGmTWFva3yLr6O mvkqdrYox15CUZlxJRjf/xmVekdXNqo/SHhUSVzLpeIO4MQ5NOqszbxz1bJZLj2cQfWHzteYsLaaS fD6kUU0w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTQt-Ix; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 06/14] mm: Remove use of folio list from folios_put() Date: Fri, 25 Aug 2023 14:59:10 +0100 Message-Id: <20230825135918.4164671-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: f4ch4cntoypmq4ngzooyzsrmj9q8hsom X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3ACEC1A0009 X-HE-Tag: 1692971963-689031 X-HE-Meta: U2FsdGVkX19DwZHFDO+Ao7E1VtP/aiWEyexCQMbQCUKhFk9cJxFnoCh+E+4pkEN6BlzOW04D01pWRlYSJKry8RNfs5PV9EXzZ0kt79N4wdOyve6MVTFiXPPQR4pyB+rc+EPth7iF/lPJPJ8gL96lb9/WFXUn0cuupHN6tyft8DPHSkb+/YGn8MmHP7klUr8gIjW9wBLb/8zHFDFbBRKmAYOvRDjfKcaOPw5ohHZNNbzvDuG7ByImiRUxB899gU8SOubxnY6ereWURvO/l7t3DPPxNI6O6sPTxFGaR7BMYGOyLHpavEAEsefBdmA/SHiu1wiuOKeBP1HBt6FLYg12NrVN9b7yuA1hL/prmZl7iNqUTjtJKxN0XtIsJWGtaxbCv1uHWPyn2bTfaLWJ7FYJbbJWrHiv0rlqXsrGbYpri/n4TSb/zgAzOnkrrg7sOLVWgkiU3TEpEJv4zbXb6xmzUFYXwEyCTp/ASzUclnA9YJnKsYkkgvf7ibwrprxl1igXTLy+VzCRaepUcAcrgjAAViBQScradNUO42RDytAyWSSTnPjM+PwtXAMRmSpwLd07YBPmdXUN7yGV3iLTaP43u43wc7XqCWN5k1ebr20OPyHkytUylsg/WFzaEnfzJtkrzYM2i8zOfSC53R0pTG04cwN7ZYNZS20wXeCH5SjzVZaUoeef7sgVVsT3xo332h/xRRGxQrE8OpTcAHmbLSmFuDSmVabYwKfllL2sIfBju63gRKYGR/5Xa/0M1TeRswulRKGlDzBV66e4vcvQIDGiSg5mA8HIHmnFRLB52XISh9cwF9JdZo1rGte+oFq36fjgFyjiKfGWSXAtC8mF/FZsDM6q7J1+BjP9rvD/cZH7z3011OGGaHOg0W+DP6Nn67N5gFn8tGLQDb+5SzwpYSJhH0jowszyZEqW1LVWQ74kQbeb3Ru57vo/yfgBRSdhvGT/C/g/FjMfhwTTCoEXeJH AU/u6+Pz XERuTIMSbV3AzGqhNYWEJ+06XRoHtKEBp2u9a0cw1dvYZS5KPtUUD8/jT8wzy5ndilJdVoMgRQl9Ql1PHoycd62E/s4ilD/3KGL+f73fiYYJ72u0fSMopVdKf9W/OHsVNcGrNXHlFgdbDMQ0rwpu7+/qSdgtLvedbkaOsS+lQzlAk3CKT4/i+zVBWqNfVgs5qX6O5GDeDbQyD6h/4aCv8JwDlk7mwwqIQYTAY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of putting the interesting folios on a list, delete the uninteresting one from the folio_batch. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index c5ea0c6669e7..6b2f3f77450c 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -959,12 +959,11 @@ void lru_cache_disable(void) */ void folios_put(struct folio_batch *folios) { - int i; - LIST_HEAD(pages_to_free); + int i, j; struct lruvec *lruvec = NULL; unsigned long flags = 0; - for (i = 0; i < folios->nr; i++) { + for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; if (is_huge_zero_page(&folio->page)) @@ -1013,14 +1012,18 @@ void folios_put(struct folio_batch *folios) count_vm_event(UNEVICTABLE_PGCLEARED); } - list_add(&folio->lru, &pages_to_free); + if (j != i) + folios->folios[j] = folio; + j++; } if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); + folios->nr = j; + if (!j) + return; - mem_cgroup_uncharge_list(&pages_to_free); - free_unref_page_list(&pages_to_free); - folios->nr = 0; + mem_cgroup_uncharge_folios(folios); + free_unref_folios(folios); } EXPORT_SYMBOL(folios_put); From patchwork Fri Aug 25 13:59:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD40FC3DA6F for ; Fri, 25 Aug 2023 13:59:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B90E48E0015; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94F492800BC; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44C8A2800C1; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EEABD2800BC for ; Fri, 25 Aug 2023 09:59:25 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C0A14140680 for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) X-FDA: 81162784290.18.9FCB7DF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id 4A336C0009 for ; Fri, 25 Aug 2023 13:59:23 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n3bum2yu; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=avt2avNOh89/3L7dWAiTvZAWsRknkyYwpYTeBU8nK8A=; b=2fwmseYbxw9j3h62vH/O1rhIbmD++dQCfmxkHyqRWJJn/QRUQYthAl09LtCm7Pz3rIBrN4 agyMsSKVuZFY1tDQvnlm89Bo19Lbcmepu7BUba/TYTJuo8JN8438gtbJBfeNoTwSKWMWm6 gVOLjQF/UChsczNIUsUcmKpHJAiLIK0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=ZUpOXnMDICUXSkoCX+Dw0JojVObJ7Z+Dlxc6lan+T2cnJRH+6rE6EWTMMELV0pCTsLUJ2X vmhc3S56eima8BA8BV3prisMZg0UT5FFdDqJbSNY4mLxpt847mH8jP84a1gXM7YbTIVP3o asmrEBZ1rA8GeDUeN9tRKJ9BrQ7oB9E= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n3bum2yu; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=avt2avNOh89/3L7dWAiTvZAWsRknkyYwpYTeBU8nK8A=; b=n3bum2yuaVGsB7EMC21buK0yN5 MTiBmF5JNn1VFyBAc6xbTTOmAcvMN6S0ja0m2e8i19i6axSwawaEyVejEtE131mx1q/4NU6aWQ5Ta KKCQMlNKi3PYJMfuUiml6Byx1sF+5PxnHn8E/fK1Kz+1dRpNA9HmDGxsDmmmwY5JGaWYCq2BykKVO iGGiGl+/KniQjRyszqmlxxhseTOZtfZzzFSTilTo/zhFJkDT8cM4niVsum41KhsQHWr9+6UPFIzZ5 0I8c5JQ44ROCmc714b1DnONHgxpHwGoPv97SvVTqBM0ABAYFHW1KAnsrsmxJcP0LX5E2Vo3WaXqR4 JgzR6T4w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTQy-Mt; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 07/14] mm: Use free_unref_folios() in put_pages_list() Date: Fri, 25 Aug 2023 14:59:11 +0100 Message-Id: <20230825135918.4164671-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4A336C0009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: g3tucf7q55hnnapxcgykepeewrbrd37c X-HE-Tag: 1692971963-43808 X-HE-Meta: U2FsdGVkX184gttDHtsrA3kv4o0ZlmPNb5VDcauynR8xNRpciYHqom7CHOTJUQ8xxN+4AOhU2NQvyWWMr/+3pWxtXpB2W9qG5YK7qyP19Ri6WlXv6xyjuJUOFHwPhAkReEjJ3nUbojiJRh+YMUIzPn4++MdSCSGAZ4Bo9a0YiW8Iorw5up6EpU/RSFs0z5WtbMS6EInum9kPjin7aaT/AMJiUnhOUmHB6S2re/5Hr/+jO36ErImxmNyZ2Fn5ki4uNB7g3cLKi1EDmpnx1dlXlYYEcGyyf9KVVJblZveXieb060QB96gpTLRW608MAqnf6sjObmIIbZTf4JDflfRcG7pabf66NBbzyF7SkM6o1fgblv7/zophLBINGob1pO8d2NqwImtdS0XisQBMzqeFQL5p/xtxhBKsz5wZ11sB008PzT7xnYwqVT8Bdon3rWp10tSMsiw2ZcMhfJHr+CMtPeaqTenyS6cI4oDQm124LomETPogk5SkS/iRbN50GJiiTF1LRDHeBD8nH/Ht9D1w5X7K0+UhvfoIHOF6uhDI1QkgKh1DA9qvCcuaOwd/jf+3SUc6Y466EDGwLR/XVIp901TsKMgar71Z2PWTTlt7GYIMsLs2hDsavQ2CF9fw0DWoTj5LDGnkPWNH0Zeb2g24FlvMwA/RHJ5LalZhxSjh/HmK/UJcj1x2CdRfjoLgQKgfKPMzDQN1C9gTpf+reBmIHhF36zbHVNFfUcURhzyqWZJxceA6lhAoL93hGEpi9b/hRPd9ui/J6LU67DPLoeez1CxvyKzu5BjB9jV9vyyyoEsp4ijIvOMaP44Ieplc7ox2wOTkK4dlt2pFEDbec6p/xp/hAEk7sxaQDsWsNj2lgasXBxA4BfT3t3LNzBF4w1H7NNCMmUcspVACP5z0mYlT9J5+W1Tgwy6lDhZpzzsdfSj4j+6x/ZL95YSayCjV/Uz860ciDhAphEXH4f426ME uoQARnRD bhq9u/cbNHM/4tduCCQ5RNPFM9ax5kpE0zqvoS/em9NAhj4+GhnK8zQ6YYlHwHO5sm7XahHyhWPOljjaVMp0iC/9PlsZEMxxLg4txFSLFa4BhiC8X6QFPC+YaEGDVUhWrTMSUhRci2NxoMyr9jH91vakqwMssASUGJ9KZGX8E+HlLvcM53fzOxw/9SOxW+DiSMFZH2J7cPLzvbmFSEQJAfbRbpmH3XQ9djwZD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Break up the list of folios into batches here so that the folios are more likely to be cache hot when doing the rest of the processing. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 6b2f3f77450c..fffdb48cfbf9 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -138,22 +138,25 @@ EXPORT_SYMBOL(__folio_put); */ void put_pages_list(struct list_head *pages) { - struct folio *folio, *next; + struct folio_batch fbatch; + struct folio *folio; - list_for_each_entry_safe(folio, next, pages, lru) { - if (!folio_put_testzero(folio)) { - list_del(&folio->lru); + folio_batch_init(&fbatch); + list_for_each_entry(folio, pages, lru) { + if (!folio_put_testzero(folio)) continue; - } if (folio_test_large(folio)) { - list_del(&folio->lru); __folio_put_large(folio); continue; } /* LRU flag must be clear because it's passed using the lru */ + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); } - free_unref_page_list(pages); + if (fbatch.nr) + free_unref_folios(&fbatch); INIT_LIST_HEAD(pages); } EXPORT_SYMBOL(put_pages_list); From patchwork Fri Aug 25 13:59:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9FBC7EE2C for ; Fri, 25 Aug 2023 13:59:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFDD42800BB; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EF422800C0; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF5692800C6; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4F2F32800C3 for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 288661606C9 for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.11.BC378B4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 662F0120019 for ; Fri, 25 Aug 2023 13:59:24 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Vov6BRAe; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qiLtYIqOEP7dbrRgh13/wEWojRMPASON5YTG2W4NUG4=; b=dk9gYzbhGtp2jx76oIIbRHuenadrVQ1qEwmRXmH+QcpwoRwOQDTeNfVB+5QP6DbbbFCfXA GyO/TfP5bZWKzZFGexuqgDr+kZMMAj0sbdm59juOWDkhNdZS8Lf2ZZGNFfy7/KNmQ5d5hk s+FYwRPTtlRTE4g7LkPOSHFCy1jMxCQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Vov6BRAe; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=jo0mycn3/pYMgcsFWxFQi105SYDcvf2Blh79XCtJl0zxfNPXdeSvhsCHG45uJbw4yi7Ldq XagqrUfNBaOe4uMGY7ce1nzPIsl5GbWXLJ9HZZ7+o5bDaXTVCF0vrlJx+csyuCsl/iRiS4 0dLrlX/OwB2mlk/TCwsBAVVooU1lz+8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=qiLtYIqOEP7dbrRgh13/wEWojRMPASON5YTG2W4NUG4=; b=Vov6BRAeQ0t1i7+WGecmjU3fDN fUuJdMgQOeCqUIbr8aX/RWKSWNCRs2f/C9GUnhBJkpiyFMhv6y7Ss7K/P0Bim6s/tYKRoqZIBkPjW EEE/fiFCL/M+0QmHws30ohZIKYlAttKwPZXVksChznGh65TGxsRlEMLhbsjw97IPpVcsYJkmo0fvU xqQbYeLVQMByAuIxFg7YUDyyTvsOo9VjrfcvXYTTKkPGaeAkVV6ZKo/owum2u3lbUrvJ4kD1ZIV5Q zdJ8twTTGkbiaZizJCgM9O1LQObWJVL+puCLaGvcn2p7+jbVJNyccKaSmUop2wJA0do1TzFRHAP8D ZMyIF9mg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTR3-RD; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 08/14] mm: use __page_cache_release() in folios_put() Date: Fri, 25 Aug 2023 14:59:12 +0100 Message-Id: <20230825135918.4164671-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 662F0120019 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: qbad9kqcshkaj4tjzc46618u17t8h1uk X-HE-Tag: 1692971964-943798 X-HE-Meta: U2FsdGVkX1+Vi00OAmgg7zIj6X9PQBOBDDlX3MxERLrv28hNktI30CoweQRu4JHE+lWa9a8c3Z0ebTNMhzcSFV55ZAUWCDGawYEmSzQ8RPH3gHj7rWaeFdygxDU7fzb1LSjMsRd7T3MVYwvkn97zjMqhAHy959myWs2bEfzxzji77oD4parncfrOBnb7xeuDLa0cni8U1y4iTEFjIX+abzU092eVRsw2pV4YNBH9g7/Bl995ZQ413KUgkVztAxRAEbWJSJ583zN01J73X5MkxO54ORsj+0SmeKSfCdCua9AaiVkq/4vSRFCZBbBWXV+nDV3G0xhpod3UQ6/aKXZvZenMZZV849mnbqgNjUq8F9Lj09mzwxfQmPbfZmm4XnIlhznSSFNnsTmblvnPb6uMYhQe32yXJgZdDlD3tVN0wPZ195pVdHvcFPzqXhVMoEDU9zHYrTJFjkaUHtLbP7HDRAiShSAdnPlcOPmFoQWBLp0GnzKDDEYjgp4+RDsgtSZs3xCS0IhOiB88DzuW7V22vTsJNHl/jzBz9jA+CTcUqRdaGyWClrxXLlPfImv5VaoS9/Q7R+Yi0dK8RC6DKOqzC3Mghy6Ds0hagu5qdoda/I+1rSnzkHI1c68K6np63j463lgxALjMww004dv80zwGXE0CE/9iC1//EOx4ife1EkaCCBI8pqBkiuUKW3JXItidjrGsQ8IWfLSR6H4g+PK9i6m5EWX+cp1PWJf77JWeUm/HAQTI1CPaXLGp27sJoqwlppDa5Qextm/RsWiMoogvwdRbitqZAhMnz+1fdINwN1wcLGeWE/oQAIxo+QjuzrZAMTf5098kNqrhhitIKlm6yk2U1VxuEz8YLHUoqxhD9QC38ntvYhVNjdM6e4531D9ugOYxGJpwlaNshm4UMSbmb7tE8YrE5Xy680JVp7+FbahtTRCWAocIHauTihp4R7FGRT/T8evCgd9SeBR+OOt L6uGKatf WA481pt3+lneQYd6KJrzhgBZabxD96cfCGxezwpm1GWMxyPaoxWZlHhvwvUPtRbGcvPyKW7xPHyYCKsN+FboK4EyZtEX6u+a1p2cSHm2RzlcxdMjIYRGJiVX7bw4uWBAprFFGlAh0A4dfnkr2vRnHB6l5PhJ2cEXEB4czaxNj58IOl97M9kv0VB6vWHgRoDPTnen3LCtZGSd3410cTV3HocCaMOwi/xjqAUsrlXEr+Ki0Na29CxSpF05HB3VImxDdMueElQ4ldbBqF78= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pass a pointer to the lruvec so we can take advantage of the folio_lruvec_relock_irqsave(). Adjust the calling convention of folio_lruvec_relock_irqsave() to suit and add a page_cache_release() wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 16 +++++----- mm/swap.c | 62 ++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d0caeb1c1136..44d3e459d16f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1666,18 +1666,18 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, return folio_lruvec_lock_irq(folio); } -/* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, - struct lruvec *locked_lruvec, unsigned long *flags) +/* Don't lock again iff folio's lruvec locked */ +static inline void folio_lruvec_relock_irqsave(struct folio *folio, + struct lruvec **lruvecp, unsigned long *flags) { - if (locked_lruvec) { - if (folio_matches_lruvec(folio, locked_lruvec)) - return locked_lruvec; + if (*lruvecp) { + if (folio_matches_lruvec(folio, *lruvecp)) + return; - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + unlock_page_lruvec_irqrestore(*lruvecp, *flags); } - return folio_lruvec_lock_irqsave(folio, flags); + *lruvecp = folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/swap.c b/mm/swap.c index fffdb48cfbf9..21c2df0f7928 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -74,22 +74,21 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), }; -/* - * This path almost never happens for VM activity - pages are normally freed - * in batches. But it gets used by networking - and for compound pages. - */ -static void __page_cache_release(struct folio *folio) +static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, + unsigned long *flagsp) { if (folio_test_lru(folio)) { - struct lruvec *lruvec; - unsigned long flags; - - lruvec = folio_lruvec_lock_irqsave(folio, &flags); - lruvec_del_folio(lruvec, folio); + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in folios_put() */ + + /* + * In rare cases, when truncation or holepunching raced with + * munlock after VM_LOCKED was cleared, Mlocked may still be + * found set here. This does not indicate a problem, unless + * "unevictable_pgs_cleared" appears worryingly large. + */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -99,9 +98,23 @@ static void __page_cache_release(struct folio *folio) } } +/* + * This path almost never happens for VM activity - pages are normally freed + * in batches. But it gets used by networking - and for compound pages. + */ +static void page_cache_release(struct folio *folio) +{ + struct lruvec *lruvec = NULL; + unsigned long flags; + + __page_cache_release(folio, &lruvec, &flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); +} + static void __folio_put_small(struct folio *folio) { - __page_cache_release(folio); + page_cache_release(folio); mem_cgroup_uncharge(folio); free_unref_page(&folio->page, 0); } @@ -115,7 +128,7 @@ static void __folio_put_large(struct folio *folio) * be called for hugetlb (it has a separate hugetlb_cgroup.) */ if (!folio_test_hugetlb(folio)) - __page_cache_release(folio); + page_cache_release(folio); destroy_large_folio(folio); } @@ -216,7 +229,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) continue; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); folio_set_lru(folio); @@ -996,24 +1009,7 @@ void folios_put(struct folio_batch *folios) continue; } - if (folio_test_lru(folio)) { - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - lruvec_del_folio(lruvec, folio); - __folio_clear_lru_flags(folio); - } - - /* - * In rare cases, when truncation or holepunching raced with - * munlock after VM_LOCKED was cleared, Mlocked may still be - * found set here. This does not indicate a problem, unless - * "unevictable_pgs_cleared" appears worryingly large. - */ - if (unlikely(folio_test_mlocked(folio))) { - __folio_clear_mlocked(folio); - zone_stat_sub_folio(folio, NR_MLOCK); - count_vm_event(UNEVICTABLE_PGCLEARED); - } + __page_cache_release(folio, &lruvec, &flags); if (j != i) folios->folios[j] = folio; From patchwork Fri Aug 25 13:59:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B589C71133 for ; Fri, 25 Aug 2023 13:59:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11A0F2800BC; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A88D42800C1; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 698978E0015; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 360002800BF for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 15472406BC for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.15.3F35468 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 7069B1C0028 for ; Fri, 25 Aug 2023 13:59:23 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C1AIhNTd; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n+gxEXWZCOB3pRmnNNsPASJ2gcVnzHVf7MDBK5BYIds=; b=F1laVAIiUvFFZpItVnnSjIzwtFY8RLfFVdxIZbH3pPzHNEQyVreD7sTMoFs5lY1U/NV52C zdtAU0DoDFtKoq7x5rUFYSIR78jnVrjOB/hm9/m1/nVTZSoJNsXZscODj0ekhGf0NA/YIK WxUoGmt92evJoTWGFBruqP4QUQU/Lk8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C1AIhNTd; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=mjQM8f3n8rqSgV6ySOMz+s2745HYxr1ojI1/M1xM8eUB2wfouB3zx58ACBnnKWBdXRSKjh TnT7M/KABvosQGl+S61Fb34kMY0d154jNx+nuC2Us0c8bmPudhGIQbWSjB6Iaj7LWdptxX y+P9IkWnHzSc8ksuL/T3+afLIUJ7ylY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=n+gxEXWZCOB3pRmnNNsPASJ2gcVnzHVf7MDBK5BYIds=; b=C1AIhNTdfXvn5iu7QjD5whStLg a7k16PfsvU0LPKKuuS0a6Jhdo32htVLuUBimitrmpIodEIpK6u/pb+RqtzUxbaZRf6zBE1n+bDe1D 6GgErNixDdHDxiOt84woNkcVe13OAI/fEsmd5eySQnm/HrPRr8an7qxFOWLWBxciPI+VuQ/hTpsY3 W4nxsS6ss78gx9YQ7+33GrLGbgSNEbIivS+HP2d/VBVJRuCeFpOXKvF1hT/xNUkcsLkocselsfpum 7sBINZvnRXmGR4pntZ6TomzPmHvOB3aPDJ/UfS+KPB+01zgeTXFbKVhBNj2p6TWyQcVp+E5T+d9ZK fI1VH4vw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTR8-V0; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 09/14] mm: Handle large folios in free_unref_folios() Date: Fri, 25 Aug 2023 14:59:13 +0100 Message-Id: <20230825135918.4164671-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: trjrc1mgzbszt7kqtk97rojuwwkdt9kc X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7069B1C0028 X-HE-Tag: 1692971963-953821 X-HE-Meta: U2FsdGVkX19lbElWZzgtCkQXEeYb7mlNVroZQgfblpFKwZNQxcYQhnQYRE5VEjp+3YeMYg4yUb7cBYnKPyUUAPNCLDSK6bttLhuAiZhAIvSgmOiY+g6A7RpNNS/wORUBmCa9Vq3BzUhAnE9FRDsz6MItks7dKJqN1kXXVefa1EB6lDJfHJiNgKqCajAKEeZ3yZchbcxHqk3yrsBammmQGUyeHWsSatw0kidyW6j1ca0PQRt03+dk/AEWrDRLVg2MCdE7wKGL6HoejaBsTs91s+SZYovX2h1U6Q0pgaNQCXR07lFVRNoEhhdf/nDql/DZT4wX3Mje67odRW8pCPVn8eJ9+ejRfXHSBi+b0EJJtLplFdS0cO7+bo8kHSY1GA1brTkMYFLOXwyotcvWviM0UKsmk2+eSNwqAyEz+iGYvS74TjXJrBoKSXx+o+zMdwGZ7tq/NiZtqFft9KHv4fYcsQKRDH9dKWCYeObhX4PYCYlJHIuMCHUa44/IgeWqsPlfTFbgCd0WubkXbaNLx/YDS30C3FHeQQ5NlfkZAk5fDKUap25Imm2hAsDEzdaPnoV/witf27wrZxgQoD7h3QfKDi66I38fzCNG4bf/txk0TZVrR9mQzfPLbLPBJQ5MkvRpcqshhYG4Ys50PfUORJ3EvJizBzb6Vlj8c9xHMr4J6uRLqUCCHbxP7ol5+AVp9E/6fBUcQq6qK4jomemz0rUCoHoPErebzQdQiwgBCuYluVyOGFd+9r3ECOYj924ZwhSiJzmkVFWBa3yW1r+4NmcPzAKYaNhbgqqyRzszlv360xJmmqBiSsFaFpEzF/K+86d04o2nmgXQIv3lou83GxCQ03frHXONEjkRkFwAYYbM0z3+mfZW+SFRbq3w5jlNY+jxAgZY5lpTnqxxmQP9TQ3owTgKAV5nVeLmkHAC03vP/MtLimZOLj/35D2+lIkrLZsPXx7/eveLetbVJxLXB3z VZ6GYw7X mQhIRZ7YZE0luc74pSO/jK+Kk30b0YGuzHs2wE6+12hBy4jijspCInEYWeby1KTpjn7VGHlQX1RZYLHP8VBSqd8VylmToWP4Rf0VDoJYHsqDRJk4qi5t8VF3B7RZ9/C7MAYI0aZ485XeZz7vmSbIujWI97ERLuxVuSHD3aCL+Z0Dmf29j6A9YUfCdEIuLhWlwJ2OZRPIaaXAQq4R92aUFgn4kBAMo/KWkbycy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call folio_undo_large_rmappable() if needed. free_unref_page_prepare() destroys the ability to call folio_order(), so stash the order in folio->private for the benefit of the second loop. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index bca5c70b5576..e586d17fb7f2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2465,7 +2465,7 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a batch of 0-order pages + * Free a batch of folios */ void free_unref_folios(struct folio_batch *folios) { @@ -2478,7 +2478,11 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) + unsigned int order = folio_order(folio); + + if (order > 0 && folio_test_large_rmappable(folio)) + folio_undo_large_rmappable(folio); + if (!free_unref_page_prepare(&folio->page, pfn, order)) continue; /* @@ -2486,11 +2490,13 @@ void free_unref_folios(struct folio_batch *folios) * comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); - if (unlikely(is_migrate_isolate(migratetype))) { + if (order > PAGE_ALLOC_COSTLY_ORDER || + is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, - 0, migratetype, FPI_NONE); + order, migratetype, FPI_NONE); continue; } + folio->private = (void *)(unsigned long)order; if (j != i) folios->folios[j] = folio; j++; @@ -2500,7 +2506,9 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned int order = (unsigned long)folio->private; + folio->private = NULL; migratetype = get_pcppage_migratetype(&folio->page); /* Different zone requires a different pcp lock */ @@ -2519,7 +2527,7 @@ void free_unref_folios(struct folio_batch *folios) if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, - folio_pfn(folio), 0, + folio_pfn(folio), order, migratetype, FPI_NONE); locked_zone = NULL; continue; @@ -2535,7 +2543,8 @@ void free_unref_folios(struct folio_batch *folios) migratetype = MIGRATE_MOVABLE; trace_mm_page_free_batched(&folio->page); - free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, + order); } if (pcp) { From patchwork Fri Aug 25 13:59:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54E1CC3DA6F for ; Fri, 25 Aug 2023 13:59:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A83958E0017; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 22AA92800C1; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 704502800C2; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B973F8E0016 for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 910A412067A for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.09.AE64896 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id CC493140017 for ; Fri, 25 Aug 2023 13:59:24 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KYTHqTRW; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971965; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4Iv8ImNVozk3626iD9HCAdyqdorH5iCsCnWPrF0esRU=; b=oi6G7aJwq28BHujb+z7wXMjUOFQDwAiOiEWEZBRVa3mWfM8IZtwLYPg734AHTnHvi/byMO R598v7e7DctRN1is3QUEqKKrg77UbY/jBrfubBLSk0qNl/NLkT7aN1LdyOt2PkoSe3TB9l C8N6wb9qgXZttVbMxLHHm4QZJQKDSNQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KYTHqTRW; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971965; a=rsa-sha256; cv=none; b=B0vyOjdyAU678RRO4rdANkio3ngr8BY64kNP7X0GlfAXP26oCc55FuUtrfRdZhcnVTw4WR pw7NMjtbomDMB6edfoUXvq75aNDGqxQszKOfkjFgoOXJT76oBXRTvwe/3CcD6/XLSbt0SL +Kh6xMJroXCuRdrFpjJNkT29nwHfQnI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=4Iv8ImNVozk3626iD9HCAdyqdorH5iCsCnWPrF0esRU=; b=KYTHqTRWvVMmdXqO7qT5wXcFdZ tceHMO2eCIDiMl3GEAEAcX8Op8M1N9CZHzRRpW8VVerQNFcI7AnbpaxCeBqN5bWllDDtzKnvognRt M/MGqRbuMZSWcZM0os2IulywTcATT458+MwDG3m2pAStVWa3BtsrQjoiEjF5UE2KMauMoq8x9E/t2 raIhy9KXGwb3D7pYRMorFEwOkG1m1FM0UoZDoEQ86ebGQzqUu0JUjjwlr3rkceucYGlzVce3Vhndq t4DKlr4ww3Dhw73eLrqv8V7MTCoj5t0NkPfmg87vP0drBRiSIDxmF2mhJPrMd+iyQml3HgOThHUjf 5PL6SQEA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLO-00HTRD-2Y; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 10/14] mm: Allow non-hugetlb large folios to be batch processed Date: Fri, 25 Aug 2023 14:59:14 +0100 Message-Id: <20230825135918.4164671-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: CC493140017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: yedhdib5pkxrzxjsnb9y9k6stu5bn3as X-HE-Tag: 1692971964-500558 X-HE-Meta: U2FsdGVkX18RU0IVuwe8lL49SUs8hs1IDb24J8xTbWGwHYEjWiEUBhnMcmhAwpJML00Trpny7CqXDgGoEbPe9rsW1BQw2B//ucLzzuYgqqfzIlhwMOZfS83zxNSJldOB9bhAVskx40uh024JViWZoIIEsDCOvnhO7wQK9R2bDDcrKT8GAQcUz/LwzL6HzScmVcYVN7VtmAbmQ6iMmkIpex1ZAq5VfzHv6Z4du4UPK3hATl/9wPK+GxfFoWzyMBukVUD9DE0Z4qgi/Z1OMKeq926snh06+WHJKr2VhK1sA+H3XFtM/3pYDvraaCdBwxsbfaBirkkVeV4ujp83nUbYgeJK4RI8dTzdzTD59zXEfOTJgswYbbxq5VW6kJpqn60xBmeZolRZNNrqQzvKeHUk43IL4lTNPrB4i9QKRxlTx09ZnCy04peJ482gcwzQddnFKOeyddQPlOmCuG6185P9RDkIUG+lgo55OnLUIWQ51zsqxgtI7Uh6RvWEjshRwGfDSUg32xWEP5dAcwxgWhIBPKccd+dfR/qlB6wKEXI3wBXUXZ4o9jz01NtAgvJD5yuiev1TzY+El7lRS8eUuFhQVlw/A6hpCDZZwFY6QKZpkh7pUt6MuI8THNGasDIO38xUX3XFHsyStimH16iu+aDxFC1Gq2uUu5XCA95yf5EEp+iyezJMl0+4++K6iKoL81N1YUfWmuuJTxkvekSrPvg4OwTz3xv1bd9NwenppLGRldMQseYAjz5UhnTB6HUrTo42L40FQMXQTmBa9Qte6GGWelpsbiw419hoD53i2KBZ5rvGlyFZzLWHR6galbfcKkNoZvKyC5uJXHcNh/SAlnN0sU9JasWbdBugfPQZxXW6C7X4DhHOxG7+ssNJ6vczGgP2HI7fKiDvY3cQ/GLKV8O1EHn5zm3/qBSfbaUbcqACwPTr9ALLjsuk+4+DPg2IbLTnCOJhdxK7fiBII3x5FKa nLCyMLQ3 muxepq2VPJU4v0oRPSMHtyWzQrls0xn3dEZfwkVFUtZecnkjhaMah01LA3t9XTKa423Mm6Ir5UT6q9UXZIurlucudPjSFDiTQH++WGIb7S3mZfRObSmHdCxyGymsuenwtpCj2UeStPPKULJRt6ZtI31WeoLk+l+v3JVBEccPHhY3ZNds83akV7G36rw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb folios still get special treatment, but normal large folios can now be freed by free_unref_folios(). This should have a reasonable performance impact, TBD. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 21c2df0f7928..8bd15402cd8f 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1000,12 +1000,13 @@ void folios_put(struct folio_batch *folios) if (!folio_put_testzero(folio)) continue; - if (folio_test_large(folio)) { + /* hugetlb has its own memcg */ + if (folio_test_hugetlb(folio)) { if (lruvec) { unlock_page_lruvec_irqrestore(lruvec, flags); lruvec = NULL; } - __folio_put_large(folio); + free_huge_folio(folio); continue; } From patchwork Fri Aug 25 13:59:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ACE4C71133 for ; Fri, 25 Aug 2023 13:59:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45A672800C2; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB0958E0018; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E6742800C4; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B917F2800CA for ; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8D10B806F7 for ; Fri, 25 Aug 2023 13:59:27 +0000 (UTC) X-FDA: 81162784374.29.E0C5E11 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id D9FA9C0009 for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IeYsemVD; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xv0w05Np1RQXNMNntL3Ow/R7WXDn51tGEj4wuerPY4E=; b=V3XkdduQMYMtTjXVgURWKuq2ZaiiIwyJ9mgFuiKhvor/9/GcT0C3XWxMw3OYhJo5o1VbSD +EdyFKai2pa6zRs70or+6ZbQhChBZWvaI6dsgcetla6GkO+n9bTH//3IZk+eo8qHkP6YMf UGjSNePMZmCTUiuRMhTj897B2mXVkWw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971966; a=rsa-sha256; cv=none; b=5gRB430xBSSw1ItU8heUBWRw6Mjb6RPpVaHM8nIGaaFAes3QdvicxQmZN30ykb8DNKE6Vr FKhzeqxdIonsMfdMrw6663LH9B9KxtwR2415qsp3gd50O5rjq8gxbnVUynTPoH5fk4qMg5 PYo2YKJ9XeqMK9ytDK3BulgV5gc4vls= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IeYsemVD; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Xv0w05Np1RQXNMNntL3Ow/R7WXDn51tGEj4wuerPY4E=; b=IeYsemVDHdHrQVgR+egV4xparP MZ94nwvfPyHwekhRRuh2X08ija5wUaTPGc80nPeFOjSqDoi4fY4wfxd/j3Kb8158CyV5WyOJnat6/ 4EyjjNB3hOuToYOjc26X0aNTVi3IUhiljJYl4BBFyoLqQgBHDeSJs3xWGpBMRLKUS4WLkwgiUtOf7 Xfen5Ylccpt/fkJ4d7Szgxtrl8begsNI0SJF2DIOTOvBz+VrSU1zNyHttmgM2jWmXXuiLpRpC3m4D bI0hIl3rkZSwI5X8AT549IeEbNaOpHQ5p/HQ7hGFWPGxFIHgD+EHJyJzmylPnRxz++Fiqc5iPG5WZ WGtyN/tg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLO-00HTRI-6q; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Mel Gorman Subject: [RFC PATCH 11/14] mm: Free folios in a batch in shrink_folio_list() Date: Fri, 25 Aug 2023 14:59:15 +0100 Message-Id: <20230825135918.4164671-12-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D9FA9C0009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 7kmkkmzf6zkau5mznyy8s3s4o4w1df5w X-HE-Tag: 1692971965-626795 X-HE-Meta: U2FsdGVkX19LAFBPBTCuKdHEBCVKpo9if3MdbL/NWSzs5M2/Ut8TM1M+9SQyEIvJ7SWMfYkOpqm8jhFUw4J+5Byus+hOSGhOfoCUHuEKwqYJb5+z/cAOcxU1mo+cr6BivA5wPCi+6lylTI2zdZA3ndzE/SEFdO4boUbHU9rvO4oBpCPJWC+kymVfd2UT2YpSwUnMw0VbiusqjLAixI0RrRAMi0TlGB/QvNs3SkWwPHK5XGLjKoO5SajQfssQhboKuldctWRXkfFtUzmYM3QOCOz6aNqO0aRChrCi17tlXl5TZX6/RiFDlDws97pAtSY/tpCShcGSu8RJumAdlFs3SdvSKjwfTgYFx9x0gr0jNsz3yFx1BRUJQCjaqMW/ghfM8+ZJFnsoMqPQy/rUktv0EYIjaJwOSkR7nh096Vu/2rW6ZvlKt0aLlGhrPsZYgaUvN3G2boC5LNUYLxw3ZyRPM+uiZRdWcxN9YOMRNIoK/SXkSmvelwSerdkUM4M1Wvp+CabaI34iQz3bRzurqWGtYFQhAZxne4IpNz+oh4KHftEACKGV1u1ix43ki3yL5EHEZmLdgSdqsn4goPmToO/We5Gtw35ZuMuKPWDOruTqkGrBCgVscsFsJ2/a8l/vUve5tiPFT9VQQF8tupUUhnPa4gROWE3vCyh+XWBAuBTudvGu1WKJ80+cFR/TYOqeHYOhjgSpPZEnBXeV4YPZB8UtPH2fe8Du9IFYKLRidDRgEnvxnICR8N7drP+676O5lsBPS/B2iR9+cqJNb8+QuwCRse89+u58iTrYS3o2qcighB40xtrbasublMvrKCl2vo9FnNhvVjfFC4piLrJrhlFBJO3EnnzpguI1UcM3yp5oECKdPX7u/9qO0nnCgoFw677L7vpAJSwR5mL2SPKIqr5DKtY8uB7rlA6kseJ3icOMllMPNKgkxO/JHcKyqJglF9I+BnNbXd1I9MxTjdtaTcT Gtz2C2vN aRCjwvnsSE24u19VmUqn90x0Hb+oztwl5V8SlGBnDP4CgCmxJHAqgOMMhbT1vbxU+xy4W1JRH0l04vLrPbVwhCuoOpSobw53kfRWSlzSm7NtxwcflN56tz19efGkPDGtybh6xl+S/Vyj3X0TEs3lvRxisFCdoMdXl9jQ2kieH2bu5WHJ46HwTeAKnz+5b+qEYbYhek5dIai2GvdJUp+aZ0ihGfeskVdRPCxqi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use free_unref_page_batch() to free the folios. This may increase the numer of IPIs from calling try_to_unmap_flush() more often, but that's going to be very workload-dependent. Signed-off-by: Matthew Wilcox (Oracle) Cc: Mel Gorman --- mm/vmscan.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6f13394b112e..965c429847fd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1706,14 +1706,15 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct pglist_data *pgdat, struct scan_control *sc, struct reclaim_stat *stat, bool ignore_references) { + struct folio_batch free_folios; LIST_HEAD(ret_folios); - LIST_HEAD(free_folios); LIST_HEAD(demote_folios); unsigned int nr_reclaimed = 0; unsigned int pgactivate = 0; bool do_demote_pass; struct swap_iocb *plug = NULL; + folio_batch_init(&free_folios); memset(stat, 0, sizeof(*stat)); cond_resched(); do_demote_pass = can_demote(pgdat->node_id, sc); @@ -2111,14 +2112,11 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, */ nr_reclaimed += nr_pages; - /* - * Is there need to periodically free_folio_list? It would - * appear not as the counts should be low - */ - if (unlikely(folio_test_large(folio))) - destroy_large_folio(folio); - else - list_add(&folio->lru, &free_folios); + if (folio_batch_add(&free_folios, folio) == 0) { + mem_cgroup_uncharge_folios(&free_folios); + try_to_unmap_flush(); + free_unref_folios(&free_folios); + } continue; activate_locked_split: @@ -2182,9 +2180,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; - mem_cgroup_uncharge_list(&free_folios); + mem_cgroup_uncharge_folios(&free_folios); try_to_unmap_flush(); - free_unref_page_list(&free_folios); + free_unref_folios(&free_folios); list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate); From patchwork Fri Aug 25 13:59:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BA2EC3DA6F for ; Fri, 25 Aug 2023 13:59:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78B8A8E0018; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 691F12800C9; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 385372800C8; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DB1E12800C9 for ; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B44C0140651 for ; Fri, 25 Aug 2023 13:59:28 +0000 (UTC) X-FDA: 81162784416.17.58F2749 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id BDA104001E for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dgiWRr1o; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971967; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=92K/vCjYxmyDWMu+BIreMC71v+Kez53bhjphz+8AFr0=; b=Y/gM0JxuHYylgR0oQVfmNPEVzvIvTSA4ktfHk2EJjBrI2m+1GBYfGUNF0DGFBSD3AyuJ0i yEqtn88+WVNTc8ZGzyzsU+XxsvEcgHyYRROI4IlaicjneYR/443MKxqnK9vVVv5MPvHMOn tA+tXKCxJOrar0R2sdpXQxGhal0eDH8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971967; a=rsa-sha256; cv=none; b=0/WG/oc6+vlp/XGGucNiPox5aStuC/EdSb1Xc1KkL0E/0ubKik2mXeYyV1OlwzfgEWI6yB QEL7nm06l/WcJ8tN15AwwVM4kXYHwgdbdJsEJ8AEUKP1fLxLMU0IgoTf929AL+KHMYxl+U xtiBi+Fj7IoLj6iAIV5HN+957e7gUl4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dgiWRr1o; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=92K/vCjYxmyDWMu+BIreMC71v+Kez53bhjphz+8AFr0=; b=dgiWRr1of/ichTG1+kGip0HMsi 8qAZKCcIk4ZXHWzX9YABfjoVDZrnaVerKkZbXidkuDfInHD82l0W59JaFyr4drmQyAo2GSIPX7ebx tVfdRKiXRWSjuNLMF8E+EWS7FZi0wNeeG8amEDbyCpy1LQKlAWjW0xEu3WiIVY1Bfg5lYH+5NsMxk ygEwCxFaIH7R3lC42lemfA2wFNKQl5Qq3JsrFIsVFGX21yZMK5qJMxo9p4vKYoB+oKBoyqGZCLIki +9lCD5i7ks/p88uENxhkHZErPORN47vxYyzB0vx0slxBfpHtUT3d/GjhjTC/h1kyJj3d0t5mx2O9L f8472d0Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLO-00HTRN-Ae; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 12/14] mm: Free folios directly in move_folios_to_lru() Date: Fri, 25 Aug 2023 14:59:16 +0100 Message-Id: <20230825135918.4164671-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BDA104001E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: eq3dobjtcnnmt5w7znhqiefw9c8qy1wu X-HE-Tag: 1692971966-426 X-HE-Meta: U2FsdGVkX194tjh3hHoACEQ/PSfLdm8Y7XlwuTOmi6ZZ3eU5W5nmecaIouGyq7SZHHnh/NsXFHbyBpr0Xks5Q35XbAIwlXjpPhOJuAyLY4abodtIsNxuTzGnhonhITjluVP5hUyKcIpWpax2wO0deq/8wt02oQI4PkYLYzuT1tKKgf5yFuPEvmXBKUs8+fwx/bwqkrTfVE/t4xdqv2vZgCUGVX+LW9NDgrl6J+lAmdLbahC+TqjFDDzCDBOW1wzPlQYt3S4+o5sWYGGk4nP7bTCtB0RnYXAjtkBvA710tjnP5LavxbTQgXIIuxcubh6NqUvPaA9ZmbNjE32s6udNL2jTXGSoERjvJ1cTC8MYw0ZfypZ1HNKuQtRwxreiUmwqhOyGmcbFquDUAbYPMoG+6PCwsa3dO5S76j0AI3COx76/WHtzsybT6CfP2VO2xb+eLnsQT/xpqVtvXwWfHwWxpg6eLvwA222ZzC7+vo9r32mkL1eqEgPBWbuYGjaGoHRKiDa8rwGlFl/QGmNgpy4bm4xH8ENep0u0a12EeJ+e3I7X9Tf7cVoi4EFL+aYvzJQsagbsTCixKOWbBVviPpFOfVrwyDz/gIWtXBfj8vhn5ngMpAWom9iSHecDMVtTWifDT0pqUiFYpN+UHNvIHAopNUwcL5nXZmz63G+YIgsmm3gGq+RI9TEQc58zHTJI6oMa/8+sbhBgx+6uy6UZFdDBqLRC7bv5IT2QKbp5NdS7QSvZW+eITz4PKJqhkdUcM4Lnb3FyZItlwWP8LaYc4tSqjVA2Jqsbk2ZB95pYROH3By+39d1ji9zVu8u1/nUbpeVQlnNXWQD1zEb1fzvLNDprPZaCtyO8KJVUlRvJ5rdDszbcqz69mYkNNybHGuFHdpCzm+lXhyBWb5dOs3Fa4gvIwW8BQwDlq4SO+hPqr0VXYxG93Tf7yigeEuhV11MRz59bcZWnfwBJiWSkjr1wcU3 VWKnBgH5 yVzS2e9/6SU9GnSmV45qwawnHblAYDhJOYarVIwZTOlWIDjI27OHU2j5fYEpnk9OeWhGB/zsoOFwESJKAZSXuWrLIGWRFzYFJt4w4amO4orFVWIv+GppFB/tHRw1mEK+NNqoJO8TNAaQtctLtDn8OoQscVrxPgrthXKCLHlH2oCoUqleHnaqWY8EameYd2ZTEa4OOlWXLeVGARVLZEdMP52sPoyc61wRMCtXt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The few folios which can't be moved to the LRU list (because their refcount dropped to zero) used to be returned to the caller to dispose of. Make this simpler to call by freeing the folios directly through free_unref_folios(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 965c429847fd..d5080510608e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2489,8 +2489,9 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, struct list_head *list) { int nr_pages, nr_moved = 0; - LIST_HEAD(folios_to_free); + struct folio_batch free_folios; + folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); @@ -2519,12 +2520,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, if (unlikely(folio_put_testzero(folio))) { __folio_clear_lru_flags(folio); - if (unlikely(folio_test_large(folio))) { + if (folio_batch_add(&free_folios, folio) == 0) { spin_unlock_irq(&lruvec->lru_lock); - destroy_large_folio(folio); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); spin_lock_irq(&lruvec->lru_lock); - } else - list_add(&folio->lru, &folios_to_free); + } continue; } @@ -2541,10 +2542,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&folios_to_free, list); + if (free_folios.nr) { + spin_unlock_irq(&lruvec->lru_lock); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; } @@ -2623,8 +2626,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - mem_cgroup_uncharge_list(&folio_list); - free_unref_page_list(&folio_list); /* * If dirty folios are scanned that are not queued for IO, it @@ -2765,8 +2766,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate = move_folios_to_lru(lruvec, &l_active); nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); - /* Keep all free folios in l_active list */ - list_splice(&l_inactive, &l_active); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); @@ -2776,8 +2775,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); - mem_cgroup_uncharge_list(&l_active); - free_unref_page_list(&l_active); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -5238,10 +5235,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap spin_unlock_irq(&lruvec->lru_lock); - mem_cgroup_uncharge_list(&list); - free_unref_page_list(&list); - - INIT_LIST_HEAD(&list); list_splice_init(&clean, &list); if (!list_empty(&list)) { From patchwork Fri Aug 25 13:59:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB04AC3DA6F for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FFBD2800BD; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 137EF8E0015; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04F512800BD; Fri, 25 Aug 2023 09:59:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DFECC8E0015 for ; Fri, 25 Aug 2023 09:59:25 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BC260406CA for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) X-FDA: 81162784290.16.AD018C1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 099594000B for ; Fri, 25 Aug 2023 13:59:23 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rIaUdkDs; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6RVMpXkIv2GEOH3GCQLoyi5Dw3649Q70CypM6RwLya8=; b=XvUGDrmYeL3YWJXE4VL2uXFrFRiqYsoVlXmuVnNC3KX5ABXqhLbERYT4grHCgQTQ+ulStO TTJXp6fpwh3Sm0C81ykWTFccF+BLgmj0flhoCfAxxHWqB/9IZTnep6Q2KSWHSDDdQHWTw9 0tGQzhz3Nn9j4DohF6v6FRwwjX1JvkM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=k6FPW3XQu115YPjwutmHhcs+s2rQMf3yGrVnLs8J4+3PRyq1ilOe8tZAzdXgdawSnh5IHi 1UjrbvNa5IbC9W8t93+p8Axzes60mkcgqirp6Kz1K8ihZnWUAT08PUn8TJ98zRCu/LdBrM kjRmtlR0gU4sanq2bqHYVbgdEOq1DLo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rIaUdkDs; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6RVMpXkIv2GEOH3GCQLoyi5Dw3649Q70CypM6RwLya8=; b=rIaUdkDsipTHUWrd20cosBqepZ Pj6RialE1H5fxXcjwZRSB22u4V4AnFjXyNMS2A8ozN1ZJI40f8iLnCVa0Z85ewwc4nwny27FEWYHO npIkOV/TRmNscymXwKLuMZBfhHN1rd4nwzMYXa4tWQt7SbdZrQSCqH9A249ds/1GYv6gGb5wmn2wp Imarje4oFYZGxaXKklXZW+oca0qkBBa8xt4GGozdZdcldkqyJ6cSeMyOmBSuMVBmnVcrKyKR5V7Vr itoI2S/ZiXpS9Oq4FUyG/N/LQE+6E+QAUEs1SkVIkNnYd79y0e2YJ1s7to0f/pSNmCcPQgIY5LaFQ 5TpNNZCg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLO-00HTRP-Dc; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 13/14] memcg: Remove mem_cgroup_uncharge_list() Date: Fri, 25 Aug 2023 14:59:17 +0100 Message-Id: <20230825135918.4164671-14-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 099594000B X-Rspam-User: X-Stat-Signature: z1q31po1ef9jtr5dsxxr3dkyddhta63f X-Rspamd-Server: rspam03 X-HE-Tag: 1692971963-153913 X-HE-Meta: U2FsdGVkX19MlTMdwjpE9zEaNVMkIg4G1gfQiDPI9BVA7lwFnLKtKdfcN7x6WzzANL1YWvL7OHazJ4UVwSsbhb6HwrI5Zzvapel5r3EcLsO6S9YrbuIJG9836pe7r9zMNRSpvQG4O2jaZqe0XrQD20AvT8UmqQ1P2euTiC3+f6r2M5SPfk+NK/QZs5zr5E9x/WkaxQTpa9wcVp099i8pqpUVkqXfkPZGewGLcaPDh8tBJn7qRIsRQq2LkU0+LM04GKr4y5D3ld/R9iL4zWMsr9WeZsJPBAhVJhI8SZKWzyuCkEiARhxCtMZLapGHSojc6JFxk05lZco5s4H2Ma6vLh11Z+9VqA2qQlCtveA4TYg4jm7bwqFESm8DXQMwRNvnuNui5QFDuprUqnMkafVvEex+qzB4sjw3SZ9KpGFD1oJQL0UhgcHMio6LS7n+BdyEEJrL4NDXuMGA1gsRq0fEnj0U53/98Qf7arM6u2cyxkRYlo4Im6pvbq+mIyxRSmqyphcgFV7nsc73iaPwaHAxC/52M6O1ZGnJhO+WIF7UgGA7F543auxLBH7d58ZpvC+nl+t9XWr6CXROYaPkzqgz5UJJs+Od5vnYsVkJ7zqdEQ1Ix3znLgG/xbNWKIacnw1+KzpoF/mKXoQRk+yxhkttoJDWTTr+rWgY9UJNePwbyyW2itTuXq6S/0CkyUtgiBJgB1arVWAnlUy3MRqM5gfXneOgLU5+ceZFzHyOabJwFErfZfi+fvx7y7VWCzma29an0EM9wH7dGjXu6uHhdh3o2NlqzwU2GPUXmZXYYUiJWSNzdlIDmec+uW4VSuM3TV7dolMmziffPFCI97TsjcrU5nQ2Rb2EC/mNMs/0kwyWSEpVrMHouDTh/GRT9F4Q3iHM9OmDzxdg21qzmdoDkgQuVbKY+O0w1ONt3MUBOFIYuKtnykCefpwijaXKN5RBkoaLu+qroC15v/9xiBT3vYs AEj7SQ3k MYh0gZki3tVHMjNOankHmJB5NiAxVpllLmDlyzMaui7/y8qmccNgru/5BGc/aViaxm+9E2RvASPdwQ3wA57VdN+GgVmybPnwgShaj0sstqf6sfbUVS27ehc66TC5vyLiB0fkC2CoZFydDirxR0fUSED3bZJb61+lIlgk3JPCrmHFFm6AYoRpXBHIJytMbPIBIqQ/Y+M7TTF4h684dS6+j2fqov7ZDcpmRTNyg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All users have been converted to mem_cgroup_uncharge_folios() so we can remove this API. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/memcontrol.h | 12 ------------ mm/memcontrol.c | 19 ------------------- 2 files changed, 31 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 44d3e459d16f..2b3966ff85f2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -705,14 +705,6 @@ static inline void mem_cgroup_uncharge(struct folio *folio) __mem_cgroup_uncharge(folio); } -void __mem_cgroup_uncharge_list(struct list_head *page_list); -static inline void mem_cgroup_uncharge_list(struct list_head *page_list) -{ - if (mem_cgroup_disabled()) - return; - __mem_cgroup_uncharge_list(page_list); -} - void __mem_cgroup_uncharge_folios(struct folio_batch *folios); static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) { @@ -1277,10 +1269,6 @@ static inline void mem_cgroup_uncharge(struct folio *folio) { } -static inline void mem_cgroup_uncharge_list(struct list_head *page_list) -{ -} - static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a0fb008afa23..c89265bd9ce6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7172,25 +7172,6 @@ void __mem_cgroup_uncharge(struct folio *folio) uncharge_batch(&ug); } -/** - * __mem_cgroup_uncharge_list - uncharge a list of page - * @page_list: list of pages to uncharge - * - * Uncharge a list of pages previously charged with - * __mem_cgroup_charge(). - */ -void __mem_cgroup_uncharge_list(struct list_head *page_list) -{ - struct uncharge_gather ug; - struct folio *folio; - - uncharge_gather_clear(&ug); - list_for_each_entry(folio, page_list, lru) - uncharge_folio(folio, &ug); - if (ug.memcg) - uncharge_batch(&ug); -} - void __mem_cgroup_uncharge_folios(struct folio_batch *folios) { struct uncharge_gather ug; From patchwork Fri Aug 25 13:59:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BCF0C3DA6F for ; Fri, 25 Aug 2023 13:59:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EDB42800C4; Fri, 25 Aug 2023 09:59:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97E452800C2; Fri, 25 Aug 2023 09:59:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE48F8E0017; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A96C32800C8 for ; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8A5C3806F5 for ; Fri, 25 Aug 2023 13:59:27 +0000 (UTC) X-FDA: 81162784374.30.115A762 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id D086F40003 for ; Fri, 25 Aug 2023 13:59:25 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="PpiVS/4D"; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7uQnZSGn1BFpbxl4yWPTOmGkXbYPGr0+sIYyneFBp5k=; b=yob3gqnPpQewGOQMHmAuXfbZ76v405yozQbcsP4vlBJQvBwnlKcp/R/is+bSq5jC/oovJ6 VAgSh71HRHAFrB7NY1e/+WzJB46BAPrBOVV6HFW7EqZZu7aZ0/1UOCRBQZcGLi90twQeD9 XGQiuqFt+mlmNFT+/MevEJtD+dq4/II= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971966; a=rsa-sha256; cv=none; b=WxLkKdtZCtiOhcG5B0w1BCJVgQey+G4stBMvERuatMWB9KiN+925cQSHgASkX+qsQqy+P9 w7lKpdgRUeSeCw+zeyHve2dKMxkUhYWvDQt8q36NqvxiavRfi15aHvvRZ8xFWYHY9UXT5+ pWAmlQ/q952Keo8bbYb9ugGWK/FIg7E= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="PpiVS/4D"; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7uQnZSGn1BFpbxl4yWPTOmGkXbYPGr0+sIYyneFBp5k=; b=PpiVS/4DF2byb7Wyv1fIX3IB4a kzbnO4OH91GwXh4in4Cp3BG6iHKQdw+gkNGN24I1y4iRkB11eNiOG4hBxzF/R1Yu3ZWV7VHJGryxR H/KhyaFwFsaW1SvZ8a/RJ7Fu+nuE4t16jXetnnoXeeZxx08TrdzuSAUgZSypZ2L+iL8e77U3w7Rh0 m8xh2mWgb4kY15fOtN+xNhmmq9s4p3ilhyEugPvXZvIe5EpYdaj/wC8RKfkxwTcnu8RfeSH3XuKC8 VEh0vMyjN71+FDzuLGkbnTrxBEFmucCQD3Vho+wf/h7qCpzIPq6PumcwKbSmiIVMlXEpvdkoPB8qq 2hYGeSmw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLO-00HTRZ-Is; Fri, 25 Aug 2023 13:59:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 14/14] mm: Remove free_unref_page_list() Date: Fri, 25 Aug 2023 14:59:18 +0100 Message-Id: <20230825135918.4164671-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D086F40003 X-Rspam-User: X-Stat-Signature: 8yn31x1ah1dwizweqi3hkmq5f3f4acf3 X-Rspamd-Server: rspam03 X-HE-Tag: 1692971965-717278 X-HE-Meta: U2FsdGVkX19H2S0pgaaUzi+mS6qesJFPoquhhUABTDrNEq2bK3MYUuO5hpERhMQ2PNXQykK3ZE/HSDuvkcgW4rOI8m48e0iUmtx7cCSR+pr85ULKGRpO3D8b63dm0YhWHcsNCVz4rsH+Pqve99J44uPiXok2Gs70ifrABcjLwh2X/c68Md3uR3pjBsq8avfeSVTAULAMQEooHzUQhFW8bbOO05e6ktFUaMRM4+EQoFwusObOMewv/SGoK2Ew2N72DeX104QEY4RSjzxE5y1q4RSr2cyM6DTml9vbJi+n6bcrTpqSvAiLQLaMg5SvmpzjkggTjvy456Yu0FPV+XUGnaxMaWs0eknXtkYz0SW1TSkeAG/td4nCQipBO1tUBf1CvBCkcTtc+AEWkHl1uT/pzd0NnkwWgg7Nz2F7pNTmr6JaGPj09VJA4+AGHm3r1W4yRTEfjImbUXHP3O5jrR3aOt3alUNC7SvlgR2BBRIcD7nrmPJERnhYep9HKdPphqlxy94PqXNF1hgJhrF37MaF2QEqvwqpyc1Z3p+/z9GrGpAApps8zH5G5trCrFIeg+THyI4efgvqn81fvBtQg5HQz04qsni9qV8GE08IO62xqyQUX43tKyyshBsZvCddccthoxbKbSjHx411+Sh/SjiNZjznxjz4j3LJqdljJgfGSLx4mjuybJHfldu8NANp0tlchNNr0nVh8chgti41RWHgeUhSLh+MwafNla16j0GteYlIl4enbE5qhR/JT6paGG92xyJWZFyCA11/oobtNmNLlSYTofn0qCMXQM49tOLiDznn+hzN2GmeA/xytkVKCd/c4p6XbckwuHCZpktoQifsOJqxl27df/Egx+bTW/OKW9oI+FM54YIVmSAcCbkOSXMwJ9eZwZ039y9pd5Mpp91FbBxUPCPdPfwTgt2guCmmjUDzacFsYWsC2h/w3TN6S1cYoUfoEVZdPc+CWDxAe/Z Nvm7Tiyo mGUwNb+oomdoBl95/dzwZ2tHNrF/lj/2eoNlgVSlsppnYjkH7D5OXyCL5Tctwlffm8RCxEJq5i4cCM9DltqJn3uXMvBIt8xuobpGdAizii2mdwT2nxrLQ1gI58wLJjz7g6aC6vc69o0nLFy/jsjKSWcXRFEMRXYT6vzjS/jtriCOoi0L9Qb9XP4XweUVu8fdJmKqNKlmu+8QObd2/MXga3yGiCP+ztsSyNNOd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers now use free_unref_folios() so we can delete this function. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/internal.h | 1 - mm/page_alloc.c | 18 ------------------ 2 files changed, 19 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 5c6a53371aeb..42feb3f59ee5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -443,7 +443,6 @@ extern int user_min_free_kbytes; void free_unref_page(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); -void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e586d17fb7f2..496304014ec0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2554,24 +2554,6 @@ void free_unref_folios(struct folio_batch *folios) folios->nr = 0; } -void free_unref_page_list(struct list_head *list) -{ - struct folio_batch fbatch; - - folio_batch_init(&fbatch); - while (!list_empty(list)) { - struct folio *folio = list_first_entry(list, struct folio, lru); - - list_del(&folio->lru); - if (folio_batch_add(&fbatch, folio) > 0) - continue; - free_unref_folios(&fbatch); - } - - if (fbatch.nr) - free_unref_folios(&fbatch); -} - /* * split_page takes a non-compound higher-order page, and splits it into * n (1< X-Patchwork-Id: 13370637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0DF5C83F17 for ; Wed, 30 Aug 2023 18:51:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E383440180; Wed, 30 Aug 2023 14:50:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96BC2440178; Wed, 30 Aug 2023 14:50:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BD04440180; Wed, 30 Aug 2023 14:50:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5F935440178 for ; Wed, 30 Aug 2023 14:50:52 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1BBAE80368 for ; Wed, 30 Aug 2023 18:50:52 +0000 (UTC) X-FDA: 81181662744.09.34F11ED Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 748C0C0025 for ; Wed, 30 Aug 2023 18:50:50 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RkUUtf2n; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693421450; a=rsa-sha256; cv=none; b=tXnr37uzJ7MOI5frjnKvzlaViI05uT5thqxIFnyuEYlX+A/37zTcxm6qm9GFrfQQXidfHp r9d8Ahae6rGWvmceFchr7UrH2tpXoMxhUSyTP6LzCyc+uZAwbSzumqrQkzQq4DzeFJ2//1 zP7SUbFbrVT8mdxTpRTaBpKaWiz1Ml8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RkUUtf2n; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693421450; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7ZjXrp9mEKX41BRmyGl7l73x/r4SiJJJz5MiqUGm6og=; b=QrrA9kLj5Ws9MTVn+azBTD7mwwohzfxahQePYLmvVagIN3aj7oDZLHfXVh9oKxyY6X7s5S Qvs8SohkiE/n0az9X4AKqTyHWtZlvbPTBk2nVZWDFMBhc/mHxi7VAgDKOEW5NkMKlDOteq 5fSMNJPVkrAECJ7GBns4Mo7Hvy6K66Y= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7ZjXrp9mEKX41BRmyGl7l73x/r4SiJJJz5MiqUGm6og=; b=RkUUtf2nd9Fdmf0xmO2wp9xQW1 gdb6MnIk0m7hwPxCu6+AgwgUWpDXJiXU7ZPdnCvi9HinXLwWc2Sw3KWnNkXdwFER7TxyirXB5MU+C zOp4cAytg4X2ubcNJlSG2j22Ymu9WN9i1p7UnDZanPqqoruJ7n2skQGYhBr7g3buHWr85h6XudekH bgBvUUKvIvIuEaezMlC704RpwJldjYyKswDKKCMNIqXlOAUlEs0XDOBUMVxq8za/f45X9Wf4IiDrf 7Mx2BQq8AXZOe548yNG/2t3LGUuy9YeSA6VouaBNAvFulw5Rebr7JJ9jPcWjpSz1+pH8nnGcJKHEp /QurJKjg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qbQH6-00ENfU-U8; Wed, 30 Aug 2023 18:50:44 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Ryan Roberts Subject: [RFC PATCH 15/18] mm: Convert free_pages_and_swap_cache() to use folios_put() Date: Wed, 30 Aug 2023 19:50:38 +0100 Message-Id: <20230830185041.3427464-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 748C0C0025 X-Stat-Signature: 5zunkz71xy4t9y5zi6du3199pxyiknf3 X-HE-Tag: 1693421450-828471 X-HE-Meta: U2FsdGVkX1/4wmuxmp6VlDss0OOvEC1t7YzDWBUy/09Uif0srNozeo2cNTvJsFh4GEdabX+kQzxmZzaDKmsP1ymzU9AZKEwlZwWyJXoRKfbYqxi8zUG6MgZtTNlbfm5Cdy1IJNS6zkMc+g0SLq+FsVzOIWiAyhIn75ZT5I8rkEDGuM+DKsVQW8+a45aV2zwRto7TFu16ZiGBpHgLH7p3BV4jZSQeIfhvCKhQ7jOdpdZw4ds4CtzJzgv3cV+d02AHL160x4m4eqHOEWb8ofx1ZR//TpZxLArGUSu+Q0iiouAs48yiA4HEInJAsdYmNRzykNzMI8V6mrku4ar9t/Y6qTi2MyKPnWPDCWexOsdJCtw5zY5z7hjuGL3w1iTAhCRB0Q4/zmIrsnNdukGY8lQCHYB310or2KRQ1mRDhfW1PEpzBJ70QGkzHzYjwN0tF/vQe80ZxhYCIajJChGVRJRUrQ2iNeRRQytzCQi2Vnlr1igkcHRcrzY7vkB3d/cpzBR8wUW/SmGpZyF2Z2YyR5I43Yp/owiQ8kaoofmucsJaHeDM9OUz//V4jCCSB3ZG89JtC9B80DoDMRpCCXYcrut9qJySJ5og1X8ja621WCs3eltZ+ReG+cOVaJU5oG9MmvG+oSZxZUhxG6/eFYQzxbUSpeEqfedNdX0BuL6icK6oGSZU2k5KQdOGHUEp9Ze3ba7TmkbGKuxQk2Kz/Auy/SPo7VBxcwVMWmAwSuFf4+NrId21sh7KlkPgkp+pPuQLQ5PKZDVKICBl8/GKXlmFr9ZWZ+oneMjv9SYTY0k6WAR2/d7yEz9768PvouH6wNKIyoz4BqwtCI6sXVBepiIY82ZUj0trGguzGyGmLSOfm1yqN7ZNrr6TJL9eknfPG7CtEUM8rtH1m8QuCVuBp2jkzlkvZe9oE8DbIDrg2Gr1AbBVPESIZynDzomDgH6ur8Tig/sgaxgMAEL+5MYCrGFD1vA CvRI58px q6A5GC7aPAZA42MdSBpu/wX8ZggvVvBekstJTCHtB8Asrn3xkJgl5kLhUAqNBtmwVW4jR2UazCuSCnMGRF2a99Q7rH6eqQnBwiLjZLbkl8CJAYCKT9c8b849ibBdnbWjUGiFXtQvHxiS4dCGIBBcGOo+/ga+F5ziIk+4LihnVPHyeaqHC+fu8+jn0GZIWXXALo5wSbKC6n0R0KQOd96+xu3Aks6jkIY5PWP02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Process the pages in batch-sized quantities instead of all-at-once. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index b3b14bd0dd64..f68ddeb93698 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -309,10 +310,18 @@ void free_page_and_swap_cache(struct page *page) */ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) { + struct folio_batch folios; + lru_add_drain(); - for (int i = 0; i < nr; i++) - free_swap_cache(encoded_page_ptr(pages[i])); - release_pages(pages, nr); + folio_batch_init(&folios); + for (int i = 0; i < nr; i++) { + struct folio *folio = page_folio(encoded_page_ptr(pages[i])); + free_swap_cache(&folio->page); + if (folio_batch_add(&folios, folio) == 0) + folios_put(&folios); + } + if (folios.nr) + folios_put(&folios); } static inline bool swap_use_vma_readahead(void) From patchwork Wed Aug 30 18:50:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13370636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BF94C83F01 for ; Wed, 30 Aug 2023 18:50:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80C6B44017F; Wed, 30 Aug 2023 14:50:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BB36440178; Wed, 30 Aug 2023 14:50:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 636FF44017F; Wed, 30 Aug 2023 14:50:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4D4D3440178 for ; Wed, 30 Aug 2023 14:50:50 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2DC75160163 for ; Wed, 30 Aug 2023 18:50:50 +0000 (UTC) X-FDA: 81181662660.25.8F33D0E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 09548100012 for ; Wed, 30 Aug 2023 18:50:47 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ZSJuCTff; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693421448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aMVqL4Ehyjcg7Nve1A90aSK4UDcsnIcdpQ7y3+6SNlc=; b=T5ytr5Wt/muFNJwRfGOmndamQCeo7dFlQ4E+INmKvAPvdYMEjPAn5iVx+fHXQ6WLtAxTXn +FwLaeOdWejcOr63h/SgapUHtkX9xMAMIi1OKpiRRYwmHUlXoXg2Smo0vERSPwIeqeCEAO dcT465pglYW9GbZP5Rsjc7P15PXG1Ug= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ZSJuCTff; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693421448; a=rsa-sha256; cv=none; b=dF3SGAiEpUoYA0BmlFTwwNrS7pGGag6qQOO0l9IhDi/eZnITPahSPRQy7SC6Hj+9pZdO0c C2b8ThNipQgaduf2++3UVb32UARcumx56z0bR3DLF63teC0RjkXGKkxNF0R7MyhSnJ0u5a R+kz23wq7Yl3SVIYB/lDLlLnOxzyIGk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=aMVqL4Ehyjcg7Nve1A90aSK4UDcsnIcdpQ7y3+6SNlc=; b=ZSJuCTff4SjyU768LyDk0256Ip wEVPxTECHoUW1qcrQsjBfTVcCSzVxzXpx5AUvv1ojzKbj9XzuL3d7Ntp4Q3CLdmunmxmBGbz9A0+h 4VMPoVZzHQHDFsTSJRBoEpYXyqgIguPFQZ5LVhqI32IWQDwhNaRUJILqVjYUJr590WwHNfqWvDIW+ r1y8BJ6kPeyE4CwA3aSxExLauOH5O6cN5YDXyoB11dFv44p8nxntkW5F23d5T658S8Gu2Os6GGZky ysn7B9ek/svHSqct4yHwDIl0ZHNDW3TFoCujrxa4EXE79BjCTp8E1ZGdFnSzriqCN9cxIzQdWILTq uZh5WtJw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qbQH7-00ENfW-26; Wed, 30 Aug 2023 18:50:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Ryan Roberts Subject: [RFC PATCH 16/18] mm: Use a folio in __collapse_huge_page_copy_succeeded() Date: Wed, 30 Aug 2023 19:50:39 +0100 Message-Id: <20230830185041.3427464-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 09548100012 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: asx7ni56qe3j85ezgeu5xxd8qp3m3ikt X-HE-Tag: 1693421447-311595 X-HE-Meta: U2FsdGVkX1+W8ayEdeiJc8YU64iH7i8McNKuEySo0y+qhYoS6DTw+snAhBfFaDnEEFwTQn7ncqQ0RMWY+FogF4KnDmIzGdHld4yzYE1Eo6XJS5WwED3pcXnhMAIsrsf63fNJE+yRAnUWaO6NFrwjYk4qXICTQBmSuzi/sxRVnKrFfPxWdtlwlZZ+zIrmd/HsdskUzzzjEVKusvSdf9ZSFYAAew2PTTzz7MWw83m94GCEPDL9t2YlyBt0qVySX6G8iXlO4Wr1K1CX76eHFbsS+OIkK1WWwWSjQRGputk9u0kmOu+J4FUHxOd1A/vtQLhUBg1hRWeg2/zR90fC9LDuN+QAhuw+9ld/siFM14Pg0WzwD6subT0Gdi8ygmpUIj4+Bb7MzNuy9C25vtLj+59/ctWwQMUuE4e6JILgLmsQ5Veigf3LXR83IWVFXEjK8UVfyGMyEVNbJQVs2MJVvQd7o57wH2a/esCNe3WwFligTko0Y84DdR5db+irYkfnl77ze/6G+WXGm6E/szHcWHHtmg7HktoJzN3wRLmMFHxK/rCO4qErXebokqWCE3RlKPuW4wDi6Zze6LLkN0pUls4TwLkroWyyIwiybowvcv7rbNj2oR7EnT8z0EZuUq/StnUEtmnNNS9JFA6tvyp6qSQPKUOQt4jrw9q+NuA7Zt4mZZ79Cy2F79bWzFGnWFrtpGk+o1RqYYAsaWdNkiMqog8Qym4HOYYDnmrEJo6EVjWo6uuaAQWVZoN5RKGxGPBgOEg8LGnUzpBQtbtuTxB2v8exa75g4DdagkQ4VfY/iH3TNUPmUx7r4OsWCuh7Xq3bk6aUnsKJeezf21tpRU+08PVgt2rS4R5SWT2EpO2/VyQ2HjvpCHL0wedehin5O8ecWkmn5dV7w/tg0SsGoOJt75Z45Qp7+3Qb/vT9taScWjB655r875KgOe/pxiEf1uVx8KH0ZaeDkRAKNHtQ7Xjwk5J xPAw8qZV yMgtdsOKSa6ybiXMOvsCO9vHq3I03fxq6/sFLYQQy11lqLaHupokuciNFdE7LENW1w/zUiOMsxbt7QTZJDAiObVkaA3tfGUrZcEhg7RAP7Fn+Eo56YZvops4DmsXpCGJwmfz3cwOgAceuJ+YaFmTW0yBQdhL6LeHGGs1j7lqE+B7i7D5f9pfqkIx/rlFr3nbnJc0OmSmDBcl5JPlR2j3c7Jtht2ujkXqCH2zH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These pages are all chained together through the lru list, so we know they're folios. Use the folio APIs to save three hidden calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88433cc25d8a..afc94c281035 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -685,8 +685,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { - struct page *src_page; - struct page *tmp; + struct folio *src, *tmp; pte_t *_pte; pte_t pteval; @@ -705,7 +704,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, ksm_might_unmap_zero_page(vma->vm_mm, pteval); } } else { - src_page = pte_page(pteval); + struct page *src_page = pte_page(pteval); if (!PageCompound(src_page)) release_pte_page(src_page); /* @@ -721,14 +720,13 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, } } - list_for_each_entry_safe(src_page, tmp, compound_pagelist, lru) { - list_del(&src_page->lru); - mod_node_page_state(page_pgdat(src_page), - NR_ISOLATED_ANON + page_is_file_lru(src_page), - -compound_nr(src_page)); - unlock_page(src_page); - free_swap_cache(src_page); - putback_lru_page(src_page); + list_for_each_entry_safe(src, tmp, compound_pagelist, lru) { + list_del(&src->lru); + node_stat_sub_folio(src, NR_ISOLATED_ANON + + folio_is_file_lru(src)); + folio_unlock(src); + free_swap_cache(&src->page); + folio_putback_lru(src); } } From patchwork Wed Aug 30 18:50:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13370638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35C6BC6FA8F for ; Wed, 30 Aug 2023 18:51:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F3FC440181; Wed, 30 Aug 2023 14:50:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 17DFA440178; Wed, 30 Aug 2023 14:50:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC2F7440181; Wed, 30 Aug 2023 14:50:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CE9AB440178 for ; Wed, 30 Aug 2023 14:50:54 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AADCEB2B9F for ; Wed, 30 Aug 2023 18:50:54 +0000 (UTC) X-FDA: 81181662828.11.C566561 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id EFB5B20015 for ; Wed, 30 Aug 2023 18:50:52 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NYwRBPij; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693421453; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k6rqhfRbjD57Ec5QR1BxCnZ0WUyQvcR2ScEtufFdKsQ=; b=yiB+m7k1izPPLI4gbbqLLwaHflRdpjJBpZMrbn3A4YdfqoaQdn0ZJAAdaYo+gWKhPuZDBF GtAIZVoQljPV1IaGsC2RUWX6TUW3sLhAkw3WjUS0x6HTF64MfqvVeMwXAWRoOgpSQcdMWZ Ivn1KxlLuSC+ed7/6cgbkP5vYTzGvOM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NYwRBPij; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693421453; a=rsa-sha256; cv=none; b=cgezUzBymIUl78/TI1jso4KbJKoIWO0aHkdHjxMYIP4C/BUhILe3IZ+j/ytD58xyBriqW1 lPzxYZl+02D4C9voV5NLDFM2YUr8jrRFb6WTK0DK11qUWi6FF4BV+vEw/PjpJmeGxj19xa ID8gx5vQBLQjH/1KQTosYS0wBL4US/M= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=k6rqhfRbjD57Ec5QR1BxCnZ0WUyQvcR2ScEtufFdKsQ=; b=NYwRBPijPyuX2bSUPBgFLrE6q/ b/eVFDiBNj3PXDrVx7mIxlz2FJqsH2E4wThbAmFpYfoE5chXDBx0H7ahZMiDgHIbgOlOZnRIiksiT m0XlTeRQlEo5L1amWrMoE+kgLag/f4rSEIWZ8x8VLpW2GqqR8b2Ga3zCa7WD9nrZI+aanwuKy3c4X yjXd/WpuKNvLbHCWFyh8X8IMSC4pVfJAiueEamFtNpVRydLqL12ifM+R0fnvLwCTTH/zRIIVR9jRf 4z2pupJQMXqVOpg69LCvR3pytb9OspexFNdvfPbKZ5X/0U3WPNAaNA8FD619Hufi/UnfJDqAudM3z huo+suVQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qbQH7-00ENfY-5c; Wed, 30 Aug 2023 18:50:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Ryan Roberts Subject: [RFC PATCH 17/18] mm: Convert free_swap_cache() to take a folio Date: Wed, 30 Aug 2023 19:50:40 +0100 Message-Id: <20230830185041.3427464-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: EFB5B20015 X-Stat-Signature: 85bohfn39s9nfyto4jjr1zwg6xsbk8px X-HE-Tag: 1693421452-146076 X-HE-Meta: U2FsdGVkX18Mj84wbixcoR7qftmGe0nBTQM+GZ0qkzOSNJB6wJ0ic4qpaZ9roGA62bO/p2toK3RTmMDnrfpbhLVPby5bhw0t0OWqztCnv4E+s1k1AvPW17LkjPsQ+yf2kxTECsdZ2xKrYLBIIbd8mHQjnZAZZNOQtyImYo8zyUqkgUpcwHVdOTFedl9KCqU35DsMi7UvOyWjkfcTM3nbRI3G3zH20qeI0Zt8v/WFcu2t9iC9Pcu/NzZyIlveOrHo5Hgh6YPqqUm2ZVNTMY1JTblTq1LPC4H2eiz9OrJPg2HE08Ytj25MQrF3bB0MhLUL6uQIHHebUWa30eo/P7V85ym2p3JhV7EeLIADB84sD9KWLx7+X7nCR3TYwI+h++l+curC09fWI/G8CVl29LEK1WmhHHOvSGRthj2Za7N+BzXJQkYwns+AjDE9XlwJnYBnP5U0EG0zWS0AOyPO7TzbQeZc7wgRa2YVxM9HDi/UEaHhu37FAdEJF9U2h9Ga6RowYywoUszv/p/rrlhprmkzc5M/UO3H6oefo6cfZmpzF2LhDjzgRH+wRqqzslacYB+dF92qEgEmOuRfBVUyZPcTBkffVnrkDAtAbGDOLGASJqbY4AcOn4pUZklG4bIv6bZ/CpSki6/0QLF9QH23n8QDkPNPo+F5naM+2lXzSaIdxE7a/223+USVUh7hbPP60guLUNAPk4sTByUPAD+ExXBJp9/3xQ7DHBHls8dUkS0RqNsQYFcpY6FDn9CE8baalYAcqF7v0Rj0/lMJfD04HNabNwEObCo8lx8px0MNDbqVJGF4xcjzdQFtpfHBAMgeerAwT7cX6HSLmMP7jaZThds7zvKSRTR4sYe7/jYUN5YxBhVETRK8IXYMvBG68k+tvG3zJBW+27z57H4YieaaF8otPudpCiqJMaVBTc9cQbIoxsiRhvXqEAi283/ov40xfCzFFk9cQA6/hBErrwCaJ6e v/2RF1m1 PQYhn/ewf8WYpjw5F8KnU3kgoxYhsgXglsWrM27aJ8YWTDNkyjcAMYEE0Q0Riwmj40zk6+Uzn98EJEFOyQKVayX3JDCRtWcsBgo5f7IHhnspGaWioabWsOMMRGriu5q+vx1/Bp4eRhmvBwGtzWQSDn/6UzdZX9aC5avLYc1vvz5nkIEFzAQhvULIFk/iY9IhvpGSNVdd2fBWtShvAXs97Tp8plrX4ke6Z9HsdwMG3eMkGNXN+OiN8Azm3EE31W3tsL05tcvjr2SQ4BH0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All but one caller already has a folio, so convert free_page_and_swap_cache() to have a folio and remove the call to page_folio(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/swap.h | 8 ++++---- mm/khugepaged.c | 2 +- mm/memory.c | 2 +- mm/swap_state.c | 12 ++++++------ 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 493487ed7c38..3536595e3bda 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -449,9 +449,9 @@ static inline unsigned long total_swapcache_pages(void) return global_node_page_state(NR_SWAPCACHE); } -extern void free_swap_cache(struct page *page); -extern void free_page_and_swap_cache(struct page *); -extern void free_pages_and_swap_cache(struct encoded_page **, int); +void free_swap_cache(struct folio *folio); +void free_page_and_swap_cache(struct page *); +void free_pages_and_swap_cache(struct encoded_page **, int); /* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; extern long total_swap_pages; @@ -534,7 +534,7 @@ static inline void put_swap_device(struct swap_info_struct *si) /* used to sanity check ptes in zap_pte_range when CONFIG_SWAP=0 */ #define free_swap_and_cache(e) is_pfn_swap_entry(e) -static inline void free_swap_cache(struct page *page) +static inline void free_swap_cache(struct folio *folio) { } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index afc94c281035..7b83bb6a1199 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -725,7 +725,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, node_stat_sub_folio(src, NR_ISOLATED_ANON + folio_is_file_lru(src)); folio_unlock(src); - free_swap_cache(&src->page); + free_swap_cache(src); folio_putback_lru(src); } } diff --git a/mm/memory.c b/mm/memory.c index e35328c2f76e..2611d0fa4465 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3194,7 +3194,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(new_folio); if (old_folio) { if (page_copied) - free_swap_cache(&old_folio->page); + free_swap_cache(old_folio); folio_put(old_folio); } diff --git a/mm/swap_state.c b/mm/swap_state.c index f68ddeb93698..e6b4a00d3655 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -282,10 +282,8 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, * folio_free_swap() _with_ the lock. * - Marcelo */ -void free_swap_cache(struct page *page) +void free_swap_cache(struct folio *folio) { - struct folio *folio = page_folio(page); - if (folio_test_swapcache(folio) && !folio_mapped(folio) && folio_trylock(folio)) { folio_free_swap(folio); @@ -299,9 +297,11 @@ void free_swap_cache(struct page *page) */ void free_page_and_swap_cache(struct page *page) { - free_swap_cache(page); + struct folio *folio = page_folio(page); + + free_swap_cache(folio); if (!is_huge_zero_page(page)) - put_page(page); + folio_put(folio); } /* @@ -316,7 +316,7 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) folio_batch_init(&folios); for (int i = 0; i < nr; i++) { struct folio *folio = page_folio(encoded_page_ptr(pages[i])); - free_swap_cache(&folio->page); + free_swap_cache(folio); if (folio_batch_add(&folios, folio) == 0) folios_put(&folios); } From patchwork Wed Aug 30 18:50:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13370639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AD78C83F01 for ; Wed, 30 Aug 2023 18:51:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65EB7440182; Wed, 30 Aug 2023 14:50:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60E7E440178; Wed, 30 Aug 2023 14:50:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 488B6440182; Wed, 30 Aug 2023 14:50:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 31518440178 for ; Wed, 30 Aug 2023 14:50:58 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 00E30120375 for ; Wed, 30 Aug 2023 18:50:57 +0000 (UTC) X-FDA: 81181662996.09.0E38F59 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 5A195100018 for ; Wed, 30 Aug 2023 18:50:56 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=CZer6hyz; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693421456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AmmffUWi4DDP1uL4h7Uj4Rayaqkr0URd3IDmGP6kFS8=; b=Ig1E2KAU34CZb1W7fMW4Bg50G1XugugcpypiIMxMdHTVCnw74nt1k+qsBje9l0JYsvBbdc Qy0PVPij7HICTRDMaAq2W27OTusmWv4JzJjQ3HBbItaKm7CHE4PHzynLXvWUotiSRxrijG zEAttCA/wq/B6USJ0w9+4iHq/S4cAo4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=CZer6hyz; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693421456; a=rsa-sha256; cv=none; b=1FRk0SyS+ntbMng1QLAMrqs0kEhZVkBx2fAclGFY98ZN+MByTd+HHmkX/dZNGVI+wke5/+ QoIxCc8zvVhTPt2wDWtOgfIwPUzoLHDw7BrENFxSeBVQ83Rg+ru4edxEW9rcCBXeGqnSRv r1hACcIan4H2Hakeo9LSjGs4EKA1qXM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AmmffUWi4DDP1uL4h7Uj4Rayaqkr0URd3IDmGP6kFS8=; b=CZer6hyzIpnLeFvk7OZ9qZ5xVe wpZXsu+pN8c3RtnEt35mg4N4idC8D8WIWzF/DMrBkSor14+zxeaun3M9gc0hcrhxnxe/9oAQ61SFW BxbfophGEA23n1utvDLLquDw4aYQ3x17R4nfigFGb79l30ohpPiriVjyBnWPb8VTw8FQsvbYCx7jZ mMrkVQkgkRbN0HaJc4h8pQvWVurZNeWGyhznS9tTa6Krgl7QV2CXnsifN/vVQyNxZ4N1Eq47320LE AhjMVHw5F0Pc6BkzcDQZ8kbR3TVTc4rp+XtE6XeDt7NX+kex99z/8L4iMHFyrQMHpAOQYhelyJoRW TvcofpWQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qbQH7-00ENfb-8Q; Wed, 30 Aug 2023 18:50:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Ryan Roberts Subject: [RFC PATCH 18/18] mm: Add pfn_range_put() Date: Wed, 30 Aug 2023 19:50:41 +0100 Message-Id: <20230830185041.3427464-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5A195100018 X-Stat-Signature: k8wh3bucfgi1cpa5uyqrxw8j6e6xjxeu X-HE-Tag: 1693421456-803791 X-HE-Meta: U2FsdGVkX182pHJ5aJnr6F+x91JJ8ZluMXob3S3CuByFuIHw5YR+NVePaSqy3FCFoJt8ZPL40dejxdOZorSznUiXlsUG8Cs9U7yVVhjH1hWm15NL3Y3UrbCGQnbWSA06BhoUzwU0XAX7uc245sEGv6PISj1ptVArWV3SGZz9fz44ViIDQMk5RmHPwd9QQFt1YMZgOrpItbwFaG/3taLpuj5WGp3NR7CbJOeSw4UaMojvgFxKlpuaROVWwdABMDqrSRVGLzhwSJKbsOzA3mDLoyStTJwt3lEdfKN9q3PZ/jSpnQnvS+rqCJUB9mk+oGIqx6qYCOUyA1jLCrxpuaS20ecKun2T1GJb7YsC4APDLmIPu5ttvNguUMXsDsCVkT6UPEq5Zp4GnD2AvF9HndkF3EXhrZwVtEPmj6psyTOduRO08+IaYx7yjD3Gu4mqLP6x4Mo4pO+OEoyeQxcmhfz90LkqT25Vy8gBpxqd9HPut8us6C6xkfbtD4VLgWYorzF0nK+WJJ6NgF5UdglmQsyugcJVVuToF8oJdo1GCAbr56krOijjC0MkUEQQHfbY7yo80xF9bGPhSVX1fP1eIF+b4LB7cViMLR2TGd1i3r5a6Zl8RWRXjHeTYWpU1l+dHSTsGO9EmZZnky+hhbPmdSW7skCtF1V3N/9fiHQ4PHoxdhnuNxJfWND0kSgsHcBbrpJlO6/jIpXDnpbeJ4NNmfa0/P85R759MAxSd9PH2S69JSAsGBmiUqH4i6/lOYmFzDQYnko6KQtIREtVddLNQTRzjvTCLDjGRZcdL2BZ2L1OEWrpmEajZiYVdH9h4w4vdy7B4NWMfzUt6A0yEP4DCcIBl7RDGI6WZvxXUq6n7QGdmF5E+s9IQ1OnV+lEmGQ3MK2SahesdNaN+vQvDrPwQAHWRVrVMndYjTMs2QG6OTosz46furXAkgSPBroiDG36MvFLtItAw+hLAsk/3IPmWK3 pAf8NQgQ pf56Cjzxt/fIcHjYjjDlyq2nX8Bi/xCR37NCCeE2fB//wtVDK7oXqujN+H1KO+eYioF0M6Xi1gMWKHunbaM+K9M37wYUtT3v/hOfv79Cxg3ST2uOMF8T5K2j14laVETgAvAuNGURdyeKmHszje/KQ2nA7MA+mVLchg5Ms6ig7B+CYN5b0+YYF1u7I/UHUYR+8zFiy0FLDIH0aSRtQaKEGcPwCAB3wQOZx8LcX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This function will be used imminently. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 7 ++++++ mm/swap.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a517ef8d2386..4f9e2cfb372e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1515,6 +1515,13 @@ typedef union { void release_pages(release_pages_arg, int nr); void folios_put(struct folio_batch *folios); +struct pfn_range { + unsigned long start; + unsigned long end; +}; + +void pfn_range_put(struct pfn_range *range, unsigned int nr); + static inline void put_page(struct page *page) { struct folio *folio = page_folio(page); diff --git a/mm/swap.c b/mm/swap.c index 8bd15402cd8f..218d2cc4c6f4 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1027,6 +1027,62 @@ void folios_put(struct folio_batch *folios) } EXPORT_SYMBOL(folios_put); +void pfn_range_put(struct pfn_range *range, unsigned int nr) +{ + struct folio_batch folios; + unsigned int i; + struct lruvec *lruvec = NULL; + unsigned long flags = 0; + + folio_batch_init(&folios); + for (i = 0; i < nr; i++) { + struct folio *folio = pfn_folio(range[i].start); + unsigned int refs = range[i].end - range[i].start; + + if (folio_is_zone_device(folio)) { + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; + } + if (put_devmap_managed_page_refs(&folio->page, refs)) + continue; + if (folio_ref_sub_and_test(folio, refs)) + free_zone_device_page(&folio->page); + continue; + } + + if (!folio_ref_sub_and_test(folio, refs)) + continue; + + /* hugetlb has its own memcg */ + if (folio_test_hugetlb(folio)) { + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; + } + free_huge_folio(folio); + continue; + } + + __page_cache_release(folio, &lruvec, &flags); + if (folio_batch_add(&folios, folio) == 0) { + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; + } + mem_cgroup_uncharge_folios(&folios); + free_unref_folios(&folios); + } + } + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); + + if (folios.nr) { + mem_cgroup_uncharge_folios(&folios); + free_unref_folios(&folios); + } +} + /** * release_pages - batched put_page() * @arg: array of pages to release