From patchwork Tue Feb 27 17:42:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7B66C54E41 for ; Tue, 27 Feb 2024 17:43:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7696E6B00EC; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 162C66B00F0; Tue, 27 Feb 2024 12:43:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C39AE6B00EB; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 633FC6B00ED for ; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D30951A0C3E for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.07.945E2C9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 400164001F for ; Tue, 27 Feb 2024 17:43:02 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QoNWEjen; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=d+kutsoFda/NSdsTOP1r0c/mceouc0WN/VuQjxo+B7M=; b=7v5AeUe6iFN84Jt8e4OzwL0keDWtZDEX/KvbQ8uCry6c3LVdlcGLBufRyoRuhBsyylPSvC 2eWDT7vDVMcfxoD5PmQ7+uzJl6KWIfw7WiSCUQZ0D+UJBpnxAjJiFS7HR2NfFgd2XN227M KbMzX8zWPjPHe2IyTwSuTAqgoOhNgAQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=tAbOXs8k3ZaM/JRtgM02mWQcv9FmzymCuLGWcSf9H1fgpjn8yU0nV/fIxiRUwR+vKsQYwb XZBQhCUx1QQF1F1UbuzvkGymg+TlNWjfF5Lszj/yYfTNXZDRTg5XVtVvRfUF6xwfSUgIzk KhwF7m6w+pwlPO8fy3ynq84yw9g00dI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QoNWEjen; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:In-Reply-To:References; bh=d+kutsoFda/NSdsTOP1r0c/mceouc0WN/VuQjxo+B7M=; b=QoNWEjenC9B0zYez3+G6CnZHeZ rSPocvsG6L02153VRXGfoxA4s4Y0pZAQhSd4lEh59LTjPgdJG8v3s3XF4IobpY2+nXZAQRM68Ie7j yzdrAVs/qS/bfkzd7zH9R2nU2d/X028//CYlDNe7/cgdJAg83FyKug4OIbJh35TK8Eia4HWFtX76O nY9206TawC/1w5ptYZsc8bLXokJJs+Hqb8TpCay6W6DBtJ7oUjBthmu942gm0rFcVROQ3FaIISEdA VRyIbFJpvObm8mmH/OsUWdc9VgIwp5C12sqmlUhesYWLl1kDrGnuF1w+5IdfftQ/LTi4tp7X4ynxu qpddpDwQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tk-00000002yqm-034X; Tue, 27 Feb 2024 17:42:56 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 00/18] Rearrange batched folio freeing Date: Tue, 27 Feb 2024 17:42:34 +0000 Message-ID: <20240227174254.710559-1-willy@infradead.org> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Stat-Signature: u4ot1m587x6izby35ckz355k9xby3er8 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 400164001F X-Rspam-User: X-HE-Tag: 1709055782-797391 X-HE-Meta: U2FsdGVkX190nEJ7w+tlFKZdF1J9CepcXqR7frDXHpd0SDQ/3qpyJRARdly+m+Ts5qeLYo8p7ecE01ekbIOA6oTQcV8WnEMzocEb3vnrOyNA5kYukwXyBUDfFf35c125/jD2HoUxY+lR0oNKG5SjCZXiS0s/icpLkUN00pAVbo4qJuKsCy3UnGP7DP+W1Ao4nbCR7lnxJp1+qfShfz2KR21Hjq/Akwm5N7pZteMTAbZq7h1iwU7EcAnzh9dZwUdxsKVlOPD9OO7K3eDVKExy19UvMWVh3a8Axoj2En1O+xSxvo30y/X+rB4X+zMU+vaFX0qNPiy6JDMRCQthBcdEmPpz2/hRUYUk6e1eepYDDzkWxZoO0CKKU4VaG5GR8+Wgj1Z4p0IA9vwpJ7AkJEWLGumFb0hfd00DZxgFJxp9uHPaDElYMKdYsfAcfKLg41mza22FP7oK4TZ4zjR5jFYrvQneKz/lx3azKwLR/m0qL25p5wt1Vwnuz3gmY+icRZx11da4VLX/esP67CUb8MiqmKGV1TiOUYq0we6K1gLLr7qHXdyiJSY15Zf7qWhJs7y5nTJk4bcgLPbZOr3cCMBD7i83GFGCVGZ3wBYSKDypuC5j2jQp3xlt8GK7QDS2KrkEAlyAQNGQXv5X/JA+5gduDqxFz+TupcAu98A6GdnrFYBrGX3N3Ntg4Qs0gsLqO1e8zvuPdFY5iSFfirpFDXgbPzgzkihJvGBp33hwuUaOXvaQe6aIFcIBaOL6rGXDRJNdI+DraSrroLmLfQAR5O7gmYMxzyu/m20d67d51efIgg5zmPXNCu/Ax2Z9YATzpKI08fmJOPBzpG12DJ8i5qGxxgUkHfS92sGJIX9eA+XvA8NljdvAyY4do7Qdt2vmxuhweJwkCyd9qzxWvAqVxfT7HCSzZrcEw8v+Zqg39EQgcF6eZLSb6L9KvM5J/ovczmytdRHyydyGNPr9u7FjQiz 1T1JGg85 VMO/a97rKyqsGjLWTXiCYfx11cWrjcmXSbNnvZAapVGnauvE6yOAgM3DH0H6c33p/Tp/2u9x18GssMfZBsyFzbosdJU1NyNs61zyUgtSXOvZCGgkFJGY4ZMYdlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Other than the obvious "remove calls to compound_head" changes, the fundamental belief here is that iterating a linked list is much slower than iterating an array (5-15x slower in my testing). There's also an associated belief that since we iterate the batch of folios three times, we do better when the array is small (ie 15 entries) than we do with a batch that is hundreds of entries long, which only gives us the opportunity for the first pages to fall out of cache by the time we get to the end. It is possible we should increase the size of folio_batch. Hopefully the bots let us know if this introduces any performance regressions. v3: - Rebased on next-20240227 - Add folios_put_refs() to support unmapping large PTE-mapped folios - Used folio_batch_reinit() instead of assigning 0 to fbatch->nr. This makes sure the iterator is correctly reset. v2: - Redo the shrink_folio_list() patch to free the mapped folios at the end instead of calling try_to_unmap_flush() more often. - Improve a number of commit messages - Use pcp_allowed_order() instead of PAGE_ALLOC_COSTLY_ORDER (Ryan) - Fix move_folios_to_lru() comment (Ryan) - Add patches 15-18 - Collect R-b tags from Ryan Matthew Wilcox (Oracle) (18): mm: Make folios_put() the basis of release_pages() mm: Convert free_unref_page_list() to use folios mm: Add free_unref_folios() mm: Use folios_put() in __folio_batch_release() memcg: Add mem_cgroup_uncharge_folios() mm: Remove use of folio list from folios_put() mm: Use free_unref_folios() in put_pages_list() mm: use __page_cache_release() in folios_put() mm: Handle large folios in free_unref_folios() mm: Allow non-hugetlb large folios to be batch processed mm: Free folios in a batch in shrink_folio_list() mm: Free folios directly in move_folios_to_lru() memcg: Remove mem_cgroup_uncharge_list() mm: Remove free_unref_page_list() mm: Remove lru_to_page() mm: Convert free_pages_and_swap_cache() to use folios_put() mm: Use a folio in __collapse_huge_page_copy_succeeded() mm: Convert free_swap_cache() to take a folio include/linux/memcontrol.h | 26 +++-- include/linux/mm.h | 17 ++-- include/linux/swap.h | 8 +- mm/internal.h | 4 +- mm/khugepaged.c | 30 +++--- mm/memcontrol.c | 16 +-- mm/memory.c | 2 +- mm/mlock.c | 3 +- mm/page_alloc.c | 76 +++++++------- mm/swap.c | 198 ++++++++++++++++++++----------------- mm/swap_state.c | 33 ++++--- mm/vmscan.c | 52 ++++------ 12 files changed, 240 insertions(+), 225 deletions(-)