[100/114] mm: optimize do_wp_page() for fresh pages in local LRU pagevecs

Commit Message

Andrew Morton March 25, 2022, 1:13 a.m. UTC

From: David Hildenbrand <[e-mail]>
Subject: mm: optimize do_wp_page() for fresh pages in local LRU pagevecs

For example, if a page just got swapped in via a read fault, the LRU
pagevecs might still hold a reference to the page.  If we trigger a write
fault on such a page, the additional reference from the LRU pagevecs will
prohibit reusing the page.

Let's conditionally drain the local LRU pagevecs when we stumble over a
!PageLRU() page.  We cannot easily drain remote LRU pagevecs and it might
not be desirable performance-wise.  Consequently, this will only avoid
copying in some cases.

Add a simple "page_count(page) > 3" check first but keep the
"page_count(page) > 1 + PageSwapCache(page)" check in place, as we want to
minimize cases where we remove a page from the swapcache but won't be able
to reuse it, for example, because another process has it mapped R/O, to
not affect reclaim.

We cannot easily handle the following cases and we will always have to
copy:

(1) The page is referenced in the LRU pagevecs of other CPUs. We really
    would have to drain the LRU pagevecs of all CPUs -- most probably
    copying is much cheaper.

(2) The page is already PageLRU() but is getting moved between LRU
    lists, for example, for activation (e.g., mark_page_accessed()),
    deactivation (MADV_COLD), or lazyfree (MADV_FREE). We'd have to
    drain mostly unconditionally, which might be bad performance-wise.
    Most probably this won't happen too often in practice.

Note that there are other reasons why an anon page might temporarily not
be PageLRU(): for example, compaction and migration have to isolate LRU
pages from the LRU lists first (isolate_lru_page()), moving them to
temporary local lists and clearing PageLRU() and holding an additional
reference on the page.  In that case, we'll always copy.

This change seems to be fairly effective with the reproducer [1] shared by
Nadav, as long as writeback is done synchronously, for example, using
zram.  However, with asynchronous writeback, we'll usually fail to free
the swapcache because the page is still under writeback: something we
cannot easily optimize for, and maybe it's not really relevant in
practice.

[1] https://lkml.kernel.org/r/[e-mail]

Link: https://lkml.kernel.org/r/[e-mail]
Signed-off-by: David Hildenbrand <[e-mail]>
Acked-by: Vlastimil Babka <[e-mail]>
Cc: Andrea Arcangeli <[e-mail]>
Cc: Christoph Hellwig <[e-mail]>
Cc: David Rientjes <[e-mail]>
Cc: Don Dutile <[e-mail]>
Cc: Hugh Dickins <[e-mail]>
Cc: Jan Kara <[e-mail]>
Cc: Jann Horn <[e-mail]>
Cc: Jason Gunthorpe <[e-mail]>
Cc: John Hubbard <[e-mail]>
Cc: Kirill A. Shutemov <[e-mail]>
Cc: Liang Zhang <[e-mail]>
Cc: Matthew Wilcox (Oracle) <[e-mail]>
Cc: Michal Hocko <[e-mail]>
Cc: Mike Kravetz <[e-mail]>
Cc: Mike Rapoport <[e-mail]>
Cc: Nadav Amit <[e-mail]>
Cc: Oleg Nesterov <[e-mail]>
Cc: Peter Xu <[e-mail]>
Cc: Rik van Riel <[e-mail]>
Cc: Roman Gushchin <[e-mail]>
Cc: Shakeel Butt <[e-mail]>
Cc: Yang Shi <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>
---

 mm/memory.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

Patch

--- a/mm/memory.c~mm-optimize-do_wp_page-for-fresh-pages-in-local-lru-pagevecs
+++ a/mm/memory.c
@@ -3294,7 +3294,15 @@  static vm_fault_t do_wp_page(struct vm_f
 		 *
 		 * PageKsm() doesn't necessarily raise the page refcount.
 		 */
-		if (PageKsm(page) || page_count(page) > 1 + PageSwapCache(page))
+		if (PageKsm(page) || page_count(page) > 3)
+			goto copy;
+		if (!PageLRU(page))
+			/*
+			 * Note: We cannot easily detect+handle references from
+			 * remote LRU pagevecs or references to PageLRU() pages.
+			 */
+			lru_add_drain();
+		if (page_count(page) > 1 + PageSwapCache(page))
 			goto copy;
 		if (!trylock_page(page))
 			goto copy;