From patchwork Tue Aug 20 09:48:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103489 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D9421398 for ; Tue, 20 Aug 2019 09:49:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2B8DB22CF7 for ; Tue, 20 Aug 2019 09:49:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2B8DB22CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 797906B0269; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7234F6B026A; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 526FD6B026B; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0150.hostedemail.com [216.40.44.150]) by kanga.kvack.org (Postfix) with ESMTP id 200E86B0269 for ; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id BD9958248AB7 for ; Tue, 20 Aug 2019 09:49:42 +0000 (UTC) X-FDA: 75842334204.09.scent80_548eab20a095f X-Spam-Summary: 2,0,0,95e5db907b7158a3,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:hannes@cmpxchg.org:mhocko@kernel.org:vdavydov.dev@gmail.com:vbabka@suse.cz:dan.j.williams@intel.com:osalvador@suse.de:richard.weiyang@gmail.com:pasha.tatashin@oracle.com:arunks@codeaurora.org:cai@lca.pw:aryabinin@virtuozzo.com:kirill.shutemov@linux.intel.com:hughd@google.com:jglisse@redhat.com:aarcange@redhat.com:aneesh.kumar@linux.ibm.com:rientjes@google.com:jrdr.linux@gmail.com:swkhack@gmail.com:stefan.potyra@elektrobit.com:rppt@linux.vnet.ibm.com:alexander.h.duyck@linux.intel.com:sfr@canb.auug.org.au:colin.king@canonical.com:jgg@ziepe.ca:mchehab+samsung@kernel.org:willy@infradead.org:peng.fan@nxp.com:ira.weiny@intel.com:ktkhai@virtuozzo.com:daniel.m.jordan@oracle.com:laoar.shao@gmail.com:yang.shi@linux.alibaba.com,RULES_HIT:41:69:152:327:355:37 9:541:80 X-HE-Tag: scent80_548eab20a095f X-Filterd-Recvd-Size: 23643 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:40 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R991e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07486;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=40;SR=0;TI=SMTPD_---0Ta-AHFA_1566294571; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-AHFA_1566294571) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:32 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Johannes Weiner , Michal Hocko , Vladimir Davydov , Vlastimil Babka , Dan Williams , Oscar Salvador , Wei Yang , Pavel Tatashin , Arun KS , Qian Cai , Andrey Ryabinin , "Kirill A. Shutemov" , Hugh Dickins , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrea Arcangeli , "Aneesh Kumar K.V" , David Rientjes , Souptick Joarder , swkhack , "Potyra, Stefan" , Mike Rapoport , Alexander Duyck , Stephen Rothwell , Colin Ian King , Jason Gunthorpe , Mauro Carvalho Chehab , Matthew Wilcox , Peng Fan , Ira Weiny , Kirill Tkhai , Daniel Jordan , Yafang Shao , Yang Shi Subject: [PATCH 01/14] mm/lru: move pgdat lru_lock into lruvec Date: Tue, 20 Aug 2019 17:48:24 +0800 Message-Id: <1566294517-86418-2-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch move lru_lock into lruvec, give a lru_lock for each of lruvec, thus bring a lru_lock for each of memcg. Per memcg lru_lock would ease the lru_lock contention a lot in this patch series. In some data center, containers are used widely to deploy different kind of services, then multiple memcgs share per node pgdat->lru_lock which cause heavy lock contentions when doing lru operations. On my 2 socket * 6 cores E5-2630 platform, 24 containers run aim9 simultaneously with mmtests' config: # AIM9 export AIM9_TESTTIME=180 export AIM9_TESTLIST=page_test,brk_test perf lock report show much contentions on lru_lock in 20 second snapshot: Name acquired contended avg wait (ns) total wait (ns) max wait (ns) min wait (ns) &(ptlock_ptr(pag... 22 0 0 0 0 0 ... &(&pgdat->lru_lo... 9 7 12728 89096 26656 1597 With this patch series, lruvec->lru_lock show no contentions &(&lruvec->lru_l... 8 0 0 0 0 0 and aim9 page_test/brk_test performance increased 5%~50%. Now this patch still using per pgdat lru_lock, no function changes yet. Signed-off-by: Alex Shi Cc: Johannes Weiner Cc: Michal Hocko Cc: Vladimir Davydov Cc: Andrew Morton Cc: Vlastimil Babka Cc: Dan Williams Cc: Oscar Salvador Cc: Mel Gorman Cc: Wei Yang Cc: Pavel Tatashin Cc: Arun KS Cc: Qian Cai Cc: Andrey Ryabinin Cc: "Kirill A. Shutemov" Cc: Hugh Dickins Cc: "Jérôme Glisse" Cc: Andrea Arcangeli Cc: "Aneesh Kumar K.V" Cc: David Rientjes Cc: Souptick Joarder Cc: swkhack Cc: "Potyra, Stefan" Cc: Mike Rapoport Cc: Alexander Duyck Cc: Stephen Rothwell Cc: Colin Ian King Cc: Jason Gunthorpe Cc: Mauro Carvalho Chehab Cc: Matthew Wilcox Cc: Peng Fan Cc: Ira Weiny Cc: Kirill Tkhai Cc: Daniel Jordan Cc: Yafang Shao Cc: Yang Shi Cc: Tejun Heo Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- include/linux/mmzone.h | 4 +++- mm/compaction.c | 10 +++++----- mm/huge_memory.c | 6 +++--- mm/memcontrol.c | 6 +++--- mm/mlock.c | 10 +++++----- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 4 ++-- mm/swap.c | 28 ++++++++++++++-------------- mm/vmscan.c | 38 +++++++++++++++++++------------------- 10 files changed, 55 insertions(+), 53 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d77d717c620c..8d0076d084be 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -295,6 +295,9 @@ struct zone_reclaim_stat { struct lruvec { struct list_head lists[NR_LRU_LISTS]; + /* move lru_lock to per lruvec for memcg */ + spinlock_t lru_lock; + struct zone_reclaim_stat reclaim_stat; /* Evictions & activations on the inactive file list */ atomic_long_t inactive_age; @@ -744,7 +747,6 @@ struct zonelist { /* Write-intensive fields used by page reclaim */ ZONE_PADDING(_pad1_) - spinlock_t lru_lock; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* diff --git a/mm/compaction.c b/mm/compaction.c index 952dc2fb24e5..9a737f343183 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -846,7 +846,7 @@ static bool too_many_isolated(pg_data_t *pgdat) * a fatal signal is pending. */ if (!(low_pfn % SWAP_CLUSTER_MAX) - && compact_unlock_should_abort(&pgdat->lru_lock, + && compact_unlock_should_abort(&pgdat->lruvec.lru_lock, flags, &locked, cc)) { low_pfn = 0; goto fatal_pending; @@ -919,7 +919,7 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(__PageMovable(page)) && !PageIsolated(page)) { if (locked) { - spin_unlock_irqrestore(&pgdat->lru_lock, + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); locked = false; } @@ -949,7 +949,7 @@ static bool too_many_isolated(pg_data_t *pgdat) /* If we already hold the lock, we can skip some rechecking */ if (!locked) { - locked = compact_lock_irqsave(&pgdat->lru_lock, + locked = compact_lock_irqsave(&pgdat->lruvec.lru_lock, &flags, cc); /* Try get exclusive access under lock */ @@ -1016,7 +1016,7 @@ static bool too_many_isolated(pg_data_t *pgdat) */ if (nr_isolated) { if (locked) { - spin_unlock_irqrestore(&pgdat->lru_lock, flags); + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); locked = false; } putback_movable_pages(&cc->migratepages); @@ -1043,7 +1043,7 @@ static bool too_many_isolated(pg_data_t *pgdat) isolate_abort: if (locked) - spin_unlock_irqrestore(&pgdat->lru_lock, flags); + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); /* * Updated the cached scanner pfn once the pageblock has been scanned diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 738065f765ab..3a483deee807 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2529,7 +2529,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, xa_unlock(&head->mapping->i_pages); } - spin_unlock_irqrestore(&pgdat->lru_lock, flags); + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); remap_page(head); @@ -2740,7 +2740,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) lru_add_drain(); /* prevent PageLRU to go away from under us, and freeze lru stats */ - spin_lock_irqsave(&pgdata->lru_lock, flags); + spin_lock_irqsave(&pgdata->lruvec.lru_lock, flags); if (mapping) { XA_STATE(xas, &mapping->i_pages, page_index(head)); @@ -2785,7 +2785,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) spin_unlock(&pgdata->split_queue_lock); fail: if (mapping) xa_unlock(&mapping->i_pages); - spin_unlock_irqrestore(&pgdata->lru_lock, flags); + spin_unlock_irqrestore(&pgdata->lruvec.lru_lock, flags); remap_page(head); ret = -EBUSY; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6f5c0c517c49..2792b8ed405f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2557,7 +2557,7 @@ static void lock_page_lru(struct page *page, int *isolated) { pg_data_t *pgdat = page_pgdat(page); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); if (PageLRU(page)) { struct lruvec *lruvec; @@ -2581,7 +2581,7 @@ static void unlock_page_lru(struct page *page, int isolated) SetPageLRU(page); add_page_to_lru_list(page, lruvec, page_lru(page)); } - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); } static void commit_charge(struct page *page, struct mem_cgroup *memcg, @@ -2901,7 +2901,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) /* * Because tail pages are not marked as "used", set it. We're under - * pgdat->lru_lock and migration entries setup in all page mappings. + * pgdat->lruvec.lru_lock and migration entries setup in all page mappings. */ void mem_cgroup_split_huge_fixup(struct page *head) { diff --git a/mm/mlock.c b/mm/mlock.c index a90099da4fb4..1279684bada0 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -194,7 +194,7 @@ unsigned int munlock_vma_page(struct page *page) * might otherwise copy PageMlocked to part of the tail pages before * we clear it in the head page. It also stabilizes hpage_nr_pages(). */ - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); if (!TestClearPageMlocked(page)) { /* Potentially, PTE-mapped THP: do not skip the rest PTEs */ @@ -206,14 +206,14 @@ unsigned int munlock_vma_page(struct page *page) __mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages); if (__munlock_isolate_lru_page(page, true)) { - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); __munlock_isolated_page(page); goto out; } __munlock_isolation_failed(page); unlock_out: - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); out: return nr_pages - 1; @@ -298,7 +298,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) pagevec_init(&pvec_putback); /* Phase 1: page isolation */ - spin_lock_irq(&zone->zone_pgdat->lru_lock); + spin_lock_irq(&zone->zone_pgdat->lruvec.lru_lock); for (i = 0; i < nr; i++) { struct page *page = pvec->pages[i]; @@ -325,7 +325,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) pvec->pages[i] = NULL; } __mod_zone_page_state(zone, NR_MLOCK, delta_munlocked); - spin_unlock_irq(&zone->zone_pgdat->lru_lock); + spin_unlock_irq(&zone->zone_pgdat->lruvec.lru_lock); /* Now we can release pins of pages that we are not munlocking */ pagevec_release(&pvec_putback); diff --git a/mm/mmzone.c b/mm/mmzone.c index 4686fdc23bb9..3750a90ed4a0 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -91,6 +91,7 @@ void lruvec_init(struct lruvec *lruvec) enum lru_list lru; memset(lruvec, 0, sizeof(struct lruvec)); + spin_lock_init(&lruvec->lru_lock); for_each_lru(lru) INIT_LIST_HEAD(&lruvec->lists[lru]); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 272c6de1bf4e..1b07dcaabbd7 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6677,7 +6677,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) init_waitqueue_head(&pgdat->pfmemalloc_wait); pgdat_page_ext_init(pgdat); - spin_lock_init(&pgdat->lru_lock); lruvec_init(node_lruvec(pgdat)); } diff --git a/mm/page_idle.c b/mm/page_idle.c index 295512465065..420bc0ac8c1e 100644 --- a/mm/page_idle.c +++ b/mm/page_idle.c @@ -42,12 +42,12 @@ static struct page *page_idle_get_page(unsigned long pfn) return NULL; pgdat = page_pgdat(page); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); if (unlikely(!PageLRU(page))) { put_page(page); page = NULL; } - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); return page; } diff --git a/mm/swap.c b/mm/swap.c index ae300397dfda..63f4782af57a 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -63,12 +63,12 @@ static void __page_cache_release(struct page *page) struct lruvec *lruvec; unsigned long flags; - spin_lock_irqsave(&pgdat->lru_lock, flags); + spin_lock_irqsave(&pgdat->lruvec.lru_lock, flags); lruvec = mem_cgroup_page_lruvec(page, pgdat); VM_BUG_ON_PAGE(!PageLRU(page), page); __ClearPageLRU(page); del_page_from_lru_list(page, lruvec, page_off_lru(page)); - spin_unlock_irqrestore(&pgdat->lru_lock, flags); + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); } __ClearPageWaiters(page); mem_cgroup_uncharge(page); @@ -201,16 +201,16 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, if (pagepgdat != pgdat) { if (pgdat) - spin_unlock_irqrestore(&pgdat->lru_lock, flags); + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); pgdat = pagepgdat; - spin_lock_irqsave(&pgdat->lru_lock, flags); + spin_lock_irqsave(&pgdat->lruvec.lru_lock, flags); } lruvec = mem_cgroup_page_lruvec(page, pgdat); (*move_fn)(page, lruvec, arg); } if (pgdat) - spin_unlock_irqrestore(&pgdat->lru_lock, flags); + spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); release_pages(pvec->pages, pvec->nr); pagevec_reinit(pvec); } @@ -326,9 +326,9 @@ void activate_page(struct page *page) pg_data_t *pgdat = page_pgdat(page); page = compound_head(page); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); __activate_page(page, mem_cgroup_page_lruvec(page, pgdat), NULL); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); } #endif @@ -733,7 +733,7 @@ void release_pages(struct page **pages, int nr) * same pgdat. The lock is held only if pgdat != NULL. */ if (locked_pgdat && ++lock_batch == SWAP_CLUSTER_MAX) { - spin_unlock_irqrestore(&locked_pgdat->lru_lock, flags); + spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); locked_pgdat = NULL; } @@ -742,7 +742,7 @@ void release_pages(struct page **pages, int nr) if (is_zone_device_page(page)) { if (locked_pgdat) { - spin_unlock_irqrestore(&locked_pgdat->lru_lock, + spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); locked_pgdat = NULL; } @@ -762,7 +762,7 @@ void release_pages(struct page **pages, int nr) if (PageCompound(page)) { if (locked_pgdat) { - spin_unlock_irqrestore(&locked_pgdat->lru_lock, flags); + spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); locked_pgdat = NULL; } __put_compound_page(page); @@ -774,11 +774,11 @@ void release_pages(struct page **pages, int nr) if (pgdat != locked_pgdat) { if (locked_pgdat) - spin_unlock_irqrestore(&locked_pgdat->lru_lock, + spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); lock_batch = 0; locked_pgdat = pgdat; - spin_lock_irqsave(&locked_pgdat->lru_lock, flags); + spin_lock_irqsave(&locked_pgdat->lruvec.lru_lock, flags); } lruvec = mem_cgroup_page_lruvec(page, locked_pgdat); @@ -794,7 +794,7 @@ void release_pages(struct page **pages, int nr) list_add(&page->lru, &pages_to_free); } if (locked_pgdat) - spin_unlock_irqrestore(&locked_pgdat->lru_lock, flags); + spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); @@ -832,7 +832,7 @@ void lru_add_page_tail(struct page *page, struct page *page_tail, VM_BUG_ON_PAGE(!PageHead(page), page); VM_BUG_ON_PAGE(PageCompound(page_tail), page); VM_BUG_ON_PAGE(PageLRU(page_tail), page); - lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock); + lockdep_assert_held(&lruvec->lru_lock); if (!list) SetPageLRU(page_tail); diff --git a/mm/vmscan.c b/mm/vmscan.c index c77d1e3761a7..c7a228525df0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1805,7 +1805,7 @@ int isolate_lru_page(struct page *page) pg_data_t *pgdat = page_pgdat(page); struct lruvec *lruvec; - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); lruvec = mem_cgroup_page_lruvec(page, pgdat); if (PageLRU(page)) { int lru = page_lru(page); @@ -1814,7 +1814,7 @@ int isolate_lru_page(struct page *page) del_page_from_lru_list(page, lruvec, lru); ret = 0; } - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); } return ret; } @@ -1890,9 +1890,9 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, VM_BUG_ON_PAGE(PageLRU(page), page); if (unlikely(!page_evictable(page))) { list_del(&page->lru); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); putback_lru_page(page); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); continue; } lruvec = mem_cgroup_page_lruvec(page, pgdat); @@ -1910,10 +1910,10 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, del_page_from_lru_list(page, lruvec, lru); if (unlikely(PageCompound(page))) { - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); mem_cgroup_uncharge(page); (*get_compound_page_dtor(page))(page); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); } else list_add(&page->lru, &pages_to_free); } else { @@ -1976,7 +1976,7 @@ static int current_may_throttle(void) lru_add_drain(); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &page_list, &nr_scanned, sc, lru); @@ -1988,7 +1988,7 @@ static int current_may_throttle(void) if (global_reclaim(sc)) __count_vm_events(item, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); if (nr_taken == 0) return 0; @@ -1996,7 +1996,7 @@ static int current_may_throttle(void) nr_reclaimed = shrink_page_list(&page_list, pgdat, sc, 0, &stat, false); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; if (global_reclaim(sc)) @@ -2009,7 +2009,7 @@ static int current_may_throttle(void) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); mem_cgroup_uncharge_list(&page_list); free_unref_page_list(&page_list); @@ -2062,7 +2062,7 @@ static void shrink_active_list(unsigned long nr_to_scan, lru_add_drain(); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, lru); @@ -2073,7 +2073,7 @@ static void shrink_active_list(unsigned long nr_to_scan, __count_vm_events(PGREFILL, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); while (!list_empty(&l_hold)) { cond_resched(); @@ -2119,7 +2119,7 @@ static void shrink_active_list(unsigned long nr_to_scan, /* * Move pages back to the lru list. */ - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); /* * Count referenced pages from currently used mappings as rotated, * even though only some of them are actually re-activated. This @@ -2137,7 +2137,7 @@ static void shrink_active_list(unsigned long nr_to_scan, __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); mem_cgroup_uncharge_list(&l_active); free_unref_page_list(&l_active); @@ -2373,7 +2373,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE, MAX_NR_ZONES) + lruvec_lru_size(lruvec, LRU_INACTIVE_FILE, MAX_NR_ZONES); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); if (unlikely(reclaim_stat->recent_scanned[0] > anon / 4)) { reclaim_stat->recent_scanned[0] /= 2; reclaim_stat->recent_rotated[0] /= 2; @@ -2394,7 +2394,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, fp = file_prio * (reclaim_stat->recent_scanned[1] + 1); fp /= reclaim_stat->recent_rotated[1] + 1; - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); fraction[0] = ap; fraction[1] = fp; @@ -4263,9 +4263,9 @@ void check_move_unevictable_pages(struct pagevec *pvec) pgscanned++; if (pagepgdat != pgdat) { if (pgdat) - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); pgdat = pagepgdat; - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&pgdat->lruvec.lru_lock); } lruvec = mem_cgroup_page_lruvec(page, pgdat); @@ -4286,7 +4286,7 @@ void check_move_unevictable_pages(struct pagevec *pvec) if (pgdat) { __count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); __count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&pgdat->lruvec.lru_lock); } } EXPORT_SYMBOL_GPL(check_move_unevictable_pages); From patchwork Tue Aug 20 09:48:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0F5B14DE for ; Tue, 20 Aug 2019 09:49:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8821722CF9 for ; Tue, 20 Aug 2019 09:49:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8821722CF9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B19B46B000D; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AA3C06B000C; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7D16B0010; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id 6884D6B000C for ; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 03F768248AB7 for ; Tue, 20 Aug 2019 09:49:40 +0000 (UTC) X-FDA: 75842334120.05.ducks15_544a095b41032 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:hannes@cmpxchg.org:mhocko@kernel.org:vdavydov.dev@gmail.com:guro@fb.com:shakeelb@google.com:chris@chrisdown.name:ktkhai@virtuozzo.com:tglx@linutronix.de,RULES_HIT:30054:30064,0,RBL:47.88.44.36:@linux.alibaba.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: ducks15_544a095b41032 X-Filterd-Recvd-Size: 4893 Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com [47.88.44.36]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:38 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R301e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04446;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0Ta-6uiR_1566294572; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-6uiR_1566294572) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:32 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Johannes Weiner , Michal Hocko , Vladimir Davydov , Roman Gushchin , Shakeel Butt , Chris Down , Kirill Tkhai , Thomas Gleixner Subject: [PATCH 02/14] lru/memcg: move the lruvec->pgdat sync out lru_lock Date: Tue, 20 Aug 2019 17:48:25 +0800 Message-Id: <1566294517-86418-3-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We are going to move lruvec getting out of lru_lock, the only unsafe part is lruvec->pgdat syncing when memory node hot pluging. Splitting out the lruvec->pgdat assignment now and will put it in lruvec lru_lock protection. No function changes in this patch now. Signed-off-by: Alex Shi Cc: Johannes Weiner Cc: Michal Hocko Cc: Vladimir Davydov Cc: Roman Gushchin Cc: Shakeel Butt Cc: Chris Down Cc: Kirill Tkhai Cc: Thomas Gleixner Cc: Andrew Morton Cc: Tejun Heo Cc: cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/memcontrol.h | 24 +++++++++++++++++------- mm/memcontrol.c | 8 +------- 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 2cd4359cb38c..95b3d9885ab6 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -359,6 +359,17 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg, return memcg->nodeinfo[nid]; } +static void sync_lruvec_pgdat(struct lruvec *lruvec, struct pglist_data *pgdat) +{ + /* + * Since a node can be onlined after the mem_cgroup was created, + * we have to be prepared to initialize lruvec->pgdat here; + * and if offlined then reonlined, we need to reinitialize it. + */ + if (!mem_cgroup_disabled() && unlikely(lruvec->pgdat != pgdat)) + lruvec->pgdat = pgdat; +} + /** * mem_cgroup_lruvec - get the lru list vector for a node or a memcg zone * @node: node of the wanted lruvec @@ -382,13 +393,7 @@ static inline struct lruvec *mem_cgroup_lruvec(struct pglist_data *pgdat, mz = mem_cgroup_nodeinfo(memcg, pgdat->node_id); lruvec = &mz->lruvec; out: - /* - * Since a node can be onlined after the mem_cgroup was created, - * we have to be prepared to initialize lruvec->pgdat here; - * and if offlined then reonlined, we need to reinitialize it. - */ - if (unlikely(lruvec->pgdat != pgdat)) - lruvec->pgdat = pgdat; + sync_lruvec_pgdat(lruvec, pgdat); return lruvec; } @@ -857,6 +862,11 @@ static inline void mem_cgroup_migrate(struct page *old, struct page *new) { } +static inline void sync_lruvec_pgdat(struct lruvec *lruvec, + struct pglist_data *pgdat) +{ +} + static inline struct lruvec *mem_cgroup_lruvec(struct pglist_data *pgdat, struct mem_cgroup *memcg) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2792b8ed405f..e8a1b0d95ba8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1257,13 +1257,7 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd mz = mem_cgroup_page_nodeinfo(memcg, page); lruvec = &mz->lruvec; out: - /* - * Since a node can be onlined after the mem_cgroup was created, - * we have to be prepared to initialize lruvec->zone here; - * and if offlined then reonlined, we need to reinitialize it. - */ - if (unlikely(lruvec->pgdat != pgdat)) - lruvec->pgdat = pgdat; + sync_lruvec_pgdat(lruvec, pgdat); return lruvec; } From patchwork Tue Aug 20 09:48:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103487 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA4BD14DE for ; Tue, 20 Aug 2019 09:49:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B130922CF9 for ; Tue, 20 Aug 2019 09:49:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B130922CF9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 85C6A6B0010; Tue, 20 Aug 2019 05:49:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7E69D6B0269; Tue, 20 Aug 2019 05:49:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6385C6B026A; Tue, 20 Aug 2019 05:49:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2FB226B0010 for ; Tue, 20 Aug 2019 05:49:42 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id D06C08419 for ; Tue, 20 Aug 2019 09:49:41 +0000 (UTC) X-FDA: 75842334162.08.able95_548b0b7aaf123 X-Spam-Summary: 2,0,0,4e0a182ceb395857,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:hannes@cmpxchg.org:mhocko@kernel.org:vdavydov.dev@gmail.com,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1261:1345:1359:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:2693:3138:3139:3140:3141:3142:3352:3865:3870:4321:5007:6261:7514:8957:9592:10004:11026:11473:11658:11914:12043:12048:12296:12297:12438:12555:12895:12986:13069:13311:13357:13846:14096:14181:14384:14394:14721:21060:21080:21451:21627:30054,0,RBL:115.124.30.54:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: able95_548b0b7aaf123 X-Filterd-Recvd-Size: 3363 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:40 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Ta-6uif_1566294573; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-6uif_1566294573) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:33 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Johannes Weiner , Michal Hocko , Vladimir Davydov Subject: [PATCH 03/14] lru/memcg: using per lruvec lock in un/lock_page_lru Date: Tue, 20 Aug 2019 17:48:26 +0800 Message-Id: <1566294517-86418-4-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now we repeatly assign the lruvec->pgdat in memcg. Will remove the assignment in lruvec getting function after very points are protected. Signed-off-by: Alex Shi Cc: Johannes Weiner Cc: Michal Hocko Cc: Vladimir Davydov Cc: Andrew Morton Cc: Tejun Heo Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/memcontrol.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8a1b0d95ba8..19fd911e8098 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2550,12 +2550,12 @@ static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) static void lock_page_lru(struct page *page, int *isolated) { pg_data_t *pgdat = page_pgdat(page); + struct lruvec *lruvec = mem_cgroup_page_lruvec(page, pgdat); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); + sync_lruvec_pgdat(lruvec, pgdat); if (PageLRU(page)) { - struct lruvec *lruvec; - lruvec = mem_cgroup_page_lruvec(page, pgdat); ClearPageLRU(page); del_page_from_lru_list(page, lruvec, page_lru(page)); *isolated = 1; @@ -2566,16 +2566,14 @@ static void lock_page_lru(struct page *page, int *isolated) static void unlock_page_lru(struct page *page, int isolated) { pg_data_t *pgdat = page_pgdat(page); + struct lruvec *lruvec = mem_cgroup_page_lruvec(page, pgdat); if (isolated) { - struct lruvec *lruvec; - - lruvec = mem_cgroup_page_lruvec(page, pgdat); VM_BUG_ON_PAGE(PageLRU(page), page); SetPageLRU(page); add_page_to_lru_list(page, lruvec, page_lru(page)); } - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); } static void commit_charge(struct page *page, struct mem_cgroup *memcg, From patchwork Tue Aug 20 09:48:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 043C31813 for ; Tue, 20 Aug 2019 09:49:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF0D922DA7 for ; Tue, 20 Aug 2019 09:49:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF0D922DA7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D3DFC6B026B; Tue, 20 Aug 2019 05:49:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CC8326B026C; Tue, 20 Aug 2019 05:49:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A31B86B026D; Tue, 20 Aug 2019 05:49:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 740EB6B026B for ; Tue, 20 Aug 2019 05:49:44 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id F0516181AC9B6 for ; Tue, 20 Aug 2019 09:49:43 +0000 (UTC) X-FDA: 75842334246.23.drink96_54d1c83a68216 X-Spam-Summary: 2,0,0,05d81090e6318038,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:vbabka@suse.cz:cai@lca.pw:aryabinin@virtuozzo.com,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1437:1535:1543:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2895:2901:3138:3139:3140:3141:3142:3355:3865:3867:3868:3871:4321:5007:6119:6261:7903:8957:10004:11026:11473:11658:11914:12043:12048:12114:12291:12296:12297:12438:12555:12683:12895:12986:13846:13868:14096:14181:14394:14721:21060:21080:21451:21627:21810:30054,0,RBL:115.124.30.44:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: drink96_54d1c83a68216 X-Filterd-Recvd-Size: 5313 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:42 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04446;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0TZztT4P_1566294573; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZztT4P_1566294573) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:34 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Vlastimil Babka , Qian Cai , Andrey Ryabinin Subject: [PATCH 04/14] lru/compaction: use per lruvec lock in isolate_migratepages_block Date: Tue, 20 Aug 2019 17:48:27 +0800 Message-Id: <1566294517-86418-5-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using lruvec locking to replace pgdat lru_lock. and then unfold compact_unlock_should_abort() to fit the replacement. Signed-off-by: Alex Shi Cc: Mel Gorman Cc: Vlastimil Babka Cc: Qian Cai Cc: Andrey Ryabinin Cc: Andrew Morton Cc: Tejun Heo Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/compaction.c | 48 ++++++++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9a737f343183..8877f38410d8 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -785,7 +785,7 @@ static bool too_many_isolated(pg_data_t *pgdat) unsigned long nr_scanned = 0, nr_isolated = 0; struct lruvec *lruvec; unsigned long flags = 0; - bool locked = false; + struct lruvec *locked_lruvec = NULL; struct page *page = NULL, *valid_page = NULL; unsigned long start_pfn = low_pfn; bool skip_on_failure = false; @@ -845,11 +845,20 @@ static bool too_many_isolated(pg_data_t *pgdat) * contention, to give chance to IRQs. Abort completely if * a fatal signal is pending. */ - if (!(low_pfn % SWAP_CLUSTER_MAX) - && compact_unlock_should_abort(&pgdat->lruvec.lru_lock, - flags, &locked, cc)) { - low_pfn = 0; - goto fatal_pending; + if (!(low_pfn % SWAP_CLUSTER_MAX)) { + if (locked_lruvec) { + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); + locked_lruvec = NULL; + } + + if (fatal_signal_pending(current)) { + cc->contended = true; + + low_pfn = 0; + goto fatal_pending; + } + + cond_resched(); } if (!pfn_valid_within(low_pfn)) @@ -918,10 +927,10 @@ static bool too_many_isolated(pg_data_t *pgdat) */ if (unlikely(__PageMovable(page)) && !PageIsolated(page)) { - if (locked) { - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, + if (locked_lruvec) { + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); - locked = false; + locked_lruvec = NULL; } if (!isolate_movable_page(page, isolate_mode)) @@ -947,10 +956,14 @@ static bool too_many_isolated(pg_data_t *pgdat) if (!(cc->gfp_mask & __GFP_FS) && page_mapping(page)) goto isolate_fail; + lruvec = mem_cgroup_page_lruvec(page, pgdat); + /* If we already hold the lock, we can skip some rechecking */ - if (!locked) { - locked = compact_lock_irqsave(&pgdat->lruvec.lru_lock, - &flags, cc); + if (lruvec != locked_lruvec) { + if (compact_lock_irqsave(&lruvec->lru_lock, &flags, cc)) + locked_lruvec = lruvec; + + sync_lruvec_pgdat(lruvec, pgdat); /* Try get exclusive access under lock */ if (!skip_updated) { @@ -974,7 +987,6 @@ static bool too_many_isolated(pg_data_t *pgdat) } } - lruvec = mem_cgroup_page_lruvec(page, pgdat); /* Try isolate the page */ if (__isolate_lru_page(page, isolate_mode) != 0) @@ -1015,9 +1027,9 @@ static bool too_many_isolated(pg_data_t *pgdat) * page anyway. */ if (nr_isolated) { - if (locked) { - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); - locked = false; + if (locked_lruvec) { + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); + locked_lruvec = NULL; } putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; @@ -1042,8 +1054,8 @@ static bool too_many_isolated(pg_data_t *pgdat) low_pfn = end_pfn; isolate_abort: - if (locked) - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); + if (locked_lruvec) + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); /* * Updated the cached scanner pfn once the pageblock has been scanned From patchwork Tue Aug 20 09:48:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1FD1B1398 for ; Tue, 20 Aug 2019 09:49:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E0A5D22CF7 for ; Tue, 20 Aug 2019 09:49:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0A5D22CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DB2C56B000C; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CED186B0269; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E03D6B000E; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0207.hostedemail.com [216.40.44.207]) by kanga.kvack.org (Postfix) with ESMTP id 751EE6B000D for ; Tue, 20 Aug 2019 05:49:40 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 14E56181AC9B6 for ; Tue, 20 Aug 2019 09:49:40 +0000 (UTC) X-FDA: 75842334120.02.boys69_544bb53dec360 X-Spam-Summary: 2,0,0,334d6f6f86b4012c,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:kirill.shutemov@linux.intel.com:hughd@google.com:vbabka@suse.cz:jglisse@redhat.com:aarcange@redhat.com:aneesh.kumar@linux.ibm.com:rientjes@google.com:jrdr.linux@gmail.com,RULES_HIT:41:152:355:379:541:800:960:966:973:988:989:1260:1261:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1534:1542:1593:1594:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3353:3867:3868:4385:5007:6261:7514:9010:10004:10400:11026:11658:11914:12043:12048:12296:12297:12438:12555:12679:12895:13846:13894:14096:14097:14181:14394:14659:14721:21060:21080:21451:21627:30054,0,RBL:115.124.30.42:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF :not bul X-HE-Tag: boys69_544bb53dec360 X-Filterd-Recvd-Size: 4132 Received: from out30-42.freemail.mail.aliyun.com (out30-42.freemail.mail.aliyun.com [115.124.30.42]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:38 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R731e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0TZzk.Bf_1566294574; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZzk.Bf_1566294574) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:34 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , "Kirill A. Shutemov" , Hugh Dickins , Vlastimil Babka , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrea Arcangeli , "Aneesh Kumar K.V" , David Rientjes , Souptick Joarder Subject: [PATCH 05/14] lru/huge_page: use per lruvec lock in __split_huge_page Date: Tue, 20 Aug 2019 17:48:28 +0800 Message-Id: <1566294517-86418-6-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using lruvec lock to replace pgdat lru_lock. Signed-off-by: Alex Shi Cc: Michal Hocko Cc: "Kirill A. Shutemov" Cc: Hugh Dickins Cc: Vlastimil Babka Cc: "Jérôme Glisse" Cc: Andrea Arcangeli Cc: "Aneesh Kumar K.V" Cc: David Rientjes Cc: Souptick Joarder Cc: Andrew Morton Cc: Tejun Heo Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/huge_memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3a483deee807..9a96c0944b4d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2529,7 +2529,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, xa_unlock(&head->mapping->i_pages); } - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); + spin_unlock_irqrestore(&lruvec->lru_lock, flags); remap_page(head); @@ -2671,6 +2671,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) struct pglist_data *pgdata = NODE_DATA(page_to_nid(head)); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; + struct lruvec *lruvec; int count, mapcount, extra_pins, ret; bool mlocked; unsigned long flags; @@ -2739,8 +2740,10 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (mlocked) lru_add_drain(); + lruvec = mem_cgroup_page_lruvec(head, pgdata); /* prevent PageLRU to go away from under us, and freeze lru stats */ - spin_lock_irqsave(&pgdata->lruvec.lru_lock, flags); + spin_lock_irqsave(&lruvec->lru_lock, flags); + sync_lruvec_pgdat(lruvec, pgdata); if (mapping) { XA_STATE(xas, &mapping->i_pages, page_index(head)); @@ -2785,7 +2788,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) spin_unlock(&pgdata->split_queue_lock); fail: if (mapping) xa_unlock(&mapping->i_pages); - spin_unlock_irqrestore(&pgdata->lruvec.lru_lock, flags); + spin_unlock_irqrestore(&lruvec->lru_lock, flags); remap_page(head); ret = -EBUSY; } From patchwork Tue Aug 20 09:48:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103499 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B35F61813 for ; Tue, 20 Aug 2019 09:50:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 89BE022DA7 for ; Tue, 20 Aug 2019 09:50:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 89BE022DA7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1556E6B026C; Tue, 20 Aug 2019 05:49:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0E1FF6B0270; Tue, 20 Aug 2019 05:49:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7B1E6B026C; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0003.hostedemail.com [216.40.44.3]) by kanga.kvack.org (Postfix) with ESMTP id 7E6A96B026C for ; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 217C5180AD809 for ; Tue, 20 Aug 2019 09:49:45 +0000 (UTC) X-FDA: 75842334330.15.shock76_54fe6ab4ad02d X-Spam-Summary: 2,0,0,8c5210855c0a99ea,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:vbabka@suse.cz:mhocko@suse.com:aryabinin@virtuozzo.com:swkhack@gmail.com:stefan.potyra@elektrobit.com,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1437:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2898:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3872:3874:4250:4321:5007:6261:7514:7903:8957:9592:10004:11026:11658:11914:12043:12048:12296:12297:12438:12555:12679:12683:12895:13846:14181:14394:14721:21060:21080:21451:21627:21796:30036:30054:30070:30079,0,RBL:115.124.30.130:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: shock76_54fe6ab4ad02d X-Filterd-Recvd-Size: 5797 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:43 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07486;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0TZztT4i_1566294574; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZztT4i_1566294574) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:34 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Vlastimil Babka , Michal Hocko , Andrey Ryabinin , swkhack , "Potyra, Stefan" Subject: [PATCH 06/14] lru/mlock: using per lruvec lock in munlock Date: Tue, 20 Aug 2019 17:48:29 +0800 Message-Id: <1566294517-86418-7-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch using sperate lruvec lock for each of pages in __munlock_pagevec(). Also this patch pass a lruvec in __munlock_isolate_lru_page() as parameter to reduce a repeat lruvec locating. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Vlastimil Babka Cc: Michal Hocko Cc: Mel Gorman Cc: Andrey Ryabinin Cc: swkhack Cc: "Potyra, Stefan" Cc: Tejun Heo Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/mlock.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 1279684bada0..9915968d490a 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -106,12 +106,10 @@ void mlock_vma_page(struct page *page) * Isolate a page from LRU with optional get_page() pin. * Assumes lru_lock already held and page already pinned. */ -static bool __munlock_isolate_lru_page(struct page *page, bool getpage) +static bool __munlock_isolate_lru_page(struct page *page, + struct lruvec *lruvec, bool getpage) { if (PageLRU(page)) { - struct lruvec *lruvec; - - lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page)); if (getpage) get_page(page); ClearPageLRU(page); @@ -183,6 +181,9 @@ unsigned int munlock_vma_page(struct page *page) { int nr_pages; pg_data_t *pgdat = page_pgdat(page); + struct lruvec *lruvec; + + lruvec = mem_cgroup_page_lruvec(page, pgdat); /* For try_to_munlock() and to serialize with page migration */ BUG_ON(!PageLocked(page)); @@ -194,7 +195,8 @@ unsigned int munlock_vma_page(struct page *page) * might otherwise copy PageMlocked to part of the tail pages before * we clear it in the head page. It also stabilizes hpage_nr_pages(). */ - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); + sync_lruvec_pgdat(lruvec, pgdat); if (!TestClearPageMlocked(page)) { /* Potentially, PTE-mapped THP: do not skip the rest PTEs */ @@ -205,15 +207,15 @@ unsigned int munlock_vma_page(struct page *page) nr_pages = hpage_nr_pages(page); __mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages); - if (__munlock_isolate_lru_page(page, true)) { - spin_unlock_irq(&pgdat->lruvec.lru_lock); + if (__munlock_isolate_lru_page(page, lruvec, true)) { + spin_unlock_irq(&lruvec->lru_lock); __munlock_isolated_page(page); goto out; } __munlock_isolation_failed(page); unlock_out: - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); out: return nr_pages - 1; @@ -291,28 +293,30 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) { int i; int nr = pagevec_count(pvec); - int delta_munlocked = -nr; struct pagevec pvec_putback; int pgrescued = 0; pagevec_init(&pvec_putback); /* Phase 1: page isolation */ - spin_lock_irq(&zone->zone_pgdat->lruvec.lru_lock); for (i = 0; i < nr; i++) { struct page *page = pvec->pages[i]; + pg_data_t *pgdat = page_pgdat(page); + struct lruvec *lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock_irq(&lruvec->lru_lock); + sync_lruvec_pgdat(lruvec, pgdat); if (TestClearPageMlocked(page)) { /* * We already have pin from follow_page_mask() * so we can spare the get_page() here. */ - if (__munlock_isolate_lru_page(page, false)) + if (__munlock_isolate_lru_page(page, lruvec, false)) { + __mod_zone_page_state(zone, NR_MLOCK, -1); + spin_unlock_irq(&lruvec->lru_lock); continue; - else + } else __munlock_isolation_failed(page); - } else { - delta_munlocked++; } /* @@ -323,9 +327,8 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) */ pagevec_add(&pvec_putback, pvec->pages[i]); pvec->pages[i] = NULL; + spin_unlock_irq(&lruvec->lru_lock); } - __mod_zone_page_state(zone, NR_MLOCK, delta_munlocked); - spin_unlock_irq(&zone->zone_pgdat->lruvec.lru_lock); /* Now we can release pins of pages that we are not munlocking */ pagevec_release(&pvec_putback); From patchwork Tue Aug 20 09:48:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103505 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3BA814DE for ; Tue, 20 Aug 2019 09:50:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C183522CF7 for ; Tue, 20 Aug 2019 09:50:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C183522CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A08336B0271; Tue, 20 Aug 2019 05:50:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B77C6B0272; Tue, 20 Aug 2019 05:50:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F4866B0274; Tue, 20 Aug 2019 05:50:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id 6914F6B0271 for ; Tue, 20 Aug 2019 05:50:27 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 1A288269DE for ; Tue, 20 Aug 2019 09:50:27 +0000 (UTC) X-FDA: 75842336094.17.loss67_5b2242519fc0d X-Spam-Summary: 2,0,0,773f652fba64fe21,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:vbabka@suse.cz:jgg@ziepe.ca:willy@infradead.org:mchehab+samsung@kernel.org:peng.fan@nxp.com:nborisov@suse.com:ira.weiny@intel.com:aryabinin@virtuozzo.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3167:3352:3865:5007:6261:8957:10004:11026:11473:11658:11914:12043:12048:12297:12438:12555:12895:13069:13311:13357:13846:14096:14181:14384:14394:14721:21060:21080:21451:21627:30054:30064,0,RBL:115.124.30.54:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:56,LUA_SUMMARY:none X-HE-Tag: loss67_5b2242519fc0d X-Filterd-Recvd-Size: 3047 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:50:25 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01422;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0TZzk.Bv_1566294575; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZzk.Bv_1566294575) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:35 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Vlastimil Babka , Jason Gunthorpe , Matthew Wilcox , Mauro Carvalho Chehab , Peng Fan , Nikolay Borisov , Ira Weiny , Andrey Ryabinin Subject: [PATCH 07/14] lru/swap: using per lruvec lock in page_cache_release Date: Tue, 20 Aug 2019 17:48:30 +0800 Message-Id: <1566294517-86418-8-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Also cares the lruvec->pgdat syncing. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Vlastimil Babka Cc: Jason Gunthorpe Cc: Matthew Wilcox Cc: Mauro Carvalho Chehab Cc: Peng Fan Cc: Nikolay Borisov Cc: Ira Weiny Cc: Andrey Ryabinin Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/swap.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 63f4782af57a..2a8fe6df08fc 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -63,12 +63,13 @@ static void __page_cache_release(struct page *page) struct lruvec *lruvec; unsigned long flags; - spin_lock_irqsave(&pgdat->lruvec.lru_lock, flags); lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock_irqsave(&lruvec->lru_lock, flags); + sync_lruvec_pgdat(lruvec, pgdat); VM_BUG_ON_PAGE(!PageLRU(page), page); __ClearPageLRU(page); del_page_from_lru_list(page, lruvec, page_off_lru(page)); - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); + spin_unlock_irqrestore(&lruvec->lru_lock, flags); } __ClearPageWaiters(page); mem_cgroup_uncharge(page); From patchwork Tue Aug 20 09:48:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103501 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7A4A31398 for ; Tue, 20 Aug 2019 09:50:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4794322CF7 for ; Tue, 20 Aug 2019 09:50:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4794322CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7965A6B0270; Tue, 20 Aug 2019 05:50:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7466D6B0271; Tue, 20 Aug 2019 05:50:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 635C96B0272; Tue, 20 Aug 2019 05:50:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id 3B9E46B0270 for ; Tue, 20 Aug 2019 05:50:21 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id DCE83180AD806 for ; Tue, 20 Aug 2019 09:50:20 +0000 (UTC) X-FDA: 75842335800.27.money62_5a3d9e89fde1d X-Spam-Summary: 2,0,0,b7a368ebec256949,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:vbabka@suse.cz:jgg@ziepe.ca:willy@infradead.org:mchehab+samsung@kernel.org:peng.fan@nxp.com:nborisov@suse.com:ira.weiny@intel.com:aryabinin@virtuozzo.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3167:3352:3867:4321:5007:6261:6642:10004:11026:11658:11914:12043:12048:12296:12297:12438:12555:12895:13069:13311:13357:13846:14096:14181:14384:14394:14721:21060:21080:21451:21627:30054:30064,0,RBL:115.124.30.42:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: money62_5a3d9e89fde1d X-Filterd-Recvd-Size: 2949 Received: from out30-42.freemail.mail.aliyun.com (out30-42.freemail.mail.aliyun.com [115.124.30.42]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:50:19 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07417;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0Ta-BP32_1566294575; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-BP32_1566294575) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:35 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Vlastimil Babka , Jason Gunthorpe , Matthew Wilcox , Mauro Carvalho Chehab , Peng Fan , Nikolay Borisov , Ira Weiny , Andrey Ryabinin Subject: [PATCH 08/14] lru/swap: uer lruvec lock in activate_page Date: Tue, 20 Aug 2019 17:48:31 +0800 Message-Id: <1566294517-86418-9-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: to replace pgdat lru_lock. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Vlastimil Babka Cc: Jason Gunthorpe Cc: Matthew Wilcox Cc: Mauro Carvalho Chehab Cc: Peng Fan Cc: Nikolay Borisov Cc: Ira Weiny Cc: Andrey Ryabinin Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/swap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 2a8fe6df08fc..d2dad08fcfd0 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -325,11 +325,14 @@ static inline void activate_page_drain(int cpu) void activate_page(struct page *page) { pg_data_t *pgdat = page_pgdat(page); + struct lruvec *lruvec; page = compound_head(page); - spin_lock_irq(&pgdat->lruvec.lru_lock); + lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock_irq(&lruvec->lru_lock); + sync_lruvec_pgdat(lruvec, pgdat); __activate_page(page, mem_cgroup_page_lruvec(page, pgdat), NULL); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); } #endif From patchwork Tue Aug 20 09:48:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103485 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 85D2314DE for ; Tue, 20 Aug 2019 09:49:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5C41B22CF7 for ; Tue, 20 Aug 2019 09:49:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C41B22CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 749676B000E; Tue, 20 Aug 2019 05:49:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6ADDB6B0010; Tue, 20 Aug 2019 05:49:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D8DE6B0269; Tue, 20 Aug 2019 05:49:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id 27D406B000E for ; Tue, 20 Aug 2019 05:49:41 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id B7CC78419 for ; Tue, 20 Aug 2019 09:49:40 +0000 (UTC) X-FDA: 75842334120.26.stew41_5468ad16c0d47 X-Spam-Summary: 2,0,0,eb8c2fe58d9eb8ff,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:vbabka@suse.cz:jgg@ziepe.ca:willy@infradead.org:mchehab+samsung@kernel.org:peng.fan@nxp.com:nborisov@suse.com:ira.weiny@intel.com:aryabinin@virtuozzo.com,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3167:3352:3867:4321:5007:6261:6642:9592:10004:11026:11658:11914:12043:12048:12296:12297:12438:12555:12895:13846:14096:14181:14394:14721:21060:21080:21451:21627:30054:30064,0,RBL:115.124.30.54:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: stew41_5468ad16c0d47 X-Filterd-Recvd-Size: 3689 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:39 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07486;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0Ta-BP3A_1566294575; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-BP3A_1566294575) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:36 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Vlastimil Babka , Jason Gunthorpe , Matthew Wilcox , Mauro Carvalho Chehab , Peng Fan , Nikolay Borisov , Ira Weiny , Andrey Ryabinin Subject: [PATCH 09/14] lru/swap: uer per lruvec lock in pagevec_lru_move_fn Date: Tue, 20 Aug 2019 17:48:32 +0800 Message-Id: <1566294517-86418-10-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: to replace pgdat lru_lock. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Vlastimil Babka Cc: Jason Gunthorpe Cc: Matthew Wilcox Cc: Mauro Carvalho Chehab Cc: Peng Fan Cc: Nikolay Borisov Cc: Ira Weiny Cc: Andrey Ryabinin Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/swap.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index d2dad08fcfd0..24a2b3456e10 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -192,26 +192,27 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, void *arg) { int i; - struct pglist_data *pgdat = NULL; - struct lruvec *lruvec; + struct lruvec *locked_lruvec = NULL; unsigned long flags = 0; for (i = 0; i < pagevec_count(pvec); i++) { struct page *page = pvec->pages[i]; - struct pglist_data *pagepgdat = page_pgdat(page); - - if (pagepgdat != pgdat) { - if (pgdat) - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); - pgdat = pagepgdat; - spin_lock_irqsave(&pgdat->lruvec.lru_lock, flags); + struct pglist_data *pgdat = page_pgdat(page); + struct lruvec *lruvec = mem_cgroup_page_lruvec(page, pgdat); + + if (locked_lruvec != lruvec) { + if (locked_lruvec) + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); + locked_lruvec = lruvec; + spin_lock_irqsave(&lruvec->lru_lock, flags); + sync_lruvec_pgdat(lruvec, pgdat); } - lruvec = mem_cgroup_page_lruvec(page, pgdat); (*move_fn)(page, lruvec, arg); } - if (pgdat) - spin_unlock_irqrestore(&pgdat->lruvec.lru_lock, flags); + if (locked_lruvec) + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); + release_pages(pvec->pages, pvec->nr); pagevec_reinit(pvec); } From patchwork Tue Aug 20 09:48:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA3C81398 for ; Tue, 20 Aug 2019 09:49:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF94422CF7 for ; Tue, 20 Aug 2019 09:49:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF94422CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E510F6B026A; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D140F6B026B; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB6E66B026C; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0102.hostedemail.com [216.40.44.102]) by kanga.kvack.org (Postfix) with ESMTP id 800096B026B for ; Tue, 20 Aug 2019 05:49:43 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 25948181AC9B6 for ; Tue, 20 Aug 2019 09:49:43 +0000 (UTC) X-FDA: 75842334246.17.night49_54b2154014148 X-Spam-Summary: 2,0,0,ad25e7666eed6aba,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:vbabka@suse.cz:jgg@ziepe.ca:willy@infradead.org:mchehab+samsung@kernel.org:peng.fan@nxp.com:nborisov@suse.com:ira.weiny@intel.com:aryabinin@virtuozzo.com,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1261:1345:1359:1431:1437:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2898:3138:3139:3140:3141:3142:3167:3354:3865:3866:3871:3874:4385:4605:5007:6261:8957:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:13846:14096:14181:14394:14721:21060:21080:21451:21627:30054:30064:30070,0,RBL:115.124.30.57:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_r ules:0:0 X-HE-Tag: night49_54b2154014148 X-Filterd-Recvd-Size: 5342 Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:41 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R561e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04446;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0Ta-6ujV_1566294576; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-6ujV_1566294576) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:36 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Vlastimil Babka , Jason Gunthorpe , Matthew Wilcox , Mauro Carvalho Chehab , Peng Fan , Nikolay Borisov , Ira Weiny , Andrey Ryabinin Subject: [PATCH 10/14] lru/swap: use per lruvec lock in release_pages Date: Tue, 20 Aug 2019 17:48:33 +0800 Message-Id: <1566294517-86418-11-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace pgdat lru_lock with lruvec lru_lock. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Vlastimil Babka Cc: Jason Gunthorpe Cc: Matthew Wilcox Cc: Mauro Carvalho Chehab Cc: Peng Fan Cc: Nikolay Borisov Cc: Ira Weiny Cc: Andrey Ryabinin Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/swap.c | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 24a2b3456e10..798bffe7875d 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -724,8 +724,7 @@ void release_pages(struct page **pages, int nr) { int i; LIST_HEAD(pages_to_free); - struct pglist_data *locked_pgdat = NULL; - struct lruvec *lruvec; + struct lruvec *locked_lruvec = NULL; unsigned long uninitialized_var(flags); unsigned int uninitialized_var(lock_batch); @@ -737,19 +736,19 @@ void release_pages(struct page **pages, int nr) * excessive with a continuous string of pages from the * same pgdat. The lock is held only if pgdat != NULL. */ - if (locked_pgdat && ++lock_batch == SWAP_CLUSTER_MAX) { - spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); - locked_pgdat = NULL; + if (locked_lruvec && ++lock_batch == SWAP_CLUSTER_MAX) { + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); + locked_lruvec = NULL; } if (is_huge_zero_page(page)) continue; if (is_zone_device_page(page)) { - if (locked_pgdat) { - spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, + if (locked_lruvec) { + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); - locked_pgdat = NULL; + locked_lruvec = NULL; } /* * ZONE_DEVICE pages that return 'false' from @@ -766,27 +765,30 @@ void release_pages(struct page **pages, int nr) continue; if (PageCompound(page)) { - if (locked_pgdat) { - spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); - locked_pgdat = NULL; + if (locked_lruvec) { + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); + locked_lruvec = NULL; } __put_compound_page(page); continue; } if (PageLRU(page)) { + struct lruvec *lruvec; struct pglist_data *pgdat = page_pgdat(page); - if (pgdat != locked_pgdat) { - if (locked_pgdat) - spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, + lruvec = mem_cgroup_page_lruvec(page, pgdat); + + if (lruvec != locked_lruvec) { + if (locked_lruvec) + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); lock_batch = 0; - locked_pgdat = pgdat; - spin_lock_irqsave(&locked_pgdat->lruvec.lru_lock, flags); + locked_lruvec = lruvec; + spin_lock_irqsave(&locked_lruvec->lru_lock, flags); + sync_lruvec_pgdat(lruvec, pgdat); } - lruvec = mem_cgroup_page_lruvec(page, locked_pgdat); VM_BUG_ON_PAGE(!PageLRU(page), page); __ClearPageLRU(page); del_page_from_lru_list(page, lruvec, page_off_lru(page)); @@ -798,8 +800,8 @@ void release_pages(struct page **pages, int nr) list_add(&page->lru, &pages_to_free); } - if (locked_pgdat) - spin_unlock_irqrestore(&locked_pgdat->lruvec.lru_lock, flags); + if (locked_lruvec) + spin_unlock_irqrestore(&locked_lruvec->lru_lock, flags); mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); From patchwork Tue Aug 20 09:48:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 233C91398 for ; Tue, 20 Aug 2019 09:50:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E1E2822DA7 for ; Tue, 20 Aug 2019 09:50:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E1E2822DA7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C80A36B026D; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B70076B0272; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DF746B026D; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0078.hostedemail.com [216.40.44.78]) by kanga.kvack.org (Postfix) with ESMTP id 63CA26B026E for ; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id D784F180AD806 for ; Tue, 20 Aug 2019 09:49:44 +0000 (UTC) X-FDA: 75842334288.26.lift28_550091b34cf40 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:ktkhai@virtuozzo.com:hannes@cmpxchg.org:daniel.m.jordan@oracle.com:laoar.shao@gmail.com:yang.shi@linux.alibaba.com,RULES_HIT:30054:30064:30070,0,RBL:47.88.44.36:@linux.alibaba.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: lift28_550091b34cf40 X-Filterd-Recvd-Size: 6995 Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com [47.88.44.36]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:43 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R551e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04446;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0Ta-AHGG_1566294576; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0Ta-AHGG_1566294576) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:37 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Kirill Tkhai , Johannes Weiner , Daniel Jordan , Yafang Shao , Yang Shi Subject: [PATCH 11/14] lru/vmscan: using per lruvec lock in lists shrinking. Date: Tue, 20 Aug 2019 17:48:34 +0800 Message-Id: <1566294517-86418-12-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The involoving functions includes isolate_lru_page, move_pages_to_lru and shrink_in/active_list. also remove unnecessary pgdat. And remove unnecessary pgdat accordingly. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Daniel Jordan Cc: Yafang Shao Cc: Yang Shi Cc: Mel Gorman Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/vmscan.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c7a228525df0..defc2c4778eb 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1805,8 +1805,9 @@ int isolate_lru_page(struct page *page) pg_data_t *pgdat = page_pgdat(page); struct lruvec *lruvec; - spin_lock_irq(&pgdat->lruvec.lru_lock); lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock_irq(&lruvec->lru_lock); + sync_lruvec_pgdat(lruvec, pgdat); if (PageLRU(page)) { int lru = page_lru(page); get_page(page); @@ -1814,7 +1815,7 @@ int isolate_lru_page(struct page *page) del_page_from_lru_list(page, lruvec, lru); ret = 0; } - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); } return ret; } @@ -1879,7 +1880,6 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, struct list_head *list) { - struct pglist_data *pgdat = lruvec_pgdat(lruvec); int nr_pages, nr_moved = 0; LIST_HEAD(pages_to_free); struct page *page; @@ -1890,12 +1890,11 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, VM_BUG_ON_PAGE(PageLRU(page), page); if (unlikely(!page_evictable(page))) { list_del(&page->lru); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); putback_lru_page(page); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); continue; } - lruvec = mem_cgroup_page_lruvec(page, pgdat); SetPageLRU(page); lru = page_lru(page); @@ -1910,10 +1909,10 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, del_page_from_lru_list(page, lruvec, lru); if (unlikely(PageCompound(page))) { - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); mem_cgroup_uncharge(page); (*get_compound_page_dtor(page))(page); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); } else list_add(&page->lru, &pages_to_free); } else { @@ -1976,7 +1975,7 @@ static int current_may_throttle(void) lru_add_drain(); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &page_list, &nr_scanned, sc, lru); @@ -1988,7 +1987,7 @@ static int current_may_throttle(void) if (global_reclaim(sc)) __count_vm_events(item, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); if (nr_taken == 0) return 0; @@ -1996,7 +1995,7 @@ static int current_may_throttle(void) nr_reclaimed = shrink_page_list(&page_list, pgdat, sc, 0, &stat, false); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; if (global_reclaim(sc)) @@ -2009,7 +2008,7 @@ static int current_may_throttle(void) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); mem_cgroup_uncharge_list(&page_list); free_unref_page_list(&page_list); @@ -2062,7 +2061,7 @@ static void shrink_active_list(unsigned long nr_to_scan, lru_add_drain(); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, lru); @@ -2073,7 +2072,7 @@ static void shrink_active_list(unsigned long nr_to_scan, __count_vm_events(PGREFILL, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); while (!list_empty(&l_hold)) { cond_resched(); @@ -2119,7 +2118,7 @@ static void shrink_active_list(unsigned long nr_to_scan, /* * Move pages back to the lru list. */ - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); /* * Count referenced pages from currently used mappings as rotated, * even though only some of them are actually re-activated. This @@ -2137,7 +2136,7 @@ static void shrink_active_list(unsigned long nr_to_scan, __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); mem_cgroup_uncharge_list(&l_active); free_unref_page_list(&l_active); From patchwork Tue Aug 20 09:48:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103507 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78C131813 for ; Tue, 20 Aug 2019 09:50:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4ED5622DCC for ; Tue, 20 Aug 2019 09:50:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4ED5622DCC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 832B96B0272; Tue, 20 Aug 2019 05:50:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7E38A6B0274; Tue, 20 Aug 2019 05:50:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D2176B0275; Tue, 20 Aug 2019 05:50:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 4B1026B0272 for ; Tue, 20 Aug 2019 05:50:40 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id F1DC28248AB7 for ; Tue, 20 Aug 2019 09:50:39 +0000 (UTC) X-FDA: 75842336598.12.grade22_5d0197d2e2559 X-Spam-Summary: 2,0,0,3322037d3cf8c3f9,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:ktkhai@virtuozzo.com:hannes@cmpxchg.org:daniel.m.jordan@oracle.com:laoar.shao@gmail.com:yang.shi@linux.alibaba.com,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1534:1542:1711:1730:1747:1777:1792:2393:2559:2562:2898:3138:3139:3140:3141:3142:3353:3867:5007:6261:6642:7514:8957:10004:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12555:12895:13846:14096:14181:14394:14721:21060:21080:21451:21627:30054:30064:30070,0,RBL:115.124.30.44:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: grade22_5d0197d2e2559 X-Filterd-Recvd-Size: 3733 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:50:38 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R971e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07417;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0TZztT5U_1566294577; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZztT5U_1566294577) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:37 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Kirill Tkhai , Johannes Weiner , Daniel Jordan , Yafang Shao , Yang Shi Subject: [PATCH 12/14] lru/vmscan: use pre lruvec lock in check_move_unevictable_pages Date: Tue, 20 Aug 2019 17:48:35 +0800 Message-Id: <1566294517-86418-13-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: to replace per pgdat lru_lock. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Daniel Jordan Cc: Yafang Shao Cc: Yang Shi Cc: Mel Gorman Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/vmscan.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index defc2c4778eb..123447b9beda 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4249,24 +4249,24 @@ int page_evictable(struct page *page) */ void check_move_unevictable_pages(struct pagevec *pvec) { - struct lruvec *lruvec; - struct pglist_data *pgdat = NULL; + struct lruvec *locked_lruvec = NULL; int pgscanned = 0; int pgrescued = 0; int i; for (i = 0; i < pvec->nr; i++) { struct page *page = pvec->pages[i]; - struct pglist_data *pagepgdat = page_pgdat(page); + struct pglist_data *pgdat = page_pgdat(page); + struct lruvec *lruvec = mem_cgroup_page_lruvec(page, pgdat); pgscanned++; - if (pagepgdat != pgdat) { - if (pgdat) - spin_unlock_irq(&pgdat->lruvec.lru_lock); - pgdat = pagepgdat; - spin_lock_irq(&pgdat->lruvec.lru_lock); + if (lruvec != locked_lruvec) { + if (locked_lruvec) + spin_unlock_irq(&locked_lruvec->lru_lock); + locked_lruvec = lruvec; + spin_lock_irq(&lruvec->lru_lock); + sync_lruvec_pgdat(lruvec, pgdat); } - lruvec = mem_cgroup_page_lruvec(page, pgdat); if (!PageLRU(page) || !PageUnevictable(page)) continue; @@ -4282,10 +4282,10 @@ void check_move_unevictable_pages(struct pagevec *pvec) } } - if (pgdat) { + if (locked_lruvec) { __count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); __count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&locked_lruvec->lru_lock); } } EXPORT_SYMBOL_GPL(check_move_unevictable_pages); From patchwork Tue Aug 20 09:48:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E5FF814DE for ; Tue, 20 Aug 2019 09:49:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B9CF722DA8 for ; Tue, 20 Aug 2019 09:49:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9CF722DA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9E3756B026E; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 945946B0271; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7218A6B026F; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id 397426B026C for ; Tue, 20 Aug 2019 05:49:45 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id DB0598248AB7 for ; Tue, 20 Aug 2019 09:49:44 +0000 (UTC) X-FDA: 75842334288.17.hose08_54fe073c9203e X-Spam-Summary: 2,0,0,b13a5baea6458245,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:mhocko@suse.com:ktkhai@virtuozzo.com:hannes@cmpxchg.org:daniel.m.jordan@oracle.com:laoar.shao@gmail.com:yang.shi@linux.alibaba.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3871:4321:5007:6261:7514:10004:11026:11473:11658:11914:12043:12048:12296:12297:12555:12895:12986:13069:13161:13229:13311:13357:13846:14181:14384:14394:14721:21060:21080:21451:21627:30054:30064,0,RBL:115.124.30.130:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: hose08_54fe073c9203e X-Filterd-Recvd-Size: 3082 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:49:43 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R751e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01422;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0TZznPmz_1566294577; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZznPmz_1566294577) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:38 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Michal Hocko , Kirill Tkhai , Johannes Weiner , Daniel Jordan , Yafang Shao , Yang Shi Subject: [PATCH 13/14] lru/vmscan: using per lruvec lru_lock in get_scan_count Date: Tue, 20 Aug 2019 17:48:36 +0800 Message-Id: <1566294517-86418-14-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The lruvec is passed as parameter, so no lruvec->pgdat syncing needed. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Michal Hocko Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Daniel Jordan Cc: Yafang Shao Cc: Yang Shi Cc: Mel Gorman Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 123447b9beda..ea5c2f3f2567 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2372,7 +2372,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE, MAX_NR_ZONES) + lruvec_lru_size(lruvec, LRU_INACTIVE_FILE, MAX_NR_ZONES); - spin_lock_irq(&pgdat->lruvec.lru_lock); + spin_lock_irq(&lruvec->lru_lock); if (unlikely(reclaim_stat->recent_scanned[0] > anon / 4)) { reclaim_stat->recent_scanned[0] /= 2; reclaim_stat->recent_rotated[0] /= 2; @@ -2393,7 +2393,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, fp = file_prio * (reclaim_stat->recent_scanned[1] + 1); fp /= reclaim_stat->recent_rotated[1] + 1; - spin_unlock_irq(&pgdat->lruvec.lru_lock); + spin_unlock_irq(&lruvec->lru_lock); fraction[0] = ap; fraction[1] = fp; From patchwork Tue Aug 20 09:48:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11103503 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C29081398 for ; Tue, 20 Aug 2019 09:50:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8B21722CF7 for ; Tue, 20 Aug 2019 09:50:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B21722CF7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 21BCA6B0273; Tue, 20 Aug 2019 05:50:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1A59B6B0272; Tue, 20 Aug 2019 05:50:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F11866B0273; Tue, 20 Aug 2019 05:50:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id C614D6B0271 for ; Tue, 20 Aug 2019 05:50:22 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 2D2AB269CE for ; Tue, 20 Aug 2019 09:50:22 +0000 (UTC) X-FDA: 75842335884.04.field72_5a5d0154ce702 X-Spam-Summary: 2,0,0,a792d8d320fba3a4,d41d8cd98f00b204,alex.shi@linux.alibaba.com,:cgroups@vger.kernel.org:linux-kernel@vger.kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:tj@kernel.org:alex.shi@linux.alibaba.com:jgg@ziepe.ca:dan.j.williams@intel.com:vbabka@suse.cz:ira.weiny@intel.com:brouer@redhat.com:aryabinin@virtuozzo.com:jannh@google.com:logang@deltatee.com:jrdr.linux@gmail.com:rcampbell@nvidia.com:tobin@kernel.org:mhocko@suse.com:osalvador@suse.de:richard.weiyang@gmail.com:hannes@cmpxchg.org:pasha.tatashin@oracle.com:arunks@codeaurora.org:willy@infradead.org:darrick.wong@oracle.com:amir73il@gmail.com:dchinner@redhat.com:josef@toxicpanda.com:kirill.shutemov@linux.intel.com:jglisse@redhat.com:mike.kravetz@oracle.com:hughd@google.com:ktkhai@virtuozzo.com:daniel.m.jordan@oracle.com:laoar.shao@gmail.com:yang.shi@linux.alibaba.com,RULES_HIT:2:41:152:355:379:541:800:960:966:968:973:988:989:1260:1261:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:153 5:1593:1 X-HE-Tag: field72_5a5d0154ce702 X-Filterd-Recvd-Size: 8492 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Aug 2019 09:50:20 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0TZzk.Cg_1566294578; Received: from localhost(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0TZzk.Cg_1566294578) by smtp.aliyun-inc.com(127.0.0.1); Tue, 20 Aug 2019 17:49:38 +0800 From: Alex Shi To: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Tejun Heo Cc: Alex Shi , Jason Gunthorpe , Dan Williams , Vlastimil Babka , Ira Weiny , Jesper Dangaard Brouer , Andrey Ryabinin , Jann Horn , Logan Gunthorpe , Souptick Joarder , Ralph Campbell , "Tobin C. Harding" , Michal Hocko , Oscar Salvador , Wei Yang , Johannes Weiner , Pavel Tatashin , Arun KS , Matthew Wilcox , "Darrick J. Wong" , Amir Goldstein , Dave Chinner , Josef Bacik , "Kirill A. Shutemov" , =?utf-8?b?SsOpcsO0?= =?utf-8?b?bWUgR2xpc3Nl?= , Mike Kravetz , Hugh Dickins , Kirill Tkhai , Daniel Jordan , Yafang Shao , Yang Shi Subject: [PATCH 14/14] mm/lru: fix the comments of lru_lock Date: Tue, 20 Aug 2019 17:48:37 +0800 Message-Id: <1566294517-86418-15-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> References: <1566294517-86418-1-git-send-email-alex.shi@linux.alibaba.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since we changed the pgdat->lru_lock to lruvec->lru_lock, have to fix the incorrect comments in code. Also fixed some zone->lru_lock comment error in ancient time. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Jason Gunthorpe Cc: Dan Williams Cc: Vlastimil Babka Cc: Ira Weiny Cc: Jesper Dangaard Brouer Cc: Andrey Ryabinin Cc: Jann Horn Cc: Logan Gunthorpe Cc: Souptick Joarder Cc: Ralph Campbell Cc: "Tobin C. Harding" Cc: Michal Hocko Cc: Oscar Salvador Cc: Mel Gorman Cc: Wei Yang Cc: Johannes Weiner Cc: Pavel Tatashin Cc: Arun KS Cc: Matthew Wilcox Cc: "Darrick J. Wong" Cc: Amir Goldstein Cc: Dave Chinner Cc: Josef Bacik Cc: "Kirill A. Shutemov" Cc: "Jérôme Glisse" Cc: Mike Kravetz Cc: Hugh Dickins Cc: Kirill Tkhai Cc: Daniel Jordan Cc: Yafang Shao Cc: Yang Shi Cc: cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 4 ++-- mm/filemap.c | 4 ++-- mm/rmap.c | 2 +- mm/vmscan.c | 6 +++--- 5 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a7a1083b6fb..f9f990d8f08f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -79,7 +79,7 @@ struct page { struct { /* Page cache and anonymous pages */ /** * @lru: Pageout list, eg. active_list protected by - * pgdat->lru_lock. Sometimes used as a generic list + * lruvec->lru_lock. Sometimes used as a generic list * by the page owner. */ struct list_head lru; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8d0076d084be..d2f782263e42 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -159,7 +159,7 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) struct pglist_data; /* - * zone->lock and the zone lru_lock are two of the hottest locks in the kernel. + * zone->lock and the lru_lock are two of the hottest locks in the kernel. * So add a wild amount of padding here to ensure that they fall into separate * cachelines. There are very few zone structures in the machine, so space * consumption is not a concern here. @@ -295,7 +295,7 @@ struct zone_reclaim_stat { struct lruvec { struct list_head lists[NR_LRU_LISTS]; - /* move lru_lock to per lruvec for memcg */ + /* perf lruvec lru_lock for memcg */ spinlock_t lru_lock; struct zone_reclaim_stat reclaim_stat; diff --git a/mm/filemap.c b/mm/filemap.c index d0cf700bf201..0a604c8284f2 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -100,8 +100,8 @@ * ->swap_lock (try_to_unmap_one) * ->private_lock (try_to_unmap_one) * ->i_pages lock (try_to_unmap_one) - * ->pgdat->lru_lock (follow_page->mark_page_accessed) - * ->pgdat->lru_lock (check_pte_range->isolate_lru_page) + * ->lruvec->lru_lock (follow_page->mark_page_accessed) + * ->lruvec->lru_lock (check_pte_range->isolate_lru_page) * ->private_lock (page_remove_rmap->set_page_dirty) * ->i_pages lock (page_remove_rmap->set_page_dirty) * bdi.wb->list_lock (page_remove_rmap->set_page_dirty) diff --git a/mm/rmap.c b/mm/rmap.c index 003377e24232..6bee4aebced6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -27,7 +27,7 @@ * mapping->i_mmap_rwsem * anon_vma->rwsem * mm->page_table_lock or pte_lock - * pgdat->lru_lock (in mark_page_accessed, isolate_lru_page) + * lruvec->lru_lock (in mark_page_accessed, isolate_lru_page) * swap_lock (in swap_duplicate, swap_info_get) * mmlist_lock (in mmput, drain_mmlist and others) * mapping->private_lock (in __set_page_dirty_buffers) diff --git a/mm/vmscan.c b/mm/vmscan.c index ea5c2f3f2567..1328eb182a3e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1662,7 +1662,7 @@ static __always_inline void update_lru_sizes(struct lruvec *lruvec, } /** - * pgdat->lru_lock is heavily contended. Some of the functions that + * lruvec->lru_lock is heavily contended. Some of the functions that * shrink the lists perform better by taking out a batch of pages * and working on them outside the LRU lock. * @@ -1864,9 +1864,9 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, * processes, from rmap. * * If the pages are mostly unmapped, the processing is fast and it is - * appropriate to hold zone_lru_lock across the whole operation. But if + * appropriate to hold lru_lock across the whole operation. But if * the pages are mapped, the processing is slow (page_referenced()) so we - * should drop zone_lru_lock around each page. It's impossible to balance + * should drop lru_lock around each page. It's impossible to balance * this, so instead we remove the pages from the LRU while processing them. * It is safe to rely on PG_active against the non-LRU pages in here because * nobody will play with that bit on a non-LRU page.