From patchwork Tue Mar 9 05:16:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 12124107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6E9FC433E0 for ; Tue, 9 Mar 2021 05:17:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9875265272 for ; Tue, 9 Mar 2021 05:17:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229623AbhCIFQu (ORCPT ); Tue, 9 Mar 2021 00:16:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229475AbhCIFQe (ORCPT ); Tue, 9 Mar 2021 00:16:34 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F01FCC06174A; Mon, 8 Mar 2021 21:16:33 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id kr3-20020a17090b4903b02900c096fc01deso4508636pjb.4; Mon, 08 Mar 2021 21:16:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=4V4MZz42xx1VLoP7TP10uY71kWCxhoB48xeGZheZ9nc=; b=tfxsVUrwWlUxMEfunHfoS0YSUrrnZDVA/XHNPDt1qz70CQKD4qKdBIZkPVzK7/JhrA YRBkYB8/2QBcOZUnUOg8MSY1TGxjp4SYtDSJ4+IhbsxYHhVGE7uGN2V/DDSdiZhm0708 /45j/oYpq//uFU8V9iwvT2BXQR5cdDCO7UfjV7kxhNCUVDVLiY5e58/PZAVQtT0kYYVL y8qkjg7i0s+lYW040l6+DR0RhYMV4lOXnuKNCCN/iQMQhGVdqmH3BTjwnElojoc15CFc N8cgJmwvyZLLKSkM2u9WDum1SH8Y3L2kMsdJe9KjtKsrytWxyk6P5kmjW22uVoRYyaBT Kahg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :mime-version:content-transfer-encoding; bh=4V4MZz42xx1VLoP7TP10uY71kWCxhoB48xeGZheZ9nc=; b=EWlgH+0XZ1OUkq5M727+3vbCSA/QbAWvfHobCIqDeqDzPO3HpnZTts8He4EXpWGYZ3 yyI/K0EftkSfUjF7Bzyenw7aerUM/3Z+q7bDZabMG4s+RvvREjBUW8g4m+zW3kcml/rl oHhUq3lWQEZYAD8tZvCkHU5fP9IcU2In75IMqEq5FHVu8JmWYfL9BAplf6tvM5Ad1IBp 1j5M/4Q05Xn3tLJbrNpq5jGehRZO9R7+eac4m3q19GR4v4NoRrveugEcWdSuk4yEL3cv pAjRaijMjvkAwWSe8jd8A6J96Pgl7ydX4P7fW8WKYSky7neJSHQ7QYYTSNGsyb77mo6k 9FOw== X-Gm-Message-State: AOAM533tI6VNehSZltRDmAUe5ivQDSeTTwltf9zA4G4mpqTHh+0iB0rt +iH9kwu1iw8QTH4C9BShdbo= X-Google-Smtp-Source: ABdhPJxQqdUv5sskQWjEWKWv529xmH6AU0zMc288xT3dmzwNiQT0O5UbGGP0L8x0CteBeZxnu9ssHQ== X-Received: by 2002:a17:902:9897:b029:e6:3e0a:b44a with SMTP id s23-20020a1709029897b02900e63e0ab44amr2315268plp.30.1615266993403; Mon, 08 Mar 2021 21:16:33 -0800 (PST) Received: from bbox-1.mtv.corp.google.com ([2620:15c:211:201:4ccc:acdd:25da:14d1]) by smtp.gmail.com with ESMTPSA id b14sm1166576pji.14.2021.03.08.21.16.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Mar 2021 21:16:32 -0800 (PST) Sender: Minchan Kim From: Minchan Kim To: Andrew Morton Cc: linux-mm , LKML , joaodias@google.com, surenb@google.com, cgoldswo@codeaurora.org, willy@infradead.org, mhocko@suse.com, david@redhat.com, vbabka@suse.cz, linux-fsdevel@vger.kernel.org, Minchan Kim Subject: [PATCH v2 1/2] mm: disable LRU pagevec during the migration temporarily Date: Mon, 8 Mar 2021 21:16:27 -0800 Message-Id: <20210309051628.3105973-1-minchan@kernel.org> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org LRU pagevec holds refcount of pages until the pagevec are drained. It could prevent migration since the refcount of the page is greater than the expection in migration logic. To mitigate the issue, callers of migrate_pages drains LRU pagevec via migrate_prep or lru_add_drain_all before migrate_pages call. However, it's not enough because pages coming into pagevec after the draining call still could stay at the pagevec so it could keep preventing page migration. Since some callers of migrate_pages have retrial logic with LRU draining, the page would migrate at next trail but it is still fragile in that it doesn't close the fundamental race between upcoming LRU pages into pagvec and migration so the migration failure could cause contiguous memory allocation failure in the end. To close the race, this patch disables lru caches(i.e, pagevec) during ongoing migration until migrate is done. Since it's really hard to reproduce, I measured how many times migrate_pages retried with force mode below debug code. int migrate_pages(struct list_head *from, new_page_t get_new_page, .. .. if (rc && reason == MR_CONTIG_RANGE && pass > 2) { printk(KERN_ERR, "pfn 0x%lx reason %d\n", page_to_pfn(page), rc); dump_page(page, "fail to migrate"); } The test was repeating android apps launching with cma allocation in background every five seconds. Total cma allocation count was about 500 during the testing. With this patch, the dump_page count was reduced from 400 to 30. It would be also useful for memory-hotplug. Signed-off-by: Minchan Kim --- * from v1 - https://lore.kernel.org/lkml/20210302210949.2440120-1-minchan@kernel.org/ * introduce __lru_add_drain_all to minimize changes - mhocko * use lru_cache_disable for memory-hotplug * schedule for every cpu at force_all_cpus * from RFC - http://lore.kernel.org/linux-mm/20210216170348.1513483-1-minchan@kernel.org * use atomic and lru_add_drain_all for strict ordering - mhocko * lru_cache_disable/enable - mhocko include/linux/migrate.h | 6 ++- include/linux/swap.h | 2 + mm/memory_hotplug.c | 3 +- mm/mempolicy.c | 6 +++ mm/migrate.c | 13 ++++--- mm/page_alloc.c | 3 ++ mm/swap.c | 82 +++++++++++++++++++++++++++++++++-------- 7 files changed, 91 insertions(+), 24 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3a389633b68f..6a23174ea081 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -47,6 +47,7 @@ extern void putback_movable_page(struct page *page); extern void migrate_prep(void); extern void migrate_prep_local(void); +extern void migrate_finish(void); extern void migrate_page_states(struct page *newpage, struct page *page); extern void migrate_page_copy(struct page *newpage, struct page *page); extern int migrate_huge_page_move_mapping(struct address_space *mapping, @@ -66,8 +67,9 @@ static inline struct page *alloc_migration_target(struct page *page, static inline int isolate_movable_page(struct page *page, isolate_mode_t mode) { return -EBUSY; } -static inline int migrate_prep(void) { return -ENOSYS; } -static inline int migrate_prep_local(void) { return -ENOSYS; } +static inline void migrate_prep(void) { return -ENOSYS; } +static inline void migrate_prep_local(void) { return -ENOSYS; } +static inline void migrate_done(void) {} static inline void migrate_page_states(struct page *newpage, struct page *page) { diff --git a/include/linux/swap.h b/include/linux/swap.h index 71166bc10d17..aaa6b9cc3f8a 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -339,6 +339,8 @@ extern void lru_note_cost(struct lruvec *lruvec, bool file, extern void lru_note_cost_page(struct page *); extern void lru_cache_add(struct page *); extern void mark_page_accessed(struct page *); +extern void lru_cache_disable(void); +extern void lru_cache_enable(void); extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index a969463bdda4..9a5af00802f9 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1571,6 +1571,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages) * in a way that pages from isolated pageblock are left on pcplists. */ zone_pcp_disable(zone); + lru_cache_disable(); /* set above range as isolated */ ret = start_isolate_page_range(start_pfn, end_pfn, @@ -1602,7 +1603,6 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages) } cond_resched(); - lru_add_drain_all(); ret = scan_movable_pages(pfn, end_pfn, &pfn); if (!ret) { @@ -1647,6 +1647,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages) zone->nr_isolate_pageblock -= nr_pages / pageblock_nr_pages; spin_unlock_irqrestore(&zone->lock, flags); + lru_cache_enable(); zone_pcp_enable(zone); /* removal success */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 6961238c7ef5..46d9986c7bf0 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1208,6 +1208,8 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, break; } mmap_read_unlock(mm); + migrate_finish(); + if (err < 0) return err; return busy; @@ -1371,6 +1373,10 @@ static long do_mbind(unsigned long start, unsigned long len, mmap_write_unlock(mm); mpol_out: mpol_put(new); + + if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) + migrate_finish(); + return err; } diff --git a/mm/migrate.c b/mm/migrate.c index a69da8aaeccd..3de67c5f70bc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -65,12 +65,9 @@ void migrate_prep(void) { /* - * Clear the LRU lists so pages can be isolated. - * Note that pages may be moved off the LRU after we have - * drained them. Those pages will fail to migrate like other - * pages that may be busy. + * Clear the LRU pcp lists so pages can be isolated. */ - lru_add_drain_all(); + lru_cache_disable(); } /* Do the necessary work of migrate_prep but not if it involves other CPUs */ @@ -79,6 +76,11 @@ void migrate_prep_local(void) lru_add_drain(); } +void migrate_finish(void) +{ + lru_cache_enable(); +} + int isolate_movable_page(struct page *page, isolate_mode_t mode) { struct address_space *mapping; @@ -1837,6 +1839,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, if (err >= 0) err = err1; out: + migrate_finish(); return err; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9811663fcf0b..6c0b5c6a4779 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8491,6 +8491,9 @@ static int __alloc_contig_migrate_range(struct compact_control *cc, ret = migrate_pages(&cc->migratepages, alloc_migration_target, NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE); } + + migrate_finish(); + if (ret < 0) { putback_movable_pages(&cc->migratepages); return ret; diff --git a/mm/swap.c b/mm/swap.c index 31b844d4ed94..fc8acccb882b 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -235,6 +235,18 @@ static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec) } } +/* return true if pagevec needs to drain */ +static bool pagevec_add_and_need_flush(struct pagevec *pvec, struct page *page) +{ + bool ret = false; + + if (!pagevec_add(pvec, page) || PageCompound(page) || + lru_cache_disabled()) + ret = true; + + return ret; +} + /* * Writeback is about to end against a page which has been marked for immediate * reclaim. If it still appears to be reclaimable, move it to the tail of the @@ -252,7 +264,7 @@ void rotate_reclaimable_page(struct page *page) get_page(page); local_lock_irqsave(&lru_rotate.lock, flags); pvec = this_cpu_ptr(&lru_rotate.pvec); - if (!pagevec_add(pvec, page) || PageCompound(page)) + if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, pagevec_move_tail_fn); local_unlock_irqrestore(&lru_rotate.lock, flags); } @@ -343,7 +355,7 @@ static void activate_page(struct page *page) local_lock(&lru_pvecs.lock); pvec = this_cpu_ptr(&lru_pvecs.activate_page); get_page(page); - if (!pagevec_add(pvec, page) || PageCompound(page)) + if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, __activate_page); local_unlock(&lru_pvecs.lock); } @@ -458,7 +470,7 @@ void lru_cache_add(struct page *page) get_page(page); local_lock(&lru_pvecs.lock); pvec = this_cpu_ptr(&lru_pvecs.lru_add); - if (!pagevec_add(pvec, page) || PageCompound(page)) + if (pagevec_add_and_need_flush(pvec, page)) __pagevec_lru_add(pvec); local_unlock(&lru_pvecs.lock); } @@ -654,7 +666,7 @@ void deactivate_file_page(struct page *page) local_lock(&lru_pvecs.lock); pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate_file); - if (!pagevec_add(pvec, page) || PageCompound(page)) + if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, lru_deactivate_file_fn); local_unlock(&lru_pvecs.lock); } @@ -676,7 +688,7 @@ void deactivate_page(struct page *page) local_lock(&lru_pvecs.lock); pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate); get_page(page); - if (!pagevec_add(pvec, page) || PageCompound(page)) + if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, lru_deactivate_fn); local_unlock(&lru_pvecs.lock); } @@ -698,7 +710,7 @@ void mark_page_lazyfree(struct page *page) local_lock(&lru_pvecs.lock); pvec = this_cpu_ptr(&lru_pvecs.lru_lazyfree); get_page(page); - if (!pagevec_add(pvec, page) || PageCompound(page)) + if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, lru_lazyfree_fn); local_unlock(&lru_pvecs.lock); } @@ -728,14 +740,7 @@ static void lru_add_drain_per_cpu(struct work_struct *dummy) lru_add_drain(); } -/* - * Doesn't need any cpu hotplug locking because we do rely on per-cpu - * kworkers being shut down before our page_alloc_cpu_dead callback is - * executed on the offlined cpu. - * Calling this function with cpu hotplug locks held can actually lead - * to obscure indirect dependencies via WQ context. - */ -void lru_add_drain_all(void) +void __lru_add_drain_all(bool force_all_cpus) { /* * lru_drain_gen - Global pages generation number @@ -780,7 +785,7 @@ void lru_add_drain_all(void) * (C) Exit the draining operation if a newer generation, from another * lru_add_drain_all(), was already scheduled for draining. Check (A). */ - if (unlikely(this_gen != lru_drain_gen)) + if (unlikely(this_gen != lru_drain_gen && !force_all_cpus)) goto done; /* @@ -810,7 +815,8 @@ void lru_add_drain_all(void) for_each_online_cpu(cpu) { struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); - if (pagevec_count(&per_cpu(lru_pvecs.lru_add, cpu)) || + if (force_all_cpus || + pagevec_count(&per_cpu(lru_pvecs.lru_add, cpu)) || data_race(pagevec_count(&per_cpu(lru_rotate.pvec, cpu))) || pagevec_count(&per_cpu(lru_pvecs.lru_deactivate_file, cpu)) || pagevec_count(&per_cpu(lru_pvecs.lru_deactivate, cpu)) || @@ -828,6 +834,18 @@ void lru_add_drain_all(void) done: mutex_unlock(&lock); } + +/* + * Doesn't need any cpu hotplug locking because we do rely on per-cpu + * kworkers being shut down before our page_alloc_cpu_dead callback is + * executed on the offlined cpu. + * Calling this function with cpu hotplug locks held can actually lead + * to obscure indirect dependencies via WQ context. + */ +void lru_add_drain_all(void) +{ + __lru_add_drain_all(false); +} #else void lru_add_drain_all(void) { @@ -835,6 +853,38 @@ void lru_add_drain_all(void) } #endif /* CONFIG_SMP */ +static atomic_t lru_disable_count = ATOMIC_INIT(0); + +bool lru_cache_disabled(void) +{ + return atomic_read(&lru_disable_count); +} + +void lru_cache_enable(void) +{ + atomic_dec(&lru_disable_count); +} +/* + * Clear the LRU lists so pages can be isolated. + */ +void lru_cache_disable(void) +{ + atomic_inc(&lru_disable_count); +#ifdef CONFIG_SMP + /* + * lru_add_drain_all in the force mode will schedule draining on + * all online CPUs so any calls of lru_cache_disabled wrapped by + * local_lock or preemption disabled would be ordered by that. + * The atomic operation doesn't need to have stronger ordering + * requirements because that is enforeced by the scheduling + * guarantees. + */ + __lru_add_drain_all(true); +#else + lru_add_drain(); +#endif +} + /** * release_pages - batched put_page() * @pages: array of pages to release From patchwork Tue Mar 9 05:16:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 12124109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ADE9C433E9 for ; Tue, 9 Mar 2021 05:17:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA0DC65290 for ; Tue, 9 Mar 2021 05:17:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229652AbhCIFQu (ORCPT ); Tue, 9 Mar 2021 00:16:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229495AbhCIFQf (ORCPT ); Tue, 9 Mar 2021 00:16:35 -0500 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1A16C06174A; Mon, 8 Mar 2021 21:16:35 -0800 (PST) Received: by mail-pf1-x42c.google.com with SMTP id t29so8570718pfg.11; Mon, 08 Mar 2021 21:16:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HORwLZBItmmfsgXHtTtePPTuVp6GLP2tVSIDV66WVpQ=; b=eYdb52HNI0m9k1dSzBzYPe/yrsDUeQf5/V4fbqkyN1tOZia9b3PlIFcT+8IAiPqp31 0EnqPEurKW/f5Wz2ZEOljJE6VotAKSGo8eyKcHfa3tdLyuSwT3vBgX9CBiUtempawwfM Fkq47xN1Ln/etM3HJSMe7V1aa7YjE4kAiXWRxfVHKTNN50lKeOlXszTj/gZvDwCycLpK en4dtCuovMRi7yLuj8x7cXw0K4BYIXtErvrP0b5CQMXjZfMqCbWpqUe9YSxbVcoHMc4W FF3kymnBFMVj60tpEl4e+kuk3bKIEqZdJywogZHISlt1opQjO6Pxytc4nS+ZTBXAK5gP L1nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=HORwLZBItmmfsgXHtTtePPTuVp6GLP2tVSIDV66WVpQ=; b=J40vmkKVL7d/gEjIvOrscHV8DgbytlUNfJUHyooPoDHMyBLOCygxwshGx/p9bhKd+H Z+kk3yjO8oJ4gMPr8j2iyNVoAbo3hvUmjoMLzW1eh4p8spPNKuoXGxxutLEdmTzIKL/v 2Thqb8ojUCMO2jgpA0Uiu0N5SweidG8xyduI4z8RrVGMJzizLe0YkG1cyEklpGAQyYiO cYEi4l10HXQCAZkbJyby378xKNvJQH6qfX27FRPgC022B4TRV4iEkmwK98sSRGjTsgsQ 2NUQEU2ZTrokdWoOi8z38yT/+9+vYPn38Pnys+fDZMuNpJpubQYO6Q7SnQ+FXOR/Zn9c 2Xvg== X-Gm-Message-State: AOAM531jDPLl/MLEsNcUf+miymixs304YBAf0F+PuXunJohb+BXYIW4d TNzcLPrgFOqNipgYRKngR750NHJwdQE= X-Google-Smtp-Source: ABdhPJxd5K5oogovLzgD2OZCi1I8c5sIi7DN9DXg2E/knjxADowEPx0V6diq32qsKGpQ1QeBNfYM4A== X-Received: by 2002:aa7:95b5:0:b029:1ef:272f:920c with SMTP id a21-20020aa795b50000b02901ef272f920cmr22146023pfk.21.1615266995451; Mon, 08 Mar 2021 21:16:35 -0800 (PST) Received: from bbox-1.mtv.corp.google.com ([2620:15c:211:201:4ccc:acdd:25da:14d1]) by smtp.gmail.com with ESMTPSA id b14sm1166576pji.14.2021.03.08.21.16.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Mar 2021 21:16:34 -0800 (PST) Sender: Minchan Kim From: Minchan Kim To: Andrew Morton Cc: linux-mm , LKML , joaodias@google.com, surenb@google.com, cgoldswo@codeaurora.org, willy@infradead.org, mhocko@suse.com, david@redhat.com, vbabka@suse.cz, linux-fsdevel@vger.kernel.org, Minchan Kim Subject: [PATCH v2 2/2] mm: fs: Invalidate BH LRU during page migration Date: Mon, 8 Mar 2021 21:16:28 -0800 Message-Id: <20210309051628.3105973-2-minchan@kernel.org> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog In-Reply-To: <20210309051628.3105973-1-minchan@kernel.org> References: <20210309051628.3105973-1-minchan@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Pages containing buffer_heads that are in one of the per-CPU buffer_head LRU caches will be pinned and thus cannot be migrated. This can prevent CMA allocations from succeeding, which are often used on platforms with co-processors (such as a DSP) that can only use physically contiguous memory. It can also prevent memory hot-unplugging from succeeding, which involves migrating at least MIN_MEMORY_BLOCK_SIZE bytes of memory, which ranges from 8 MiB to 1 GiB based on the architecture in use. Correspondingly, invalidate the BH LRU caches before a migration starts and stop any buffer_head from being cached in the LRU caches, until migration has finished. Signed-off-by: Chris Goldsworthy Signed-off-by: Minchan Kim --- fs/buffer.c | 12 ++++++++++-- include/linux/buffer_head.h | 3 +++ include/linux/swap.h | 1 + mm/swap.c | 5 ++++- 4 files changed, 18 insertions(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 96c7604f69b3..4492e9d4c9d3 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1301,6 +1301,14 @@ static void bh_lru_install(struct buffer_head *bh) int i; check_irqs_on(); + /* + * buffer_head in bh_lru could increase refcount of the page + * until it will be invalidated. It causes page migraion failure. + * Skip putting upcoming bh into bh_lru until migration is done. + */ + if (lru_cache_disabled()) + return; + bh_lru_lock(); b = this_cpu_ptr(&bh_lrus); @@ -1446,7 +1454,7 @@ EXPORT_SYMBOL(__bread_gfp); * This doesn't race because it runs in each cpu either in irq * or with preempt disabled. */ -static void invalidate_bh_lru(void *arg) +void invalidate_bh_lru(void *arg) { struct bh_lru *b = &get_cpu_var(bh_lrus); int i; @@ -1458,7 +1466,7 @@ static void invalidate_bh_lru(void *arg) put_cpu_var(bh_lrus); } -static bool has_bh_in_lru(int cpu, void *dummy) +bool has_bh_in_lru(int cpu, void *dummy) { struct bh_lru *b = per_cpu_ptr(&bh_lrus, cpu); int i; diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 6b47f94378c5..3ae62f3f788e 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -194,6 +194,8 @@ void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size, struct buffer_head *__bread_gfp(struct block_device *, sector_t block, unsigned size, gfp_t gfp); void invalidate_bh_lrus(void); +void invalidate_bh_lru(void *arg); +bool has_bh_in_lru(int cpu, void *dummy); struct buffer_head *alloc_buffer_head(gfp_t gfp_flags); void free_buffer_head(struct buffer_head * bh); void unlock_buffer(struct buffer_head *bh); @@ -406,6 +408,7 @@ static inline int inode_has_buffers(struct inode *inode) { return 0; } static inline void invalidate_inode_buffers(struct inode *inode) {} static inline int remove_inode_buffers(struct inode *inode) { return 1; } static inline int sync_mapping_buffers(struct address_space *mapping) { return 0; } +static inline void invalidate_bh_lru(void *arg) {} #define buffer_heads_over_limit 0 #endif /* CONFIG_BLOCK */ diff --git a/include/linux/swap.h b/include/linux/swap.h index aaa6b9cc3f8a..5386cce1a26d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -341,6 +341,7 @@ extern void lru_cache_add(struct page *); extern void mark_page_accessed(struct page *); extern void lru_cache_disable(void); extern void lru_cache_enable(void); +extern bool lru_cache_disabled(void); extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); diff --git a/mm/swap.c b/mm/swap.c index fc8acccb882b..d599d6449154 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "internal.h" @@ -641,6 +642,7 @@ void lru_add_drain_cpu(int cpu) pagevec_lru_move_fn(pvec, lru_lazyfree_fn); activate_page_drain(cpu); + invalidate_bh_lru(NULL); } /** @@ -821,7 +823,8 @@ void __lru_add_drain_all(bool force_all_cpus) pagevec_count(&per_cpu(lru_pvecs.lru_deactivate_file, cpu)) || pagevec_count(&per_cpu(lru_pvecs.lru_deactivate, cpu)) || pagevec_count(&per_cpu(lru_pvecs.lru_lazyfree, cpu)) || - need_activate_page_drain(cpu)) { + need_activate_page_drain(cpu) || + has_bh_in_lru(cpu, NULL)) { INIT_WORK(work, lru_add_drain_per_cpu); queue_work_on(cpu, mm_percpu_wq, work); __cpumask_set_cpu(cpu, &has_work);