| Message ID | 20220921060616.73086-2-ying.huang@intel.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | migrate_pages(): batch TLB flushing | expand |
On 21 Sep 2022, at 2:06, Huang Ying wrote: > This is a preparation patch to batch the page unmapping and moving for > the normal pages and THPs. Based on that we can batch the TLB > shootdown during the page migration and make it possible to use some > hardware accelerator for the page copying. > > In this patch the huge page (PageHuge()) and normal page and THP > migration is separated in migrate_pages() to make it easy to change > the normal page and THP migration implementation. > > Signed-off-by: "Huang, Ying" <ying.huang@intel.com> > Cc: Zi Yan <ziy@nvidia.com> > Cc: Yang Shi <shy828301@gmail.com> > Cc: Baolin Wang <baolin.wang@linux.alibaba.com> > Cc: Oscar Salvador <osalvador@suse.de> > Cc: Matthew Wilcox <willy@infradead.org> > --- > mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 64 insertions(+), 9 deletions(-) Maybe it would be better to have two subroutines for hugetlb migration and normal page migration respectively. migrate_pages() becomes very large at this point. > > diff --git a/mm/migrate.c b/mm/migrate.c > index 571d8c9fd5bc..117134f1c6dc 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > > trace_mm_migrate_pages_start(mode, reason); > > + for (pass = 0; pass < 10 && retry; pass++) { > + retry = 0; > + > + list_for_each_entry_safe(page, page2, from, lru) { > + nr_subpages = compound_nr(page); > + cond_resched(); > + > + if (!PageHuge(page)) > + continue; > + > + rc = unmap_and_move_huge_page(get_new_page, > + put_new_page, private, page, > + pass > 2, mode, reason, > + &ret_pages); > + /* > + * The rules are: > + * Success: hugetlb page will be put back > + * -EAGAIN: stay on the from list > + * -ENOMEM: stay on the from list > + * -ENOSYS: stay on the from list > + * Other errno: put on ret_pages list then splice to > + * from list > + */ > + switch(rc) { > + case -ENOSYS: > + /* Hugetlb migration is unsupported */ > + nr_failed++; > + nr_failed_pages += nr_subpages; > + list_move_tail(&page->lru, &ret_pages); > + break; > + case -ENOMEM: > + /* > + * When memory is low, don't bother to try to migrate > + * other pages, just exit. > + */ > + nr_failed++; > + nr_failed_pages += nr_subpages + nr_retry_pages; > + goto out; > + case -EAGAIN: > + retry++; > + nr_retry_pages += nr_subpages; > + break; > + case MIGRATEPAGE_SUCCESS: > + nr_succeeded += nr_subpages; > + break; > + default: > + /* > + * Permanent failure (-EBUSY, etc.): > + * unlike -EAGAIN case, the failed page is > + * removed from migration page list and not > + * retried in the next outer loop. > + */ > + nr_failed++; > + nr_failed_pages += nr_subpages; > + break; > + } > + } > + } > + nr_failed += retry; > + retry = 1; > thp_subpage_migration: > for (pass = 0; pass < 10 && (retry || thp_retry); pass++) { > retry = 0; > @@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > cond_resched(); > > if (PageHuge(page)) > - rc = unmap_and_move_huge_page(get_new_page, > - put_new_page, private, page, > - pass > 2, mode, reason, > - &ret_pages); > - else > - rc = unmap_and_move(get_new_page, put_new_page, > + continue; > + > + rc = unmap_and_move(get_new_page, put_new_page, > private, page, pass > 2, mode, > reason, &ret_pages); > /* > * The rules are: > - * Success: non hugetlb page will be freed, hugetlb > - * page will be put back > + * Success: page will be freed > * -EAGAIN: stay on the from list > * -ENOMEM: stay on the from list > * -ENOSYS: stay on the from list > @@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > nr_thp_split++; > break; > } > - /* Hugetlb migration is unsupported */ > } else if (!no_subpage_counting) { > nr_failed++; > } > -- > 2.35.1 -- Best Regards, Yan, Zi
Hi, Zi, Thank you for comments! Zi Yan <ziy@nvidia.com> writes: > On 21 Sep 2022, at 2:06, Huang Ying wrote: > >> This is a preparation patch to batch the page unmapping and moving for >> the normal pages and THPs. Based on that we can batch the TLB >> shootdown during the page migration and make it possible to use some >> hardware accelerator for the page copying. >> >> In this patch the huge page (PageHuge()) and normal page and THP >> migration is separated in migrate_pages() to make it easy to change >> the normal page and THP migration implementation. >> >> Signed-off-by: "Huang, Ying" <ying.huang@intel.com> >> Cc: Zi Yan <ziy@nvidia.com> >> Cc: Yang Shi <shy828301@gmail.com> >> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> >> Cc: Oscar Salvador <osalvador@suse.de> >> Cc: Matthew Wilcox <willy@infradead.org> >> --- >> mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++------- >> 1 file changed, 64 insertions(+), 9 deletions(-) > > Maybe it would be better to have two subroutines for hugetlb migration > and normal page migration respectively. migrate_pages() becomes very > large at this point. Yes. migrate_pages() becomes even larger with this patchset. I will consider more about how to deal with that. I will try the method in your comments in [3/6] for that too. Best Regards, Huang, Ying >> >> diff --git a/mm/migrate.c b/mm/migrate.c >> index 571d8c9fd5bc..117134f1c6dc 100644 >> --- a/mm/migrate.c >> +++ b/mm/migrate.c >> @@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> >> trace_mm_migrate_pages_start(mode, reason); >> >> + for (pass = 0; pass < 10 && retry; pass++) { >> + retry = 0; >> + >> + list_for_each_entry_safe(page, page2, from, lru) { >> + nr_subpages = compound_nr(page); >> + cond_resched(); >> + >> + if (!PageHuge(page)) >> + continue; >> + >> + rc = unmap_and_move_huge_page(get_new_page, >> + put_new_page, private, page, >> + pass > 2, mode, reason, >> + &ret_pages); >> + /* >> + * The rules are: >> + * Success: hugetlb page will be put back >> + * -EAGAIN: stay on the from list >> + * -ENOMEM: stay on the from list >> + * -ENOSYS: stay on the from list >> + * Other errno: put on ret_pages list then splice to >> + * from list >> + */ >> + switch(rc) { >> + case -ENOSYS: >> + /* Hugetlb migration is unsupported */ >> + nr_failed++; >> + nr_failed_pages += nr_subpages; >> + list_move_tail(&page->lru, &ret_pages); >> + break; >> + case -ENOMEM: >> + /* >> + * When memory is low, don't bother to try to migrate >> + * other pages, just exit. >> + */ >> + nr_failed++; >> + nr_failed_pages += nr_subpages + nr_retry_pages; >> + goto out; >> + case -EAGAIN: >> + retry++; >> + nr_retry_pages += nr_subpages; >> + break; >> + case MIGRATEPAGE_SUCCESS: >> + nr_succeeded += nr_subpages; >> + break; >> + default: >> + /* >> + * Permanent failure (-EBUSY, etc.): >> + * unlike -EAGAIN case, the failed page is >> + * removed from migration page list and not >> + * retried in the next outer loop. >> + */ >> + nr_failed++; >> + nr_failed_pages += nr_subpages; >> + break; >> + } >> + } >> + } >> + nr_failed += retry; >> + retry = 1; >> thp_subpage_migration: >> for (pass = 0; pass < 10 && (retry || thp_retry); pass++) { >> retry = 0; >> @@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> cond_resched(); >> >> if (PageHuge(page)) >> - rc = unmap_and_move_huge_page(get_new_page, >> - put_new_page, private, page, >> - pass > 2, mode, reason, >> - &ret_pages); >> - else >> - rc = unmap_and_move(get_new_page, put_new_page, >> + continue; >> + >> + rc = unmap_and_move(get_new_page, put_new_page, >> private, page, pass > 2, mode, >> reason, &ret_pages); >> /* >> * The rules are: >> - * Success: non hugetlb page will be freed, hugetlb >> - * page will be put back >> + * Success: page will be freed >> * -EAGAIN: stay on the from list >> * -ENOMEM: stay on the from list >> * -ENOSYS: stay on the from list >> @@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> nr_thp_split++; >> break; >> } >> - /* Hugetlb migration is unsupported */ >> } else if (!no_subpage_counting) { >> nr_failed++; >> } >> -- >> 2.35.1 > > > -- > Best Regards, > Yan, Zi
On 9/21/2022 2:06 PM, Huang Ying wrote: > This is a preparation patch to batch the page unmapping and moving for > the normal pages and THPs. Based on that we can batch the TLB > shootdown during the page migration and make it possible to use some > hardware accelerator for the page copying. > > In this patch the huge page (PageHuge()) and normal page and THP > migration is separated in migrate_pages() to make it easy to change > the normal page and THP migration implementation. > > Signed-off-by: "Huang, Ying" <ying.huang@intel.com> > Cc: Zi Yan <ziy@nvidia.com> > Cc: Yang Shi <shy828301@gmail.com> > Cc: Baolin Wang <baolin.wang@linux.alibaba.com> > Cc: Oscar Salvador <osalvador@suse.de> > Cc: Matthew Wilcox <willy@infradead.org> > --- > mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 64 insertions(+), 9 deletions(-) > > diff --git a/mm/migrate.c b/mm/migrate.c > index 571d8c9fd5bc..117134f1c6dc 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > > trace_mm_migrate_pages_start(mode, reason); > > + for (pass = 0; pass < 10 && retry; pass++) { > + retry = 0; > + > + list_for_each_entry_safe(page, page2, from, lru) { > + nr_subpages = compound_nr(page); > + cond_resched(); > + > + if (!PageHuge(page)) > + continue; > + > + rc = unmap_and_move_huge_page(get_new_page, > + put_new_page, private, page, > + pass > 2, mode, reason, > + &ret_pages); > + /* > + * The rules are: > + * Success: hugetlb page will be put back > + * -EAGAIN: stay on the from list > + * -ENOMEM: stay on the from list > + * -ENOSYS: stay on the from list > + * Other errno: put on ret_pages list then splice to > + * from list > + */ > + switch(rc) { > + case -ENOSYS: > + /* Hugetlb migration is unsupported */ > + nr_failed++; > + nr_failed_pages += nr_subpages; > + list_move_tail(&page->lru, &ret_pages); > + break; > + case -ENOMEM: > + /* > + * When memory is low, don't bother to try to migrate > + * other pages, just exit. > + */ > + nr_failed++; > + nr_failed_pages += nr_subpages + nr_retry_pages; > + goto out; > + case -EAGAIN: > + retry++; > + nr_retry_pages += nr_subpages; > + break; > + case MIGRATEPAGE_SUCCESS: > + nr_succeeded += nr_subpages; > + break; > + default: > + /* > + * Permanent failure (-EBUSY, etc.): > + * unlike -EAGAIN case, the failed page is > + * removed from migration page list and not > + * retried in the next outer loop. > + */ > + nr_failed++; > + nr_failed_pages += nr_subpages; > + break; > + } > + } > + } > + nr_failed += retry; Seems we should also record the nr_retry_pages? since the second loop will reset the nr_retry_pages. nr_failed_pages += nr_retry_pages; Besides, I also agree with Zi Yan's comment to simplify this larger function.
Baolin Wang <baolin.wang@linux.alibaba.com> writes: > On 9/21/2022 2:06 PM, Huang Ying wrote: >> This is a preparation patch to batch the page unmapping and moving for >> the normal pages and THPs. Based on that we can batch the TLB >> shootdown during the page migration and make it possible to use some >> hardware accelerator for the page copying. >> In this patch the huge page (PageHuge()) and normal page and THP >> migration is separated in migrate_pages() to make it easy to change >> the normal page and THP migration implementation. >> Signed-off-by: "Huang, Ying" <ying.huang@intel.com> >> Cc: Zi Yan <ziy@nvidia.com> >> Cc: Yang Shi <shy828301@gmail.com> >> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> >> Cc: Oscar Salvador <osalvador@suse.de> >> Cc: Matthew Wilcox <willy@infradead.org> >> --- >> mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++------- >> 1 file changed, 64 insertions(+), 9 deletions(-) >> diff --git a/mm/migrate.c b/mm/migrate.c >> index 571d8c9fd5bc..117134f1c6dc 100644 >> --- a/mm/migrate.c >> +++ b/mm/migrate.c >> @@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> trace_mm_migrate_pages_start(mode, reason); >> + for (pass = 0; pass < 10 && retry; pass++) { >> + retry = 0; >> + >> + list_for_each_entry_safe(page, page2, from, lru) { >> + nr_subpages = compound_nr(page); >> + cond_resched(); >> + >> + if (!PageHuge(page)) >> + continue; >> + >> + rc = unmap_and_move_huge_page(get_new_page, >> + put_new_page, private, page, >> + pass > 2, mode, reason, >> + &ret_pages); >> + /* >> + * The rules are: >> + * Success: hugetlb page will be put back >> + * -EAGAIN: stay on the from list >> + * -ENOMEM: stay on the from list >> + * -ENOSYS: stay on the from list >> + * Other errno: put on ret_pages list then splice to >> + * from list >> + */ >> + switch(rc) { >> + case -ENOSYS: >> + /* Hugetlb migration is unsupported */ >> + nr_failed++; >> + nr_failed_pages += nr_subpages; >> + list_move_tail(&page->lru, &ret_pages); >> + break; >> + case -ENOMEM: >> + /* >> + * When memory is low, don't bother to try to migrate >> + * other pages, just exit. >> + */ >> + nr_failed++; >> + nr_failed_pages += nr_subpages + nr_retry_pages; >> + goto out; >> + case -EAGAIN: >> + retry++; >> + nr_retry_pages += nr_subpages; >> + break; >> + case MIGRATEPAGE_SUCCESS: >> + nr_succeeded += nr_subpages; >> + break; >> + default: >> + /* >> + * Permanent failure (-EBUSY, etc.): >> + * unlike -EAGAIN case, the failed page is >> + * removed from migration page list and not >> + * retried in the next outer loop. >> + */ >> + nr_failed++; >> + nr_failed_pages += nr_subpages; >> + break; >> + } >> + } >> + } >> + nr_failed += retry; > > Seems we should also record the nr_retry_pages? since the second loop > will reset the nr_retry_pages. > > nr_failed_pages += nr_retry_pages; Good catch! Will do that in the next version. > Besides, I also agree with Zi Yan's comment to simplify this larger > function. Yes. I think so too. Best Regards, Huang, Ying
diff --git a/mm/migrate.c b/mm/migrate.c index 571d8c9fd5bc..117134f1c6dc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); + for (pass = 0; pass < 10 && retry; pass++) { + retry = 0; + + list_for_each_entry_safe(page, page2, from, lru) { + nr_subpages = compound_nr(page); + cond_resched(); + + if (!PageHuge(page)) + continue; + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, page, + pass > 2, mode, reason, + &ret_pages); + /* + * The rules are: + * Success: hugetlb page will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_pages list then splice to + * from list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + nr_failed_pages += nr_subpages; + list_move_tail(&page->lru, &ret_pages); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other pages, just exit. + */ + nr_failed++; + nr_failed_pages += nr_subpages + nr_retry_pages; + goto out; + case -EAGAIN: + retry++; + nr_retry_pages += nr_subpages; + break; + case MIGRATEPAGE_SUCCESS: + nr_succeeded += nr_subpages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed page is + * removed from migration page list and not + * retried in the next outer loop. + */ + nr_failed++; + nr_failed_pages += nr_subpages; + break; + } + } + } + nr_failed += retry; + retry = 1; thp_subpage_migration: for (pass = 0; pass < 10 && (retry || thp_retry); pass++) { retry = 0; @@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, cond_resched(); if (PageHuge(page)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, page, - pass > 2, mode, reason, - &ret_pages); - else - rc = unmap_and_move(get_new_page, put_new_page, + continue; + + rc = unmap_and_move(get_new_page, put_new_page, private, page, pass > 2, mode, reason, &ret_pages); /* * The rules are: - * Success: non hugetlb page will be freed, hugetlb - * page will be put back + * Success: page will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_thp_split++; break; } - /* Hugetlb migration is unsupported */ } else if (!no_subpage_counting) { nr_failed++; }
This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying. In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation. Signed-off-by: "Huang, Ying" <ying.huang@intel.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Yang Shi <shy828301@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Matthew Wilcox <willy@infradead.org> --- mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 64 insertions(+), 9 deletions(-)