Message ID | 20210204202500.26474-4-joao.m.martins@oracle.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | mm/gup: page unpining improvements | expand |
On 2/4/21 12:24 PM, Joao Martins wrote: > Add a unpin_user_page_range_dirty_lock() API which takes a starting page > and how many consecutive pages we want to unpin and optionally dirty. > > Given that we won't be iterating on a list of changes, change > compound_next() to receive a bool, whether to calculate from the starting Thankfully, that claim is stale and can now be removed from this commit description. > page, or walk the page array. Finally add a separate iterator, > for_each_compound_range() that just operate in page ranges as opposed > to page array. > > For users (like RDMA mr_dereg) where each sg represents a > contiguous set of pages, we're able to more efficiently unpin > pages without having to supply an array of pages much of what > happens today with unpin_user_pages(). > > Suggested-by: Jason Gunthorpe <jgg@nvidia.com> > Signed-off-by: Joao Martins <joao.m.martins@oracle.com> > --- > include/linux/mm.h | 2 ++ > mm/gup.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 66 insertions(+) > > diff --git a/include/linux/mm.h b/include/linux/mm.h > index a608feb0d42e..b76063f7f18a 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) > void unpin_user_page(struct page *page); > void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, > bool make_dirty); > +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, > + bool make_dirty); > void unpin_user_pages(struct page **pages, unsigned long npages); > > /** > diff --git a/mm/gup.c b/mm/gup.c > index 5a3dd235017a..3426736a01b2 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -215,6 +215,34 @@ void unpin_user_page(struct page *page) > } > EXPORT_SYMBOL(unpin_user_page); > > +static inline void range_next(unsigned long i, unsigned long npages, > + struct page **list, struct page **head, > + unsigned int *ntails) Would compound_range_next() be a better name? > +{ > + struct page *next, *page; > + unsigned int nr = 1; > + > + if (i >= npages) > + return; > + > + npages -= i; > + next = *list + i; > + > + page = compound_head(next); > + if (PageCompound(page) && compound_order(page) > 1) > + nr = min_t(unsigned int, > + page + compound_nr(page) - next, npages); This pointer arithmetic will involve division. Which may be unnecessarily expensive, if there is a way to calculate this with indices instead of pointer arithmetic. I'm not sure if there is, off hand, but thought it worth mentioning because the point is sometimes overlooked. > + > + *head = page; > + *ntails = nr; > +} > + > +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ > + for (__i = 0, \ > + range_next(__i, __npages, __list, &(__head), &(__ntails)); \ > + __i < __npages; __i += __ntails, \ > + range_next(__i, __npages, __list, &(__head), &(__ntails))) > + > static inline void compound_next(unsigned long i, unsigned long npages, > struct page **list, struct page **head, > unsigned int *ntails) > @@ -306,6 +334,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, > } > EXPORT_SYMBOL(unpin_user_pages_dirty_lock); > > +/** > + * unpin_user_page_range_dirty_lock() - release and optionally dirty > + * gup-pinned page range > + * > + * @page: the starting page of a range maybe marked dirty, and definitely released. > + * @npages: number of consecutive pages to release. > + * @make_dirty: whether to mark the pages dirty > + * > + * "gup-pinned page range" refers to a range of pages that has had one of the > + * get_user_pages() variants called on that page. > + * > + * For the page ranges defined by [page .. page+npages], make that range (or > + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the > + * page range was previously listed as clean. > + * > + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is > + * required, then the caller should a) verify that this is really correct, > + * because _lock() is usually required, and b) hand code it: > + * set_page_dirty_lock(), unpin_user_page(). > + * > + */ > +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, > + bool make_dirty) > +{ > + unsigned long index; > + struct page *head; > + unsigned int ntails; > + > + for_each_compound_range(index, &page, npages, head, ntails) { > + if (make_dirty && !PageDirty(head)) > + set_page_dirty_lock(head); > + put_compound_head(head, ntails, FOLL_PIN); > + } > +} > +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); > + > /** > * unpin_user_pages() - release an array of gup-pinned pages. > * @pages: array of pages to be marked dirty and released. > Didn't spot any actual problems with how this works. thanks,
On 2/5/21 4:49 AM, John Hubbard wrote: > On 2/4/21 12:24 PM, Joao Martins wrote: >> Add a unpin_user_page_range_dirty_lock() API which takes a starting page >> and how many consecutive pages we want to unpin and optionally dirty. >> >> Given that we won't be iterating on a list of changes, change >> compound_next() to receive a bool, whether to calculate from the starting > > Thankfully, that claim is stale and can now be removed from this commit > description. > Yes, I'll delete it. >> page, or walk the page array. Finally add a separate iterator, >> for_each_compound_range() that just operate in page ranges as opposed >> to page array. >> >> For users (like RDMA mr_dereg) where each sg represents a >> contiguous set of pages, we're able to more efficiently unpin >> pages without having to supply an array of pages much of what >> happens today with unpin_user_pages(). >> >> Suggested-by: Jason Gunthorpe <jgg@nvidia.com> >> Signed-off-by: Joao Martins <joao.m.martins@oracle.com> >> --- >> include/linux/mm.h | 2 ++ >> mm/gup.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 66 insertions(+) >> >> diff --git a/include/linux/mm.h b/include/linux/mm.h >> index a608feb0d42e..b76063f7f18a 100644 >> --- a/include/linux/mm.h >> +++ b/include/linux/mm.h >> @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) >> void unpin_user_page(struct page *page); >> void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, >> bool make_dirty); >> +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, >> + bool make_dirty); >> void unpin_user_pages(struct page **pages, unsigned long npages); >> >> /** >> diff --git a/mm/gup.c b/mm/gup.c >> index 5a3dd235017a..3426736a01b2 100644 >> --- a/mm/gup.c >> +++ b/mm/gup.c >> @@ -215,6 +215,34 @@ void unpin_user_page(struct page *page) >> } >> EXPORT_SYMBOL(unpin_user_page); >> >> +static inline void range_next(unsigned long i, unsigned long npages, >> + struct page **list, struct page **head, >> + unsigned int *ntails) > > Would compound_range_next() be a better name? > Yeah, will change to that instead. range_next() might actually get confused for operations done on struct range *. One other thing about my naming is that unpin_user_page_range_dirty_lock() is *huge*. But it seems to adhere to the rest of unpin_* family of functions naming. Couldn't find a better alternative :/ >> +{ >> + struct page *next, *page; >> + unsigned int nr = 1; >> + >> + if (i >= npages) >> + return; >> + >> + npages -= i; I will remove this @npages subtraction into the min_t() calculation as it's the only placed that's used. >> + next = *list + i; >> + >> + page = compound_head(next); >> + if (PageCompound(page) && compound_order(page) > 1) I am not handling compound_order == 1 so will change to >= in the condition above. @compound_nr is placed on the second page. >> + nr = min_t(unsigned int, >> + page + compound_nr(page) - next, npages); > > This pointer arithmetic will involve division. Which may be unnecessarily > expensive, if there is a way to calculate this with indices instead of > pointer arithmetic. I'm not sure if there is, off hand, but thought it > worth mentioning because the point is sometimes overlooked. > Sadly, can't think of :( hence had to adhere to what seems to be the pattern today. Any conversion to PFNs (page_to_pfn) will do same said arithmetic, and I don't think we can reliably use page_index (and even that is only available on the head page). >> + >> + *head = page; >> + *ntails = nr; >> +} >> + >> +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ >> + for (__i = 0, \ >> + range_next(__i, __npages, __list, &(__head), &(__ntails)); \ >> + __i < __npages; __i += __ntails, \ >> + range_next(__i, __npages, __list, &(__head), &(__ntails))) >> + >> static inline void compound_next(unsigned long i, unsigned long npages, >> struct page **list, struct page **head, >> unsigned int *ntails) >> @@ -306,6 +334,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, >> } >> EXPORT_SYMBOL(unpin_user_pages_dirty_lock); >> >> +/** >> + * unpin_user_page_range_dirty_lock() - release and optionally dirty >> + * gup-pinned page range >> + * >> + * @page: the starting page of a range maybe marked dirty, and definitely released. >> + * @npages: number of consecutive pages to release. >> + * @make_dirty: whether to mark the pages dirty >> + * >> + * "gup-pinned page range" refers to a range of pages that has had one of the >> + * get_user_pages() variants called on that page. >> + * >> + * For the page ranges defined by [page .. page+npages], make that range (or >> + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the >> + * page range was previously listed as clean. >> + * >> + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is >> + * required, then the caller should a) verify that this is really correct, >> + * because _lock() is usually required, and b) hand code it: >> + * set_page_dirty_lock(), unpin_user_page(). >> + * >> + */ >> +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, >> + bool make_dirty) >> +{ >> + unsigned long index; >> + struct page *head; >> + unsigned int ntails; >> + >> + for_each_compound_range(index, &page, npages, head, ntails) { >> + if (make_dirty && !PageDirty(head)) >> + set_page_dirty_lock(head); >> + put_compound_head(head, ntails, FOLL_PIN); >> + } >> +} >> +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); >> + >> /** >> * unpin_user_pages() - release an array of gup-pinned pages. >> * @pages: array of pages to be marked dirty and released. >> > > Didn't spot any actual problems with how this works. /me nods
diff --git a/include/linux/mm.h b/include/linux/mm.h index a608feb0d42e..b76063f7f18a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) void unpin_user_page(struct page *page); void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty); void unpin_user_pages(struct page **pages, unsigned long npages); /** diff --git a/mm/gup.c b/mm/gup.c index 5a3dd235017a..3426736a01b2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,34 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void range_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *next, *page; + unsigned int nr = 1; + + if (i >= npages) + return; + + npages -= i; + next = *list + i; + + page = compound_head(next); + if (PageCompound(page) && compound_order(page) > 1) + nr = min_t(unsigned int, + page + compound_nr(page) - next, npages); + + *head = page; + *ntails = nr; +} + +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + range_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + range_next(__i, __npages, __list, &(__head), &(__ntails))) + static inline void compound_next(unsigned long i, unsigned long npages, struct page **list, struct page **head, unsigned int *ntails) @@ -306,6 +334,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); +/** + * unpin_user_page_range_dirty_lock() - release and optionally dirty + * gup-pinned page range + * + * @page: the starting page of a range maybe marked dirty, and definitely released. + * @npages: number of consecutive pages to release. + * @make_dirty: whether to mark the pages dirty + * + * "gup-pinned page range" refers to a range of pages that has had one of the + * get_user_pages() variants called on that page. + * + * For the page ranges defined by [page .. page+npages], make that range (or + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the + * page range was previously listed as clean. + * + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is + * required, then the caller should a) verify that this is really correct, + * because _lock() is usually required, and b) hand code it: + * set_page_dirty_lock(), unpin_user_page(). + * + */ +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty) +{ + unsigned long index; + struct page *head; + unsigned int ntails; + + for_each_compound_range(index, &page, npages, head, ntails) { + if (make_dirty && !PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); + } +} +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); + /** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released.
Add a unpin_user_page_range_dirty_lock() API which takes a starting page and how many consecutive pages we want to unpin and optionally dirty. Given that we won't be iterating on a list of changes, change compound_next() to receive a bool, whether to calculate from the starting page, or walk the page array. Finally add a separate iterator, for_each_compound_range() that just operate in page ranges as opposed to page array. For users (like RDMA mr_dereg) where each sg represents a contiguous set of pages, we're able to more efficiently unpin pages without having to supply an array of pages much of what happens today with unpin_user_pages(). Suggested-by: Jason Gunthorpe <jgg@nvidia.com> Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- include/linux/mm.h | 2 ++ mm/gup.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+)