| Message ID | 20210205204127.29441-4-joao.m.martins@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | mm/gup: page unpining improvements | expand |
On Fri, Feb 05, 2021 at 08:41:26PM +0000, Joao Martins wrote: > Add a unpin_user_page_range_dirty_lock() API which takes a starting page > and how many consecutive pages we want to unpin and optionally dirty. > > To that end, define another iterator for_each_compound_range() > that operates in page ranges as opposed to page array. > > For users (like RDMA mr_dereg) where each sg represents a > contiguous set of pages, we're able to more efficiently unpin > pages without having to supply an array of pages much of what > happens today with unpin_user_pages(). > > Suggested-by: Jason Gunthorpe <jgg@nvidia.com> > Signed-off-by: Joao Martins <joao.m.martins@oracle.com> > --- > include/linux/mm.h | 2 ++ > mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 64 insertions(+) Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> > +/** > + * unpin_user_page_range_dirty_lock() - release and optionally dirty > + * gup-pinned page range > + * > + * @page: the starting page of a range maybe marked dirty, and definitely released. > + * @npages: number of consecutive pages to release. > + * @make_dirty: whether to mark the pages dirty > + * > + * "gup-pinned page range" refers to a range of pages that has had one of the > + * get_user_pages() variants called on that page. Tidy this language though, this only works with the pin_user_pages variants because it hardwires FOLL_PIN Jason
On 2/5/21 12:41 PM, Joao Martins wrote: > Add a unpin_user_page_range_dirty_lock() API which takes a starting page > and how many consecutive pages we want to unpin and optionally dirty. > > To that end, define another iterator for_each_compound_range() > that operates in page ranges as opposed to page array. > > For users (like RDMA mr_dereg) where each sg represents a > contiguous set of pages, we're able to more efficiently unpin > pages without having to supply an array of pages much of what > happens today with unpin_user_pages(). > > Suggested-by: Jason Gunthorpe <jgg@nvidia.com> > Signed-off-by: Joao Martins <joao.m.martins@oracle.com> > --- > include/linux/mm.h | 2 ++ > mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 64 insertions(+) > > diff --git a/include/linux/mm.h b/include/linux/mm.h > index a608feb0d42e..b76063f7f18a 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) > void unpin_user_page(struct page *page); > void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, > bool make_dirty); > +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, > + bool make_dirty); > void unpin_user_pages(struct page **pages, unsigned long npages); > > /** > diff --git a/mm/gup.c b/mm/gup.c > index 467a11df216d..938964d31494 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -215,6 +215,32 @@ void unpin_user_page(struct page *page) > } > EXPORT_SYMBOL(unpin_user_page); > > +static inline void compound_range_next(unsigned long i, unsigned long npages, > + struct page **list, struct page **head, > + unsigned int *ntails) Yes, the new names look good, and I have failed to find any logic errors, so: Reviewed-by: John Hubbard <jhubbard@nvidia.com> thanks,
On 2/10/21 11:15 PM, Jason Gunthorpe wrote: > On Fri, Feb 05, 2021 at 08:41:26PM +0000, Joao Martins wrote: >> Add a unpin_user_page_range_dirty_lock() API which takes a starting page >> and how many consecutive pages we want to unpin and optionally dirty. >> >> To that end, define another iterator for_each_compound_range() >> that operates in page ranges as opposed to page array. >> >> For users (like RDMA mr_dereg) where each sg represents a >> contiguous set of pages, we're able to more efficiently unpin >> pages without having to supply an array of pages much of what >> happens today with unpin_user_pages(). >> >> Suggested-by: Jason Gunthorpe <jgg@nvidia.com> >> Signed-off-by: Joao Martins <joao.m.martins@oracle.com> >> --- >> include/linux/mm.h | 2 ++ >> mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 64 insertions(+) > > Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> > Thanks! >> +/** >> + * unpin_user_page_range_dirty_lock() - release and optionally dirty >> + * gup-pinned page range >> + * >> + * @page: the starting page of a range maybe marked dirty, and definitely released. >> + * @npages: number of consecutive pages to release. >> + * @make_dirty: whether to mark the pages dirty >> + * >> + * "gup-pinned page range" refers to a range of pages that has had one of the >> + * get_user_pages() variants called on that page. > > Tidy this language though, this only works with the pin_user_pages > variants because it hardwires FOLL_PIN > Yes, I can respin a v4 with that adjustment.
On 2/10/21 11:19 PM, John Hubbard wrote: > On 2/5/21 12:41 PM, Joao Martins wrote: >> Add a unpin_user_page_range_dirty_lock() API which takes a starting page >> and how many consecutive pages we want to unpin and optionally dirty. >> >> To that end, define another iterator for_each_compound_range() >> that operates in page ranges as opposed to page array. >> >> For users (like RDMA mr_dereg) where each sg represents a >> contiguous set of pages, we're able to more efficiently unpin >> pages without having to supply an array of pages much of what >> happens today with unpin_user_pages(). >> >> Suggested-by: Jason Gunthorpe <jgg@nvidia.com> >> Signed-off-by: Joao Martins <joao.m.martins@oracle.com> >> --- >> include/linux/mm.h | 2 ++ >> mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 64 insertions(+) >> >> diff --git a/include/linux/mm.h b/include/linux/mm.h >> index a608feb0d42e..b76063f7f18a 100644 >> --- a/include/linux/mm.h >> +++ b/include/linux/mm.h >> @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) >> void unpin_user_page(struct page *page); >> void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, >> bool make_dirty); >> +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, >> + bool make_dirty); >> void unpin_user_pages(struct page **pages, unsigned long npages); >> >> /** >> diff --git a/mm/gup.c b/mm/gup.c >> index 467a11df216d..938964d31494 100644 >> --- a/mm/gup.c >> +++ b/mm/gup.c >> @@ -215,6 +215,32 @@ void unpin_user_page(struct page *page) >> } >> EXPORT_SYMBOL(unpin_user_page); >> >> +static inline void compound_range_next(unsigned long i, unsigned long npages, >> + struct page **list, struct page **head, >> + unsigned int *ntails) > > Yes, the new names look good, and I have failed to find any logic errors, so: > > Reviewed-by: John Hubbard <jhubbard@nvidia.com> > Thanks again for all the input!
diff --git a/include/linux/mm.h b/include/linux/mm.h index a608feb0d42e..b76063f7f18a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) void unpin_user_page(struct page *page); void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty); void unpin_user_pages(struct page **pages, unsigned long npages); /** diff --git a/mm/gup.c b/mm/gup.c index 467a11df216d..938964d31494 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,32 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void compound_range_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *next, *page; + unsigned int nr = 1; + + if (i >= npages) + return; + + next = *list + i; + page = compound_head(next); + if (PageCompound(page) && compound_order(page) >= 1) + nr = min_t(unsigned int, + page + compound_nr(page) - next, npages - i); + + *head = page; + *ntails = nr; +} + +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + compound_range_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + compound_range_next(__i, __npages, __list, &(__head), &(__ntails))) + static inline void compound_next(unsigned long i, unsigned long npages, struct page **list, struct page **head, unsigned int *ntails) @@ -303,6 +329,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); +/** + * unpin_user_page_range_dirty_lock() - release and optionally dirty + * gup-pinned page range + * + * @page: the starting page of a range maybe marked dirty, and definitely released. + * @npages: number of consecutive pages to release. + * @make_dirty: whether to mark the pages dirty + * + * "gup-pinned page range" refers to a range of pages that has had one of the + * get_user_pages() variants called on that page. + * + * For the page ranges defined by [page .. page+npages], make that range (or + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the + * page range was previously listed as clean. + * + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is + * required, then the caller should a) verify that this is really correct, + * because _lock() is usually required, and b) hand code it: + * set_page_dirty_lock(), unpin_user_page(). + * + */ +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty) +{ + unsigned long index; + struct page *head; + unsigned int ntails; + + for_each_compound_range(index, &page, npages, head, ntails) { + if (make_dirty && !PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); + } +} +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); + /** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released.
Add a unpin_user_page_range_dirty_lock() API which takes a starting page and how many consecutive pages we want to unpin and optionally dirty. To that end, define another iterator for_each_compound_range() that operates in page ranges as opposed to page array. For users (like RDMA mr_dereg) where each sg represents a contiguous set of pages, we're able to more efficiently unpin pages without having to supply an array of pages much of what happens today with unpin_user_pages(). Suggested-by: Jason Gunthorpe <jgg@nvidia.com> Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- include/linux/mm.h | 2 ++ mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)