| Message ID | 20230524153311.3625329-5-dhowells@redhat.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | splice, net: Replace sendpage with sendmsg(MSG_SPLICE_PAGES), part 3 | expand |
On 2023/5/24 23:33, David Howells wrote: > Change the page_frag_cache allocator to use multipage folios rather than > groups of pages. This reduces page_frag_free to just a folio_put() or > put_page(). Hi, David put_page() is not used in this patch, perhaps remove it to avoid the confusion? Also, Is there any significant difference between __free_pages() and folio_put()? IOW, what does the 'reduces' part means here? I followed some disscusion about folio before, but have not really understood about real difference between 'multipage folios' and 'groups of pages' yet. Is folio mostly used to avoid the confusion about whether a page is 'headpage of compound page', 'base page' or 'tailpage of compound page'? Or is there any abvious benefit about folio that I missed? > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > index 306a3d1a0fa6..d7c52a5979cc 100644 > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) > } > > struct page_frag_cache { > - void * va; > -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) > - __u16 offset; > - __u16 size; > -#else > - __u32 offset; > -#endif > + struct folio *folio; > + unsigned int offset; > /* we maintain a pagecount bias, so that we dont dirty cache line > * containing page->_refcount every time we allocate a fragment. > */ > - unsigned int pagecnt_bias; > - bool pfmemalloc; > + unsigned int pagecnt_bias; > + bool pfmemalloc; > }; It seems 'va' and 'size' field is used to avoid touching 'stuct page' to avoid possible cache bouncing when there is more frag can be allocated from the page while other frags is freed at the same time before this patch? It might be worth calling that out in the commit log or split it into another patch to make it clearer and easier to review?
Yunsheng Lin <linyunsheng@huawei.com> wrote: > > Change the page_frag_cache allocator to use multipage folios rather than > > groups of pages. This reduces page_frag_free to just a folio_put() or > > put_page(). > > put_page() is not used in this patch, perhaps remove it to avoid > the confusion? Will do if I need to respin the patches. > Also, Is there any significant difference between __free_pages() > and folio_put()? IOW, what does the 'reduces' part means here? I meant that the folio code handles page compounding for us and we don't need to work out how big the page is for ourselves. If you look at __free_pages(), you can see a PageHead() call. folio_put() doesn't need that. > I followed some disscusion about folio before, but have not really > understood about real difference between 'multipage folios' and > 'groups of pages' yet. Is folio mostly used to avoid the confusion > about whether a page is 'headpage of compound page', 'base page' or > 'tailpage of compound page'? Or is there any abvious benefit about > folio that I missed? There is a benefit: a folio pointer always points to the head page and so we never need to do "is this compound? where's the head?" logic to find it. When going from a page pointer, we still have to find the head. Ultimately, the aim is to reduce struct page to a typed pointer to massively reduce the amount of space consumed by mem_map[]. A page struct will then point at a folio or a slab struct or one of a number of different types. But to get to that point, we have to stop a whole lot of things from using page structs, but rather use some other type, such as folio. Eventually, there won't be a need for head pages and tail pages per se - just memory objects of different sizes. > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index 306a3d1a0fa6..d7c52a5979cc 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) > > } > > > > struct page_frag_cache { > > - void * va; > > -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) > > - __u16 offset; > > - __u16 size; > > -#else > > - __u32 offset; > > -#endif > > + struct folio *folio; > > + unsigned int offset; > > /* we maintain a pagecount bias, so that we dont dirty cache line > > * containing page->_refcount every time we allocate a fragment. > > */ > > - unsigned int pagecnt_bias; > > - bool pfmemalloc; > > + unsigned int pagecnt_bias; > > + bool pfmemalloc; > > }; > > It seems 'va' and 'size' field is used to avoid touching 'stuct page' to > avoid possible cache bouncing when there is more frag can be allocated > from the page while other frags is freed at the same time before this patch? Hmmm... fair point, though va is calculated from the page pointer on most arches without the need to dereference struct page (only arc, m68k and sparc define WANT_PAGE_VIRTUAL). David
Hi, On 26.5.2023 15.47, David Howells wrote: > Yunsheng Lin <linyunsheng@huawei.com> wrote: > >>> Change the page_frag_cache allocator to use multipage folios rather than >>> groups of pages. This reduces page_frag_free to just a folio_put() or >>> put_page(). >> >> put_page() is not used in this patch, perhaps remove it to avoid >> the confusion? > > Will do if I need to respin the patches. > >> Also, Is there any significant difference between __free_pages() >> and folio_put()? IOW, what does the 'reduces' part means here? > > I meant that the folio code handles page compounding for us and we don't need > to work out how big the page is for ourselves. > > If you look at __free_pages(), you can see a PageHead() call. folio_put() > doesn't need that. > >> I followed some disscusion about folio before, but have not really >> understood about real difference between 'multipage folios' and >> 'groups of pages' yet. Is folio mostly used to avoid the confusion >> about whether a page is 'headpage of compound page', 'base page' or >> 'tailpage of compound page'? Or is there any abvious benefit about >> folio that I missed? > > There is a benefit: a folio pointer always points to the head page and so we > never need to do "is this compound? where's the head?" logic to find it. When > going from a page pointer, we still have to find the head. > But page_frag_free() uses folio_put(virt_to_folio(addr)) and virt_to_folio() depends on the compound infrastructure to get the head page and folio. > Ultimately, the aim is to reduce struct page to a typed pointer to massively > reduce the amount of space consumed by mem_map[]. A page struct will then > point at a folio or a slab struct or one of a number of different types. But > to get to that point, we have to stop a whole lot of things from using page > structs, but rather use some other type, such as folio. > > Eventually, there won't be a need for head pages and tail pages per se - just > memory objects of different sizes. > >>> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h >>> index 306a3d1a0fa6..d7c52a5979cc 100644 >>> --- a/include/linux/mm_types.h >>> +++ b/include/linux/mm_types.h >>> @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) >>> } >>> >>> struct page_frag_cache { >>> - void * va; >>> -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) >>> - __u16 offset; >>> - __u16 size; >>> -#else >>> - __u32 offset; >>> -#endif >>> + struct folio *folio; >>> + unsigned int offset; >>> /* we maintain a pagecount bias, so that we dont dirty cache line >>> * containing page->_refcount every time we allocate a fragment. >>> */ >>> - unsigned int pagecnt_bias; >>> - bool pfmemalloc; >>> + unsigned int pagecnt_bias; >>> + bool pfmemalloc; >>> }; >> >> It seems 'va' and 'size' field is used to avoid touching 'stuct page' to >> avoid possible cache bouncing when there is more frag can be allocated >> from the page while other frags is freed at the same time before this patch? > > Hmmm... fair point, though va is calculated from the page pointer on most > arches without the need to dereference struct page (only arc, m68k and sparc > define WANT_PAGE_VIRTUAL). > > David > --Mika
On Wed, 24 May 2023 16:33:03 +0100 David Howells wrote: > - offset = nc->offset - fragsz; > - if (unlikely(offset < 0)) { > - page = virt_to_page(nc->va); > - > - if (page_ref_count(page) != nc->pagecnt_bias) > + offset = nc->offset; > + if (unlikely(fragsz > offset)) { > + /* Reuse the folio if everyone we gave it to has finished with > + * it. > + */ > + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { > + nc->folio = NULL; > goto refill; > + } > + > if (unlikely(nc->pfmemalloc)) { > - page_ref_sub(page, nc->pagecnt_bias - 1); > - __free_pages(page, compound_order(page)); > + __folio_put(folio); This is not a pure 1:1 page -> folio conversion. Why mix conversion with other code changes?
On Fri, 2023-05-26 at 19:56 +0800, Yunsheng Lin wrote: > On 2023/5/24 23:33, David Howells wrote: > > Change the page_frag_cache allocator to use multipage folios rather than > > groups of pages. This reduces page_frag_free to just a folio_put() or > > put_page(). > > Hi, David > > put_page() is not used in this patch, perhaps remove it to avoid > the confusion? > Also, Is there any significant difference between __free_pages() > and folio_put()? IOW, what does the 'reduces' part means here? > > I followed some disscusion about folio before, but have not really > understood about real difference between 'multipage folios' and > 'groups of pages' yet. Is folio mostly used to avoid the confusion > about whether a page is 'headpage of compound page', 'base page' or > 'tailpage of compound page'? Or is there any abvious benefit about > folio that I missed? > > > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index 306a3d1a0fa6..d7c52a5979cc 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) > > } > > > > struct page_frag_cache { > > - void * va; > > -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) > > - __u16 offset; > > - __u16 size; > > -#else > > - __u32 offset; > > -#endif > > + struct folio *folio; > > + unsigned int offset; > > /* we maintain a pagecount bias, so that we dont dirty cache line > > * containing page->_refcount every time we allocate a fragment. > > */ > > - unsigned int pagecnt_bias; > > - bool pfmemalloc; > > + unsigned int pagecnt_bias; > > + bool pfmemalloc; > > }; > > It seems 'va' and 'size' field is used to avoid touching 'stuct page' to > avoid possible cache bouncing when there is more frag can be allocated > from the page while other frags is freed at the same time before this patch? > It might be worth calling that out in the commit log or split it into another > patch to make it clearer and easier to review? Yes, there is a cost for going from page to virtual address. That is why we only use the page when we finally get to freeing or resetting the pagecnt_bias. Also I have some concerns about going from page to folio as it seems like the folio_alloc setups the transparent hugepage destructor instead of using the compound page destructor. I would think that would slow down most users as it looks like there is a spinlock that is taken in the hugepage destructor that isn't there in the compound page destructor.
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..d7c52a5979cc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) } struct page_frag_cache { - void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif + struct folio *folio; + unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; - bool pfmemalloc; + unsigned int pagecnt_bias; + bool pfmemalloc; }; typedef unsigned long vm_flags_t; diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index 9d3f6fbd9a07..ffd68bfb677d 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,33 +16,34 @@ #include <linux/init.h> #include <linux/mm.h> -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +/* + * Allocate a new folio for the frag cache. + */ +static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) { - struct page *page = NULL; + struct folio *folio = NULL; gfp_t gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; + gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); #endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + if (unlikely(!folio)) + folio = folio_alloc(gfp, 0); - nc->va = page ? page_address(page) : NULL; - - return page; + if (folio) + nc->folio = folio; + return folio; } void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + struct folio *folio = page_folio(page); + + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); - if (page_ref_sub_and_test(page, count - 1)) - __free_pages(page, compound_order(page)); + folio_put_refs(folio, count); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -54,11 +55,12 @@ EXPORT_SYMBOL(__page_frag_cache_drain); */ void page_frag_cache_clear(struct page_frag_cache *nc) { - if (nc->va) { - struct page *page = virt_to_head_page(nc->va); + struct folio *folio = nc->folio; - __page_frag_cache_drain(page, nc->pagecnt_bias); - nc->va = NULL; + if (folio) { + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + folio_put_refs(folio, nc->pagecnt_bias); + nc->folio = NULL; } } EXPORT_SYMBOL(page_frag_cache_clear); @@ -67,56 +69,51 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align) { - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; + struct folio *folio = nc->folio; + size_t offset; WARN_ON_ONCE(!is_power_of_2(align)); - if (unlikely(!nc->va)) { + if (unlikely(!folio)) { refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) + folio = page_frag_cache_refill(nc, gfp_mask); + if (!folio) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pfmemalloc = folio_is_pfmemalloc(folio); nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; + nc->offset = folio_size(folio); } - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (page_ref_count(page) != nc->pagecnt_bias) + offset = nc->offset; + if (unlikely(fragsz > offset)) { + /* Reuse the folio if everyone we gave it to has finished with + * it. + */ + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { + nc->folio = NULL; goto refill; + } + if (unlikely(nc->pfmemalloc)) { - page_ref_sub(page, nc->pagecnt_bias - 1); - __free_pages(page, compound_order(page)); + __folio_put(folio); + nc->folio = NULL; goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { + offset = folio_size(folio); + if (unlikely(fragsz > offset)) { /* * The caller is trying to allocate a fragment * with fragsz > PAGE_SIZE but the cache isn't big @@ -126,15 +123,17 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, * it could make memory pressure worse * so we simply return NULL here. */ + nc->offset = offset; return NULL; } } nc->pagecnt_bias--; + offset -= fragsz; offset &= ~(align - 1); nc->offset = offset; - return nc->va + offset; + return folio_address(folio) + offset; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -143,8 +142,6 @@ EXPORT_SYMBOL(page_frag_alloc_align); */ void page_frag_free(void *addr) { - struct page *page = virt_to_head_page(addr); - - __free_pages(page, compound_order(page)); + folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free);
Change the page_frag_cache allocator to use multipage folios rather than groups of pages. This reduces page_frag_free to just a folio_put() or put_page(). Signed-off-by: David Howells <dhowells@redhat.com> cc: "David S. Miller" <davem@davemloft.net> cc: Eric Dumazet <edumazet@google.com> cc: Jakub Kicinski <kuba@kernel.org> cc: Paolo Abeni <pabeni@redhat.com> cc: Jens Axboe <axboe@kernel.dk> cc: Jeroen de Borst <jeroendb@google.com> cc: Catherine Sullivan <csully@google.com> cc: Shailend Chand <shailend@google.com> cc: Felix Fietkau <nbd@nbd.name> cc: John Crispin <john@phrozen.org> cc: Sean Wang <sean.wang@mediatek.com> cc: Mark Lee <Mark-MC.Lee@mediatek.com> cc: Lorenzo Bianconi <lorenzo@kernel.org> cc: Matthias Brugger <matthias.bgg@gmail.com> cc: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com> cc: Keith Busch <kbusch@kernel.org> cc: Jens Axboe <axboe@fb.com> cc: Christoph Hellwig <hch@lst.de> cc: Sagi Grimberg <sagi@grimberg.me> cc: Chaitanya Kulkarni <kch@nvidia.com> cc: Andrew Morton <akpm@linux-foundation.org> cc: Matthew Wilcox <willy@infradead.org> cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- include/linux/mm_types.h | 13 ++---- mm/page_frag_alloc.c | 99 +++++++++++++++++++--------------------- 2 files changed, 52 insertions(+), 60 deletions(-)