| Message ID | 20231020095952.11055-2-linyunsheng@huawei.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 58d53d8f7da63dd13903bec0a40b3009a841b61b |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | introduce page_pool_alloc() related API | expand |
Hi Yunsheng, [...] > + * 1. 'n == 1': no need to actually overwrite it. > + * 2. 'n != 1': overwrite it with one, which is the rare case > + * for pp_frag_count draining. > * > - * The main advantage to doing this is that an atomic_read is > - * generally a much cheaper operation than an atomic update, > - * especially when dealing with a page that may be partitioned > - * into only 2 or 3 pieces. > + * The main advantage to doing this is that not only we avoid a atomic > + * update, as an atomic_read is generally a much cheaper operation than > + * an atomic update, especially when dealing with a page that may be > + * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count > + * handling by ensuring all pages have partitioned into only 1 piece > + * initially, and only overwrite it when the page is partitioned into > + * more than one piece. > */ > - if (atomic_long_read(&page->pp_frag_count) == nr) > + if (atomic_long_read(&page->pp_frag_count) == nr) { > + /* As we have ensured nr is always one for constant case using > + * the BUILD_BUG_ON(), only need to handle the non-constant case > + * here for pp_frag_count draining, which is a rare case. > + */ > + BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); > + if (!__builtin_constant_p(nr)) > + atomic_long_set(&page->pp_frag_count, 1); Aren't we changing the behaviour of the current code here? IIRC is atomic_long_read(&page->pp_frag_count) == nr we never updated the atomic pp_frag_count and the reasoning was that the next caller can set it properly. > + > return 0; > + } > > ret = atomic_long_sub_return(nr, &page->pp_frag_count); > WARN_ON(ret < 0); > + > + /* We are the last user here too, reset pp_frag_count back to 1 to > + * ensure all pages have been partitioned into 1 piece initially, > + * this should be the rare case when the last two fragment users call > + * page_pool_defrag_page() currently. > + */ > + if (unlikely(!ret)) > + atomic_long_set(&page->pp_frag_count, 1); > + > return ret; > } > [....] Thanks /Ilias
On 2023/10/23 19:43, Ilias Apalodimas wrote: > Hi Yunsheng, > > [...] > >> + * 1. 'n == 1': no need to actually overwrite it. >> + * 2. 'n != 1': overwrite it with one, which is the rare case >> + * for pp_frag_count draining. >> * >> - * The main advantage to doing this is that an atomic_read is >> - * generally a much cheaper operation than an atomic update, >> - * especially when dealing with a page that may be partitioned >> - * into only 2 or 3 pieces. >> + * The main advantage to doing this is that not only we avoid a atomic >> + * update, as an atomic_read is generally a much cheaper operation than >> + * an atomic update, especially when dealing with a page that may be >> + * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count >> + * handling by ensuring all pages have partitioned into only 1 piece >> + * initially, and only overwrite it when the page is partitioned into >> + * more than one piece. >> */ >> - if (atomic_long_read(&page->pp_frag_count) == nr) >> + if (atomic_long_read(&page->pp_frag_count) == nr) { >> + /* As we have ensured nr is always one for constant case using >> + * the BUILD_BUG_ON(), only need to handle the non-constant case >> + * here for pp_frag_count draining, which is a rare case. >> + */ >> + BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); >> + if (!__builtin_constant_p(nr)) >> + atomic_long_set(&page->pp_frag_count, 1); > > Aren't we changing the behaviour of the current code here? IIRC is > atomic_long_read(&page->pp_frag_count) == nr we never updated the atomic > pp_frag_count and the reasoning was that the next caller can set it > properly. If the next caller is calling the page_pool_alloc_frag(), then yes, because page_pool_fragment_page() will be used to reset the page->pp_frag_count, so it does not really matter what is the value of page->pp_frag_count when we are recycling a page. If the next caller is calling page_pool_alloc_pages() directly without fragmenting a page, the above code is used to ensure that pp_frag_count is always one when page_pool_alloc_pages() fetches a page from pool->alloc or pool->ring, because page_pool_fragment_page() is not used to reset the page->pp_frag_count for page_pool_alloc_pages() and we have removed the per page_pool PP_FLAG_PAGE_FRAG in page_pool_is_last_frag(). As we don't know if the caller is page_pool_alloc_frag() or page_pool_alloc_pages(), so the above code ensure the page in pool->alloc or pool->ring always have the pp_frag_count being one. > >> + >> return 0; >> + } >> >> ret = atomic_long_sub_return(nr, &page->pp_frag_count); >> WARN_ON(ret < 0); >> + >> + /* We are the last user here too, reset pp_frag_count back to 1 to >> + * ensure all pages have been partitioned into 1 piece initially, >> + * this should be the rare case when the last two fragment users call >> + * page_pool_defrag_page() currently. >> + */ >> + if (unlikely(!ret)) >> + atomic_long_set(&page->pp_frag_count, 1); >> + >> return ret; >> } >> > > [....] > > Thanks > /Ilias > > . >
On Mon, 23 Oct 2023 at 15:27, Yunsheng Lin <linyunsheng@huawei.com> wrote: > > On 2023/10/23 19:43, Ilias Apalodimas wrote: > > Hi Yunsheng, > > > > [...] > > > >> + * 1. 'n == 1': no need to actually overwrite it. > >> + * 2. 'n != 1': overwrite it with one, which is the rare case > >> + * for pp_frag_count draining. > >> * > >> - * The main advantage to doing this is that an atomic_read is > >> - * generally a much cheaper operation than an atomic update, > >> - * especially when dealing with a page that may be partitioned > >> - * into only 2 or 3 pieces. > >> + * The main advantage to doing this is that not only we avoid a atomic > >> + * update, as an atomic_read is generally a much cheaper operation than > >> + * an atomic update, especially when dealing with a page that may be > >> + * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count > >> + * handling by ensuring all pages have partitioned into only 1 piece > >> + * initially, and only overwrite it when the page is partitioned into > >> + * more than one piece. > >> */ > >> - if (atomic_long_read(&page->pp_frag_count) == nr) > >> + if (atomic_long_read(&page->pp_frag_count) == nr) { > >> + /* As we have ensured nr is always one for constant case using > >> + * the BUILD_BUG_ON(), only need to handle the non-constant case > >> + * here for pp_frag_count draining, which is a rare case. > >> + */ > >> + BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); > >> + if (!__builtin_constant_p(nr)) > >> + atomic_long_set(&page->pp_frag_count, 1); > > > > Aren't we changing the behaviour of the current code here? IIRC is > > atomic_long_read(&page->pp_frag_count) == nr we never updated the atomic > > pp_frag_count and the reasoning was that the next caller can set it > > properly. > > If the next caller is calling the page_pool_alloc_frag(), then yes, > because page_pool_fragment_page() will be used to reset the > page->pp_frag_count, so it does not really matter what is the value > of page->pp_frag_count when we are recycling a page. > > If the next caller is calling page_pool_alloc_pages() directly without > fragmenting a page, the above code is used to ensure that pp_frag_count > is always one when page_pool_alloc_pages() fetches a page from pool->alloc > or pool->ring, because page_pool_fragment_page() is not used to reset the > page->pp_frag_count for page_pool_alloc_pages() and we have removed the > per page_pool PP_FLAG_PAGE_FRAG in page_pool_is_last_frag(). > > As we don't know if the caller is page_pool_alloc_frag() or > page_pool_alloc_pages(), so the above code ensure the page in pool->alloc > or pool->ring always have the pp_frag_count being one. Fair enough, Jakub pulled the series before I managed to ack them, but that's okay. It's been long overdue apologies. FWIW I went through the patches and didn't find anything wrong coding-wise Thanks /Ilias > > > > > > >> + > >> return 0; > >> + } > >> > >> ret = atomic_long_sub_return(nr, &page->pp_frag_count); > >> WARN_ON(ret < 0); > >> + > >> + /* We are the last user here too, reset pp_frag_count back to 1 to > >> + * ensure all pages have been partitioned into 1 piece initially, > >> + * this should be the rare case when the last two fragment users call > >> + * page_pool_defrag_page() currently. > >> + */ > >> + if (unlikely(!ret)) > >> + atomic_long_set(&page->pp_frag_count, 1); > >> + > >> return ret; > >> } > >> > > > > [....] > > > > Thanks > > /Ilias > > > > . > >
diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 8f64adf86f5b..759489c037c7 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -115,28 +115,49 @@ static inline long page_pool_defrag_page(struct page *page, long nr) long ret; /* If nr == pp_frag_count then we have cleared all remaining - * references to the page. No need to actually overwrite it, instead - * we can leave this to be overwritten by the calling function. + * references to the page: + * 1. 'n == 1': no need to actually overwrite it. + * 2. 'n != 1': overwrite it with one, which is the rare case + * for pp_frag_count draining. * - * The main advantage to doing this is that an atomic_read is - * generally a much cheaper operation than an atomic update, - * especially when dealing with a page that may be partitioned - * into only 2 or 3 pieces. + * The main advantage to doing this is that not only we avoid a atomic + * update, as an atomic_read is generally a much cheaper operation than + * an atomic update, especially when dealing with a page that may be + * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count + * handling by ensuring all pages have partitioned into only 1 piece + * initially, and only overwrite it when the page is partitioned into + * more than one piece. */ - if (atomic_long_read(&page->pp_frag_count) == nr) + if (atomic_long_read(&page->pp_frag_count) == nr) { + /* As we have ensured nr is always one for constant case using + * the BUILD_BUG_ON(), only need to handle the non-constant case + * here for pp_frag_count draining, which is a rare case. + */ + BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); + if (!__builtin_constant_p(nr)) + atomic_long_set(&page->pp_frag_count, 1); + return 0; + } ret = atomic_long_sub_return(nr, &page->pp_frag_count); WARN_ON(ret < 0); + + /* We are the last user here too, reset pp_frag_count back to 1 to + * ensure all pages have been partitioned into 1 piece initially, + * this should be the rare case when the last two fragment users call + * page_pool_defrag_page() currently. + */ + if (unlikely(!ret)) + atomic_long_set(&page->pp_frag_count, 1); + return ret; } -static inline bool page_pool_is_last_frag(struct page_pool *pool, - struct page *page) +static inline bool page_pool_is_last_frag(struct page *page) { - /* If fragments aren't enabled or count is 0 we were the last user */ - return !(pool->p.flags & PP_FLAG_PAGE_FRAG) || - (page_pool_defrag_page(page, 1) == 0); + /* If page_pool_defrag_page() returns 0, we were the last user */ + return page_pool_defrag_page(page, 1) == 0; } /** @@ -161,7 +182,7 @@ static inline void page_pool_put_page(struct page_pool *pool, * allow registering MEM_TYPE_PAGE_POOL, but shield linker. */ #ifdef CONFIG_PAGE_POOL - if (!page_pool_is_last_frag(pool, page)) + if (!page_pool_is_last_frag(page)) return; page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 8a9868ea5067..953535cab081 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -376,6 +376,14 @@ static void page_pool_set_pp_info(struct page_pool *pool, { page->pp = pool; page->pp_magic |= PP_SIGNATURE; + + /* Ensuring all pages have been split into one fragment initially: + * page_pool_set_pp_info() is only called once for every page when it + * is allocated from the page allocator and page_pool_fragment_page() + * is dirtying the same cache line as the page->pp_magic above, so + * the overhead is negligible. + */ + page_pool_fragment_page(page, 1); if (pool->p.init_callback) pool->p.init_callback(page, pool->p.init_arg); } @@ -672,7 +680,7 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, struct page *page = virt_to_head_page(data[i]); /* It is not the last user for the page frag case */ - if (!page_pool_is_last_frag(pool, page)) + if (!page_pool_is_last_frag(page)) continue; page = __page_pool_put_page(pool, page, -1, false);
Currently when page_pool_create() is called with PP_FLAG_PAGE_FRAG flag, page_pool_alloc_pages() is only allowed to be called under the below constraints: 1. page_pool_fragment_page() need to be called to setup page->pp_frag_count immediately. 2. page_pool_defrag_page() often need to be called to drain the page->pp_frag_count when there is no more user will be holding on to that page. Those constraints exist in order to support a page to be split into multi fragments. And those constraints have some overhead because of the cache line dirtying/bouncing and atomic update. Those constraints are unavoidable for case when we need a page to be split into more than one fragment, but there is also case that we want to avoid the above constraints and their overhead when a page can't be split as it can only hold a fragment as requested by user, depending on different use cases: use case 1: allocate page without page splitting. use case 2: allocate page with page splitting. use case 3: allocate page with or without page splitting depending on the fragment size. Currently page pool only provide page_pool_alloc_pages() and page_pool_alloc_frag() API to enable the 1 & 2 separately, so we can not use a combination of 1 & 2 to enable 3, it is not possible yet because of the per page_pool flag PP_FLAG_PAGE_FRAG. So in order to allow allocating unsplit page without the overhead of split page while still allow allocating split page we need to remove the per page_pool flag in page_pool_is_last_frag(), as best as I can think of, it seems there are two methods as below: 1. Add per page flag/bit to indicate a page is split or not, which means we might need to update that flag/bit everytime the page is recycled, dirtying the cache line of 'struct page' for use case 1. 2. Unify the page->pp_frag_count handling for both split and unsplit page by assuming all pages in the page pool is split into a big fragment initially. As page pool already supports use case 1 without dirtying the cache line of 'struct page' whenever a page is recyclable, we need to support the above use case 3 with minimal overhead, especially not adding any noticeable overhead for use case 1, and we are already doing an optimization by not updating pp_frag_count in page_pool_defrag_page() for the last fragment user, this patch chooses to unify the pp_frag_count handling to support the above use case 3. There is no noticeable performance degradation and some justification for unifying the frag_count handling with this patch applied using a micro-benchmark testing in [1]. 1. https://lore.kernel.org/all/bf2591f8-7b3c-4480-bb2c-31dc9da1d6ac@huawei.com/ Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com> CC: Lorenzo Bianconi <lorenzo@kernel.org> CC: Alexander Duyck <alexander.duyck@gmail.com> CC: Liang Chen <liangchen.linux@gmail.com> CC: Alexander Lobakin <aleksander.lobakin@intel.com> --- include/net/page_pool/helpers.h | 47 ++++++++++++++++++++++++--------- net/core/page_pool.c | 10 ++++++- 2 files changed, 43 insertions(+), 14 deletions(-)