[v7,0/4] block: add larger order folio instead of pages

Message

Kundan Kumar July 4, 2024, 7:03 a.m. UTC

User space memory is mapped in kernel in form of pages array. These pages
are iterated and added to BIO. In process, pages are also checked for
contiguity and merged.

When mTHP is enabled the pages generally belong to larger order folio. This
patch series enables adding large folio to bio. It fetches folio for
page in the page array. The page might start from an offset in the folio
which could be multiples of PAGE_SIZE. Subsequent pages in page array
might belong to same folio. Using the length of folio, folio_offset and
remaining size, determine length in folio which can be added to the bio.
Check if pages are contiguous and belong to same folio. If yes then skip
further processing for the contiguous pages.

This complete scheme reduces the overhead of iterating through pages.

perf diff before and after this change(with mTHP enabled):

Perf diff for write I/O with 128K block size:
    1.24%     -0.20%  [kernel.kallsyms]  [k] bio_iov_iter_get_pages
    1.71%             [kernel.kallsyms]  [k] bvec_try_merge_page
Perf diff for read I/O with 128K block size:
    4.03%     -1.59%  [kernel.kallsyms]  [k] bio_iov_iter_get_pages
    5.14%             [kernel.kallsyms]  [k] bvec_try_merge_page

Patch 1: Adds folio-lized version of bvec_try_merge_hw_page()
Patch 2: Adds folio-lized version of bio_add_hw_page()
Patch 3: Adds bigger size from folio to BIO
Patch 4: Unpin user pages belonging to folio at once

Changes since v6:
- folio-lize bvec_try_merge_hw_page() to bvec_try_merge_hw_folio()
- restructured the code as per comments
- typecast with size_t while calculating the offset in folio.

Changes since v5:
- Made offset and len as size_t in function bio_add_hw_folio()
- Avoid unpinning skipped pages at submission, rather unpin all pages at
  once on IO completion

Changes since v4:
- folio-lize bio_add_hw_page() to bio_add_hw_folio()
- make bio_add_hw_page() as a wrapper around bio_add_hw_folio()
- make new functions bio_release_folio() and unpin_user_folio()
- made a helper function to check for contiguous pages of folio
- changed &folio->page to folio_page(folio, 0)
- reworded comments

Changes since v3:
- Added change to see if pages are contiguous and belong to same folio.
  If not then avoid skipping of pages.(Suggested by Matthew Wilcox)

Changes since v2:
- Made separate patches
- Corrected code as per kernel coding style
- Removed size_folio variable

Changes since v1:
- Changed functions bio_iov_add_page() and bio_iov_add_zone_append_page()
  to accept a folio
- Removed branch and calculate folio_offset and len in same fashion for
  both 0 order and larger folios
- Added change in NVMe driver to use nvme_setup_prp_simple() by
  ignoring multiples of PAGE_SIZE in offset
- Added a change to unpin_user_pages which were added as folios. Also
  stopped the unpin of pages one by one from __bio_release_pages()
  (Suggested by Keith)

Kundan Kumar (4):
  block: Added folio-lized version of bvec_try_merge_hw_page()
  block: Added folio-lized version of bio_add_hw_page()
  block: introduce folio awareness and add a bigger size from folio
  block: unpin user pages belonging to a folio at once

 block/bio.c        | 133 ++++++++++++++++++++++++++++++++++++---------
 block/blk.h        |  15 +++++
 include/linux/mm.h |   1 +
 mm/gup.c           |  13 +++++
 4 files changed, 135 insertions(+), 27 deletions(-)