[v2,0/5] Optimize mmap_exit for large folios

Message

Ryan Roberts Aug. 30, 2023, 9:50 a.m. UTC

Hi All,

This is v2 of a series to improve performance of process teardown,
taking advantage of the fact that large folios are increasingly
regularly pte-mapped in user space; supporting filesystems already use
large folios for pagecache memory, and large folios for anonymous memory
are (hopefully) on the horizon.

See last patch for performance numbers, including measurements that show
this approach doesn't regress (and actually improves a little bit) when
all folios are small.

The basic approach is to accumulate contiguous ranges of pages in the
mmu_gather structure (instead of storing each individual page pointer),
then take advantage of this internal format to efficiently batch rmap
removal, swapcache removal and page release - see the commit messages
for more details.

This series replaces the previous approach I took at [2], which was much
smaller in scope, only attempting to batch rmap removal for anon pages.
Feedback was that I should do something more general that would also
batch-remove pagecache pages from the rmap. But while designing that, I
found it was also possible to improve swapcache removal and page
release. Hopefully I haven't gone too far the other way now! Note that
patch 1 is unchanged from that originl series.

Note that this series will conflict with Matthew's series at [3]. I
figure we both race to mm-unstable and the loser has to do the conflict
resolution?

This series is based on mm-unstable (b93868dbf9bc).


Changes since v1 [1]
--------------------

- Now using pfns for start and end of page ranges within a folio.
  `struct page`s may not be contiguous on some setups so using pointers
  breaks these systems. (Thanks to Zi Yan).
- Fixed zone_device folio reference putting. (Thanks to Matthew and
  David).
- Refactored release_pages() and folios_put_refs() so that they now
  share a common implementation.


[1] https://lore.kernel.org/linux-mm/[e-mail]/
[2] https://lore.kernel.org/linux-mm/[e-mail]/
[3] https://lore.kernel.org/linux-mm/[e-mail]/


Thanks,
Ryan

Ryan Roberts (5):
  mm: Implement folio_remove_rmap_range()
  mm/mmu_gather: generalize mmu_gather rmap removal mechanism
  mm/mmu_gather: Remove encoded_page infrastructure
  mm: Refector release_pages()
  mm/mmu_gather: Store and process pages in contig ranges

 arch/s390/include/asm/tlb.h |   9 +-
 include/asm-generic/tlb.h   |  49 ++++-----
 include/linux/mm.h          |  11 +-
 include/linux/mm_types.h    |  34 +-----
 include/linux/rmap.h        |   2 +
 include/linux/swap.h        |   6 +-
 mm/memory.c                 |  24 +++--
 mm/mmu_gather.c             | 114 ++++++++++++++------
 mm/rmap.c                   | 125 ++++++++++++++++------
 mm/swap.c                   | 201 ++++++++++++++++++++++--------------
 mm/swap_state.c             |  11 +-
 11 files changed, 367 insertions(+), 219 deletions(-)

--
2.25.1