[RFC,v2,0/6] Reduce TLB flushes under some specific conditions

Message

Byungchul Park Aug. 17, 2023, 8:05 a.m. UTC

Hi,

While I'm working with CXL memory, I have been facing migraion overhead
esp. TLB shootdown on promotion or demotion between different tiers.
Yeah.. most TLB shootdowns on migration through hinting fault can be
avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid
flushing TLB in batch if PTE is inaccessible").

However, it's only for ones using hinting fault. I thought it'd be much
better if we have a general mechanism to reduce # of TLB flushes that
we can apply to any type of migration. I tried it only for tiering
migration for now tho.

I'm suggesting a mechanism to reduce TLB flushes by keeping source and
destination of folios participated in the migrations until all TLB
flushes required are done, only if those folios are not mapped with
write permission PTE entries at all.

I saw the number of TLB full flush reduced over 50% and the performance
a little bit improved but not that big with the workload I tested with,
XSBench. However, I believe that it would help more with other ones or
any real ones. It'd be appreciated to tell me if I'm missing something.

	Byungchul

---

Changes from RFC:

	1. Fix a bug triggered when a destination folio at the previous
	   migration becomes a source folio at the next migration,
	   before the folio gets handled properly so that the folio can
	   play with another migration. There was inconsistency in the
	   folio's state. Fixed it.

	2. Split the patch set into more pieces so that the folks can
	   review better. (Feedbacked by Nadav Amit)

	3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic().
	   (Feedbacked by Nadav Amit)

	4. Tried to add sufficient comments to explain the patch set
	   better. (Feedbacked by Nadav Amit)

Byungchul Park (6):
  mm/rmap: Recognize non-writable TLB entries during TLB batch flush
  mm: Defer TLB flush by keeping both src and dst folios at migration
  mm, migrc: Skip TLB flushes at the CPUs that already have been done
  mm, migrc: Ajust __zone_watermark_ok() with the amount of pending
    folios
  mm, migrc: Add a sysctl knob to enable/disable MIGRC mechanism
  mm, migrc: Implement internal allocator to minimize impact onto vm

 arch/x86/include/asm/tlbflush.h |   9 +
 arch/x86/mm/tlb.c               |  67 ++++++
 include/linux/mm.h              |  30 +++
 include/linux/mm_types.h        |  47 ++++
 include/linux/mm_types_task.h   |   4 +-
 include/linux/mmzone.h          |   6 +
 include/linux/sched.h           |   5 +
 init/Kconfig                    |  13 ++
 mm/internal.h                   |  14 ++
 mm/memory.c                     |  17 +-
 mm/migrate.c                    | 381 +++++++++++++++++++++++++++++++-
 mm/mm_init.c                    |   1 +
 mm/page_alloc.c                 |  19 ++
 mm/rmap.c                       | 133 ++++++++++-
 14 files changed, 734 insertions(+), 12 deletions(-)