[v3,0/6] mm/list_lru: Split list_lru lock into per-cgroup scope

Message

Kairui Song Nov. 4, 2024, 5:52 p.m. UTC

From: Kairui Song <[e-mail]>

Currently, every list_lru has a per-node lock that protects adding,
deletion, isolation, and reparenting of all list_lru_one instances
belonging to this list_lru on this node. This lock contention is heavy
when multiple cgroups modify the same list_lru.

This can be alleviated by splitting the lock into per-cgroup scope.

To achieve this, this series reworked and optimized the reparenting
process step by step, making it possible to have a stable list_lru_one,
and making it possible to pin the list_lru_one. Then split the lock
into per-cgroup scope.

The result is ~15% performance gain for simple multi-cgroup tar test
of small files, and reduced LOC. See PATCH 5/6 for test details.

V2: https://lore.kernel.org/linux-mm/[e-mail]/
Updates from V2:
- Collect Acked-by.
- Fix a WARN_ON issue caused by potential compiler optimization issue.
  [Dan Carpenter, Naresh Kamboju]
  https://lore.kernel.org/linux-mm/[e-mail]/
  [Applied to "mm/list_lru: split the lock to per-cgroup scope"]
- Fix a BUG_ON issue, V2 forgot to cover user of LRU_STOP. [Usama Arif]
  https://lore.kernel.org/linux-mm/CAMgjq7D_OA=vYf5SnNnKXjppPFhDqsbYF--6=[e-mail]/

V1: https://lore.kernel.org/linux-mm/[e-mail]/
Updates from V1:
- Collect Review-by.
- Fix a race of initialization issue that may lead to mem leak [Muchun
  Song]
- Drop a unrelated and incorrect fix [Shakeel Butt]
- Use VM_WARN_ON instead of WARN_ON for several sanity checks.

Kairui Song (6):
  mm/list_lru: don't pass unnecessary key parameters
  mm/list_lru: don't export list_lru_add
  mm/list_lru: code clean up for reparenting
  mm/list_lru: simplify reparenting and initial allocation
  mm/list_lru: split the lock to per-cgroup scope
  mm/list_lru: simplify the list_lru walk callback function

 drivers/android/binder_alloc.c |   8 +-
 drivers/android/binder_alloc.h |   2 +-
 fs/dcache.c                    |   4 +-
 fs/gfs2/quota.c                |   2 +-
 fs/inode.c                     |   5 +-
 fs/nfs/nfs42xattr.c            |   4 +-
 fs/nfsd/filecache.c            |   5 +-
 fs/xfs/xfs_buf.c               |   2 -
 fs/xfs/xfs_qm.c                |   6 +-
 include/linux/list_lru.h       |  26 ++-
 mm/list_lru.c                  | 383 ++++++++++++++++-----------------
 mm/memcontrol.c                |  10 +-
 mm/workingset.c                |  20 +-
 mm/zswap.c                     |  12 +-
 14 files changed, 241 insertions(+), 248 deletions(-)