[v3,0/6] Enable THP for text section of non-shmem files

Message

Song Liu June 19, 2019, 6:24 a.m. UTC

Changes v2 => v3:
1. Removed the limitation (cannot write to file with THP) by truncating
   whole file during sys_open (see 6/6);
2. Fixed a VM_BUG_ON_PAGE() in filemap_fault() (see 2/6);
3. Split function rename to a separate patch (Rik);
4. Updated condition in hugepage_vma_check() (Rik).

Changes v1 => v2:
1. Fixed a missing mem_cgroup_commit_charge() for non-shmem case.

This set follows up discussion at LSF/MM 2019. The motivation is to put
text section of an application in THP, and thus reduces iTLB miss rate and
improves performance. Both Facebook and Oracle showed strong interests to
this feature.

To make reviews easier, this set aims a mininal valid product. Current
version of the work does not have any changes to file system specific
code. This comes with some limitations (discussed later).

This set enables an application to "hugify" its text section by simply
running something like:

          madvise(0x600000, 0x80000, MADV_HUGEPAGE);

Before this call, the /proc/<pid>/maps looks like:

    00400000-074d0000 r-xp 00000000 00:27 2006927     app

After this call, part of the text section is split out and mapped to
THP:

    00400000-00425000 r-xp 00000000 00:27 2006927     app
    00600000-00e00000 r-xp 00200000 00:27 2006927     app   <<< on THP
    00e00000-074d0000 r-xp 00a00000 00:27 2006927     app

Limitations:

1. This only works for text section (vma with VM_DENYWRITE).
2. Original limitation #2 is removed in v3.

We gated this feature with an experimental config, READ_ONLY_THP_FOR_FS.
Once we get better support on the write path, we can remove the config and
enable it by default.

Tested cases:
1. Tested with btrfs and ext4.
2. Tested with real work application (memcache like caching service).
3. Tested with "THP aware uprobe":
   https://patchwork.kernel.org/project/linux-mm/list/?series=131339

Please share your comments and suggestions on this.

Thanks!

Song Liu (6):
  filemap: check compound_head(page)->mapping in filemap_fault()
  filemap: update offset check in filemap_fault()
  mm,thp: stats for file backed THP
  khugepaged: rename collapse_shmem() and khugepaged_scan_shmem()
  mm,thp: add read-only THP support for (non-shmem) FS
  mm,thp: handle writes to file with THP in pagecache

 fs/inode.c             |   3 ++
 fs/proc/meminfo.c      |   4 ++
 include/linux/fs.h     |  31 ++++++++++++
 include/linux/mmzone.h |   2 +
 mm/Kconfig             |  11 +++++
 mm/filemap.c           |   9 ++--
 mm/khugepaged.c        | 104 +++++++++++++++++++++++++++++++++--------
 mm/rmap.c              |  12 +++--
 mm/truncate.c          |   7 ++-
 mm/vmstat.c            |   2 +
 10 files changed, 156 insertions(+), 29 deletions(-)

--
2.17.1

Comments

Michal Hocko June 19, 2019, 6:51 a.m. UTC | #1

[Cc fsdevel and lkml]

On Tue 18-06-19 23:24:18, Song Liu wrote:
> Changes v2 => v3:
> 1. Removed the limitation (cannot write to file with THP) by truncating
>    whole file during sys_open (see 6/6);
> 2. Fixed a VM_BUG_ON_PAGE() in filemap_fault() (see 2/6);
> 3. Split function rename to a separate patch (Rik);
> 4. Updated condition in hugepage_vma_check() (Rik).
> 
> Changes v1 => v2:
> 1. Fixed a missing mem_cgroup_commit_charge() for non-shmem case.
> 
> This set follows up discussion at LSF/MM 2019. The motivation is to put
> text section of an application in THP, and thus reduces iTLB miss rate and
> improves performance. Both Facebook and Oracle showed strong interests to
> this feature.
> 
> To make reviews easier, this set aims a mininal valid product. Current
> version of the work does not have any changes to file system specific
> code. This comes with some limitations (discussed later).
> 
> This set enables an application to "hugify" its text section by simply
> running something like:
> 
>           madvise(0x600000, 0x80000, MADV_HUGEPAGE);
> 
> Before this call, the /proc/<pid>/maps looks like:
> 
>     00400000-074d0000 r-xp 00000000 00:27 2006927     app
> 
> After this call, part of the text section is split out and mapped to
> THP:
> 
>     00400000-00425000 r-xp 00000000 00:27 2006927     app
>     00600000-00e00000 r-xp 00200000 00:27 2006927     app   <<< on THP
>     00e00000-074d0000 r-xp 00a00000 00:27 2006927     app
> 
> Limitations:
> 
> 1. This only works for text section (vma with VM_DENYWRITE).
> 2. Original limitation #2 is removed in v3.
> 
> We gated this feature with an experimental config, READ_ONLY_THP_FOR_FS.
> Once we get better support on the write path, we can remove the config and
> enable it by default.
> 
> Tested cases:
> 1. Tested with btrfs and ext4.
> 2. Tested with real work application (memcache like caching service).
> 3. Tested with "THP aware uprobe":
>    https://patchwork.kernel.org/project/linux-mm/list/?series=131339
> 
> Please share your comments and suggestions on this.
> 
> Thanks!
> 
> Song Liu (6):
>   filemap: check compound_head(page)->mapping in filemap_fault()
>   filemap: update offset check in filemap_fault()
>   mm,thp: stats for file backed THP
>   khugepaged: rename collapse_shmem() and khugepaged_scan_shmem()
>   mm,thp: add read-only THP support for (non-shmem) FS
>   mm,thp: handle writes to file with THP in pagecache
> 
>  fs/inode.c             |   3 ++
>  fs/proc/meminfo.c      |   4 ++
>  include/linux/fs.h     |  31 ++++++++++++
>  include/linux/mmzone.h |   2 +
>  mm/Kconfig             |  11 +++++
>  mm/filemap.c           |   9 ++--
>  mm/khugepaged.c        | 104 +++++++++++++++++++++++++++++++++--------
>  mm/rmap.c              |  12 +++--
>  mm/truncate.c          |   7 ++-
>  mm/vmstat.c            |   2 +
>  10 files changed, 156 insertions(+), 29 deletions(-)
> 
> --
> 2.17.1