[v2,0/8] convert hugetlb fault functions to folios

Message

Sidhartha Kumar Jan. 25, 2023, 5:05 p.m. UTC

============== CHANGE LOG ===========================
v1 -> v2
	- Add r-b to patches 1,3

	- Refactor hugetlbfs_pagecache_present() to use rcu +
	  page_cache_next_miss() per Matthew in patch 2.

	- Change function name from putback_active_hugetlb_folio()
	  to folio_putback_active_hugetlb() per Matthew in patch 3.

	- Squashed v1 patches 4-5 into one patch to make for easier
	  review as did not need to handle null page case from
	  alloc_huge_page().
	  
	  Refactor hugetlbfs_fallocate() to use same rcu +
	  page_cache_next_miss() that Matthew suggested for 
	  hugetlbfs_pagecache_present() per Mike in patch 4.

	  Initialize h_cg variable to NULL in alloc_hugetlb_folio() to 
	  suppress compiler warning when CONFIG_CGROUP_HUGETLB is not set
	  per [e-mail] in patch 4.

	  Changed variable page_in_pagecache to folio_in_pagecache in
	  hugetlb_mcopy_atomic_pte() as the variable is set to true once
	  the folio variable is added to the page cache in patch 4.

	- Fixed wrong change from get_page(page) ->
	  folio_get(pagecache_folio) reported by Gerald Schaefer in
	  patch 7.


============== OVERVIEW ===========================
This series converts the hugetlb page faulting functions to operate on
folios. These include hugetlb_no_page(), hugetlb_wp(),
copy_hugetlb_page_range(), and hugetlb_mcopy_atomic_pte().

patch 1-3:
        - convert prerequisite helper functions to folios.

patch 4:
        - add a folio variable to the hugetlb fault functions to complete
	  conversion to folios.

patch 5-7:
        - convert three functions to take in a folio rather than a page as
          all callers now use folios.

patch 8:
	-update documentation that references alloc_huge_page

============== TEST COVERAGE ============================

Linux Test Project Hugetlb Test Fault Coverage

[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlb_no_page'
FUNC                              COUNT
hugetlb_no_page                    7796

[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlb_wp'
FUNC                              COUNT
hugetlb_wp                         4623

Using fallocate commands to create files on hugeltbfs

[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlbfs*'
FUNC                              COUNT
hugetlbfs_fallocate                   1


Userfaultfd selftest
./userfaultfd hugetlb 256 50 /dev/hugepages/hugefile

[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlb_mcopy_atomic_pte'
FUNC                              COUNT
hugetlb_mcopy_atomic_pte           6240


[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'copy_hugetlb_page_range'
FUNC                              COUNT
copy_hugetlb_page_range               3

============== PERFORMANCE ============================
using bpftrace to track time spent in fault functions over 10 rounds of
the LTP hugetlb tests

pre-patch:

@hugetlb_wp_nsecs:
[256, 512)          3675 |@@@@@@@@@@                                          |
[512, 1K)          18875 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)            1366 |@@@                                                 |
[2K, 4K)              77 |                                                    |
[4K, 8K)              12 |                                                    |
[8K, 16K)             10 |                                                    |
[16K, 32K)             2 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               0 |                                                    |
[2G, 4G)               1 |                                                    |

@hugetlb_no_page_nsecs:                                                                                                                                                                                                                   
[64, 128)              1 |                                                    |                                                                                                                                                           
[128, 256)             0 |                                                    |                                                                                                                                                           
[256, 512)            67 |                                                    |                                                                                                                                                           
[512, 1K)             66 |                                                    |                                                                                                                                                           
[1K, 2K)              65 |                                                    |                                                                                                                                                           
[2K, 4K)             198 |                                                    |                                                                                                                                                           
[4K, 8K)              97 |                                                    |                                                                                                                                                           
[8K, 16K)              3 |                                                    |                                                                                                                                                           
[16K, 32K)             4 |                                                    |                                                                                                                                                           
[32K, 64K)           678 |                                                    |                                                                                                                                                           
[64K, 128K)         3401 |@                                                   |                                                                                                                                                           
[128K, 256K)       96746 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|                                                                                                                                                           
[256K, 512K)         107 |                                                    |                                                                                                                                                           
[512K, 1M)             0 |                                                    |                                                                                                                                                           
[1M, 2M)               1 |                                                    |                                                                                                                                                           
[2M, 4M)               0 |                                                    |                                                                                                                                                           
[4M, 8M)               1 |                                                    |                                                                                                                                                           
[8M, 16M)              0 |                                                    |                                                                                                                                                           
[16M, 32M)             0 |                                                    |                                                                                                                                                           
[32M, 64M)             0 |                                                    |                                                                                                                                                           
[64M, 128M)            0 |                                                    |                                                                                                                                                           
[128M, 256M)           0 |                                                    |                                                                                                                                                           
[256M, 512M)           0 |                                                    |                                                                                                                                                           
[512M, 1G)             0 |                                                    |                                                                                                                                                           
[1G, 2G)               0 |                                                    |                                                                                                                                                           
[2G, 4G)             283 |                                                    |

post-patch:

@hugetlb_wp_nsecs: 
[256, 512)         22282 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)          16242 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@               |
[1K, 2K)            1579 |@@@                                                 |
[2K, 4K)              76 |                                                    |
[4K, 8K)              18 |                                                    |
[8K, 16K)             30 |                                                    |
[16K, 32K)             4 |                                                    |

@hugetlb_no_page_nsecs:                                                                                                                                                                                                                    
[256, 512)            92 |                                                    |                                                                                                                                                            
[512, 1K)             46 |                                                    |                                                                                                                                                            
[1K, 2K)              83 |                                                    |                                                                                                                                                            
[2K, 4K)             188 |                                                    |                                                                                                                                                            
[4K, 8K)              80 |                                                    |                                                                                                                                                            
[8K, 16K)             13 |                                                    |                                                                                                                                                            
[16K, 32K)             8 |                                                    |                                                                                                                                                            
[32K, 64K)          2332 |@                                                   |                                                                                                                                                            
[64K, 128K)         7992 |@@@                                                 |                                                                                                                                                            
[128K, 256K)      106347 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[256K, 512K)         143 |                                                    |
[512K, 1M)             1 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               0 |                                                    |
[2G, 4G)             606 |                                                    |

rebased on 01/24/23 mm-unstable

Sidhartha Kumar (8):
  mm/hugetlb: convert hugetlb_install_page to folios
  mm/hugetlb: convert hugetlbfs_pagecache_present() to folios
  mm/hugetlb: convert putback_active_hugepage to take in a folio
  mm/hugetlb: convert hugetlb fault paths to use alloc_hugetlb_folio()
  mm/hugetlb: convert restore_reserve_on_error to take in a folio
  mm/hugetlb: convert hugetlb_add_to_page_cache to take in a folio
  mm/hugetlb: convert hugetlb_wp() to take in a folio
  Documentation/mm: update hugetlbfs documentation to mention
    alloc_hugetlb_folio

 Documentation/mm/hugetlbfs_reserv.rst         |  21 +-
 .../zh_CN/mm/hugetlbfs_reserv.rst             |  14 +-
 fs/hugetlbfs/inode.c                          |  40 +--
 include/linux/hugetlb.h                       |  16 +-
 include/linux/rmap.h                          |   2 +-
 mm/hugetlb.c                                  | 279 +++++++++---------
 mm/mempolicy.c                                |   6 +-
 mm/migrate.c                                  |   8 +-
 mm/rmap.c                                     |   6 +-
 9 files changed, 196 insertions(+), 196 deletions(-)