| Message ID | 20190531064313.193437-4-minchan@kernel.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | introduce memory hinting API for external process | expand |
On Fri 31-05-19 15:43:10, Minchan Kim wrote: > When a process expects no accesses to a certain memory range > for a long time, it could hint kernel that the pages can be > reclaimed instantly but data should be preserved for future use. > This could reduce workingset eviction so it ends up increasing > performance. > > This patch introduces the new MADV_PAGEOUT hint to madvise(2) > syscall. MADV_PAGEOUT can be used by a process to mark a memory > range as not expected to be used for a long time so that kernel > reclaims the memory instantly. The hint can help kernel in deciding > which pages to evict proactively. Again, are there any restictions on what kind of memory can be paged out? Private/Shared, anonymous/file backed. Any restrictions on mapping type. Etc. Please make sure all that is in the changelog. What are the failure modes? E.g. what if the swap is full, does the call fails or it silently ignores the error? Thanks!
On Fri, May 31, 2019 at 10:50:44AM +0200, Michal Hocko wrote: > On Fri 31-05-19 15:43:10, Minchan Kim wrote: > > When a process expects no accesses to a certain memory range > > for a long time, it could hint kernel that the pages can be > > reclaimed instantly but data should be preserved for future use. > > This could reduce workingset eviction so it ends up increasing > > performance. > > > > This patch introduces the new MADV_PAGEOUT hint to madvise(2) > > syscall. MADV_PAGEOUT can be used by a process to mark a memory > > range as not expected to be used for a long time so that kernel > > reclaims the memory instantly. The hint can help kernel in deciding > > which pages to evict proactively. > > Again, are there any restictions on what kind of memory can be paged out? > Private/Shared, anonymous/file backed. Any restrictions on mapping type. > Etc. Please make sure all that is in the changelog. It's same with MADV_COLD. Yes, I will include all detail in the description. > > What are the failure modes? E.g. what if the swap is full, does the call > fails or it silently ignores the error? In such case, just ignore the swapout. It returns -EINVAL only if the vma is one of (VM_LOCKED|VM_HUGETLB|VM_PFNMAP) at this moment. > > Thanks! > -- > Michal Hocko > SUSE Labs
Hi Michan, this looks pretty straight-forward to me, only one kink: On Fri, May 31, 2019 at 03:43:10PM +0900, Minchan Kim wrote: > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -2126,6 +2126,83 @@ static void shrink_active_list(unsigned long nr_to_scan, > nr_deactivate, nr_rotated, sc->priority, file); > } > > +unsigned long reclaim_pages(struct list_head *page_list) > +{ > + int nid = -1; > + unsigned long nr_isolated[2] = {0, }; > + unsigned long nr_reclaimed = 0; > + LIST_HEAD(node_page_list); > + struct reclaim_stat dummy_stat; > + struct scan_control sc = { > + .gfp_mask = GFP_KERNEL, > + .priority = DEF_PRIORITY, > + .may_writepage = 1, > + .may_unmap = 1, > + .may_swap = 1, > + }; > + > + while (!list_empty(page_list)) { > + struct page *page; > + > + page = lru_to_page(page_list); > + if (nid == -1) { > + nid = page_to_nid(page); > + INIT_LIST_HEAD(&node_page_list); > + nr_isolated[0] = nr_isolated[1] = 0; > + } > + > + if (nid == page_to_nid(page)) { > + list_move(&page->lru, &node_page_list); > + nr_isolated[!!page_is_file_cache(page)] += > + hpage_nr_pages(page); > + continue; > + } > + > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > + nr_isolated[0]); > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > + nr_isolated[1]); > + nr_reclaimed += shrink_page_list(&node_page_list, > + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS, > + &dummy_stat, true); > + while (!list_empty(&node_page_list)) { > + struct page *page = lru_to_page(&node_page_list); > + > + list_del(&page->lru); > + putback_lru_page(page); > + } > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > + -nr_isolated[0]); > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > + -nr_isolated[1]); > + nid = -1; > + } > + > + if (!list_empty(&node_page_list)) { > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > + nr_isolated[0]); > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > + nr_isolated[1]); > + nr_reclaimed += shrink_page_list(&node_page_list, > + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS, > + &dummy_stat, true); > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > + -nr_isolated[0]); > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > + -nr_isolated[1]); > + > + while (!list_empty(&node_page_list)) { > + struct page *page = lru_to_page(&node_page_list); > + > + list_del(&page->lru); > + putback_lru_page(page); > + } > + > + } The NR_ISOLATED accounting, nid parsing etc. is really awkward and makes it hard to see what the function actually does. Can you please make those ISOLATED counters part of the isolation API? Your patch really shows this is an overdue cleanup. These are fast local percpu counters, we don't need the sprawling batching we do all over vmscan.c, migrate.c, khugepaged.c, compaction.c etc. Isolation can increase the counter page by page, and reclaim or putback can likewise decrease them one by one. It looks like mlock is the only user of the isolation api that does not participate in the NR_ISOLATED_* counters protocol, but I don't see why it wouldn't, or why doing so would hurt. There are also seem to be quite a few callsites that use the atomic versions of the counter API when they're clearly under the irqsafe lru_lock. That would be fixed automatically by this work as well.
Hey Johannes, On Fri, May 31, 2019 at 12:59:27PM -0400, Johannes Weiner wrote: > Hi Michan, > > this looks pretty straight-forward to me, only one kink: > > On Fri, May 31, 2019 at 03:43:10PM +0900, Minchan Kim wrote: > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -2126,6 +2126,83 @@ static void shrink_active_list(unsigned long nr_to_scan, > > nr_deactivate, nr_rotated, sc->priority, file); > > } > > > > +unsigned long reclaim_pages(struct list_head *page_list) > > +{ > > + int nid = -1; > > + unsigned long nr_isolated[2] = {0, }; > > + unsigned long nr_reclaimed = 0; > > + LIST_HEAD(node_page_list); > > + struct reclaim_stat dummy_stat; > > + struct scan_control sc = { > > + .gfp_mask = GFP_KERNEL, > > + .priority = DEF_PRIORITY, > > + .may_writepage = 1, > > + .may_unmap = 1, > > + .may_swap = 1, > > + }; > > + > > + while (!list_empty(page_list)) { > > + struct page *page; > > + > > + page = lru_to_page(page_list); > > + if (nid == -1) { > > + nid = page_to_nid(page); > > + INIT_LIST_HEAD(&node_page_list); > > + nr_isolated[0] = nr_isolated[1] = 0; > > + } > > + > > + if (nid == page_to_nid(page)) { > > + list_move(&page->lru, &node_page_list); > > + nr_isolated[!!page_is_file_cache(page)] += > > + hpage_nr_pages(page); > > + continue; > > + } > > + > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > > + nr_isolated[0]); > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > > + nr_isolated[1]); > > + nr_reclaimed += shrink_page_list(&node_page_list, > > + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS, > > + &dummy_stat, true); > > + while (!list_empty(&node_page_list)) { > > + struct page *page = lru_to_page(&node_page_list); > > + > > + list_del(&page->lru); > > + putback_lru_page(page); > > + } > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > > + -nr_isolated[0]); > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > > + -nr_isolated[1]); > > + nid = -1; > > + } > > + > > + if (!list_empty(&node_page_list)) { > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > > + nr_isolated[0]); > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > > + nr_isolated[1]); > > + nr_reclaimed += shrink_page_list(&node_page_list, > > + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS, > > + &dummy_stat, true); > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, > > + -nr_isolated[0]); > > + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, > > + -nr_isolated[1]); > > + > > + while (!list_empty(&node_page_list)) { > > + struct page *page = lru_to_page(&node_page_list); > > + > > + list_del(&page->lru); > > + putback_lru_page(page); > > + } > > + > > + } > > The NR_ISOLATED accounting, nid parsing etc. is really awkward and > makes it hard to see what the function actually does. > > Can you please make those ISOLATED counters part of the isolation API? > Your patch really shows this is an overdue cleanup. Yeah, that was very painful. > > These are fast local percpu counters, we don't need the sprawling > batching we do all over vmscan.c, migrate.c, khugepaged.c, > compaction.c etc. Isolation can increase the counter page by page, and > reclaim or putback can likewise decrease them one by one. > > It looks like mlock is the only user of the isolation api that does > not participate in the NR_ISOLATED_* counters protocol, but I don't > see why it wouldn't, or why doing so would hurt. > > There are also seem to be quite a few callsites that use the atomic > versions of the counter API when they're clearly under the irqsafe > lru_lock. That would be fixed automatically by this work as well. I agree all points so will prepare clean up patch.
diff --git a/include/linux/swap.h b/include/linux/swap.h index 0ce997edb8bb..063c0c1e112b 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -365,6 +365,7 @@ extern int vm_swappiness; extern int remove_mapping(struct address_space *mapping, struct page *page); extern unsigned long vm_total_pages; +extern unsigned long reclaim_pages(struct list_head *page_list); #ifdef CONFIG_NUMA extern int node_reclaim_mode; extern int sysctl_min_unmapped_ratio; diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h index 1190f4e7f7b9..92e347a89ddc 100644 --- a/include/uapi/asm-generic/mman-common.h +++ b/include/uapi/asm-generic/mman-common.h @@ -44,6 +44,7 @@ #define MADV_WILLNEED 3 /* will need these pages */ #define MADV_DONTNEED 4 /* don't need these pages */ #define MADV_COLD 5 /* deactivatie these pages */ +#define MADV_PAGEOUT 6 /* reclaim these pages */ /* common parameters: try to keep these consistent across architectures */ #define MADV_FREE 8 /* free pages only if memory pressure */ diff --git a/mm/madvise.c b/mm/madvise.c index bff150eab6da..9d749a1420b4 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -41,6 +41,7 @@ static int madvise_need_mmap_write(int behavior) case MADV_WILLNEED: case MADV_DONTNEED: case MADV_COLD: + case MADV_PAGEOUT: case MADV_FREE: return 0; default: @@ -415,6 +416,128 @@ static long madvise_cold(struct vm_area_struct *vma, return 0; } +static int madvise_pageout_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + pte_t *orig_pte, *pte, ptent; + spinlock_t *ptl; + LIST_HEAD(page_list); + struct page *page; + int isolated = 0; + struct vm_area_struct *vma = walk->vma; + unsigned long next; + + if (fatal_signal_pending(current)) + return -EINTR; + + next = pmd_addr_end(addr, end); + if (pmd_trans_huge(*pmd)) { + ptl = pmd_trans_huge_lock(pmd, vma); + if (!ptl) + return 0; + + if (is_huge_zero_pmd(*pmd)) + goto huge_unlock; + + page = pmd_page(*pmd); + if (page_mapcount(page) > 1) + goto huge_unlock; + + if (next - addr != HPAGE_PMD_SIZE) { + int err; + + get_page(page); + spin_unlock(ptl); + lock_page(page); + err = split_huge_page(page); + unlock_page(page); + put_page(page); + if (!err) + goto regular_page; + return 0; + } + + if (isolate_lru_page(page)) + goto huge_unlock; + + list_add(&page->lru, &page_list); +huge_unlock: + spin_unlock(ptl); + reclaim_pages(&page_list); + return 0; + } + + if (pmd_trans_unstable(pmd)) + return 0; +regular_page: + orig_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + for (pte = orig_pte; addr < end; pte++, addr += PAGE_SIZE) { + ptent = *pte; + if (!pte_present(ptent)) + continue; + + page = vm_normal_page(vma, addr, ptent); + if (!page) + continue; + + if (page_mapcount(page) > 1) + continue; + + if (isolate_lru_page(page)) + continue; + + isolated++; + list_add(&page->lru, &page_list); + if (isolated >= SWAP_CLUSTER_MAX) { + pte_unmap_unlock(orig_pte, ptl); + reclaim_pages(&page_list); + isolated = 0; + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + orig_pte = pte; + } + } + + pte_unmap_unlock(orig_pte, ptl); + reclaim_pages(&page_list); + cond_resched(); + + return 0; +} + +static void madvise_pageout_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + struct mm_walk warm_walk = { + .pmd_entry = madvise_pageout_pte_range, + .mm = vma->vm_mm, + }; + + tlb_start_vma(tlb, vma); + walk_page_range(addr, end, &warm_walk); + tlb_end_vma(tlb, vma); +} + + +static long madvise_pageout(struct vm_area_struct *vma, + struct vm_area_struct **prev, + unsigned long start_addr, unsigned long end_addr) +{ + struct mm_struct *mm = vma->vm_mm; + struct mmu_gather tlb; + + *prev = vma; + if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP)) + return -EINVAL; + + lru_add_drain(); + tlb_gather_mmu(&tlb, mm, start_addr, end_addr); + madvise_pageout_page_range(&tlb, vma, start_addr, end_addr); + tlb_finish_mmu(&tlb, start_addr, end_addr); + + return 0; +} + static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -805,6 +928,8 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, return madvise_willneed(vma, prev, start, end); case MADV_COLD: return madvise_cold(vma, prev, start, end); + case MADV_PAGEOUT: + return madvise_pageout(vma, prev, start, end); case MADV_FREE: case MADV_DONTNEED: return madvise_dontneed_free(vma, prev, start, end, behavior); @@ -827,6 +952,7 @@ madvise_behavior_valid(int behavior) case MADV_DONTNEED: case MADV_FREE: case MADV_COLD: + case MADV_PAGEOUT: #ifdef CONFIG_KSM case MADV_MERGEABLE: case MADV_UNMERGEABLE: diff --git a/mm/vmscan.c b/mm/vmscan.c index 0973a46a0472..280dd808fb91 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2126,6 +2126,83 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_deactivate, nr_rotated, sc->priority, file); } +unsigned long reclaim_pages(struct list_head *page_list) +{ + int nid = -1; + unsigned long nr_isolated[2] = {0, }; + unsigned long nr_reclaimed = 0; + LIST_HEAD(node_page_list); + struct reclaim_stat dummy_stat; + struct scan_control sc = { + .gfp_mask = GFP_KERNEL, + .priority = DEF_PRIORITY, + .may_writepage = 1, + .may_unmap = 1, + .may_swap = 1, + }; + + while (!list_empty(page_list)) { + struct page *page; + + page = lru_to_page(page_list); + if (nid == -1) { + nid = page_to_nid(page); + INIT_LIST_HEAD(&node_page_list); + nr_isolated[0] = nr_isolated[1] = 0; + } + + if (nid == page_to_nid(page)) { + list_move(&page->lru, &node_page_list); + nr_isolated[!!page_is_file_cache(page)] += + hpage_nr_pages(page); + continue; + } + + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, + nr_isolated[0]); + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, + nr_isolated[1]); + nr_reclaimed += shrink_page_list(&node_page_list, + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS, + &dummy_stat, true); + while (!list_empty(&node_page_list)) { + struct page *page = lru_to_page(&node_page_list); + + list_del(&page->lru); + putback_lru_page(page); + } + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, + -nr_isolated[0]); + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, + -nr_isolated[1]); + nid = -1; + } + + if (!list_empty(&node_page_list)) { + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, + nr_isolated[0]); + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, + nr_isolated[1]); + nr_reclaimed += shrink_page_list(&node_page_list, + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS, + &dummy_stat, true); + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON, + -nr_isolated[0]); + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE, + -nr_isolated[1]); + + while (!list_empty(&node_page_list)) { + struct page *page = lru_to_page(&node_page_list); + + list_del(&page->lru); + putback_lru_page(page); + } + + } + + return nr_reclaimed; +} + /* * The inactive anon list should be small enough that the VM never has * to do too much work.
When a process expects no accesses to a certain memory range for a long time, it could hint kernel that the pages can be reclaimed instantly but data should be preserved for future use. This could reduce workingset eviction so it ends up increasing performance. This patch introduces the new MADV_PAGEOUT hint to madvise(2) syscall. MADV_PAGEOUT can be used by a process to mark a memory range as not expected to be used for a long time so that kernel reclaims the memory instantly. The hint can help kernel in deciding which pages to evict proactively. * RFCv1 * rename from MADV_COLD to MADV_PAGEOUT - hannes * bail out if process is being killed - Hillf * fix reclaim_pages bugs - Hillf Signed-off-by: Minchan Kim <minchan@kernel.org> --- include/linux/swap.h | 1 + include/uapi/asm-generic/mman-common.h | 1 + mm/madvise.c | 126 +++++++++++++++++++++++++ mm/vmscan.c | 77 +++++++++++++++ 4 files changed, 205 insertions(+)