From patchwork Mon Jun 3 05:36:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 10972323 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C51A06C5 for ; Mon, 3 Jun 2019 05:37:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B279328650 for ; Mon, 3 Jun 2019 05:37:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A6860288B6; Mon, 3 Jun 2019 05:37:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F3EC28650 for ; Mon, 3 Jun 2019 05:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E4F96B026B; Mon, 3 Jun 2019 01:37:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6939B6B026C; Mon, 3 Jun 2019 01:37:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55BBA6B026D; Mon, 3 Jun 2019 01:37:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 17B466B026B for ; Mon, 3 Jun 2019 01:37:17 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id r12so12610266pfl.2 for ; Sun, 02 Jun 2019 22:37:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:sender:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=QINx6pc+u7G6MZku2LRFGj1TYkzHOt+jmXPT4uDcbyI=; b=isiHovSs+L1Yi0OS5WopLVKTkMQHmMwXvM6opXdEBXv7+wvLYL9DEUTeIXMr3RMh6D LbEkY5JQf3ZDAFBu0WC3dFBbX195vEstCUD4rxJZz/4A5jYcWZ8Q/BFwVZx4c+LbzvpL 0TMZT13z7wKu9hHZv3OjOhyv4Mdr72758eH0mLDBSZZKRg2ya2j/Hj9W2pUfCI9R7YHW NR6Tp/h3mU5Mqv3Db+VOlttgnXwnFUjpkOf7G4OP5hN2SqTsZRALl0I8qTVxDnd4sB4w 9wSDNuT7cEpZ5CdhxgniuISE9JZn4RR2vS+ov/roa7Sbefp1RA8UtmGPAH515GOBKoBH 3H5Q== X-Gm-Message-State: APjAAAXs7cFdBqwuxpoJu/Q7cKAsX85SrvV/k467aTyvaVO3vWzqntCB DwDB43tSjpne7AyYsGa/pr+3AYR0tIR9Q08Wc0how202/3d/YW5BsQNPMmm53863Yaq4lMvs/xj wqdkAbvvRpcjauqGiPwHZWE0N8Jw6V97bziTX90HKgHL6qDHKZ8AIjLaFnyGSdgY= X-Received: by 2002:a17:902:ba8c:: with SMTP id k12mr26460696pls.229.1559540236445; Sun, 02 Jun 2019 22:37:16 -0700 (PDT) X-Received: by 2002:a17:902:ba8c:: with SMTP id k12mr26460644pls.229.1559540235095; Sun, 02 Jun 2019 22:37:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559540235; cv=none; d=google.com; s=arc-20160816; b=OUvcNFJNW730PivGG2pmAqh/MUrjZlcoy2/YhTMeUmRbnNknmxnDaHBTDC040VF2Ju cssVvTjGesBP/Y93oiGo7KIPxsM7ljRTNl98PJlgemHvuhgHpSqTB3W+CKGoy29cHmkX 5PFAXJgr8yKXdsZ038cKFv7GTrAiS49Aa/1BhpLOpoH2j9i5X/U77FNWRyWv5MIIIMlx VQq7pP01aj8ttdrBCtGwkoPDRU/KqsadyOZdIAA34XsSPLGWFZVkj4Mt2ZU9/+0+Pr7E 47yWpzThreTznLPvX7PMUYmRnHy1PZJjm5H5iU3okqMOT7hquFdelz5tIlspUfeZZanj IMsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=QINx6pc+u7G6MZku2LRFGj1TYkzHOt+jmXPT4uDcbyI=; b=gXNdpkrUy2W8GjxalCtWa9fw2BxdLz8w4zE3qL9EUgEZHq7r4onSvnAgdSAvT9b94Q 7eWz3Owl3Bl5+bAQIRF1/9+ns+5gYizaFTLpcZGA3VsqkkKr5YhlaZ1zHF1YjzwubP08 x1tFuXm/M0mDIylJU+2XIkA6DDcLd/GzjtLvL2KSXpNhPaSv5NeCr30MIvqtwPjXuX0x w5E9tzEp20jn4FivxQh4O1T2hbtqKmubjP82XfvYLH3e7VgNArhnYCiLC8raf/kyedas 2sBcW6XOOQGgPXzNJ5MCXAFSpt4mF/oCFXBMHe0Cgi/MPJF9cGaFqJWrgnWzfv5Tn2+8 pLhQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ufaZOuUq; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v10sor15731919plg.28.2019.06.02.22.37.14 for (Google Transport Security); Sun, 02 Jun 2019 22:37:15 -0700 (PDT) Received-SPF: pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ufaZOuUq; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QINx6pc+u7G6MZku2LRFGj1TYkzHOt+jmXPT4uDcbyI=; b=ufaZOuUqMTgOmRUAaC8szMySjRN8xjP/S+UVtUe9i963vnxK8XV/H7BcR8d57rqIdT se4D2zRcp+51NYNxRmljaP/H0uxuy/TcEXdbWvPgB9i1oEMsaFK7hBRcFQCTUqvmB/sK DT1Q/z//tp5EEOOoFeVHdwRWYB7eRMVsqLYhXrBg1LeIUcOLJ5a/Ej9U569HOecFqyXB 3GWnKFf+szY80jarHnJxI3RvXlRgQ9KEydsOVqeP5/BhkupHpvWTbsptOXzxCvnIC9K9 GJsaDd89lCYPTC+7j2UJtTyapST5nG5ivbYYDQP6fAlLqtq47Tg1B+y2fcCr35XrTfQa FD0A== X-Google-Smtp-Source: APXvYqydyq+MAblN2aWsCApQJrSKcDOSW1ecP2q+rxdQ2AiRWjcyp2qMg2i6j8P10fcHFgO5Vwq4cA== X-Received: by 2002:a17:902:a415:: with SMTP id p21mr27724890plq.46.1559540234640; Sun, 02 Jun 2019 22:37:14 -0700 (PDT) Received: from bbox-2.seo.corp.google.com ([2401:fa00:d:0:98f1:8b3d:1f37:3e8]) by smtp.gmail.com with ESMTPSA id a18sm5986222pjq.0.2019.06.02.22.37.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 22:37:13 -0700 (PDT) From: Minchan Kim To: Andrew Morton Cc: linux-mm , LKML , linux-api@vger.kernel.org, Michal Hocko , Johannes Weiner , Tim Murray , Joel Fernandes , Suren Baghdasaryan , Daniel Colascione , Shakeel Butt , Sonny Rao , Brian Geffon , jannh@google.com, oleg@redhat.com, christian@brauner.io, oleksandr@redhat.com, hdanton@sina.com, Minchan Kim Subject: [PATCH v1 1/4] mm: introduce MADV_COLD Date: Mon, 3 Jun 2019 14:36:52 +0900 Message-Id: <20190603053655.127730-2-minchan@kernel.org> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog In-Reply-To: <20190603053655.127730-1-minchan@kernel.org> References: <20190603053655.127730-1-minchan@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a process expects no accesses to a certain memory range, it could give a hint to kernel that the pages can be reclaimed when memory pressure happens but data should be preserved for future use. This could reduce workingset eviction so it ends up increasing performance. This patch introduces the new MADV_COLD hint to madvise(2) syscall. MADV_COLD can be used by a process to mark a memory range as not expected to be used in the near future. The hint can help kernel in deciding which pages to evict early during memory pressure. It works for every LRU pages like MADV_[DONTNEED|FREE]. IOW, It moves active file page -> inactive file LRU active anon page -> inacdtive anon LRU Unlike MADV_FREE, it doesn't move active anonymous pages to inactive files's head because MADV_COLD is a little bit different symantic. MADV_FREE means it's okay to discard when the memory pressure because the content of the page is *garbage* so freeing such pages is almost zero overhead since we don't need to swap out and access afterward causes just minor fault. Thus, it would make sense to put those freeable pages in inactive file LRU to compete other used-once pages. Even, it could give a bonus to make them be reclaimed on swapless system. However, MADV_COLD doesn't mean garbage so reclaiming them requires swap-out/in in the end. So it's better to move inactive anon's LRU list, not file LRU. Furthermore, it would help to avoid unnecessary scanning of cold anonymous if system doesn't have a swap device. All of error rule is same with MADV_DONTNEED. Note: This hint works with only private pages(IOW, page_mapcount(page) < 2) because shared page could have more chance to be accessed from other processes sharing the page although the caller reset the reference bits. It ends up preventing the reclaim of the page and wastes CPU cycle. * RFCv2 * add more description - mhocko * RFCv1 * renaming from MADV_COOL to MADV_COLD - hannes * internal review * use clear_page_youn in deactivate_page - joelaf * Revise the description - surenb * Renaming from MADV_WARM to MADV_COOL - surenb Signed-off-by: Minchan Kim --- include/linux/page-flags.h | 1 + include/linux/page_idle.h | 15 ++++ include/linux/swap.h | 1 + include/uapi/asm-generic/mman-common.h | 1 + mm/internal.h | 2 +- mm/madvise.c | 115 ++++++++++++++++++++++++- mm/oom_kill.c | 2 +- mm/swap.c | 43 +++++++++ 8 files changed, 176 insertions(+), 4 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 9f8712a4b1a5..58b06654c8dd 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -424,6 +424,7 @@ static inline bool set_hwpoison_free_buddy_page(struct page *page) TESTPAGEFLAG(Young, young, PF_ANY) SETPAGEFLAG(Young, young, PF_ANY) TESTCLEARFLAG(Young, young, PF_ANY) +CLEARPAGEFLAG(Young, young, PF_ANY) PAGEFLAG(Idle, idle, PF_ANY) #endif diff --git a/include/linux/page_idle.h b/include/linux/page_idle.h index 1e894d34bdce..f3f43b317150 100644 --- a/include/linux/page_idle.h +++ b/include/linux/page_idle.h @@ -19,6 +19,11 @@ static inline void set_page_young(struct page *page) SetPageYoung(page); } +static inline void clear_page_young(struct page *page) +{ + ClearPageYoung(page); +} + static inline bool test_and_clear_page_young(struct page *page) { return TestClearPageYoung(page); @@ -65,6 +70,16 @@ static inline void set_page_young(struct page *page) set_bit(PAGE_EXT_YOUNG, &page_ext->flags); } +static void clear_page_young(struct page *page) +{ + struct page_ext *page_ext = lookup_page_ext(page); + + if (unlikely(!page_ext)) + return; + + clear_bit(PAGE_EXT_YOUNG, &page_ext->flags); +} + static inline bool test_and_clear_page_young(struct page *page) { struct page_ext *page_ext = lookup_page_ext(page); diff --git a/include/linux/swap.h b/include/linux/swap.h index de2c67a33b7e..0ce997edb8bb 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -340,6 +340,7 @@ extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_all(void); extern void rotate_reclaimable_page(struct page *page); extern void deactivate_file_page(struct page *page); +extern void deactivate_page(struct page *page); extern void mark_page_lazyfree(struct page *page); extern void swap_setup(void); diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h index bea0278f65ab..1190f4e7f7b9 100644 --- a/include/uapi/asm-generic/mman-common.h +++ b/include/uapi/asm-generic/mman-common.h @@ -43,6 +43,7 @@ #define MADV_SEQUENTIAL 2 /* expect sequential page references */ #define MADV_WILLNEED 3 /* will need these pages */ #define MADV_DONTNEED 4 /* don't need these pages */ +#define MADV_COLD 5 /* deactivatie these pages */ /* common parameters: try to keep these consistent across architectures */ #define MADV_FREE 8 /* free pages only if memory pressure */ diff --git a/mm/internal.h b/mm/internal.h index 9eeaf2b95166..75a4f96ec0fb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -43,7 +43,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, unsigned long floor, unsigned long ceiling); -static inline bool can_madv_dontneed_vma(struct vm_area_struct *vma) +static inline bool can_madv_lru_vma(struct vm_area_struct *vma) { return !(vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP)); } diff --git a/mm/madvise.c b/mm/madvise.c index 628022e674a7..ab158766858a 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -40,6 +40,7 @@ static int madvise_need_mmap_write(int behavior) case MADV_REMOVE: case MADV_WILLNEED: case MADV_DONTNEED: + case MADV_COLD: case MADV_FREE: return 0; default: @@ -307,6 +308,113 @@ static long madvise_willneed(struct vm_area_struct *vma, return 0; } +static int madvise_cold_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + pte_t *orig_pte, *pte, ptent; + spinlock_t *ptl; + struct page *page; + struct vm_area_struct *vma = walk->vma; + unsigned long next; + + next = pmd_addr_end(addr, end); + if (pmd_trans_huge(*pmd)) { + ptl = pmd_trans_huge_lock(pmd, vma); + if (!ptl) + return 0; + + if (is_huge_zero_pmd(*pmd)) + goto huge_unlock; + + page = pmd_page(*pmd); + if (page_mapcount(page) > 1) + goto huge_unlock; + + if (next - addr != HPAGE_PMD_SIZE) { + int err; + + get_page(page); + spin_unlock(ptl); + lock_page(page); + err = split_huge_page(page); + unlock_page(page); + put_page(page); + if (!err) + goto regular_page; + return 0; + } + + pmdp_test_and_clear_young(vma, addr, pmd); + deactivate_page(page); +huge_unlock: + spin_unlock(ptl); + return 0; + } + + if (pmd_trans_unstable(pmd)) + return 0; + +regular_page: + orig_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + for (pte = orig_pte; addr < end; pte++, addr += PAGE_SIZE) { + ptent = *pte; + + if (pte_none(ptent)) + continue; + + if (!pte_present(ptent)) + continue; + + page = vm_normal_page(vma, addr, ptent); + if (!page) + continue; + + if (page_mapcount(page) > 1) + continue; + + ptep_test_and_clear_young(vma, addr, pte); + deactivate_page(page); + } + + pte_unmap_unlock(orig_pte, ptl); + cond_resched(); + + return 0; +} + +static void madvise_cold_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + struct mm_walk cool_walk = { + .pmd_entry = madvise_cold_pte_range, + .mm = vma->vm_mm, + }; + + tlb_start_vma(tlb, vma); + walk_page_range(addr, end, &cool_walk); + tlb_end_vma(tlb, vma); +} + +static long madvise_cold(struct vm_area_struct *vma, + struct vm_area_struct **prev, + unsigned long start_addr, unsigned long end_addr) +{ + struct mm_struct *mm = vma->vm_mm; + struct mmu_gather tlb; + + *prev = vma; + if (!can_madv_lru_vma(vma)) + return -EINVAL; + + lru_add_drain(); + tlb_gather_mmu(&tlb, mm, start_addr, end_addr); + madvise_cold_page_range(&tlb, vma, start_addr, end_addr); + tlb_finish_mmu(&tlb, start_addr, end_addr); + + return 0; +} + static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -519,7 +627,7 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, int behavior) { *prev = vma; - if (!can_madv_dontneed_vma(vma)) + if (!can_madv_lru_vma(vma)) return -EINVAL; if (!userfaultfd_remove(vma, start, end)) { @@ -541,7 +649,7 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, */ return -ENOMEM; } - if (!can_madv_dontneed_vma(vma)) + if (!can_madv_lru_vma(vma)) return -EINVAL; if (end > vma->vm_end) { /* @@ -695,6 +803,8 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, return madvise_remove(vma, prev, start, end); case MADV_WILLNEED: return madvise_willneed(vma, prev, start, end); + case MADV_COLD: + return madvise_cold(vma, prev, start, end); case MADV_FREE: case MADV_DONTNEED: return madvise_dontneed_free(vma, prev, start, end, behavior); @@ -716,6 +826,7 @@ madvise_behavior_valid(int behavior) case MADV_WILLNEED: case MADV_DONTNEED: case MADV_FREE: + case MADV_COLD: #ifdef CONFIG_KSM case MADV_MERGEABLE: case MADV_UNMERGEABLE: diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 5a58778c91d4..f73d5f5145f0 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -515,7 +515,7 @@ bool __oom_reap_task_mm(struct mm_struct *mm) set_bit(MMF_UNSTABLE, &mm->flags); for (vma = mm->mmap ; vma; vma = vma->vm_next) { - if (!can_madv_dontneed_vma(vma)) + if (!can_madv_lru_vma(vma)) continue; /* diff --git a/mm/swap.c b/mm/swap.c index 7b079976cbec..cebedab15aa2 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -47,6 +47,7 @@ int page_cluster; static DEFINE_PER_CPU(struct pagevec, lru_add_pvec); static DEFINE_PER_CPU(struct pagevec, lru_rotate_pvecs); static DEFINE_PER_CPU(struct pagevec, lru_deactivate_file_pvecs); +static DEFINE_PER_CPU(struct pagevec, lru_deactivate_pvecs); static DEFINE_PER_CPU(struct pagevec, lru_lazyfree_pvecs); #ifdef CONFIG_SMP static DEFINE_PER_CPU(struct pagevec, activate_page_pvecs); @@ -538,6 +539,23 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec, update_page_reclaim_stat(lruvec, file, 0); } +static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec, + void *arg) +{ + if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) { + int file = page_is_file_cache(page); + int lru = page_lru_base_type(page); + + del_page_from_lru_list(page, lruvec, lru + LRU_ACTIVE); + ClearPageActive(page); + ClearPageReferenced(page); + clear_page_young(page); + add_page_to_lru_list(page, lruvec, lru); + + __count_vm_events(PGDEACTIVATE, hpage_nr_pages(page)); + update_page_reclaim_stat(lruvec, file, 0); + } +} static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec, void *arg) @@ -590,6 +608,10 @@ void lru_add_drain_cpu(int cpu) if (pagevec_count(pvec)) pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL); + pvec = &per_cpu(lru_deactivate_pvecs, cpu); + if (pagevec_count(pvec)) + pagevec_lru_move_fn(pvec, lru_deactivate_fn, NULL); + pvec = &per_cpu(lru_lazyfree_pvecs, cpu); if (pagevec_count(pvec)) pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL); @@ -623,6 +645,26 @@ void deactivate_file_page(struct page *page) } } +/* + * deactivate_page - deactivate a page + * @page: page to deactivate + * + * deactivate_page() moves @page to the inactive list if @page was on the active + * list and was not an unevictable page. This is done to accelerate the reclaim + * of @page. + */ +void deactivate_page(struct page *page) +{ + if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) { + struct pagevec *pvec = &get_cpu_var(lru_deactivate_pvecs); + + get_page(page); + if (!pagevec_add(pvec, page) || PageCompound(page)) + pagevec_lru_move_fn(pvec, lru_deactivate_fn, NULL); + put_cpu_var(lru_deactivate_pvecs); + } +} + /** * mark_page_lazyfree - make an anon page lazyfree * @page: page to deactivate @@ -687,6 +729,7 @@ void lru_add_drain_all(void) if (pagevec_count(&per_cpu(lru_add_pvec, cpu)) || pagevec_count(&per_cpu(lru_rotate_pvecs, cpu)) || pagevec_count(&per_cpu(lru_deactivate_file_pvecs, cpu)) || + pagevec_count(&per_cpu(lru_deactivate_pvecs, cpu)) || pagevec_count(&per_cpu(lru_lazyfree_pvecs, cpu)) || need_activate_page_drain(cpu)) { INIT_WORK(work, lru_add_drain_per_cpu); From patchwork Mon Jun 3 05:36:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 10972325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E71B1398 for ; Mon, 3 Jun 2019 05:37:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E7CD28650 for ; Mon, 3 Jun 2019 05:37:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 525CD288B6; Mon, 3 Jun 2019 05:37:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D626828650 for ; Mon, 3 Jun 2019 05:37:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0E7A6B026C; Mon, 3 Jun 2019 01:37:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DBFE56B026D; Mon, 3 Jun 2019 01:37:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAEBB6B026E; Mon, 3 Jun 2019 01:37:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 96ABC6B026C for ; Mon, 3 Jun 2019 01:37:21 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id r4so12796603pfh.16 for ; Sun, 02 Jun 2019 22:37:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:sender:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=AspJoK2TnbwgjAatSkGxez0VLRm0DAxlcRNfqD2t6lc=; b=TNVf36kU4Z+FyaPEl6uyRS+Kn+ig+VWQOB05/dT7/l7d3e2m4hNMUTTf0ad57zHDPO sUdDttauQpCMO0YeFKJu6CBrPrPGj5rWnCw0hjHjkluxmlydl0CNAPPkr2vaNHcTPqGM 0AXFUf4IsvcFY2dDrhfkmbv0yvCusox4JUUXj1UzZryBjkmPRlHFMc1CVmagZNsqDIT/ 4+44+4WeMoQZ/V03QdU7kQ/l39g68tTLwi478uNzIEJ4n+VZ6r0ZpRKjyyNHrvpQ/hQv /yg3gE1ZsVW0SHSuwFaCt+dKetDodnrtYvcYCjI3iRePfPvM9JKIif6ylL36FIsrL9SI pfew== X-Gm-Message-State: APjAAAXQf6NGRQVn0kVS3Xx1AfebaMyIbcHneTl5X/w5jWx3vUzLrzzv QVGE5sBECfa3KvuIlI/9Oce0GZKDMulYZppbIHUKjMcrI0FDBNJVOYO7G3KEokNMjq1oav+sLJd e+CmzH81Q5AShC6anY2SIca6QtKTtyXHzWQYkX2rRvZBXqfBQqKj7ulmWnfGmokA= X-Received: by 2002:aa7:83d4:: with SMTP id j20mr29711611pfn.90.1559540241073; Sun, 02 Jun 2019 22:37:21 -0700 (PDT) X-Received: by 2002:aa7:83d4:: with SMTP id j20mr29711556pfn.90.1559540240147; Sun, 02 Jun 2019 22:37:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559540240; cv=none; d=google.com; s=arc-20160816; b=g9rmpHd3wmmIRVkMl1bxCg53js1pzDVAgJDohGiHfpLwPekbXJClYSXmXjj36UL9mn nTKG4Absj5E9HhaNXII8dBzgNVeHr8/fAdEN4J/sPcGXsNdmAJ2p3z9497mDAlXnGVXX 4e3i8C3tGEKdADNbjOF6er1FS+I7Ao7mNtzPsbiu0/+POhTfUzfuOQkmzBwgisR+/k91 b6mJ2ExRqgBnf3T1NwO1N7zTp9kyAdnuVdQYljSOCL/2b0OJiiBcffcKr9npSQvp10Sq dDZJnCtSo22HZ8tjiH8nPM3RGKk3a9HhKVZnhHcJije+IvyCXZGCyIhVWbHe0bZxAGWl TNtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=AspJoK2TnbwgjAatSkGxez0VLRm0DAxlcRNfqD2t6lc=; b=uE8jpWXIf7MMzdrdIspjg+OOFF8netNN5Cgiq0QicwRhNW32cfpI2nLjzGqZPVHSii 1dnvrtkPgdXRrKsUaRnHrzQnB+v1u9C05TAw0Mq2M9v4saczk4TStgHbQ8vDFuodW+yi yob2KRKvYZ8TtDaqT5bxVKhCD8SErIncAxOzOuONaUspQx9FFPhVsZJvyMAcQAhDYD3T lHSdKRtqTWPlg/87EkgdgDXZoYgSg+IsRvs4VN4dzF2mjIbvWHDeE/mk8Umi/9TxCmwB DNPIjmAym0vkCU05ikSe+mWQ0i7UoyuRUt+EiY9UvTjr9bMeSTNMkd84NWJJ7z8pJIQF 3t5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=sPcwms7L; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 194sor14004351pgc.19.2019.06.02.22.37.20 for (Google Transport Security); Sun, 02 Jun 2019 22:37:20 -0700 (PDT) Received-SPF: pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=sPcwms7L; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AspJoK2TnbwgjAatSkGxez0VLRm0DAxlcRNfqD2t6lc=; b=sPcwms7LYj4OL0DvDW4nBr5ZoY9hI81X1IP/z3cTp0vq+ISknRS5WqE6Zw8bwdF1Sg yvbs53EAb/7oq1V6nr51+aJkcyrxHNvFmHHLSNvUb6ZOgkOKpazX8IPcwl1VbQuR1hd8 Qpn/AscGh273pm+aEcqMzd3eQ64p3IcnT2e5OFQsccb3pMmblaCx37gA1Uk2W6gmH174 J36Nb/oBVslBf/K7IHRDzps+IdfImuzteUG++SbBm9HjFenHj/eh3aqLpTn6pVfugtYk HoPaCxRCQ8aicl8uWFhoOKf49ggJZvYN/PKYwPokllmdUqk2vB0MaRcq8x+dRrxtN00s hG/w== X-Google-Smtp-Source: APXvYqz0WzWbOPH7SHnTOp8MB+cFakTtiCbidmfpKSiMxWd9qnxaF17/qkGfGsSNuehNI7zsyP+OdA== X-Received: by 2002:a63:788a:: with SMTP id t132mr26741639pgc.52.1559540239802; Sun, 02 Jun 2019 22:37:19 -0700 (PDT) Received: from bbox-2.seo.corp.google.com ([2401:fa00:d:0:98f1:8b3d:1f37:3e8]) by smtp.gmail.com with ESMTPSA id a18sm5986222pjq.0.2019.06.02.22.37.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 22:37:18 -0700 (PDT) From: Minchan Kim To: Andrew Morton Cc: linux-mm , LKML , linux-api@vger.kernel.org, Michal Hocko , Johannes Weiner , Tim Murray , Joel Fernandes , Suren Baghdasaryan , Daniel Colascione , Shakeel Butt , Sonny Rao , Brian Geffon , jannh@google.com, oleg@redhat.com, christian@brauner.io, oleksandr@redhat.com, hdanton@sina.com, Minchan Kim Subject: [PATCH v1 2/4] mm: change PAGEREF_RECLAIM_CLEAN with PAGE_REFRECLAIM Date: Mon, 3 Jun 2019 14:36:53 +0900 Message-Id: <20190603053655.127730-3-minchan@kernel.org> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog In-Reply-To: <20190603053655.127730-1-minchan@kernel.org> References: <20190603053655.127730-1-minchan@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The local variable references in shrink_page_list is PAGEREF_RECLAIM_CLEAN as default. It is for preventing to reclaim dirty pages when CMA try to migrate pages. Strictly speaking, we don't need it because CMA didn't allow to write out by .may_writepage = 0 in reclaim_clean_pages_from_list. Moreover, it has a problem to prevent anonymous pages's swap out even though force_reclaim = true in shrink_page_list on upcoming patch. So this patch makes references's default value to PAGEREF_RECLAIM and rename force_reclaim with ignore_references to make it more clear. This is a preparatory work for next patch. * RFCv1 * use ignore_referecnes as parameter name - hannes Acked-by: Johannes Weiner Signed-off-by: Minchan Kim --- mm/vmscan.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 84dcb651d05c..0973a46a0472 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1102,7 +1102,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, struct scan_control *sc, enum ttu_flags ttu_flags, struct reclaim_stat *stat, - bool force_reclaim) + bool ignore_references) { LIST_HEAD(ret_pages); LIST_HEAD(free_pages); @@ -1116,7 +1116,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, struct address_space *mapping; struct page *page; int may_enter_fs; - enum page_references references = PAGEREF_RECLAIM_CLEAN; + enum page_references references = PAGEREF_RECLAIM; bool dirty, writeback; unsigned int nr_pages; @@ -1247,7 +1247,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, } } - if (!force_reclaim) + if (!ignore_references) references = page_check_references(page, sc); switch (references) { From patchwork Mon Jun 3 05:36:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 10972327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 340791398 for ; Mon, 3 Jun 2019 05:37:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20FBF28650 for ; Mon, 3 Jun 2019 05:37:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 136D6288B5; Mon, 3 Jun 2019 05:37:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E79BB288B6 for ; Mon, 3 Jun 2019 05:37:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAF386B026D; Mon, 3 Jun 2019 01:37:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C62A36B026E; Mon, 3 Jun 2019 01:37:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4F306B026F; Mon, 3 Jun 2019 01:37:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 7BD4E6B026D for ; Mon, 3 Jun 2019 01:37:27 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id f1so12825316pfb.0 for ; Sun, 02 Jun 2019 22:37:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:sender:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=6bbySnogE9Qd5QixoKSp27GOIYlcEz4Q+4n3evuFS2s=; b=A8RA3fMRLziIfXk20oeJWIJYj8cYOKJZ6geeBtbKE9A2hOixKxbvO/PZTNUCj9QxUA V0EiVZdATo6u/+oQwgmH79RN2u2haFmaEOeYxkvAsbA6uOEGlAWzwyD25F2RV/xg2Mk/ HlKlIthw37bVNKxAcqvhd/G98CKdWLtu4vEH1qyKzj+l/W5sZ4dIyZTFRrQeLlzy8ymc K8dS4/YS/+zlchNAv5hVd3ErOWUWht78KZYBWar461B9orqJ3+1MLn74tOiot2f+UMA8 sWAewwbMdpmQGH75q1rLIXZLH1D90l2pMDV+bSwzkJFZYQJ4SZ4kzCJX9I8g00tl4ZXp i8Ew== X-Gm-Message-State: APjAAAVUw/reRWJdaBpcJfViBINsWZSjJ61zLsm01S6x6ivH6O9lmqcC /rQuq+L+GSy0eFmdswOYgf+oHabDiTvTDAB4796SV3omlvg/634i5YI/JHDWLs2QfO0VDsmGM+h IbPBMQt6ra1q4jguYSXIax5qtcvDdhDCnk8jNxWoGKm0OKX8+n4VC7U3lYI6C59s= X-Received: by 2002:a17:902:12f:: with SMTP id 44mr27527469plb.137.1559540246989; Sun, 02 Jun 2019 22:37:26 -0700 (PDT) X-Received: by 2002:a17:902:12f:: with SMTP id 44mr27527407plb.137.1559540245648; Sun, 02 Jun 2019 22:37:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559540245; cv=none; d=google.com; s=arc-20160816; b=tLcpggIgxGOrD/2RgqfkWKI9mwS8hynoFiAEiEZOKtCvZBc5jA7v4SRTYEOI9qhXOp 0UfirX2J60OjWlKJsrouykkiKGAORiDdaZlAYv8t3DcCpc5Xqa0WUGQAz40kgxG/ZJSK jVV2IHJDGn6WYcWsUbpffd5L5p8t7kNC4KHsyFmQPTFyEmDA6X0LNYGTkW4RaSguLn8W gtUlM1j+j/3uxfTaByyIfN6iqQivYFfEoONqjOTZxHJaRDfGXpB0HlrzVoDpMAag4l2Z hL61Hr0h/YrlUn+ZhBNumjhctb6rIiCv6hOkgM8sJeZbguKHlonSmzXOeovLeDL5XNQh UHWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=6bbySnogE9Qd5QixoKSp27GOIYlcEz4Q+4n3evuFS2s=; b=tfU061hKy62Ngs9HuUa7jHUA/ULUyr7YADg+kCklD2R6f1FmcVE65l5cOZ/8z4Nwjy EaNJDD1DZaATkGfng/IZqoU3YZ5dTZR9loFKvFBpx0DGs1JIDrY85xSIfraMva1XlS1e Ythrh/mbmGJ+nkdt5iPCWUzftAnA0RQg8Wz44hzmjrYsY8Yq8q12GRtWU9q9buAgnjIk 5HwqDKE/NWb4smhlZd5UbW4CvmdYBQVTEyZmD7TBA/UENEnohB0vFJ+HXZVOdZ2OvBs4 17dWQi3+oF/MuCMM7MMr/1Y6GgqIZXtKInyXAF7HXW0HiIvklSut475Op5ls2qqplrml 69tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="gXWy3/vy"; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id r65sor15965723pjb.6.2019.06.02.22.37.25 for (Google Transport Security); Sun, 02 Jun 2019 22:37:25 -0700 (PDT) Received-SPF: pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="gXWy3/vy"; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6bbySnogE9Qd5QixoKSp27GOIYlcEz4Q+4n3evuFS2s=; b=gXWy3/vyQL9jBjM9azFs/8i1rMw5mx6IIlkutWndsgDgEIbWSzKgDeinCMi2NqKel+ hDW580nbbSzlryiAjrfjK6nIZg29vnRWXxhkvg1ZjXKJE0OskzZ2SOZUGMOBiIL8KhKa 4cCgVwQj8piP+aCZhqOqlN/FEeAGFx3dClMgssatrQiyNpDzdcuiQGzxDkFtJjZeGc4e ke+0a9EO0FoRFLtPYVBI5MSGu9xWNtcCQRvaDhOTb9asJ453NCFq/OmiKl9232VQT5NY C9zxdQhHtSN2ejrZPwqSIG048SAk93g2M/uTls03zqmNtTTQS6/1ra6Wpv9xp2uV41aw +iHQ== X-Google-Smtp-Source: APXvYqwTCBrgjqzRsF/8ltrK/fMDOKlL3ROdeyyjDzrnMbonhz4oO686YDoBnQmIPdZ6RVe4j0ufKQ== X-Received: by 2002:a17:90a:b78b:: with SMTP id m11mr27919789pjr.106.1559540245171; Sun, 02 Jun 2019 22:37:25 -0700 (PDT) Received: from bbox-2.seo.corp.google.com ([2401:fa00:d:0:98f1:8b3d:1f37:3e8]) by smtp.gmail.com with ESMTPSA id a18sm5986222pjq.0.2019.06.02.22.37.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 22:37:24 -0700 (PDT) From: Minchan Kim To: Andrew Morton Cc: linux-mm , LKML , linux-api@vger.kernel.org, Michal Hocko , Johannes Weiner , Tim Murray , Joel Fernandes , Suren Baghdasaryan , Daniel Colascione , Shakeel Butt , Sonny Rao , Brian Geffon , jannh@google.com, oleg@redhat.com, christian@brauner.io, oleksandr@redhat.com, hdanton@sina.com, Minchan Kim Subject: [PATCH v1 3/4] mm: account nr_isolated_xxx in [isolate|putback]_lru_page Date: Mon, 3 Jun 2019 14:36:54 +0900 Message-Id: <20190603053655.127730-4-minchan@kernel.org> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog In-Reply-To: <20190603053655.127730-1-minchan@kernel.org> References: <20190603053655.127730-1-minchan@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The isolate counting is pecpu counter so it would be not huge gain to work them by batch. Rather than complicating to make them batch, let's make it more stright-foward via adding the counting logic into [isolate|putback]_lru_page API. Link: http://lkml.kernel.org/r/20190531165927.GA20067@cmpxchg.org Suggested-by: Johannes Weiner Signed-off-by: Minchan Kim --- mm/compaction.c | 2 -- mm/gup.c | 7 +------ mm/khugepaged.c | 3 --- mm/memory-failure.c | 3 --- mm/memory_hotplug.c | 4 ---- mm/mempolicy.c | 6 +----- mm/migrate.c | 37 ++++++++----------------------------- mm/vmscan.c | 22 ++++++++++++++++------ 8 files changed, 26 insertions(+), 58 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9e1b9acb116b..c6591682deda 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -982,8 +982,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* Successfully isolated */ del_page_from_lru_list(page, lruvec, page_lru(page)); - inc_node_page_state(page, - NR_ISOLATED_ANON + page_is_file_cache(page)); isolate_success: list_add(&page->lru, &cc->migratepages); diff --git a/mm/gup.c b/mm/gup.c index 63ac50e48072..2d9a9bc358c7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1360,13 +1360,8 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk, drain_allow = false; } - if (!isolate_lru_page(head)) { + if (!isolate_lru_page(head)) list_add_tail(&head->lru, &cma_page_list); - mod_node_page_state(page_pgdat(head), - NR_ISOLATED_ANON + - page_is_file_cache(head), - hpage_nr_pages(head)); - } } } } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a335f7c1fac4..3359df994fb4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -503,7 +503,6 @@ void __khugepaged_exit(struct mm_struct *mm) static void release_pte_page(struct page *page) { - dec_node_page_state(page, NR_ISOLATED_ANON + page_is_file_cache(page)); unlock_page(page); putback_lru_page(page); } @@ -602,8 +601,6 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_DEL_PAGE_LRU; goto out; } - inc_node_page_state(page, - NR_ISOLATED_ANON + page_is_file_cache(page)); VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(PageLRU(page), page); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index bc749265a8f3..2187bad7ceff 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1796,9 +1796,6 @@ static int __soft_offline_page(struct page *page, int flags) * so use !__PageMovable instead for LRU page's mapping * cannot have PAGE_MAPPING_MOVABLE. */ - if (!__PageMovable(page)) - inc_node_page_state(page, NR_ISOLATED_ANON + - page_is_file_cache(page)); list_add(&page->lru, &pagelist); ret = migrate_pages(&pagelist, new_page, NULL, MPOL_MF_MOVE_ALL, MIGRATE_SYNC, MR_MEMORY_FAILURE); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index a88c5f334e5a..a41bea24d0c9 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1390,10 +1390,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn) ret = isolate_movable_page(page, ISOLATE_UNEVICTABLE); if (!ret) { /* Success */ list_add_tail(&page->lru, &source); - if (!__PageMovable(page)) - inc_node_page_state(page, NR_ISOLATED_ANON + - page_is_file_cache(page)); - } else { pr_warn("failed to isolate pfn %lx\n", pfn); dump_page(page, "isolation failed"); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 5b3bf1747c19..cfb0590f69bb 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -948,12 +948,8 @@ static void migrate_page_add(struct page *page, struct list_head *pagelist, * Avoid migrating a page that is shared with others. */ if ((flags & MPOL_MF_MOVE_ALL) || page_mapcount(head) == 1) { - if (!isolate_lru_page(head)) { + if (!isolate_lru_page(head)) list_add_tail(&head->lru, pagelist); - mod_node_page_state(page_pgdat(head), - NR_ISOLATED_ANON + page_is_file_cache(head), - hpage_nr_pages(head)); - } } } diff --git a/mm/migrate.c b/mm/migrate.c index 572b4bc85d76..39b95ba04d3e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -190,8 +190,6 @@ void putback_movable_pages(struct list_head *l) unlock_page(page); put_page(page); } else { - mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + - page_is_file_cache(page), -hpage_nr_pages(page)); putback_lru_page(page); } } @@ -1181,10 +1179,17 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, return -ENOMEM; if (page_count(page) == 1) { + bool is_lru = !__PageMovable(page); + /* page was freed from under us. So we are done. */ ClearPageActive(page); ClearPageUnevictable(page); - if (unlikely(__PageMovable(page))) { + if (likely(is_lru)) + mod_node_page_state(page_pgdat(page), + NR_ISOLATED_ANON + + page_is_file_cache(page), + hpage_nr_pages(page)); + else { lock_page(page); if (!PageMovable(page)) __ClearPageIsolated(page); @@ -1210,15 +1215,6 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, * restored. */ list_del(&page->lru); - - /* - * Compaction can migrate also non-LRU pages which are - * not accounted to NR_ISOLATED_*. They can be recognized - * as __PageMovable - */ - if (likely(!__PageMovable(page))) - mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + - page_is_file_cache(page), -hpage_nr_pages(page)); } /* @@ -1572,9 +1568,6 @@ static int add_page_for_migration(struct mm_struct *mm, unsigned long addr, err = 0; list_add_tail(&head->lru, pagelist); - mod_node_page_state(page_pgdat(head), - NR_ISOLATED_ANON + page_is_file_cache(head), - hpage_nr_pages(head)); } out_putpage: /* @@ -1890,8 +1883,6 @@ static struct page *alloc_misplaced_dst_page(struct page *page, static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { - int page_lru; - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); /* Avoid migrating to a node that is nearly full */ @@ -1913,10 +1904,6 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) return 0; } - page_lru = page_is_file_cache(page); - mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_lru, - hpage_nr_pages(page)); - /* * Isolating the page has taken another reference, so the * caller's reference can be safely dropped without the page @@ -1971,8 +1958,6 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, if (nr_remaining) { if (!list_empty(&migratepages)) { list_del(&page->lru); - dec_node_page_state(page, NR_ISOLATED_ANON + - page_is_file_cache(page)); putback_lru_page(page); } isolated = 0; @@ -2002,7 +1987,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, pg_data_t *pgdat = NODE_DATA(node); int isolated = 0; struct page *new_page = NULL; - int page_lru = page_is_file_cache(page); unsigned long start = address & HPAGE_PMD_MASK; new_page = alloc_pages_node(node, @@ -2048,8 +2032,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, /* Retake the callers reference and putback on LRU */ get_page(page); putback_lru_page(page); - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR); goto out_unlock; } @@ -2099,9 +2081,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, count_vm_events(PGMIGRATE_SUCCESS, HPAGE_PMD_NR); count_vm_numa_events(NUMA_PAGE_MIGRATE, HPAGE_PMD_NR); - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_lru, - -HPAGE_PMD_NR); return isolated; out_fail: diff --git a/mm/vmscan.c b/mm/vmscan.c index 0973a46a0472..56df55e8afcd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -999,6 +999,9 @@ int remove_mapping(struct address_space *mapping, struct page *page) void putback_lru_page(struct page *page) { lru_cache_add(page); + mod_node_page_state(page_pgdat(page), + NR_ISOLATED_ANON + page_is_file_cache(page), + -hpage_nr_pages(page)); put_page(page); /* drop ref from isolate */ } @@ -1464,6 +1467,9 @@ static unsigned long shrink_page_list(struct list_head *page_list, */ nr_reclaimed += nr_pages; + mod_node_page_state(pgdat, NR_ISOLATED_ANON + + page_is_file_cache(page), + -nr_pages); /* * Is there need to periodically free_page_list? It would * appear not as the counts should be low @@ -1539,7 +1545,6 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone, ret = shrink_page_list(&clean_pages, zone->zone_pgdat, &sc, TTU_IGNORE_ACCESS, &dummy_stat, true); list_splice(&clean_pages, page_list); - mod_node_page_state(zone->zone_pgdat, NR_ISOLATED_FILE, -ret); return ret; } @@ -1615,6 +1620,9 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode) */ ClearPageLRU(page); ret = 0; + __mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + + page_is_file_cache(page), + hpage_nr_pages(page)); } return ret; @@ -1746,6 +1754,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan, total_scan, skipped, nr_taken, mode, lru); update_lru_sizes(lruvec, lru, nr_zone_taken); + return nr_taken; } @@ -1794,6 +1803,9 @@ int isolate_lru_page(struct page *page) ClearPageLRU(page); del_page_from_lru_list(page, lruvec, lru); ret = 0; + mod_node_page_state(pgdat, NR_ISOLATED_ANON + + page_is_file_cache(page), + hpage_nr_pages(page)); } spin_unlock_irq(&pgdat->lru_lock); } @@ -1885,6 +1897,9 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, update_lru_size(lruvec, lru, page_zonenum(page), nr_pages); list_move(&page->lru, &lruvec->lists[lru]); + __mod_node_page_state(pgdat, NR_ISOLATED_ANON + + page_is_file_cache(page), + -hpage_nr_pages(page)); if (put_page_testzero(page)) { __ClearPageLRU(page); __ClearPageActive(page); @@ -1962,7 +1977,6 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &page_list, &nr_scanned, sc, lru); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); reclaim_stat->recent_scanned[file] += nr_taken; item = current_is_kswapd() ? PGSCAN_KSWAPD : PGSCAN_DIRECT; @@ -1988,8 +2002,6 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, move_pages_to_lru(lruvec, &page_list); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&pgdat->lru_lock); mem_cgroup_uncharge_list(&page_list); @@ -2048,7 +2060,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, lru); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); reclaim_stat->recent_scanned[file] += nr_taken; __count_vm_events(PGREFILL, nr_scanned); @@ -2117,7 +2128,6 @@ static void shrink_active_list(unsigned long nr_to_scan, __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); spin_unlock_irq(&pgdat->lru_lock); mem_cgroup_uncharge_list(&l_active); From patchwork Mon Jun 3 05:36:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 10972329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2230C6C5 for ; Mon, 3 Jun 2019 05:37:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FA8D28650 for ; Mon, 3 Jun 2019 05:37:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 035AD288B6; Mon, 3 Jun 2019 05:37:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37BC128650 for ; Mon, 3 Jun 2019 05:37:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F20DB6B026E; Mon, 3 Jun 2019 01:37:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EF7976B026F; Mon, 3 Jun 2019 01:37:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE7D26B0270; Mon, 3 Jun 2019 01:37:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id A66BE6B026E for ; Mon, 3 Jun 2019 01:37:32 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id w31so1635861pgk.23 for ; Sun, 02 Jun 2019 22:37:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:sender:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=VaNSbzaDg92x8qknqbR2PMhHNKJQW/ZUsMPGiA5Uvg8=; b=WJ7IbSUkepf+5FvV0A1Ot0LjDHc8S/smOBxL6YHNpqT1jBXqDRwvoCB4rYrdqDlf/F LlCzbE6+MYycpVp/MuSPALiv5n4L16u51HXqD8HWzkgfZSK59PAfLmt5mzME54nlL7Ds 68xBoKZLfqwOvZTEQVIs5VWVgZNkuBQOWxU2w7/pa84tsPQv001/B/iAq5gYhFKXdFn/ y6QEER5W4u004XhvG93RC0cFLhjjBGGFqJj8Re9yNgGrbLtCXTr2kxJH9Me70NkxqXVX xMI2YJMf3B5v0u/DKP6YbQpZWK3QK1dt9EyI+58UzzL45z/nTQByKVJ+r3QyfrojaIXh ZrfQ== X-Gm-Message-State: APjAAAW0o/YRf1iWLxnNSFqnljBYQV7zXPWkxCtbrE/TJJ+kDNz+NmE3 i0/2cvr7YpIPkwhGJVKZFlnXM5rrF9WhRqEelv1Wke0oVe81Z6DjjO7z9ERPm7juYdCb9taTt7N Swe79bPKdDu+73D5e5PZs7De13aKecyRLKB0nBBNlqzvVwKeFA4RJU/MXNEf9A1w= X-Received: by 2002:a63:dc09:: with SMTP id s9mr26747212pgg.425.1559540252145; Sun, 02 Jun 2019 22:37:32 -0700 (PDT) X-Received: by 2002:a63:dc09:: with SMTP id s9mr26747168pgg.425.1559540251139; Sun, 02 Jun 2019 22:37:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559540251; cv=none; d=google.com; s=arc-20160816; b=pc9ocEIOQ9JInH0OY3Ob40u5jVwKQp8w25EITG37C52sQD4wBgK7ef2XifVb8JwS5t Tw0zNYeXnU6Quvnnr1UMeuQ88Hf3ai/lDoC4ZsFZWbeckc8pUM/M2vIO/k95q/jVewv5 Ghjhv2qqbuTscehC1/NvUHGkZ+IreKvbItKoGyPwViFF5iOllRjMzgRiWQuIOrQWrKQJ +/truzzMW1eiC3GDgJvNhns95QwSkcdtrbBEQwSHo0x+5tc/ZsRmYqvwoCGKIuTeLHaW bXf51hEoGaplAoY/F4FcrXpIQ8RLD/2gw5cBEf90fldsO5YSAaI5G6CTQUjQo/NqHw3e XGzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=VaNSbzaDg92x8qknqbR2PMhHNKJQW/ZUsMPGiA5Uvg8=; b=qBiCBJYh6Qc5Zj7Ve4plFx8YCAdxcWyJSEV1bJHdoH3w9FM8Rr44gxpjV2gqf7hkJt 444OrYAuSbVfcK/7qI+e15/WSom67PxvJKbgrr4Fd/fnbRTddoVpGMqvVB+2LUm5G0UC 3ARtSfg2ZrpELlvu0WmEL67ioPt2YTgmudhKoAzAZv8q6mkUptZ+oSHFpNlVmBguL6RO o+kDuEkxWFx5oQ2sf41dTaxR2zd2kxdntcrAN8EyRHwJ7Z5LuneKgrJy4I4NYjYcfb9l hFjcyJDyJ6a/brZ/Qxzm4cZv14wCebvknBhTMwRHelGdWBjN7R4gjF3WJSGxuQZDTaXU 5r8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=RxlmGxFF; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id e7sor13887514pgs.66.2019.06.02.22.37.31 for (Google Transport Security); Sun, 02 Jun 2019 22:37:31 -0700 (PDT) Received-SPF: pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=RxlmGxFF; spf=pass (google.com: domain of minchan.kim@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VaNSbzaDg92x8qknqbR2PMhHNKJQW/ZUsMPGiA5Uvg8=; b=RxlmGxFF55pcXEOln35FcFIZu+PdPTL0LdwKA1pVLJlIkeHoSJPdwUhAQL+utkqzHo NST5Y28/3uWp49yOuAXC69lRpT8U/7wJfNKRbDccCel9sowRaghbB4/2Av3V8730h7ng Pm7zotIIs85u4FjfVCRy6k0zMVUO3GiG/EaxlkYl04/CnV5DsfOQxXUGWZPoaLZmgujS IeLaCWl8gm/8jtyv/Ilg5Ild53oSpiiG7Sg7WMCn9W82n/lGrManYfGTxKUnQ6tUH+L8 LkiFT714c3XySvK+52Sg8p24JprTvTiwhcq+t8FCE5/58tl5yjlSvyuGm1GYvZ/rdRt0 rQxg== X-Google-Smtp-Source: APXvYqzm2ZazlKrN8EN4M9IRPtD+xIiqdbjl/PDwyA9mR2zzwo/ifQNNgsq6O4hvJBv6D42bhm4NXg== X-Received: by 2002:a63:b547:: with SMTP id u7mr26726840pgo.322.1559540250731; Sun, 02 Jun 2019 22:37:30 -0700 (PDT) Received: from bbox-2.seo.corp.google.com ([2401:fa00:d:0:98f1:8b3d:1f37:3e8]) by smtp.gmail.com with ESMTPSA id a18sm5986222pjq.0.2019.06.02.22.37.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 22:37:29 -0700 (PDT) From: Minchan Kim To: Andrew Morton Cc: linux-mm , LKML , linux-api@vger.kernel.org, Michal Hocko , Johannes Weiner , Tim Murray , Joel Fernandes , Suren Baghdasaryan , Daniel Colascione , Shakeel Butt , Sonny Rao , Brian Geffon , jannh@google.com, oleg@redhat.com, christian@brauner.io, oleksandr@redhat.com, hdanton@sina.com, Minchan Kim Subject: [PATCH v1 4/4] mm: introduce MADV_PAGEOUT Date: Mon, 3 Jun 2019 14:36:55 +0900 Message-Id: <20190603053655.127730-5-minchan@kernel.org> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog In-Reply-To: <20190603053655.127730-1-minchan@kernel.org> References: <20190603053655.127730-1-minchan@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a process expects no accesses to a certain memory range for a long time, it could hint kernel that the pages can be reclaimed instantly but data should be preserved for future use. This could reduce workingset eviction so it ends up increasing performance. This patch introduces the new MADV_PAGEOUT hint to madvise(2) syscall. MADV_PAGEOUT can be used by a process to mark a memory range as not expected to be used for a long time so that kernel reclaims *any LRU* pages instantly. The hint can help kernel in deciding which pages to evict proactively. All of error rule is same with MADV_DONTNEED. Note: This hint works with only private pages(IOW, page_mapcount(page) < 2) because shared page could have more chance to be accessed from other processes sharing the page so that it could cause major fault soon, which is inefficient. * RFC v2 * make reclaim_pages simple via factoring out isolate logic - hannes * RFCv1 * rename from MADV_COLD to MADV_PAGEOUT - hannes * bail out if process is being killed - Hillf * fix reclaim_pages bugs - Hillf Signed-off-by: Minchan Kim --- include/linux/swap.h | 1 + include/uapi/asm-generic/mman-common.h | 1 + mm/madvise.c | 126 +++++++++++++++++++++++++ mm/vmscan.c | 34 +++++++ 4 files changed, 162 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 0ce997edb8bb..063c0c1e112b 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -365,6 +365,7 @@ extern int vm_swappiness; extern int remove_mapping(struct address_space *mapping, struct page *page); extern unsigned long vm_total_pages; +extern unsigned long reclaim_pages(struct list_head *page_list); #ifdef CONFIG_NUMA extern int node_reclaim_mode; extern int sysctl_min_unmapped_ratio; diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h index 1190f4e7f7b9..92e347a89ddc 100644 --- a/include/uapi/asm-generic/mman-common.h +++ b/include/uapi/asm-generic/mman-common.h @@ -44,6 +44,7 @@ #define MADV_WILLNEED 3 /* will need these pages */ #define MADV_DONTNEED 4 /* don't need these pages */ #define MADV_COLD 5 /* deactivatie these pages */ +#define MADV_PAGEOUT 6 /* reclaim these pages */ /* common parameters: try to keep these consistent across architectures */ #define MADV_FREE 8 /* free pages only if memory pressure */ diff --git a/mm/madvise.c b/mm/madvise.c index ab158766858a..b010249cb8b6 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -41,6 +41,7 @@ static int madvise_need_mmap_write(int behavior) case MADV_WILLNEED: case MADV_DONTNEED: case MADV_COLD: + case MADV_PAGEOUT: case MADV_FREE: return 0; default: @@ -415,6 +416,128 @@ static long madvise_cold(struct vm_area_struct *vma, return 0; } +static int madvise_pageout_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + pte_t *orig_pte, *pte, ptent; + spinlock_t *ptl; + LIST_HEAD(page_list); + struct page *page; + int isolated = 0; + struct vm_area_struct *vma = walk->vma; + unsigned long next; + + if (fatal_signal_pending(current)) + return -EINTR; + + next = pmd_addr_end(addr, end); + if (pmd_trans_huge(*pmd)) { + ptl = pmd_trans_huge_lock(pmd, vma); + if (!ptl) + return 0; + + if (is_huge_zero_pmd(*pmd)) + goto huge_unlock; + + page = pmd_page(*pmd); + if (page_mapcount(page) > 1) + goto huge_unlock; + + if (next - addr != HPAGE_PMD_SIZE) { + int err; + + get_page(page); + spin_unlock(ptl); + lock_page(page); + err = split_huge_page(page); + unlock_page(page); + put_page(page); + if (!err) + goto regular_page; + return 0; + } + + if (isolate_lru_page(page)) + goto huge_unlock; + + list_add(&page->lru, &page_list); +huge_unlock: + spin_unlock(ptl); + reclaim_pages(&page_list); + return 0; + } + + if (pmd_trans_unstable(pmd)) + return 0; +regular_page: + orig_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + for (pte = orig_pte; addr < end; pte++, addr += PAGE_SIZE) { + ptent = *pte; + if (!pte_present(ptent)) + continue; + + page = vm_normal_page(vma, addr, ptent); + if (!page) + continue; + + if (page_mapcount(page) > 1) + continue; + + if (isolate_lru_page(page)) + continue; + + isolated++; + list_add(&page->lru, &page_list); + if (isolated >= SWAP_CLUSTER_MAX) { + pte_unmap_unlock(orig_pte, ptl); + reclaim_pages(&page_list); + isolated = 0; + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + orig_pte = pte; + } + } + + pte_unmap_unlock(orig_pte, ptl); + reclaim_pages(&page_list); + cond_resched(); + + return 0; +} + +static void madvise_pageout_page_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + struct mm_walk warm_walk = { + .pmd_entry = madvise_pageout_pte_range, + .mm = vma->vm_mm, + }; + + tlb_start_vma(tlb, vma); + walk_page_range(addr, end, &warm_walk); + tlb_end_vma(tlb, vma); +} + + +static long madvise_pageout(struct vm_area_struct *vma, + struct vm_area_struct **prev, + unsigned long start_addr, unsigned long end_addr) +{ + struct mm_struct *mm = vma->vm_mm; + struct mmu_gather tlb; + + *prev = vma; + if (!can_madv_lru_vma(vma)) + return -EINVAL; + + lru_add_drain(); + tlb_gather_mmu(&tlb, mm, start_addr, end_addr); + madvise_pageout_page_range(&tlb, vma, start_addr, end_addr); + tlb_finish_mmu(&tlb, start_addr, end_addr); + + return 0; +} + static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -805,6 +928,8 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, return madvise_willneed(vma, prev, start, end); case MADV_COLD: return madvise_cold(vma, prev, start, end); + case MADV_PAGEOUT: + return madvise_pageout(vma, prev, start, end); case MADV_FREE: case MADV_DONTNEED: return madvise_dontneed_free(vma, prev, start, end, behavior); @@ -827,6 +952,7 @@ madvise_behavior_valid(int behavior) case MADV_DONTNEED: case MADV_FREE: case MADV_COLD: + case MADV_PAGEOUT: #ifdef CONFIG_KSM case MADV_MERGEABLE: case MADV_UNMERGEABLE: diff --git a/mm/vmscan.c b/mm/vmscan.c index 56df55e8afcd..2c2cf442db58 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2136,6 +2136,40 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_deactivate, nr_rotated, sc->priority, file); } +unsigned long reclaim_pages(struct list_head *page_list) +{ + unsigned long nr_reclaimed = 0; + LIST_HEAD(node_page_list); + struct reclaim_stat dummy_stat; + struct scan_control sc = { + .gfp_mask = GFP_KERNEL, + .priority = DEF_PRIORITY, + .may_writepage = 1, + .may_unmap = 1, + .may_swap = 1, + }; + + while (!list_empty(page_list)) { + struct page *page; + + page = lru_to_page(page_list); + list_move(&page->lru, &node_page_list); + nr_reclaimed += shrink_page_list(&node_page_list, + page_pgdat(page), + &sc, TTU_IGNORE_ACCESS, + &dummy_stat, true); + if (!list_empty(&node_page_list)) { + struct page *page = lru_to_page(&node_page_list); + + list_del(&page->lru); + putback_lru_page(page); + + } + } + + return nr_reclaimed; +} + /* * The inactive anon list should be small enough that the VM never has * to do too much work.