From patchwork Wed May 4 21:44:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12838670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55D74C433EF for ; Wed, 4 May 2022 21:45:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E524A6B0080; Wed, 4 May 2022 17:45:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB3E86B0081; Wed, 4 May 2022 17:45:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B90296B0082; Wed, 4 May 2022 17:45:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A9FEE6B0080 for ; Wed, 4 May 2022 17:45:19 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8C07A60A48 for ; Wed, 4 May 2022 21:45:19 +0000 (UTC) X-FDA: 79429391958.05.C7B755C Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf12.hostedemail.com (Postfix) with ESMTP id E2C2F4007C for ; Wed, 4 May 2022 21:45:02 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id b198-20020a6334cf000000b003ab23ccd0cbso1280006pga.14 for ; Wed, 04 May 2022 14:45:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2dkmQNcQmOGlATP5teArTW29S0qwEJDHirTfhWHFJLU=; b=VmZsQDvLrCENyv2YV2Cs6Jdzk6NHXszxT5Vv/27CiCiCNa3wTX3aptBnxuXgPhRyfl X55Cjx4b0fkpP4o+Um1erHCgwILK9tRwDF9ez7XzHshig8LdKC8WktISxL7r8XgT5EfV BsXuG/C0rH+RHptdLkuSG0XFgSrVE1ScIX6Z2Kf2ldw3kASG+mlASACVjvuUheg/hErc mXCJq3MIt4C1LjGBFD3y11VRpc22l/DpGXQvOgsPBv5xvs7rmteF9zceym05LB9Md4JD b0wwTpUmVfxTBUG9aoNWUnTGxmQ6x4UZSE4aBG1atSk7albm0eC9a9oKAZy/jUCXEBX3 Tflw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2dkmQNcQmOGlATP5teArTW29S0qwEJDHirTfhWHFJLU=; b=SC2iRJ/h6XtJMn6VmSkAxVV36QIlD/S0DmBnVDSAbgCObQJjtDPlmNcHu3TuWvlkqj ZpaasUz6MP7PjFcGvisKgjvtY4boMitiBixWuY791trd3gX95lyBFV5fwaJmaou2AmWr 0wDs8K+C6rRybzdzy6xZ3D/DaCX08MgzH6gKPfl1DNG1018Pu+Vl1gJo5X6U11tPftTD JOeyzVdFAV76HMq+EZZW0zWYKP1EYKUL+2/GQ9FyXOVPRIV0YUb1oYNf2W/qXujP1FCr OrX8/gdaZG1whXKz8nJiCS5Pp3sPU6EDZPYWfmTmHtY9xeDTSH19sfe7M2wFIuEK/mfc Q8kA== X-Gm-Message-State: AOAM5314T20skrXe6errm3prAXb7g4G8E1cniQPQ2ZwFGju/LyORDVQd 7X1YJob/eUFq920Hr8GqunOvEciEmfw7 X-Google-Smtp-Source: ABdhPJwlweA3CuSRC17P1DV0jq9EFuIYuZiVJB24sel6oKq67LS7H0mcAvgIDlwt4a/uoTSS6hwRbuvRGMW1 X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a17:90a:528f:b0:1dc:9a7c:4a3 with SMTP id w15-20020a17090a528f00b001dc9a7c04a3mr1881607pjh.112.1651700718156; Wed, 04 May 2022 14:45:18 -0700 (PDT) Date: Wed, 4 May 2022 14:44:33 -0700 In-Reply-To: <20220504214437.2850685-1-zokeefe@google.com> Message-Id: <20220504214437.2850685-10-zokeefe@google.com> Mime-Version: 1.0 References: <20220504214437.2850685-1-zokeefe@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v5 09/13] mm/khugepaged: rename prefix of shared collapse functions From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , Peter Xu , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Thomas Bogendoerfer , "Zach O'Keefe" X-Rspamd-Queue-Id: E2C2F4007C X-Stat-Signature: h9ij34usk7dqwxx81fokhmhxpx5qob7r Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VmZsQDvL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 37vNyYgcKCLs0plffgfhpphmf.dpnmjovy-nnlwbdl.psh@flex--zokeefe.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=37vNyYgcKCLs0plffgfhpphmf.dpnmjovy-nnlwbdl.psh@flex--zokeefe.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1651700702-710407 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following functions/tracepoints are shared between khugepaged and madvise collapse contexts. Replace the "khugepaged_" prefixe with generic "hpage_collapse_" prefix in such cases: huge_memory:mm_khugepaged_scan_pmd -> huge_memory:mm_hpage_collapse_scan_pmd khugepaged_test_exit() -> hpage_collapse_test_exit() khugepaged_scan_abort() -> hpage_collapse_scan_abort() khugepaged_scan_pmd() -> hpage_collapse_scan_pmd() khugepaged_find_target_node() -> hpage_collapse_find_target_node() Signed-off-by: Zach O'Keefe Acked-by: David Rientjes --- include/trace/events/huge_memory.h | 2 +- mm/khugepaged.c | 72 ++++++++++++++++-------------- 2 files changed, 39 insertions(+), 35 deletions(-) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h index 55392bf30a03..fb6c73632ff3 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -48,7 +48,7 @@ SCAN_STATUS #define EM(a, b) {a, b}, #define EMe(a, b) {a, b} -TRACE_EVENT(mm_khugepaged_scan_pmd, +TRACE_EVENT(mm_hpage_collapse_scan_pmd, TP_PROTO(struct mm_struct *mm, struct page *page, bool writable, int referenced, int none_or_zero, int status, int unmapped), diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 165c646ddb8f..44e31e072124 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -96,7 +96,7 @@ struct collapse_control { /* Num pages scanned per node */ int node_load[MAX_NUMNODES]; - /* Last target selected in khugepaged_find_target_node() */ + /* Last target selected in hpage_collapse_find_target_node() */ int last_target_node; struct page *hpage; @@ -453,7 +453,7 @@ static void insert_to_mm_slots_hash(struct mm_struct *mm, hash_add(mm_slots_hash, &mm_slot->hash, (long)mm); } -static inline int khugepaged_test_exit(struct mm_struct *mm) +static inline int hpage_collapse_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) == 0; } @@ -502,7 +502,7 @@ int __khugepaged_enter(struct mm_struct *mm) return -ENOMEM; /* __khugepaged_exit() must not run from under us */ - VM_BUG_ON_MM(khugepaged_test_exit(mm), mm); + VM_BUG_ON_MM(hpage_collapse_test_exit(mm), mm); if (unlikely(test_and_set_bit(MMF_VM_HUGEPAGE, &mm->flags))) { free_mm_slot(mm_slot); return 0; @@ -566,11 +566,10 @@ void __khugepaged_exit(struct mm_struct *mm) } else if (mm_slot) { /* * This is required to serialize against - * khugepaged_test_exit() (which is guaranteed to run - * under mmap sem read mode). Stop here (after we - * return all pagetables will be destroyed) until - * khugepaged has finished working on the pagetables - * under the mmap_lock. + * hpage_collapse_test_exit() (which is guaranteed to run + * under mmap sem read mode). Stop here (after we return all + * pagetables will be destroyed) until khugepaged has finished + * working on the pagetables under the mmap_lock. */ mmap_write_lock(mm); mmap_write_unlock(mm); @@ -807,7 +806,7 @@ static void khugepaged_alloc_sleep(void) remove_wait_queue(&khugepaged_wait, &wait); } -static bool khugepaged_scan_abort(int nid, struct collapse_control *cc) +static bool hpage_collapse_scan_abort(int nid, struct collapse_control *cc) { int i; @@ -854,7 +853,7 @@ static bool alloc_hpage(gfp_t gfp, int node, struct collapse_control *cc) } #ifdef CONFIG_NUMA -static int khugepaged_find_target_node(struct collapse_control *cc) +static int hpage_collapse_find_target_node(struct collapse_control *cc) { int nid, target_node = 0, max_value = 0; @@ -901,7 +900,7 @@ static bool khugepaged_alloc_page(gfp_t gfp, int node, return alloc_hpage(gfp, node, cc); } #else -static int khugepaged_find_target_node(struct collapse_control *cc) +static int hpage_collapse_find_target_node(struct collapse_control *cc) { return 0; } @@ -982,7 +981,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, struct vm_area_struct *vma; unsigned long hstart, hend; - if (unlikely(khugepaged_test_exit(mm))) + if (unlikely(hpage_collapse_test_exit(mm))) return SCAN_ANY_PROCESS; *vmap = vma = find_vma(mm, address); @@ -1026,7 +1025,7 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, /* * Bring missing pages in from swap, to complete THP collapse. - * Only done if khugepaged_scan_pmd believes it is worthwhile. + * Only done if hpage_collapse_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held, * but with mmap_lock held to protect against vma changes. @@ -1093,7 +1092,7 @@ static int alloc_charge_hpage(struct mm_struct *mm, struct collapse_control *cc) const struct cpumask *cpumask; #endif gfp_t gfp = alloc_hugepage_khugepaged_gfpmask() | __GFP_THISNODE; - int node = khugepaged_find_target_node(cc); + int node = hpage_collapse_find_target_node(cc); #ifdef CONFIG_NUMA /* sched to specified node before huge page memory copy */ @@ -1263,9 +1262,10 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, return result; } -static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, bool *mmap_locked, - struct collapse_control *cc) +static int hpage_collapse_scan_pmd(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long address, bool *mmap_locked, + struct collapse_control *cc) { pmd_t *pmd; pte_t *pte, *_pte; @@ -1357,7 +1357,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, * hit record. */ node = page_to_nid(page); - if (khugepaged_scan_abort(node, cc)) { + if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; goto out_unmap; } @@ -1419,8 +1419,8 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unmapped, cc); } out: - trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced, - none_or_zero, result, unmapped); + trace_mm_hpage_collapse_scan_pmd(mm, page, writable, referenced, + none_or_zero, result, unmapped); return result; } @@ -1430,7 +1430,7 @@ static void collect_mm_slot(struct mm_slot *mm_slot) lockdep_assert_held(&khugepaged_mm_lock); - if (khugepaged_test_exit(mm)) { + if (hpage_collapse_test_exit(mm)) { /* free mm_slot */ hash_del(&mm_slot->hash); list_del(&mm_slot->mm_node); @@ -1601,7 +1601,7 @@ static void khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot) if (!mmap_write_trylock(mm)) return; - if (unlikely(khugepaged_test_exit(mm))) + if (unlikely(hpage_collapse_test_exit(mm))) goto out; for (i = 0; i < mm_slot->nr_pte_mapped_thp; i++) @@ -1664,7 +1664,8 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) * it'll always mapped in small page size for uffd-wp * registered ranges. */ - if (!khugepaged_test_exit(mm) && !userfaultfd_wp(vma)) + if (!hpage_collapse_test_exit(mm) && + !userfaultfd_wp(vma)) collapse_and_free_pmd(mm, vma, addr, pmd); mmap_write_unlock(mm); } else { @@ -2089,7 +2090,7 @@ static int khugepaged_scan_file(struct mm_struct *mm, struct file *file, } node = page_to_nid(page); - if (khugepaged_scan_abort(node, cc)) { + if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; break; } @@ -2178,7 +2179,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, vma = NULL; if (unlikely(!mmap_read_trylock(mm))) goto breakouterloop_mmap_lock; - if (likely(!khugepaged_test_exit(mm))) + if (likely(!hpage_collapse_test_exit(mm))) vma = find_vma(mm, khugepaged_scan.address); progress++; @@ -2186,7 +2187,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, unsigned long hstart, hend; cond_resched(); - if (unlikely(khugepaged_test_exit(mm))) { + if (unlikely(hpage_collapse_test_exit(mm))) { progress++; break; } @@ -2212,7 +2213,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, bool mmap_locked = true; cond_resched(); - if (unlikely(khugepaged_test_exit(mm))) + if (unlikely(hpage_collapse_test_exit(mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart || @@ -2229,9 +2230,10 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, cc); fput(file); } else { - result = khugepaged_scan_pmd(mm, vma, - khugepaged_scan.address, - &mmap_locked, cc); + result = hpage_collapse_scan_pmd(mm, vma, + khugepaged_scan.address, + &mmap_locked, + cc); } if (result == SCAN_SUCCEED) ++khugepaged_pages_collapsed; @@ -2255,7 +2257,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, * Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ - if (khugepaged_test_exit(mm) || !vma) { + if (hpage_collapse_test_exit(mm) || !vma) { /* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find @@ -2495,7 +2497,8 @@ static int madvise_collapse_errno(enum scan_result r) static int madvise_alloc_charge_hpage(struct mm_struct *mm, struct collapse_control *cc) { - if (!alloc_hpage(GFP_TRANSHUGE, khugepaged_find_target_node(cc), cc)) + if (!alloc_hpage(GFP_TRANSHUGE, hpage_collapse_find_target_node(cc), + cc)) return SCAN_ALLOC_HUGE_PAGE_FAIL; if (unlikely(mem_cgroup_charge(page_folio(cc->hpage), mm, GFP_TRANSHUGE))) @@ -2542,13 +2545,14 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, cond_resched(); result = SCAN_FAIL; - if (unlikely(khugepaged_test_exit(mm))) { + if (unlikely(hpage_collapse_test_exit(mm))) { result = SCAN_ANY_PROCESS; break; } memset(cc.node_load, 0, sizeof(cc.node_load)); - result = khugepaged_scan_pmd(mm, vma, addr, &mmap_locked, &cc); + result = hpage_collapse_scan_pmd(mm, vma, addr, &mmap_locked, + &cc); if (!mmap_locked) *prev = NULL; /* tell madvise we dropped mmap_lock */