From patchwork Thu Jun 13 08:38:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13696348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7D7FC27C4F for ; Thu, 13 Jun 2024 08:39:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 645366B00A3; Thu, 13 Jun 2024 04:39:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CD2D6B00A4; Thu, 13 Jun 2024 04:39:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 420AB6B00A5; Thu, 13 Jun 2024 04:39:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 246B26B00A3 for ; Thu, 13 Jun 2024 04:39:26 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F37FA3A17 for ; Thu, 13 Jun 2024 08:39:25 +0000 (UTC) X-FDA: 82225216290.12.E2D448D Received: from mail-oa1-f53.google.com (mail-oa1-f53.google.com [209.85.160.53]) by imf11.hostedemail.com (Postfix) with ESMTP id D7DEA4000F for ; Thu, 13 Jun 2024 08:39:23 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FCJMwcMs; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.160.53 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718267962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CewtBWl7EeBPsRvYcQKXFLRJSGy2eXvxF5yP4Eq9hNE=; b=SwMCDNz0n+IHz52ElavOXgywvAOgXxUsdQyeRKeChVHrThfjoM5E80wwtmCIZY8ao5bUBb xA2y2zfiyiHqBEJT4lByaQOOPE1ETLwHfR5iuziU4IfmE1Eaykons7y2dxI/qN4LxpCYNK lJZMTUeqCx4LEgAG7+TiuRJ4vaAodEs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FCJMwcMs; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.160.53 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718267962; a=rsa-sha256; cv=none; b=bkslcOom/pI8wtullotwpV77hV+z2jBgzktygOEaGJH5R02FfXdDbtrk7zY625fzw7OsWM tlrv08vCMfGUSj5eS8ZSj7B+rKlNQdXof5mP/SmMK1T9+YJG/3beXEh3ZMlV0AThV/TDp/ kxu04znM6Mc0Lv/F1QgCCkSmHd/Pl+U= Received: by mail-oa1-f53.google.com with SMTP id 586e51a60fabf-254f646b625so135019fac.0 for ; Thu, 13 Jun 2024 01:39:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1718267963; x=1718872763; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CewtBWl7EeBPsRvYcQKXFLRJSGy2eXvxF5yP4Eq9hNE=; b=FCJMwcMsryZpffR5w7tfiSSfuaivTGyJ7eLskZrOLdNRn90Ps17JQhDAyaNBPUmw6A Z5MD2D7V7iwxsNzB3vLofYNHfVhjUt0kv3oCKK49pyw+oBUqAh3YNGsl2m1kdYBerhLM P7uptEMWeQALeFzmCBzzMO19PSVDwB2U/3N2VBDdhtG8l+o1Q3dNmpdepfjLTE4NaJ48 AxSMO3tBpjeMhbPSzKe/784BSwtRRUB6viKQmOFFiv7UcypTAeh/LBtcSfNwOKCy0Ogl 9nAYu60KMWD84BrgTW2F7tIyyC0fm3CkpgYtlL7ns7raQd8fmyPAyNG48SU7pkK41XJJ 4w1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718267963; x=1718872763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CewtBWl7EeBPsRvYcQKXFLRJSGy2eXvxF5yP4Eq9hNE=; b=esZotgw0WCW3k6CO6xJNjpjiTeZUN4q5GJWfNbYaA31gMJxNWAEwMVLqHnR5z3TTAh s2G6XvgVprH+7U2CeXWpNWxYvJnvoxaSSi6K5oP3MlK5iA76DJJJE6KTLCjnb4zsBw5z S5DBO0zVSuA297nrZHC9is50LmglGJ+49LIEYwaSbTsWW5Md5FV3M1M3RMZ70JFPDZSw EMMFu8zNsMpKR0zPeJ6PScbrLoFyIEkY9ezNdE+qOCZaAQfPCYDjeGvX0gO8DFnloV3A Tuzq/iwRZE9OvO6sPPBwQhm0eKGZIh6kkmmzZ4g8Ga6yAApR4ukTl4IpwMKjGmQSnx9q IEYw== X-Gm-Message-State: AOJu0YyHHD1DMcf3bbZy6pKqms9RzFiU/wMt0BRhs353+cq/qzBTiQdJ 0XOOQ0a/XBnoEDEzGE+UnUA0ii5iZKhEC3ekkYKdwQa8ND14yArBGVSjaoZnyLo= X-Google-Smtp-Source: AGHT+IF3BvBRGUqMq+LK1a8EUEuvL+E9w9pX4Nvfr12+FIihulto0pHvra8MZUn64vN2LmeV0o+qLQ== X-Received: by 2002:a05:6870:95a8:b0:250:702f:8bab with SMTP id 586e51a60fabf-2551505f631mr4491220fac.3.1718267962761; Thu, 13 Jun 2024 01:39:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-705ccb980bdsm820856b3a.211.2024.06.13.01.39.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jun 2024 01:39:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [RFC PATCH 3/3] mm: free empty user PTE pages Date: Thu, 13 Jun 2024 16:38:10 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: D7DEA4000F X-Stat-Signature: s6145dhn6ad3phbtnyz9kow9f3uxczs6 X-Rspam-User: X-HE-Tag: 1718267963-481624 X-HE-Meta: U2FsdGVkX18baka1BjHwezOdR1pet4WUXBrOprclBvLoMPISMeCZAAu8OS5ewnunWbFM55UrwSyLKkOXNjvth4B43lh4Gt27TVCD0IV1/2LXTixhsFp/JhrpcSyOM03+rAn1UKrK1vn608R8egI2bmUK63wjws2kFo6OFO+ck90Hy9DOLVkAqikOslHj5RVzwtFxZC0g6BgSpB8PGwLZpL1cOLKp/kV6R/vDLx5nUJc9Yli9kAP/+/6PlP5m/QPMpp3zBscQKYKOV6kUYwtev+wFiToPqNdEip/JjOR6AoHx/hrz+t2e95UI84zgd4LM9/JUsTJ98nQOdU/8wcYDpcoGMg0I6gVLw+hu1N4ZQZEArgQWEZegINALLee7qhGhaTFuGSZyIY4QnX9k3ig/S1x+rzdxqBZOJw4JGPVAMQCL9hck8vTKyvMPqlC59GTV0DjwIbXUxZEiAd4V/YiFPe3SbniL8UMyQay9iCwLnfBy24mVk/Wy86oJC//GFohqBXwQQle5cePtkKy56Y4USmPHcYRmf0kKbGU0zaW8V3XXe0bIwmlk7aexWqyC2tRaYyjJNSs2oOAgZ2su3KQlybbnv/qKGHCy9Lj7lzg6LaUPXZ0NVNAR1F3wYBtfIcptBceU+JiibMatZLHrk1TV07sTgdsg2q+YQEppzsd52X+d2X3T4iVfKmGil/SiG064QPJ7tKGjqUzNROoz/cv5inkfBYJYux+ul/+vncO2V8+GJLi01B54Wtv5/7cb1bxBZcMYWIfbZtKYg/+bLdGhk4TdT+6Cm536e4Xl0FlTIWtv4Sn3k83WwtRpk24CSvPvV6FU9wkdXWajiUl3bPK0uyo89QMb03pWsvesL7EMKuD2RkjvZjf02Vprfp9Nw1mFBEWYUFR6gmLrOo7Vqu83RhH/WaA0jCcmQ5l5rawhVA/ojZ920GAC1nEVbbm43v+tj+p0tUvd7MjleSlr/AF Om1z7Jpa D6nRHgeo8sCxW/ue0iViIu6kLkJEBGYugxkdRIPJzvwwAmlXZEMg7BJkb9cwnsy8MLnEGkwWXInUek2F68N/03/GQywmNApHHy4D8b9c7/uZ/T7KWEIVv2g1duxob3PWeoD5H1oluHWiNKOHnG+qHku7zOJ/SJ4pqWEIICVUg8dcLqKj5Ew6YAbmZ80e4cd4Py2oG0G5WvLoRIzN8TkEFercNKFVi6ETkfasoayJ8F8D655fw5eytzxvGtka30Yz7ibZTjTax+1DP1J+rEsZn18KmAxKGcXWa6KeEPquet04a8gJIcNCDfhP5IH4FYKT5ou89 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now in order to pursue high performance, applications mostly use some high-performance user-mode memory allocators, such as jemalloc or tcmalloc. These memory allocators use madvise(MADV_DONTNEED or MADV_FREE) to release physical memory, but neither MADV_DONTNEED nor MADV_FREE will release page table memory, which may cause huge page table memory usage. The following are a memory usage snapshot of one process which actually happened on our server: VIRT: 55t RES: 590g VmPTE: 110g In this case, most of the page table entries are empty. For such a PTE page where all entries are empty, we can actually free it back to the system for others to use. Similar to numa_balancing, this commit adds a task_work to scan the address space of the user process when it returns to user space. If a suitable empty PTE page is found, it will be released. The following code snippet can show the effect of optimization: mmap 50G while (1) { for (; i < 1024 * 25; i++) { touch 2M memory madvise MADV_DONTNEED 2M } } As we can see, the memory usage of VmPTE is reduced: before after VIRT 50.0 GB 50.0 GB RES 3.1 MB 3.1 MB VmPTE 102640 kB 756 kB (Even less) Signed-off-by: Qi Zheng --- include/linux/mm_types.h | 4 + include/linux/pgtable.h | 14 +++ include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 2 + mm/Makefile | 2 +- mm/freept.c | 180 +++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 18 +++- 8 files changed, 220 insertions(+), 2 deletions(-) create mode 100644 mm/freept.c diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index ef09c4eef6d3..bbc697fa4a83 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -839,6 +839,10 @@ struct mm_struct { #endif #ifdef CONFIG_MMU atomic_long_t pgtables_bytes; /* size of all page tables */ + /* Next mm_pgtable scan (in jiffies) */ + unsigned long mm_pgtable_next_scan; + /* Restart point for scanning and freeing empty user PTE pages */ + unsigned long mm_pgtable_scan_offset; #endif int map_count; /* number of VMAs */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index fbff20070ca3..4d1cfaa92422 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1589,6 +1589,20 @@ static inline unsigned long my_zero_pfn(unsigned long addr) } #endif /* CONFIG_MMU */ +#ifdef CONFIG_MMU +#define MM_PGTABLE_SCAN_DELAY 100 /* 100ms */ +#define MM_PGTABLE_SCAN_SIZE 256 /* 256MB */ +void init_mm_pgtable_work(struct task_struct *p); +void task_tick_mm_pgtable(struct task_struct *curr); +#else +static inline void init_mm_pgtable_work(struct task_struct *p) +{ +} +static inline void task_tick_mm_pgtable(struct task_struct *curr) +{ +} +#endif + #ifdef CONFIG_MMU #ifndef CONFIG_TRANSPARENT_HUGEPAGE diff --git a/include/linux/sched.h b/include/linux/sched.h index 73c874e051f7..5c0f3d96d608 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1485,6 +1485,7 @@ struct task_struct { #ifdef CONFIG_MMU struct task_struct *oom_reaper_list; struct timer_list oom_reaper_timer; + struct callback_head pgtable_work; #endif #ifdef CONFIG_VMAP_STACK struct vm_struct *stack_vm_area; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index c663075c86fb..d5f6df6f5c32 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4359,6 +4359,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending = NULL; #endif init_sched_mm_cid(p); + init_mm_pgtable_work(p); } DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 41b58387023d..bbc7cbf22eaa 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12696,6 +12696,8 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued) if (static_branch_unlikely(&sched_numa_balancing)) task_tick_numa(rq, curr); + task_tick_mm_pgtable(curr); + update_misfit_status(curr, rq); check_update_overutilized_status(task_rq(curr)); diff --git a/mm/Makefile b/mm/Makefile index 8fb85acda1b1..af1a324aa65e 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -54,7 +54,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ mm_init.o percpu.o slab_common.o \ compaction.o show_mem.o shmem_quota.o\ interval_tree.o list_lru.o workingset.o \ - debug.o gup.o mmap_lock.o $(mmu-y) + debug.o gup.o mmap_lock.o freept.o $(mmu-y) # Give 'page_alloc' its own module-parameter namespace page-alloc-y := page_alloc.o diff --git a/mm/freept.c b/mm/freept.c new file mode 100644 index 000000000000..ed1ea5535e03 --- /dev/null +++ b/mm/freept.c @@ -0,0 +1,180 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include + +void task_tick_mm_pgtable(struct task_struct *curr) +{ + struct callback_head *work = &curr->pgtable_work; + unsigned long now = jiffies; + + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || + work->next != work) + return; + + if (time_before(now, READ_ONCE(curr->mm->mm_pgtable_next_scan))) + return; + + task_work_add(curr, work, TWA_RESUME); +} + +/* + * Locking: + * - already held the mmap read lock to traverse the vma tree and pgtable + * - use pmd lock for clearing pmd entry + * - use pte lock for checking empty PTE page, and release it after clearing + * pmd entry, then we can capture the changed pmd in pte_offset_map_lock() + * etc after holding this pte lock. Thanks to this, we don't need to hold the + * rmap-related locks. + * - users of pte_offset_map_lock() etc all expect the PTE page to be stable by + * using rcu lock, so use pte_free_defer() to free PTE pages. + */ +static int freept_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + struct mmu_notifier_range range; + struct mm_struct *mm = walk->mm; + pte_t *start_pte, *pte; + pmd_t pmdval; + spinlock_t *pml = NULL, *ptl; + unsigned long haddr = addr; + int i; + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, + haddr, haddr + PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + start_pte = pte_offset_map_nolock(mm, pmd, &pmdval, haddr, &ptl); + if (!start_pte) + goto out; + + pml = pmd_lock(mm, pmd); + if (ptl != pml) + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) + goto out_ptl; + + /* Check if it is empty PTE page */ + for (i = 0, addr = haddr, pte = start_pte; + i < PTRS_PER_PTE; i++, addr += PAGE_SIZE, pte++) { + if (!pte_none(ptep_get(pte))) + goto out_ptl; + } + pte_unmap(start_pte); + + pmd_clear(pmd); + flush_tlb_range(walk->vma, haddr, haddr + PMD_SIZE); + pmdp_get_lockless_sync(); + if (ptl != pml) + spin_unlock(ptl); + spin_unlock(pml); + + mmu_notifier_invalidate_range_end(&range); + + mm_dec_nr_ptes(mm); + pte_free_defer(mm, pmd_pgtable(pmdval)); + + return 0; + +out_ptl: + pte_unmap_unlock(start_pte, ptl); + if (pml != ptl) + spin_unlock(pml); +out: + mmu_notifier_invalidate_range_end(&range); + + return 0; +} + +static const struct mm_walk_ops mm_pgtable_walk_ops = { + .pmd_entry = freept_pmd_entry, + .walk_lock = PGWALK_RDLOCK, +}; + +static void task_mm_pgtable_work(struct callback_head *work) +{ + unsigned long now = jiffies, old_scan, next_scan; + struct task_struct *p = current; + struct mm_struct *mm = p->mm; + struct vm_area_struct *vma; + unsigned long start, end; + struct vma_iterator vmi; + + work->next = work; /* Prevent double-add */ + if (p->flags & PF_EXITING) + return; + + if (!mm->mm_pgtable_next_scan) { + mm->mm_pgtable_next_scan = now + msecs_to_jiffies(MM_PGTABLE_SCAN_DELAY); + return; + } + + old_scan = mm->mm_pgtable_next_scan; + if (time_before(now, old_scan)) + return; + + next_scan = now + msecs_to_jiffies(MM_PGTABLE_SCAN_DELAY); + if (!try_cmpxchg(&mm->mm_pgtable_next_scan, &old_scan, next_scan)) + return; + + if (!mmap_read_trylock(mm)) + return; + + start = mm->mm_pgtable_scan_offset; + vma_iter_init(&vmi, mm, start); + vma = vma_next(&vmi); + if (!vma) { + mm->mm_pgtable_scan_offset = 0; + start = 0; + vma_iter_set(&vmi, start); + vma = vma_next(&vmi); + } + + do { + /* Skip hugetlb case */ + if (is_vm_hugetlb_page(vma)) + continue; + + /* Leave this to the THP path to handle */ + if (vma->vm_flags & VM_HUGEPAGE) + continue; + + /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ + if (userfaultfd_wp(vma)) + continue; + + /* Only consider PTE pages that do not cross vmas */ + start = ALIGN(vma->vm_start, PMD_SIZE); + end = ALIGN_DOWN(vma->vm_end, PMD_SIZE); + if (end - start < PMD_SIZE) + continue; + + walk_page_range_vma(vma, start, end, &mm_pgtable_walk_ops, NULL); + + if (end - mm->mm_pgtable_scan_offset >= (MM_PGTABLE_SCAN_SIZE << 20)) + goto out; + + cond_resched(); + } for_each_vma(vmi, vma); + +out: + mm->mm_pgtable_scan_offset = vma ? end : 0; + mmap_read_unlock(mm); +} + +void init_mm_pgtable_work(struct task_struct *p) +{ + struct mm_struct *mm = p->mm; + int mm_users = 0; + + if (mm) { + mm_users = atomic_read(&mm->mm_users); + if (mm_users == 1) + mm->mm_pgtable_next_scan = jiffies + msecs_to_jiffies(MM_PGTABLE_SCAN_DELAY); + } + p->pgtable_work.next = &p->pgtable_work; /* Protect against double add */ + init_task_work(&p->pgtable_work, task_mm_pgtable_work); +} diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2a8703ee876c..a2b96f4ba737 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1581,7 +1581,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, NULL, haddr, &ptl); + start_pte = pte_offset_map_nolock(mm, pmd, &pgt_pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1589,6 +1589,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + /* pmd entry may be changed by others */ + if (unlikely(!pml && !pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1636,6 +1640,11 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pml = pmd_lock(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(ptl); + goto unlock; + } } pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); @@ -1663,6 +1672,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, } if (start_pte) pte_unmap_unlock(start_pte, ptl); +unlock: if (pml && pml != ptl) spin_unlock(pml); if (notified) @@ -1722,6 +1732,12 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) mmu_notifier_invalidate_range_start(&range); pml = pmd_lock(mm, pmd); + /* check if the pmd is still valid */ + if (check_pmd_still_valid(mm, addr, pmd) != SCAN_SUCCEED) { + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING);