[v6,08/11] mm/mremap: properly flush the TLB on mremap.

Commit Message

Aneesh Kumar K.V May 24, 2021, 9:01 a.m. UTC

As explained in
commit eb66ae030829 ("mremap: properly flush TLB before releasing the page")
mremap is special in that it doesn't take ownership of the page. The
optimized version for PUD/PMD aligned mremap also doesn't hold the ptl lock.
Hence flush the TLB before we update the new page table location. This ensures
the kernel invalidates the older translation cache before it can free the page via
the newly inserted translation.

Fixes: c49dd3401802 ("mm: speedup mremap on 1GB or larger regions")
Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
Link: https://lore.kernel.org/linux-mm/CAHk-=[e-mail]
Signed-off-by: Aneesh Kumar K.V <[e-mail]>
---
 mm/mremap.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

Patch

diff --git a/mm/mremap.c b/mm/mremap.c
index 000a71917557..8967a3707332 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -266,11 +266,13 @@  static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 	/* Clear the pmd */
 	pmd = *old_pmd;
 	pmd_clear(old_pmd);
-
+	/*
+	 * flush the TLB before we move the page table entries.
+	 */
+	flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PMD_SIZE);
 	VM_BUG_ON(!pmd_none(*new_pmd));
 	pmd_populate(mm, new_pmd, pmd_pgtable(pmd));
 
-	flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PMD_SIZE);
 	if (new_ptl != old_ptl)
 		spin_unlock(new_ptl);
 	spin_unlock(old_ptl);
@@ -313,11 +315,14 @@  static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr,
 	/* Clear the pud */
 	pud = *old_pud;
 	pud_clear(old_pud);
+	/*
+	 * flush the TLB before we move the page table entries.
+	 */
+	flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PUD_SIZE);
 
 	VM_BUG_ON(!pud_none(*new_pud));
 
 	pud_populate(mm, new_pud, (pmd_t *)pud_page_vaddr(pud));
-	flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PUD_SIZE);
 	if (new_ptl != old_ptl)
 		spin_unlock(new_ptl);
 	spin_unlock(old_ptl);