[v3,02/16] mm/hugetlb: take src_mm->write_protect_seq in copy_hugetlb_page_range()

Commit Message

David Hildenbrand March 29, 2022, 4:04 p.m. UTC

Let's do it just like copy_page_range(), taking the seqlock and making
sure the mmap_lock is held in write mode.

This allows for add a VM_BUG_ON to page_needs_cow_for_dma() and
properly synchronizes cocnurrent fork() with GUP-fast of hugetlb pages,
which will be relevant for further changes.

Signed-off-by: David Hildenbrand <[e-mail]>
---
 include/linux/mm.h | 4 ++++
 mm/hugetlb.c       | 8 ++++++--
 2 files changed, 10 insertions(+), 2 deletions(-)

Comments

Vlastimil Babka April 11, 2022, 4:15 p.m. UTC | #1

On 3/29/22 18:04, David Hildenbrand wrote:
> Let's do it just like copy_page_range(), taking the seqlock and making
> sure the mmap_lock is held in write mode.
> 
> This allows for add a VM_BUG_ON to page_needs_cow_for_dma() and
> properly synchronizes cocnurrent fork() with GUP-fast of hugetlb pages,

			concurrent

> which will be relevant for further changes.
> 
> Signed-off-by: David Hildenbrand <[e-mail]>

Acked-by: Vlastimil Babka <[e-mail]>

Patch

diff --git a/include/linux/mm.h b/include/linux/mm.h
index e34edb775334..9e53458a5ed3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1576,6 +1576,8 @@  static inline bool page_maybe_dma_pinned(struct page *page)
 /*
  * This should most likely only be called during fork() to see whether we
  * should break the cow immediately for a page on the src mm.
+ *
+ * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq.
  */
 static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma,
 					  struct page *page)
@@ -1583,6 +1585,8 @@  static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma,
 	if (!is_cow_mapping(vma->vm_flags))
 		return false;
 
+	VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1));
+
 	if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags))
 		return false;
 
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index b34f50156f7e..ad5ffdc615c6 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4714,6 +4714,8 @@  int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 					vma->vm_start,
 					vma->vm_end);
 		mmu_notifier_invalidate_range_start(&range);
+		mmap_assert_write_locked(src);
+		raw_write_seqcount_begin(&src->write_protect_seq);
 	} else {
 		/*
 		 * For shared mappings i_mmap_rwsem must be held to call
@@ -4846,10 +4848,12 @@  int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 		spin_unlock(dst_ptl);
 	}
 
-	if (cow)
+	if (cow) {
+		raw_write_seqcount_end(&src->write_protect_seq);
 		mmu_notifier_invalidate_range_end(&range);
-	else
+	} else {
 		i_mmap_unlock_read(mapping);
+	}
 
 	return ret;
 }