[v5,1/2] mm,thp: recheck each page before collapsing file THP

Commit Message

Song Liu Nov. 6, 2019, 6:09 a.m. UTC

In collapse_file(), for !is_shmem case, current check cannot guarantee
the locked page is up-to-date.  Specifically, xas_unlock_irq() should
not be called before lock_page() and get_page(); and it is necessary to
recheck PageUptodate() after locking the page.

With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
may contain corrupted data.  This is because khugepaged mistakenly
collapses some not up-to-date sub pages into a huge page, and assumes
the huge page is up-to-date.  This will NOT corrupt data in the disk,
because the page is read-only and never written back.  Fix this by
properly checking PageUptodate() after locking the page.  This check
replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".

Also, move PageDirty() check after locking the page.  Current
khugepaged should not try to collapse dirty file THP, because it is
limited to read-only .text. The only case we hit a dirty page here is
when the page hasn't been written since write. Bail out and retry when
this happens.

syzbot reported bug on previous version of this patch.

Fixes: 99cb0dbd47a1 ("mm,thp: add read-only THP support for (non-shmem) FS")
Reported-by: [e-mail]
Cc: Johannes Weiner <[e-mail]>
Cc: Kirill A. Shutemov <[e-mail]>
Cc: Hugh Dickins <[e-mail]>
Cc: William Kucharski <[e-mail]>
Cc: Andrew Morton <[e-mail]>
Signed-off-by: Song Liu <[e-mail]>

---

[[e-mail]: v4]
 Link: http://lkml.kernel.org/r/[e-mail]
[[e-mail]: fix deadlock in collapse_file()]
 Link: http://lkml.kernel.org/r/[e-mail]
Link: http://lkml.kernel.org/r/[e-mail]
[[e-mail]: flush file for !is_shmem PageDirty() case in collapse_file()]
https://lkml.kernel.org/linux-mm/[e-mail]/

---
 mm/khugepaged.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)

--
2.17.1

Comments

Andrew Morton Nov. 8, 2019, 3:53 a.m. UTC | #1

On Tue, 5 Nov 2019 22:09:29 -0800 Song Liu <[e-mail]> wrote:

> In collapse_file(), for !is_shmem case, current check cannot guarantee
> the locked page is up-to-date.  Specifically, xas_unlock_irq() should
> not be called before lock_page() and get_page(); and it is necessary to
> recheck PageUptodate() after locking the page.
> 
> With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
> may contain corrupted data.  This is because khugepaged mistakenly
> collapses some not up-to-date sub pages into a huge page, and assumes
> the huge page is up-to-date.  This will NOT corrupt data in the disk,
> because the page is read-only and never written back.  Fix this by
> properly checking PageUptodate() after locking the page.  This check
> replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".
> 
> Also, move PageDirty() check after locking the page.  Current
> khugepaged should not try to collapse dirty file THP, because it is
> limited to read-only .text. The only case we hit a dirty page here is
> when the page hasn't been written since write. Bail out and retry when
> this happens.

Incorrect data is pretty serious.  Should we backport this into -stable
kernels?

(I suspect I already asked this in response to earier versions, sorry ;))

Song Liu Nov. 8, 2019, 3:58 a.m. UTC | #2

> On Nov 7, 2019, at 7:53 PM, Andrew Morton <[e-mail]> wrote:
> 
> On Tue, 5 Nov 2019 22:09:29 -0800 Song Liu <[e-mail]> wrote:
> 
>> In collapse_file(), for !is_shmem case, current check cannot guarantee
>> the locked page is up-to-date.  Specifically, xas_unlock_irq() should
>> not be called before lock_page() and get_page(); and it is necessary to
>> recheck PageUptodate() after locking the page.
>> 
>> With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
>> may contain corrupted data.  This is because khugepaged mistakenly
>> collapses some not up-to-date sub pages into a huge page, and assumes
>> the huge page is up-to-date.  This will NOT corrupt data in the disk,
>> because the page is read-only and never written back.  Fix this by
>> properly checking PageUptodate() after locking the page.  This check
>> replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".
>> 
>> Also, move PageDirty() check after locking the page.  Current
>> khugepaged should not try to collapse dirty file THP, because it is
>> limited to read-only .text. The only case we hit a dirty page here is
>> when the page hasn't been written since write. Bail out and retry when
>> this happens.
> 
> Incorrect data is pretty serious.  Should we backport this into -stable
> kernels?
> 
> (I suspect I already asked this in response to earier versions, sorry ;))

This is new feature (and new bug :( ) in 5.4. So no need to back port.

Thanks,
Song

Song Liu Nov. 13, 2019, 11:47 p.m. UTC | #3

> On Nov 5, 2019, at 10:09 PM, Song Liu <[e-mail]> wrote:
> 
> In collapse_file(), for !is_shmem case, current check cannot guarantee
> the locked page is up-to-date.  Specifically, xas_unlock_irq() should
> not be called before lock_page() and get_page(); and it is necessary to
> recheck PageUptodate() after locking the page.
> 
> With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
> may contain corrupted data.  This is because khugepaged mistakenly
> collapses some not up-to-date sub pages into a huge page, and assumes
> the huge page is up-to-date.  This will NOT corrupt data in the disk,
> because the page is read-only and never written back.  Fix this by
> properly checking PageUptodate() after locking the page.  This check
> replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".
> 
> Also, move PageDirty() check after locking the page.  Current
> khugepaged should not try to collapse dirty file THP, because it is
> limited to read-only .text. The only case we hit a dirty page here is
> when the page hasn't been written since write. Bail out and retry when
> this happens.
> 
> syzbot reported bug on previous version of this patch.
> 
> Fixes: 99cb0dbd47a1 ("mm,thp: add read-only THP support for (non-shmem) FS")
> Reported-by: [e-mail]
> Cc: Johannes Weiner <[e-mail]>
> Cc: Kirill A. Shutemov <[e-mail]>
> Cc: Hugh Dickins <[e-mail]>
> Cc: William Kucharski <[e-mail]>
> Cc: Andrew Morton <[e-mail]>
> Signed-off-by: Song Liu <[e-mail]>

I think we need this in 5.4 official, but I haven't seen it in Linus' 
master branch. 

Hi Andrew,

Could you please send patch/pull-request for it? 

Thanks,
Song

Andrew Morton Nov. 15, 2019, 12:10 a.m. UTC | #4

On Wed, 13 Nov 2019 23:47:06 +0000 Song Liu <[e-mail]> wrote:

> 
> > On Nov 5, 2019, at 10:09 PM, Song Liu <[e-mail]> wrote:
> > 
> > In collapse_file(), for !is_shmem case, current check cannot guarantee
> > the locked page is up-to-date.  Specifically, xas_unlock_irq() should
> > not be called before lock_page() and get_page(); and it is necessary to
> > recheck PageUptodate() after locking the page.
> > 
> > With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
> > may contain corrupted data.  This is because khugepaged mistakenly
> > collapses some not up-to-date sub pages into a huge page, and assumes
> > the huge page is up-to-date.  This will NOT corrupt data in the disk,
> > because the page is read-only and never written back.  Fix this by
> > properly checking PageUptodate() after locking the page.  This check
> > replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".
> > 
> > Also, move PageDirty() check after locking the page.  Current
> > khugepaged should not try to collapse dirty file THP, because it is
> > limited to read-only .text. The only case we hit a dirty page here is
> > when the page hasn't been written since write. Bail out and retry when
> > this happens.
> > 
> > syzbot reported bug on previous version of this patch.
> > 
> > Fixes: 99cb0dbd47a1 ("mm,thp: add read-only THP support for (non-shmem) FS")
> > Reported-by: [e-mail]
> > Cc: Johannes Weiner <[e-mail]>
> > Cc: Kirill A. Shutemov <[e-mail]>
> > Cc: Hugh Dickins <[e-mail]>
> > Cc: William Kucharski <[e-mail]>
> > Cc: Andrew Morton <[e-mail]>
> > Signed-off-by: Song Liu <[e-mail]>
> 
> I think we need this in 5.4 official, but I haven't seen it in Linus' 
> master branch. 
> 
> Hi Andrew,
> 
> Could you please send patch/pull-request for it? 

Yep, today or tomorrow...

Patch

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 0a1b4b484ac5..40215795d641 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1601,17 +1601,6 @@  static void collapse_file(struct mm_struct *mm,
 					result = SCAN_FAIL;
 					goto xa_unlocked;
 				}
-			} else if (!PageUptodate(page)) {
-				xas_unlock_irq(&xas);
-				wait_on_page_locked(page);
-				if (!trylock_page(page)) {
-					result = SCAN_PAGE_LOCK;
-					goto xa_unlocked;
-				}
-				get_page(page);
-			} else if (PageDirty(page)) {
-				result = SCAN_FAIL;
-				goto xa_locked;
 			} else if (trylock_page(page)) {
 				get_page(page);
 				xas_unlock_irq(&xas);
@@ -1626,7 +1615,12 @@  static void collapse_file(struct mm_struct *mm,
 		 * without racing with truncate.
 		 */
 		VM_BUG_ON_PAGE(!PageLocked(page), page);
-		VM_BUG_ON_PAGE(!PageUptodate(page), page);
+
+		/* make sure the page is up to date */
+		if (unlikely(!PageUptodate(page))) {
+			result = SCAN_FAIL;
+			goto out_unlock;
+		}

 		/*
 		 * If file was truncated then extended, or hole-punched, before
@@ -1642,6 +1636,16 @@  static void collapse_file(struct mm_struct *mm,
 			goto out_unlock;
 		}

+		if (!is_shmem && PageDirty(page)) {
+			/*
+			 * khugepaged only works on read-only fd, so this
+			 * page is dirty because it hasn't been flushed
+			 * since first write.
+			 */
+			result = SCAN_FAIL;
+			goto out_unlock;
+		}
+
 		if (isolate_lru_page(page)) {
 			result = SCAN_DEL_PAGE_LRU;
 			goto out_unlock;