From patchwork Fri Feb 17 08:54:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 13144439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A410CC636D4 for ; Fri, 17 Feb 2023 08:55:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 431536B007D; Fri, 17 Feb 2023 03:55:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 407CD6B007E; Fri, 17 Feb 2023 03:55:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CF876B0080; Fri, 17 Feb 2023 03:55:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1E3426B007D for ; Fri, 17 Feb 2023 03:55:19 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C15C114159E for ; Fri, 17 Feb 2023 08:55:18 +0000 (UTC) X-FDA: 80476174716.14.906C887 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf04.hostedemail.com (Postfix) with ESMTP id D01064000C for ; Fri, 17 Feb 2023 08:55:16 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GZkmRSsd; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf04.hostedemail.com: domain of stevensd@chromium.org designates 209.85.216.49 as permitted sender) smtp.mailfrom=stevensd@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676624116; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KjMpoZ8bcgvH2E9XkmftEhbBkctgLmEbKJ6cV9dN/tw=; b=fELT0NFGXi/bDZcaSYtV5tYNYdHE28T9YipHLHmQqhGxTUq4htcgd8NGIrN2BLGrPZd/V7 vkHcXfK2TmOixsT9lDedJ4zRK10WHT4JWjEnp5Ff35RWw+5n5hfCMVtTNbUumJXWm4wVuY r66aXQLqx+KrysybaK2rA3+0LgxzU68= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GZkmRSsd; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf04.hostedemail.com: domain of stevensd@chromium.org designates 209.85.216.49 as permitted sender) smtp.mailfrom=stevensd@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676624116; a=rsa-sha256; cv=none; b=EkjxgTohBoPUdnNVUHdRnusdEXdWDd6t3iLPJD8Mr3l3jnzHb0ZcQxgCZUhJFOG6nuD16t SiFRsXkN0p5lZQ8MGE0wiYGMAe6qeCVT9YIFXQ1rCOEbjoBElAxJSjtJtMrPKAWQBeOmsq zxywOJ112QMHmOt6w/TX9wFE/mw7QSs= Received: by mail-pj1-f49.google.com with SMTP id d15-20020a17090ae28f00b00229eec90a7fso513362pjz.0 for ; Fri, 17 Feb 2023 00:55:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KjMpoZ8bcgvH2E9XkmftEhbBkctgLmEbKJ6cV9dN/tw=; b=GZkmRSsdPBfla5qFntrp2Pw4Fv/pqdobPbGt2YMGNVPFzOczxMqoeQwL/ECUItTSwC Z2EdXnqyL4OaBFASsYyJs5KeDsJjdMoRycYji3+G03xuy31DgKDrPMgW5P/kcBQAMSNx pvFaDbymNIwo9wkkmZsc85H5601aY8DC4tZos= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KjMpoZ8bcgvH2E9XkmftEhbBkctgLmEbKJ6cV9dN/tw=; b=JpS4QOtB8rLXTydUZFAsgnR2g6uk135Magp2QTOVDZq/URXQW0bdppLgQKyyUtpgUq Nvp/f+LevkOmbCCfiSiofugH5E3S0Nk/Wjhsp5WxuTA3Vmb6ffVsZk9mBs0oRsb98OZr x4McWrBCnyr0slnFBADuncWWsMtsJ/QGKAiaP8kfjuL28KH4/03Czg7TLo7aVHWQr/9L kjlz0gdx2Rr81Jxj4tf5RfL9njTmUCNfIYZPNsGyI74JFp2KjuKtwdjpPO6q8krAq2aT 0iVut9zjN9XqsBWMs0SF/aqbjOIlHk5aQ+h/YoIGiI5uitgCuxeuv0bHMewOPMgrUjLZ VhWQ== X-Gm-Message-State: AO0yUKWJH2poKQrNVaQPIrsxU2MPVe9/t6jMaO2lQCFxpjGwaBV9BR+S MwG6ztdSXcC3domUI3WKKt5Q326hTtOoZkGP X-Google-Smtp-Source: AK7set+gdOoMg/f/eZ6vrpn2+McA1nHrAJV9kACCzgvxBVBKR4TgUlsSlLQeog5O33xrEkX2ULDZpg== X-Received: by 2002:a05:6a20:7b11:b0:c7:32b8:d6b9 with SMTP id s17-20020a056a207b1100b000c732b8d6b9mr3982808pzh.13.1676624115269; Fri, 17 Feb 2023 00:55:15 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b7bc:8cb9:1364:30fb]) by smtp.gmail.com with UTF8SMTPSA id j12-20020a62b60c000000b0059072daa002sm2550076pff.192.2023.02.17.00.55.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 17 Feb 2023 00:55:14 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org, Peter Xu , Matthew Wilcox Cc: Andrew Morton , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH v4 3/3] mm/khugepaged: maintain page cache uptodate flag Date: Fri, 17 Feb 2023 17:54:39 +0900 Message-Id: <20230217085439.2826375-4-stevensd@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230217085439.2826375-1-stevensd@google.com> References: <20230217085439.2826375-1-stevensd@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D01064000C X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: pgft3byr3enpoq7dumykmmunbnaeh8ky X-HE-Tag: 1676624116-342887 X-HE-Meta: U2FsdGVkX18XBY7txWY0IlNIDzCnwDRIm9pVfPyU3LbE+FnU/IT5NrzaxoTrqbup7RfpMPXsYM4TTs8gx/kcChJYF0TvAocKjw1CqNSJ/E9x4Hats0jbwBXHmQlvXDHCry4X1mcGXacknOvAWxGf3jwxlOQhPpy/k8DW7D6Ue6aHff9rPTqF0d83s+an//TMLrNiPu1sXaz6uOuZZy08ifSbm/FTJtp4fFey0mGPGsW1OhnKooQ12MZRlaC+Z6j+FUexpY49xCuRMVlfXY4CGO0UMVhLD+AFGScECt6J4Y8cP1eiZTtMCsKpCgujVNkau5Zux40B+jr507jYYqanBJ1GPMwJizyjz99bg6kk77PUIO6t/9uCqZU3rMQBGbYFYfI8hnhnLJ8p4EiDjctAgodw1bDi6lBzOszKvqtGJi25XlmRI+gbUvUZKWscBKF2MrlodJTLNdD/4w5ZG6gZ1H0nJ92XFA6SoCnEZJb8BmgdJMWXZwHkgfBAk35OFpBmfk0tXHW3W0JevwDCB+9VmkX3m+H17X2LSbJvAJz3cKJDo72ciNnFAEof2a15thcIBEPS0Q1DBl2trKtuzuz6/Y57mks/m9ToUpW8DuBcjZtc+Q5K2PnirYAv77onL6n+0H+dJhpPpEL9CmthjVe3MRZQhu9z3JbNo+fFYzbYwPL8GDGUZWfjeb0dR4NRIYA/aQrWAu6G0XDcUHpDdUM9PXNlR71B+aYkGJZxeCGidG0DyNnCZ4oLDxueZnlWX0qy8mRpShqrab/fAmnxag+Zvh97ojuW7IdFWsK/EolOh/STNJrZbNR+61pvaR3ELkNNbC7tW9aczwPlgcCvx3hDOV2Jl0LUYtby4xRoWcWMGE51JpzXoZdoi2la6uZOKjOugwWtCSK/TuVIwWRtVpau5RJtJRW6r2DMINeQlu+7qt94UYbFG0KrYgNhiEUtKOy/hybswlMZU6CbTB9iKPj 5ekd5J8Y NCN5vl6xyXC5cTrWql1vEyVDdMOmqll/qL9PUicDrKxXr6vMBukIDB8+xCu25I7Np6KqLJyl0hCUCUl8mWUygAP6uppAKWEuj2etmIBV/jspsPFlIUXgoHaaa90OwfRbeg0nWyOBUSjsD9HNtmAnJBd/llOLcLpKLxzW11hdSmJ+SEmflfkxT+6jKAgYA0SG285L/dJLrFgv2Beui1WpYG1zmIbsS/vrLFz+F9u3/o+qn0qD+W98IY692ZbLrkMVHKnfDuSPetTW/rlhrFhXerQu4QuNaAPxPr6WU5ODAWFVE2ivT+ytlr+dheuOYhIu6P4AM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: David Stevens Make sure that collapse_file doesn't interfere with checking the uptodate flag in the page cache by only inserting hpage into the page cache after it has been updated and marked uptodate. This is achieved by simply not replacing present pages with hpage when iterating over them target range. The present pages are already locked, so replacing the with the locked hpage before the collapse is finalized is unnecessary. This fixes a race where folio_seek_hole_data would mistake hpage for an fallocated but unwritten page. This race is visible to userspace via data temporarily disappearing from SEEK_DATA/SEEK_HOLE. Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Signed-off-by: David Stevens Acked-by: Peter Xu --- mm/khugepaged.c | 50 ++++++++++++------------------------------------- 1 file changed, 12 insertions(+), 38 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 1c37f9151345..e08cf7c5ebdf 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1908,12 +1908,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } } while (1); - /* - * At this point the hpage is locked and not up-to-date. - * It's safe to insert it into the page cache, because nobody would - * be able to map it or use it in another way until we unlock it. - */ - xas_set(&xas, start); for (index = start; index < end; index++) { page = xas_next(&xas); @@ -2082,13 +2076,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } /* - * Add the page to the list to be able to undo the collapse if - * something go wrong. + * Accumulate the pages that are being collapsed. */ list_add_tail(&page->lru, &pagelist); - - /* Finally, replace with the new page. */ - xas_store(&xas, hpage); continue; out_unlock: unlock_page(page); @@ -2127,8 +2117,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, goto rollback; /* - * Replacing old pages with new one has succeeded, now we - * attempt to copy the contents. + * The old pages are locked, so they won't change anymore. */ index = start; list_for_each_entry(page, &pagelist, lru) { @@ -2209,11 +2198,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, /* nr_none is always 0 for non-shmem. */ __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); } - /* Join all the small entries into a single multi-index entry. */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, hpage); - xas_unlock_irq(&xas); + /* + * Mark hpage as uptodate before inserting it into the page cache so + * that it isn't mistaken for an fallocated but unwritten page. + */ folio = page_folio(hpage); folio_mark_uptodate(folio); folio_ref_add(folio, HPAGE_PMD_NR - 1); @@ -2222,6 +2211,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, folio_mark_dirty(folio); folio_add_lru(folio); + /* Join all the small entries into a single multi-index entry. */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, hpage); + xas_unlock_irq(&xas); + /* * Remove pte page tables, so we can re-fault the page as huge. */ @@ -2246,36 +2240,18 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, rollback: /* Something went wrong: roll back page cache changes */ - xas_lock_irq(&xas); if (nr_none) { mapping->nrpages -= nr_none; shmem_uncharge(mapping->host, nr_none); } - xas_set(&xas, start); - end = index; - for (index = start; index < end; index++) { - xas_next(&xas); - page = list_first_entry_or_null(&pagelist, - struct page, lru); - if (!page || xas.xa_index < page->index) { - nr_none--; - continue; - } - - VM_BUG_ON_PAGE(page->index != xas.xa_index, page); - + list_for_each_entry_safe(page, tmp, &pagelist, lru) { /* Unfreeze the page. */ list_del(&page->lru); page_ref_unfreeze(page, 2); - xas_store(&xas, page); - xas_pause(&xas); - xas_unlock_irq(&xas); unlock_page(page); putback_lru_page(page); - xas_lock_irq(&xas); } - VM_BUG_ON(nr_none); /* * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. * This undo is not needed unless failure is due to SCAN_COPY_MC. @@ -2283,8 +2259,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, if (!is_shmem && result == SCAN_COPY_MC) filemap_nr_thps_dec(mapping); - xas_unlock_irq(&xas); - hpage->mapping = NULL; unlock_page(hpage);