From patchwork Tue Mar 7 05:20:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 13162797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17B5AC678DB for ; Tue, 7 Mar 2023 05:21:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B9BE6B0074; Tue, 7 Mar 2023 00:21:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96A2F280003; Tue, 7 Mar 2023 00:21:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 832C2280002; Tue, 7 Mar 2023 00:21:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7175F6B0074 for ; Tue, 7 Mar 2023 00:21:08 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EEF1F8072A for ; Tue, 7 Mar 2023 05:21:07 +0000 (UTC) X-FDA: 80540953374.17.49C615E Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf24.hostedemail.com (Postfix) with ESMTP id 04DE2180017 for ; Tue, 7 Mar 2023 05:21:05 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Q9sGR9Jp; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf24.hostedemail.com: domain of stevensd@chromium.org designates 209.85.210.174 as permitted sender) smtp.mailfrom=stevensd@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678166466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eFAeWPY2kcSCdT3O6CTUVlduCnpUJhZej+FbmoWYNkg=; b=jHR5Wyg2Q5e5AaNpPolOQL2qslwBXBFAst2WHvc+5+h689p7tVrUeRtqkW7uVkLWDksIBW EgluwPR/4YKCWh9dheGqO5U6iZBNYCprqoT6dh9GymgVUZDkhRUbGHGf+/4u5b8M9zbQk5 fF330iRnyLy4AK1bOremqg31cWqC0zc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Q9sGR9Jp; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf24.hostedemail.com: domain of stevensd@chromium.org designates 209.85.210.174 as permitted sender) smtp.mailfrom=stevensd@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678166466; a=rsa-sha256; cv=none; b=ZqCapadr/JfQB0RqZ1MX4aT3Nuq60jGztzrdyF4Cc44YWi6FmDJY6VLoMyTOA+qXquQhAF EVgrv+m5qeCavwK7YjWi27aPJGHapuwgQN/oGHrLNdP6v5cRvilaoMs5J7c6FpCIZyECWW lhBm4k7A8N3hZ7I2uKe2s1DTJjWkBLM= Received: by mail-pf1-f174.google.com with SMTP id a7so7294675pfx.10 for ; Mon, 06 Mar 2023 21:21:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678166464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eFAeWPY2kcSCdT3O6CTUVlduCnpUJhZej+FbmoWYNkg=; b=Q9sGR9JplEaeNqsSC0pRNCxNi/C7wmGhoCkA56L/loqSr4HfTSKbd24kgSyhErzSqH 0I6wjvacPrWo2mFsQrRLwwJXlg3UaeAhxOZ0HtAJPSslGOMFmzE06GPMU9FLNZYobvT/ 3o1S3aGwC8Ks5mnW1V4r0yFDwjDaYhzpIxfho= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678166464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eFAeWPY2kcSCdT3O6CTUVlduCnpUJhZej+FbmoWYNkg=; b=nQbYYeO2HKrbdPc5NuYBZK1UoAAzNIstYgz/qkRNjNUMjOxf6UaHrQWP+7elTBKK/q xMeWOq9rJ5w2tSVtOgMEl9a0PXM9/GLvcQ4kE4+nCvOchck0X+b85Iu/gBMK+WDcAcl4 Rrq/AvtMntOiRR5YB4mPKwqEYAH3e8O+cUfafSZaix5u43a38VB2/UCmtRlTHNJeT6au 34+zm4MCrPsB9XEA/9ga8qq6pHfcEPZMgtZgRzahxug5qsr5g92lY84Ay9ADvOZ6MY5Z lwZM7sFv94IW6TGYiKca0E2SvgVVQOaY0c0nZvrZaGJ3Vl/MROvrUbmR1c142fP7H0VO 7VHg== X-Gm-Message-State: AO0yUKUWxkGHQjyoEMq3zhFfCM7Q4EJ18EnVWtnOH32MDN1wsjrcOWR2 npUdOWjD3OMypHpYVHbm4zZElOMESUIvIQnLx5U= X-Google-Smtp-Source: AK7set9Qm/5FdMeOBnVeJtsCUuwHXzpeDkbq4jDrAQajKNW8Ze2GzSb1kc4cMCotGeX1k2yhybBoAw== X-Received: by 2002:a62:6542:0:b0:5a8:ad9d:83f with SMTP id z63-20020a626542000000b005a8ad9d083fmr10412674pfb.24.1678166464453; Mon, 06 Mar 2023 21:21:04 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1f73:9034:ce28:4421]) by smtp.gmail.com with UTF8SMTPSA id c26-20020aa78c1a000000b005a8f1187112sm7117378pfd.58.2023.03.06.21.21.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Mar 2023 21:21:04 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org, Andrew Morton Cc: Peter Xu , Matthew Wilcox , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , Jiaqi Yan , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH v5 3/3] mm/khugepaged: maintain page cache uptodate flag Date: Tue, 7 Mar 2023 14:20:36 +0900 Message-Id: <20230307052036.1520708-4-stevensd@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230307052036.1520708-1-stevensd@google.com> References: <20230307052036.1520708-1-stevensd@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 04DE2180017 X-Stat-Signature: m94acftmrf54k8e6rg977tyyqr3r7oye X-HE-Tag: 1678166465-309516 X-HE-Meta: U2FsdGVkX194Emk4CmFyzJI92JFzrMla4v8VLr9vtQjf2KQKIe27Oe17tzdi+rro7DVJ+ATgwL9lndi6HDZWet0SmMXHTs3fOB6A1TXXF7p2fw3TalfX1iToPE1eqPaRppzgJ+uUs/Qzb0h8PfttpGZzNSOtm9z6bnH5iEiZ2iUKgc8CEZtY1onsjrBYp6RYjs8moRKfc3/y6oPhXUhDFR4Ij3Q3bHHUQfSC8MvfyX7ETklrfYuvjqGCNSpCrRNzc08nn6Y62vKyMzffy/T1BOxUekeVDPy9WBx0aeZ7HX3stvnlk/2zWKTWSTWa5dfVFsMD6UkVlIXeLYmzgcu2PTcDaBlOUrVBCftzFU1tmL/xHdi/QrfBRlvBhp010uwjXaFlJImfZvo8JcCia8uhZWeW/Ed3ih2zY0u5dC0Ec/mWVJqOiEjkYC6czVF4N118h/CcOFlQNYqJnQxbmSVIPi93gCkIXa3iLpyxXpiGUSSs8ldlhPlSaldmlfaIkhAI+Pp+DHNGcZ4fsSobBQgWFiu3v63L197I7POXBN5qVYJ2lHTS6HbtyAf1YK6AmZ6gjDLK0aisa17IC198vacNHO3ELmZnqdeQa0bTYxjYQbfW1Kcgv5YQK832BFnXKoxwoQ/vQhNYoe86EgDphdsEGkEbHsGtJVdc0nZdNL0FAUGw8IBcYzP4lWjn1hnDA/kNgDUL+MNCbHY+2wERBXiWZUEFayX5dq/xse+MV2boS9T3xyUe562nc/9cXOvs4XDdf9gMeWyLxt3GgvKsuFUYGXyVba4IqqUTddNkyhO/qzOfAeP2p4k6GVJVDkcJaRkRAIByMTlTdI8YPgiwSB4sNsV0NSEJnnG7+K9vg+vqreEQSr9t1Fkkooib63KKtKHShcjeStDv5UI5GXwhe+tq3LYDmD6yxcVtiXCkjomQrOgCUt4qMocUtpvuKsrZSwSwqKm5mT8rIK7idhCx+dS Q+A4JtNT HHxOdkGe1GwXeY363NR8M++4eoWNOD4AD9Dilu7BPAxwxPmGbXrh7BN1oReF4w6b3sFVyk9qR41Pzmw3iAF0BXUtJsE5kDYPb1/EsA2XcjLRKlOuc57GHtiiE54emjj9+HXnRHN0JyxmY/BLF3MDEFRoyRmxFJGe91u722LXvC3uxLFd68bjtr+8Ao/1nwOQy6TjcOw7kLYE0FtR71PLS0r3bFcFTB8RLF3J0rMd57l1ghTxy1SO/H67cvv/fO1SyLEhDA49pHRWxm9Ui8r8myk/wj+NyyqSYD6bb0mmqyrEGR2HlsVuXLM3j6cHl+OD7GETX4tWo9d06h6ob9EpuBxD5DuQJJsrG0XOCbgXXd89NI3NckUuIbiGniw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: David Stevens Make sure that collapse_file doesn't interfere with checking the uptodate flag in the page cache by only inserting hpage into the page cache after it has been updated and marked uptodate. This is achieved by simply not replacing present pages with hpage when iterating over the target range. The present pages are already locked, so replacing the with the locked hpage before the collapse is finalized is unnecessary. This fixes a race where folio_seek_hole_data would mistake hpage for an fallocated but unwritten page. This race is visible to userspace via data temporarily disappearing from SEEK_DATA/SEEK_HOLE. Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Signed-off-by: David Stevens Acked-by: Peter Xu --- mm/khugepaged.c | 50 ++++++++++++------------------------------------- 1 file changed, 12 insertions(+), 38 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 51ae399f2035..bdde0a02811b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1930,12 +1930,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } } while (1); - /* - * At this point the hpage is locked and not up-to-date. - * It's safe to insert it into the page cache, because nobody would - * be able to map it or use it in another way until we unlock it. - */ - xas_set(&xas, start); for (index = start; index < end; index++) { page = xas_next(&xas); @@ -2104,13 +2098,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } /* - * Add the page to the list to be able to undo the collapse if - * something go wrong. + * Accumulate the pages that are being collapsed. */ list_add_tail(&page->lru, &pagelist); - - /* Finally, replace with the new page. */ - xas_store(&xas, hpage); continue; out_unlock: unlock_page(page); @@ -2149,8 +2139,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, goto rollback; /* - * Replacing old pages with new one has succeeded, now we - * attempt to copy the contents. + * The old pages are locked, so they won't change anymore. */ index = start; list_for_each_entry(page, &pagelist, lru) { @@ -2230,11 +2219,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, /* nr_none is always 0 for non-shmem. */ __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); } - /* Join all the small entries into a single multi-index entry. */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, hpage); - xas_unlock_irq(&xas); + /* + * Mark hpage as uptodate before inserting it into the page cache so + * that it isn't mistaken for an fallocated but unwritten page. + */ folio = page_folio(hpage); folio_mark_uptodate(folio); folio_ref_add(folio, HPAGE_PMD_NR - 1); @@ -2243,6 +2232,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, folio_mark_dirty(folio); folio_add_lru(folio); + /* Join all the small entries into a single multi-index entry. */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, hpage); + xas_unlock_irq(&xas); + /* * Remove pte page tables, so we can re-fault the page as huge. */ @@ -2267,36 +2261,18 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, rollback: /* Something went wrong: roll back page cache changes */ - xas_lock_irq(&xas); if (nr_none) { mapping->nrpages -= nr_none; shmem_uncharge(mapping->host, nr_none); } - xas_set(&xas, start); - end = index; - for (index = start; index < end; index++) { - xas_next(&xas); - page = list_first_entry_or_null(&pagelist, - struct page, lru); - if (!page || xas.xa_index < page->index) { - nr_none--; - continue; - } - - VM_BUG_ON_PAGE(page->index != xas.xa_index, page); - + list_for_each_entry_safe(page, tmp, &pagelist, lru) { /* Unfreeze the page. */ list_del(&page->lru); page_ref_unfreeze(page, 2); - xas_store(&xas, page); - xas_pause(&xas); - xas_unlock_irq(&xas); unlock_page(page); putback_lru_page(page); - xas_lock_irq(&xas); } - VM_BUG_ON(nr_none); /* * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. * This undo is not needed unless failure is due to SCAN_COPY_MC. @@ -2304,8 +2280,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, if (!is_shmem && result == SCAN_COPY_MC) filemap_nr_thps_dec(mapping); - xas_unlock_irq(&xas); - hpage->mapping = NULL; unlock_page(hpage);