From patchwork Mon Oct 10 16:01:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiaqi Yan X-Patchwork-Id: 13002708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6571C433F5 for ; Mon, 10 Oct 2022 16:02:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A0646B0074; Mon, 10 Oct 2022 12:02:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FE6C8E0001; Mon, 10 Oct 2022 12:02:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 250DE6B0078; Mon, 10 Oct 2022 12:02:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F18516B0074 for ; Mon, 10 Oct 2022 12:02:58 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AE3BA80E58 for ; Mon, 10 Oct 2022 16:02:58 +0000 (UTC) X-FDA: 80005508436.15.BCB556F Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf16.hostedemail.com (Postfix) with ESMTP id DB19718001E for ; Mon, 10 Oct 2022 16:02:57 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id q62-20020a17090a17c400b0020d334fbe19so966697pja.0 for ; Mon, 10 Oct 2022 09:02:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Q8QsjaQUWLMJIjfLpthUsdOczcqbIgg7nHMp+C9TNcY=; b=fC4EXTklc4oThw6kdhAWM5o/h+qyRo2j6zKTNWjT2QM82qLkY4A+ZXtAyBqGtnOJgQ rpg+GLmox5ojMq79bT4CSSShoCbI+pqhCmARahmuDsJ6jkENszBYsjrCYiWiJGyPB3Xl +fJuM5UOnRj8BEBTsFxlyq4sbKqWf9QFnsTocyJeYUhyYJ/0D4FRcZRc97LjRPtIYQ/z umTwiFNyOJiV9S7V253arNzryLvA2fMZJKVp/bThh4o4MU7aknW5PWVgCvb/oub+x2Rc 7eypADzPjHiLnxhqCpiX+qc+YMayl6jMzL4LBLnp1RH3Vr3u3MjzQ3yuhlAF7Psc0+PQ 6xOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q8QsjaQUWLMJIjfLpthUsdOczcqbIgg7nHMp+C9TNcY=; b=qV4WmFGuYNArL3L9MG3jEYQXXuycP3RBaaHOA1wAvhLPbnM9x0r2tvGM4TYLQiYzoM AgOxIfBAYIY17H9UY9ci50Qnsa/nCBUGe92vHxTVBq0NrmOmBk2dzT4AKNKMiECmpDgR iILWETJSXyc/OmOdIyXIyxfIkuCP53KaSuxf4kGG8takCq7cubL59CfWtwQ7lKkZXVvU 2EszX9h7TEqfViN5oJZBDo+QgETazHfI/ZkUqCNhym4UU4oI1Z/+EgE3leUJeXni6ur9 XAODgA36d568vUb+8yrSTYdIBRJkLcJ4HJiITWaBkROgxwUZnXusLrVVzSbenMY2o8Mx qoUQ== X-Gm-Message-State: ACrzQf3Nvm/T7FU4H/Z9bacyClNiA5DGy2T3Hm0lygpmrvuZaGiroSUl f8nvG/K4HyG/zqgzSKOVzTLlPkL7hDO/Bw== X-Google-Smtp-Source: AMsMyM5joyRPSg8OaWSVt9sz5vwYxYAFySMZi8rIf/5AgcfylG+5nYViYuWhZTsGCC+lEbppOh6Drsa2CgR6yQ== X-Received: from yjqkernel.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1837]) (user=jiaqiyan job=sendgmr) by 2002:aa7:8896:0:b0:556:cc53:3f43 with SMTP id z22-20020aa78896000000b00556cc533f43mr20506566pfe.19.1665417776552; Mon, 10 Oct 2022 09:02:56 -0700 (PDT) Date: Mon, 10 Oct 2022 09:01:42 -0700 In-Reply-To: <20221010160142.1087120-1-jiaqiyan@google.com> Mime-Version: 1.0 References: <20221010160142.1087120-1-jiaqiyan@google.com> X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221010160142.1087120-3-jiaqiyan@google.com> Subject: [PATCH v5 2/2] mm/khugepaged: recover from poisoned file-backed memory From: Jiaqi Yan To: shy828301@gmail.com, tongtiangen@huawei.com Cc: tony.luck@intel.com, naoya.horiguchi@nec.com, kirill.shutemov@linux.intel.com, linmiaohe@huawei.com, linux-mm@kvack.org, akpm@linux-foundation.org, jiaqiyan@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665417777; a=rsa-sha256; cv=none; b=rDkB/9T4nT0ZhiwIhD8Trouh26yZo2xJnO4acCgEGUxpbMSRiuQqEn63WdkQye3zuX+6QZ XUR8SEtbw5bQVVvGO78cJsmdqDU4SQnWgZ582xnvdaLK40gu+Bvq0mQ+e3OBwjBbWIo+7p ligOy7tcITYH+5C2Axyh2FJKyL1yLMA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fC4EXTkl; spf=pass (imf16.hostedemail.com: domain of 3MEJEYwgKCOUQPHXPfHUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jiaqiyan.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3MEJEYwgKCOUQPHXPfHUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665417777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q8QsjaQUWLMJIjfLpthUsdOczcqbIgg7nHMp+C9TNcY=; b=poQZMbZPmcp3o3lkiDGWRomTQ41WAnhC3JWCxaJ+UbbM97VEmhcymVeTOlOW4sAjRLrVfc 7ml1El0kNZfsH2uMSY54HWOr4x1+Dxhjs3AKm5Yc7UiJgiAj8OcMkh8MtYEn/EIQk0n2ZX K2elIEqieZkFBP9aHTTV9yypcjwqZ7Y= Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fC4EXTkl; spf=pass (imf16.hostedemail.com: domain of 3MEJEYwgKCOUQPHXPfHUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jiaqiyan.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3MEJEYwgKCOUQPHXPfHUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: o48mo6u36itsyek7xhgdscah5nkt8gdf X-Rspamd-Queue-Id: DB19718001E X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1665417777-374092 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make collapse_file roll back when copying pages failed. More concretely: * extract copying operations into a separate loop * postpone the updates for nr_none until both scanning and copying succeeded * postpone joining small xarray entries until both scanning and copying succeeded * postpone the update operations to NR_XXX_THPS * for non-SHMEM file, roll back filemap_nr_thps_inc if scan succeeded but copying failed Signed-off-by: Jiaqi Yan --- mm/khugepaged.c | 74 ++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 29 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 552e7cb4c8b42..ccc96ef8e46cb 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1664,7 +1664,7 @@ static void collapse_file(struct mm_struct *mm, { struct address_space *mapping = file->f_mapping; gfp_t gfp; - struct page *new_page; + struct page *new_page, *page, *tmp; pgoff_t index, end = start + HPAGE_PMD_NR; LIST_HEAD(pagelist); XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); @@ -1720,7 +1720,7 @@ static void collapse_file(struct mm_struct *mm, xas_set(&xas, start); for (index = start; index < end; index++) { - struct page *page = xas_next(&xas); + page = xas_next(&xas); VM_BUG_ON(index != xas.xa_index); if (is_shmem) { @@ -1892,10 +1892,7 @@ static void collapse_file(struct mm_struct *mm, } nr = thp_nr_pages(new_page); - if (is_shmem) - __mod_lruvec_page_state(new_page, NR_SHMEM_THPS, nr); - else { - __mod_lruvec_page_state(new_page, NR_FILE_THPS, nr); + if (!is_shmem) { filemap_nr_thps_inc(mapping); /* * Paired with smp_mb() in do_dentry_open() to ensure @@ -1906,21 +1903,10 @@ static void collapse_file(struct mm_struct *mm, smp_mb(); if (inode_is_open_for_write(mapping->host)) { result = SCAN_FAIL; - __mod_lruvec_page_state(new_page, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); goto xa_locked; } } - - if (nr_none) { - __mod_lruvec_page_state(new_page, NR_FILE_PAGES, nr_none); - /* nr_none is always 0 for non-shmem. */ - __mod_lruvec_page_state(new_page, NR_SHMEM, nr_none); - } - - /* Join all the small entries into a single multi-index entry */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, new_page); xa_locked: xas_unlock_irq(&xas); xa_unlocked: @@ -1933,20 +1919,34 @@ static void collapse_file(struct mm_struct *mm, try_to_unmap_flush(); if (result == SCAN_SUCCEED) { - struct page *page, *tmp; - /* * Replacing old pages with new one has succeeded, now we - * need to copy the content and free the old pages. + * attempt to copy the contents. */ index = start; - list_for_each_entry_safe(page, tmp, &pagelist, lru) { + list_for_each_entry(page, &pagelist, lru) { while (index < page->index) { clear_highpage(new_page + (index % HPAGE_PMD_NR)); index++; } - copy_highpage(new_page + (page->index % HPAGE_PMD_NR), - page); + if (copy_highpage_mc(new_page + (page->index % HPAGE_PMD_NR), page)) { + result = SCAN_COPY_MC; + break; + } + index++; + } + while (result == SCAN_SUCCEED && index < end) { + clear_highpage(new_page + (page->index % HPAGE_PMD_NR)); + index++; + } + } + + if (result == SCAN_SUCCEED) { + /* + * Copying old pages to huge one has succeeded, now we + * need to free the old pages. + */ + list_for_each_entry_safe(page, tmp, &pagelist, lru) { list_del(&page->lru); page->mapping = NULL; page_ref_unfreeze(page, 1); @@ -1954,12 +1954,23 @@ static void collapse_file(struct mm_struct *mm, ClearPageUnevictable(page); unlock_page(page); put_page(page); - index++; } - while (index < end) { - clear_highpage(new_page + (index % HPAGE_PMD_NR)); - index++; + + xas_lock_irq(&xas); + if (is_shmem) + __mod_lruvec_page_state(new_page, NR_SHMEM_THPS, nr); + else + __mod_lruvec_page_state(new_page, NR_FILE_THPS, nr); + + if (nr_none) { + __mod_lruvec_page_state(new_page, NR_FILE_PAGES, nr_none); + /* nr_none is always 0 for non-shmem. */ + __mod_lruvec_page_state(new_page, NR_SHMEM, nr_none); } + /* Join all the small entries into a single multi-index entry. */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, new_page); + xas_unlock_irq(&xas); SetPageUptodate(new_page); page_ref_add(new_page, HPAGE_PMD_NR - 1); @@ -1975,8 +1986,6 @@ static void collapse_file(struct mm_struct *mm, khugepaged_pages_collapsed++; } else { - struct page *page; - /* Something went wrong: roll back page cache changes */ xas_lock_irq(&xas); if (nr_none) { @@ -2010,6 +2019,13 @@ static void collapse_file(struct mm_struct *mm, xas_lock_irq(&xas); } VM_BUG_ON(nr_none); + /* + * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. + * This undo is not needed unless failure is due to SCAN_COPY_MC. + */ + if (!is_shmem && result == SCAN_COPY_MC) + filemap_nr_thps_dec(mapping); + xas_unlock_irq(&xas); new_page->mapping = NULL;