From patchwork Tue Apr 26 14:44:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12827328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF142C433F5 for ; Tue, 26 Apr 2022 14:44:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FB3E6B0081; Tue, 26 Apr 2022 10:44:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 682276B0083; Tue, 26 Apr 2022 10:44:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FDDB6B0085; Tue, 26 Apr 2022 10:44:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 430A46B0081 for ; Tue, 26 Apr 2022 10:44:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 158ED12073B for ; Tue, 26 Apr 2022 14:44:37 +0000 (UTC) X-FDA: 79399301394.20.2AC1E4F Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 40F22C0050 for ; Tue, 26 Apr 2022 14:44:29 +0000 (UTC) Received: by mail-pl1-f201.google.com with SMTP id w24-20020a170902a71800b0015d00267d74so4746186plq.6 for ; Tue, 26 Apr 2022 07:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xkNoRIjDVkHtQzAd/hGEjCQdf3OP2abrGhCTRsST8ak=; b=JM7KDsYLHyZ0+MchKRZmiMfv83rqqWv+5BHvEMU7t1ukhl1BU2nym/cFUEzDq4g4fN CF9VwbX5CNp2m4s7wKdIMzcfp3b0lX5Ir1/cP3Q1g/NRc+7k/ut8qLY6KZvagsJzn8Ow 9AR5hh0RZ60AEfzqcD/vjYv3xpk6sRz6rQ+BzI9i7cGcvBaXpEQR1vP2rGywHsmJiXaW KlZzfkDg0+7uVaxx/KTrYX/OaWqZxAkkfz/knylFOtjZ7QV9wLIeoFBi78wOKroUE+nR ntxNhrOgCvhmPBrQSG4Sr7w9UfSj8KgDfRVQDWJJjZNE5fkok/tH+Z8oRQ09vByzHH58 r16w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xkNoRIjDVkHtQzAd/hGEjCQdf3OP2abrGhCTRsST8ak=; b=ByxK2gCtdlPlNxQG1yGLRl9ZJVRqie9c+ekgMVknCLSwLJh2UL8X/mEBumACL7iTXT VRaBDUB/17cvE4CwKFU8FCiuMfrZXKLD5JMAB+mHnhqkOTjdL3Gzb5qA+XN5F6qAz+8B ALrUOL6Z3METz/Res16VyNsrybOgzkmbMsNrAmmciDwTDiR+8vLiqzwUCrKjBnahxLKj naENcbCnenWNhU7IVUIKeQOisgTj5eB49s7GvMS+yr1vedEqOkBnl50a43ucTqdAfCyS NpC6pZpMfrq3K1AFxtea5GBfO+G7nlSYq/M6Zoi0CUT6BUrbtXKVin/s5fYBkmdYhdmV HmwA== X-Gm-Message-State: AOAM533GRXeRMVTdvXZ4+V/007Nu4kv4UQsgp7CIZnlaO8iCGS4pmgPI N71d3ClqDHBMAtE+d1ZlX2dkDWRF92BB X-Google-Smtp-Source: ABdhPJxEGwL6WocYdNy7DdgsNzPCUweH8djyXvLrDbpuitZbRvma1Rljmv22JyepXsTf9Ei2io5h3kmRkrGj X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a17:902:e550:b0:15c:f4f2:814f with SMTP id n16-20020a170902e55000b0015cf4f2814fmr15334178plf.123.1650984275223; Tue, 26 Apr 2022 07:44:35 -0700 (PDT) Date: Tue, 26 Apr 2022 07:44:08 -0700 In-Reply-To: <20220426144412.742113-1-zokeefe@google.com> Message-Id: <20220426144412.742113-9-zokeefe@google.com> Mime-Version: 1.0 References: <20220426144412.742113-1-zokeefe@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 08/12] mm/khugepaged: add flag to ignore page young/referenced requirement From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , Peter Xu , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Thomas Bogendoerfer , "Zach O'Keefe" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 40F22C0050 X-Stat-Signature: 3n7uyak1f7uye9yxrdyst6wdoykq8d3g X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JM7KDsYL; spf=pass (imf28.hostedemail.com: domain of 3UwVoYgcKCBgNC822324CC492.0CA96BIL-AA8Jy08.CF4@flex--zokeefe.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3UwVoYgcKCBgNC822324CC492.0CA96BIL-AA8Jy08.CF4@flex--zokeefe.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1650984269-814076 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add enforce_young flag to struct collapse_control that allows context to ignore requirement that some pages in region being collapsed be young or referenced. Set this flag in khugepaged collapse context to preserve existing khugepaged behavior and unset the flag in madvise collapse context since the user presumably has reason to believe the collapse will be beneficial. Signed-off-by: Zach O'Keefe --- mm/khugepaged.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 57725482290d..fe6810825259 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -90,6 +90,9 @@ struct collapse_control { /* Respect khugepaged_max_ptes_[none|swap|shared] */ bool enforce_pte_scan_limits; + /* Require memory to be young */ + bool enforce_young; + /* Num pages scanned per node */ int node_load[MAX_NUMNODES]; @@ -738,9 +741,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, list_add_tail(&page->lru, compound_pagelist); next: /* There should be enough young pte to collapse the page */ - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; if (pte_write(pteval)) @@ -749,7 +753,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, if (unlikely(!writable)) { result = SCAN_PAGE_RO; - } else if (unlikely(!referenced)) { + } else if (unlikely(cc->enforce_young && !referenced)) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1409,14 +1413,16 @@ static void scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, cr->result = SCAN_PAGE_COUNT; goto out_unmap; } - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; } if (!writable) { cr->result = SCAN_PAGE_RO; - } else if (!referenced || (unmapped && referenced < HPAGE_PMD_NR/2)) { + } else if (cc->enforce_young && (!referenced || (unmapped && referenced + < HPAGE_PMD_NR / 2))) { cr->result = SCAN_LACK_REFERENCED_PAGE; } else { cr->result = SCAN_SUCCEED; @@ -2376,6 +2382,7 @@ static int khugepaged(void *none) struct mm_slot *mm_slot; struct collapse_control cc = { .enforce_pte_scan_limits = true, + .enforce_young = true, .last_target_node = NUMA_NO_NODE, .gfp = &alloc_hugepage_khugepaged_gfpmask, .alloc_hpage = &khugepaged_alloc_page, @@ -2524,6 +2531,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, { struct collapse_control cc = { .enforce_pte_scan_limits = false, + .enforce_young = false, .last_target_node = NUMA_NO_NODE, .hpage = NULL, .gfp = &alloc_hugepage_madvise_gfpmask,