From patchwork Wed May 4 21:44:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12838668 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C04CC433EF for ; Wed, 4 May 2022 21:45:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDF8A6B007D; Wed, 4 May 2022 17:45:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B8F876B007E; Wed, 4 May 2022 17:45:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A56546B0080; Wed, 4 May 2022 17:45:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 955676B007D for ; Wed, 4 May 2022 17:45:15 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 632102BDE4 for ; Wed, 4 May 2022 21:45:15 +0000 (UTC) X-FDA: 79429391790.24.5F9D34E Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf18.hostedemail.com (Postfix) with ESMTP id 439841C0084 for ; Wed, 4 May 2022 21:45:07 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id t24-20020a17090a449800b001d2d6e740c3so3487487pjg.9 for ; Wed, 04 May 2022 14:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=5AZXzap0NKDHHUVSd4ZmCTitOVST19Wp792E9GujTtg=; b=Om00ZpZ/vyrJoB0r27H42fJGPDM2niFnq21PnJWkmVOpEb3cyRwBq+HLWcjRA9nRxg Y8WvQ4BWNN9lU3Dkdl8q2yL6gic7CPcoJOeq2e+2tilPxDpomH9NWhKEP4ul8P69yhHq WwbesZM7H1ZupxMivYaPsK/P0vhTY+Qoq7hv44mY2wS2SK/VShW35CzpA3OU3Dl9gUJg HbnwSiiEo52HwSKBtbBiTFuKBvWkvD2bNf4+CrVguXeX0c299fC74lYRjp9moJWFw/bR 390aA0qZZAVX8bDAjOIMdg0A10KNxyir8CcjAgFPSBdGlVI06Zv9KT+dkUGonG6JZygV e3jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5AZXzap0NKDHHUVSd4ZmCTitOVST19Wp792E9GujTtg=; b=lqNc4MCHjPzdWY8MvyNWD3pGn+Wp6pGgYSAdyyRMJL5ejjIYwPxM2B3gL7EVAlIsNN 0ajgFgiQPKTE0aCjO6i9SbnROW8QS+MlJjxxhyJnhEzguavOVHGzLATutvXpLV77ahxd nPPJmsukroIY3+HRq0AO4ilGbAxOfntTZp9MPiRc2BmuAvphT0Pw7y7VnpmUTOVHTDLH 05Db26RheQ4RFOf0en72FnErvX5HIcE9NVMLqAvTqG7+rU600767bjXDR8aclj6iPkH5 FUTPMsThgWhkDvzuLhBaNbaYoQMSdHj8p3hKkev3le3+UlVG8msE+zqVsilXnkV8BZPn 6yJw== X-Gm-Message-State: AOAM532G5LEWhgoR7l4g6miK4VwKAyo7wpV/FtQdXkzruvVejEoKFh3z vGz7uQwOiTJWMGyYy06y9IfR7N8gqUW/ X-Google-Smtp-Source: ABdhPJxGSyt8ncT6PLYhulgEg5MMlpixCwNvmBTjxi44bb/zKMEvXyhFxKdSBtABb3lcAZKHPD3kYY1VXg7N X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a05:6a00:150a:b0:510:3ea4:4cd9 with SMTP id q10-20020a056a00150a00b005103ea44cd9mr4582007pfu.27.1651700713893; Wed, 04 May 2022 14:45:13 -0700 (PDT) Date: Wed, 4 May 2022 14:44:31 -0700 In-Reply-To: <20220504214437.2850685-1-zokeefe@google.com> Message-Id: <20220504214437.2850685-8-zokeefe@google.com> Mime-Version: 1.0 References: <20220504214437.2850685-1-zokeefe@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v5 07/13] mm/khugepaged: add flag to ignore page young/referenced requirement From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , Peter Xu , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Thomas Bogendoerfer , "Zach O'Keefe" X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 439841C0084 X-Stat-Signature: 7qpa43hwi3hk9agprj3b88ge54pf3j8e X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="Om00ZpZ/"; spf=pass (imf18.hostedemail.com: domain of 36fNyYgcKCLYvkgaabackkcha.Ykihejqt-iigrWYg.knc@flex--zokeefe.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=36fNyYgcKCLYvkgaabackkcha.Ykihejqt-iigrWYg.knc@flex--zokeefe.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1651700707-617558 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add enforce_young flag to struct collapse_control that allows context to ignore requirement that some pages in region being collapsed be young or referenced. Set this flag in khugepaged collapse context to preserve existing khugepaged behavior. This flag will be used (unset) when introducing madvise collapse context since here, the user presumably has reason to believe the collapse will be beneficial and khugepaged heuristics shouldn't tell the user they are wrong. Signed-off-by: Zach O'Keefe Acked-by: David Rientjes --- mm/khugepaged.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index ca730aec0e3e..b14807b7002e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -90,6 +90,9 @@ struct collapse_control { /* Respect khugepaged_max_ptes_[none|swap|shared] */ bool enforce_pte_scan_limits; + /* Require memory to be young */ + bool enforce_young; + /* Num pages scanned per node */ int node_load[MAX_NUMNODES]; @@ -720,9 +723,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, list_add_tail(&page->lru, compound_pagelist); next: /* There should be enough young pte to collapse the page */ - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; if (pte_write(pteval)) @@ -731,7 +735,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, if (unlikely(!writable)) { result = SCAN_PAGE_RO; - } else if (unlikely(!referenced)) { + } else if (unlikely(cc->enforce_young && !referenced)) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1387,14 +1391,16 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, result = SCAN_PAGE_COUNT; goto out_unmap; } - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; } if (!writable) { result = SCAN_PAGE_RO; - } else if (!referenced || (unmapped && referenced < HPAGE_PMD_NR/2)) { + } else if (cc->enforce_young && (!referenced || (unmapped && referenced + < HPAGE_PMD_NR / 2))) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -2343,6 +2349,7 @@ static int khugepaged(void *none) struct mm_slot *mm_slot; struct collapse_control cc = { .enforce_pte_scan_limits = true, + .enforce_young = true, .last_target_node = NUMA_NO_NODE, .alloc_charge_hpage = &alloc_charge_hpage, };