From patchwork Sun Apr 10 13:54:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12808152 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13396C433F5 for ; Sun, 10 Apr 2022 13:55:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 599A88D0003; Sun, 10 Apr 2022 09:55:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 521A48D0001; Sun, 10 Apr 2022 09:55:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 373478D0003; Sun, 10 Apr 2022 09:55:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 20D478D0001 for ; Sun, 10 Apr 2022 09:55:15 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id EC26D120B51 for ; Sun, 10 Apr 2022 13:55:14 +0000 (UTC) X-FDA: 79341116148.13.334B53C Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf09.hostedemail.com (Postfix) with ESMTP id 755AA140004 for ; Sun, 10 Apr 2022 13:55:14 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id v10-20020a17090a0c8a00b001c7a548e4f7so10842162pja.2 for ; Sun, 10 Apr 2022 06:55:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xUPk2DkNiFStSfsg7fvWt61fb/cYrWOLsUemRz+N124=; b=HFoWDjgSzNj9L6Uvwi/isbGr1e2nTud3A2DGjAl+TcBFS25Tq2DP8VoNnES2yyGThd F3h6ub5TkxIz7Qn9S/lo9jAmerKNkCmWwiegpHaCbhHsGljZZqn4P/3ORuSGtl4xfz5O DDDrD/wWnkRDijxRr1TSjTK88mvb3d4tqQfz74ALvieeChJ7T6LYOnUYbF8G1AznhR9V hBcb0dj95E0PpiF8P9ZO3TrN+zeJ/EIvAP7tOI5V6f3XgvTo9W1zhctjEwZOhxbKtN5R fuysssS+AcPx0SUfGDbmdeJiQwGFJWwA16cmCDI0fy38QkehgESBCig3Y0069P8bjcxg hxew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xUPk2DkNiFStSfsg7fvWt61fb/cYrWOLsUemRz+N124=; b=hjwj2kkLNV9xtpSlnpkCRdwvDnRVujf/4/7VRLPUT7dOF3wVElvE/JVtiuqlZDIDNh kPdohClOLqKLaNfU0ctnMHwB5kUcIPlDIXAZgOXR//j5pkzGyENo0Kk2wBd+j8yWFwXc gDJztMqGA7EZ1ens/6LvbXuAsiLSPf1KjqvtgIUsHocSrqeIFyXCJxntoTrGgIouUPib epBFtwCAv6zRyrNsB+1KIqckQaPM16Hp1DKRgHP+YgNhgG/6zQ25M5pp1boCIX9GbjrJ zenKQfJayMOqWOLMBQMIzJ9H0RWLug8GY/etM5NjU0OSibjPe+8KfCHpVSMWzQKDNefM bhoA== X-Gm-Message-State: AOAM530ddvd19hclxRqSF3cCTHJ7YCs24unSuZvMckQQ9zIRTanTQa6v 9wdqOXBS0NNk7AQPkhdR9c3XTaY2czEm X-Google-Smtp-Source: ABdhPJxFD/emTUVdxm1W48JbMpdh4+SM3AhHa3RUGl8AhJP0WayL24fz5XJa6Bl/vIn56R+lRbdWsMvPTWDV X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a05:6a00:174f:b0:4fd:aed5:b5e4 with SMTP id j15-20020a056a00174f00b004fdaed5b5e4mr28379544pfc.39.1649598913462; Sun, 10 Apr 2022 06:55:13 -0700 (PDT) Date: Sun, 10 Apr 2022 06:54:41 -0700 In-Reply-To: <20220410135445.3897054-1-zokeefe@google.com> Message-Id: <20220410135445.3897054-9-zokeefe@google.com> Mime-Version: 1.0 References: <20220410135445.3897054-1-zokeefe@google.com> X-Mailer: git-send-email 2.35.1.1178.g4f1659d476-goog Subject: [PATCH 08/12] mm/khugepaged: add flag to ignore page young/referenced requirement From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Peter Xu , Thomas Bogendoerfer , "Zach O'Keefe" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 755AA140004 X-Stat-Signature: uwkxmorc5c746bsbanb4uwtmizgbo98m Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=HFoWDjgS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3weFSYgcKCOgjYUOOPOQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--zokeefe.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3weFSYgcKCOgjYUOOPOQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--zokeefe.bounces.google.com X-Rspam-User: X-HE-Tag: 1649598914-251137 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add enforce_young flag to struct collapse_control that allows context to ignore requirement that some pages in region being collapsed be young or referenced. Set this flag in khugepaged collapse context to preserve existing khugepaged behavior and unset the flag in madvise collapse context since the user presumably has reason to believe the collapse will be beneficial. Signed-off-by: Zach O'Keefe --- mm/khugepaged.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7f555da26fdc..8e5e45355c6d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -90,6 +90,9 @@ struct collapse_control { /* Respect khugepaged_max_ptes_[none|swap|shared] */ bool enforce_pte_scan_limits; + /* Require memory to be young */ + bool enforce_young; + /* Num pages scanned per node */ int node_load[MAX_NUMNODES]; @@ -737,9 +740,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, list_add_tail(&page->lru, compound_pagelist); next: /* There should be enough young pte to collapse the page */ - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; if (pte_write(pteval)) @@ -748,7 +752,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, if (unlikely(!writable)) { result = SCAN_PAGE_RO; - } else if (unlikely(!referenced)) { + } else if (unlikely(cc->enforce_young && !referenced)) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1408,14 +1412,16 @@ static void scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, cr->result = SCAN_PAGE_COUNT; goto out_unmap; } - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; } if (!writable) { cr->result = SCAN_PAGE_RO; - } else if (!referenced || (unmapped && referenced < HPAGE_PMD_NR/2)) { + } else if (cc->enforce_young && (!referenced || (unmapped && referenced + < HPAGE_PMD_NR / 2))) { cr->result = SCAN_LACK_REFERENCED_PAGE; } else { cr->result = SCAN_SUCCEED; @@ -2362,6 +2368,7 @@ static int khugepaged(void *none) struct mm_slot *mm_slot; struct collapse_control cc = { .enforce_pte_scan_limits = true, + .enforce_young = true, .last_target_node = NUMA_NO_NODE, .alloc_hpage = &khugepaged_alloc_page, }; @@ -2504,6 +2511,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, { struct collapse_control cc = { .enforce_pte_scan_limits = false, + .enforce_young = false, .last_target_node = NUMA_NO_NODE, .hpage = NULL, .alloc_hpage = &alloc_hpage,