From patchwork Tue Feb 11 11:13:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13969531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08B76C0219B for ; Tue, 11 Feb 2025 11:16:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DB2E280004; Tue, 11 Feb 2025 06:16:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76385280001; Tue, 11 Feb 2025 06:16:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DD49280004; Tue, 11 Feb 2025 06:16:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 363C6280001 for ; Tue, 11 Feb 2025 06:16:30 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E904D4481A for ; Tue, 11 Feb 2025 11:16:29 +0000 (UTC) X-FDA: 83107410498.28.7387E6A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 4A864C0012 for ; Tue, 11 Feb 2025 11:16:28 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739272588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rPHWL81e3X5xJLqPXe38UP90M/1a3mLjWx5Vl/7I9dk=; b=va25EK3sZDFejaxGNYWTfPFOqT9d/ha6/g/YqKI2VmuxDXXVb0fsGuYMqqMiqC5BHxqovt 8B7GO4Yj/zWXrNzgnIGKs2mgndYggfWhQN7IEWVyyg5fZGyctJ2V6mszylKReZxSs7aej/ E7E/FF3UdcMiHSsyD6YJZAAwEKaT64s= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739272588; a=rsa-sha256; cv=none; b=hhwz7cNxvOJsoNijLhjwkyFxcC5WtA5Rj/nxBIorTwG7LeQ6/d9h9t5gsKT42+bY+qUIUs Jt2gx/m6f3TL6qxZVSa31tRzoJmOQfDwOhgg75iQkCgaxZBFmO7EZgNSAwe7U13yL4t3g/ YqeQlPPC6jkB42N2jADtLuBqVvUrCgs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1C70313D5; Tue, 11 Feb 2025 03:16:49 -0800 (PST) Received: from K4MQJ0H1H2.emea.arm.com (K4MQJ0H1H2.blr.arm.com [10.162.40.80]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D52D43F5A1; Tue, 11 Feb 2025 03:16:17 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: npache@redhat.com, ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [PATCH v2 16/17] khugepaged: Implement strict policy for mTHP collapse Date: Tue, 11 Feb 2025 16:43:25 +0530 Message-Id: <20250211111326.14295-17-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250211111326.14295-1-dev.jain@arm.com> References: <20250211111326.14295-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4A864C0012 X-Stat-Signature: ku4b5gsd7hirmycfhtrmtuu88bzjozfs X-HE-Tag: 1739272588-823375 X-HE-Meta: U2FsdGVkX1/3aaevMOIKtlfgW8eRupFlJGFQrHA+PibYjZfCGu6TeCwyFxEYbcGARz1tZx4uJyzaGTjR37eOJum5jtJkCGRaPtdW8DCs60nIrFzoR8KuOyg5zFwMG+6xH7hTPwf5UNKU8y+hxcUqoU9QRxq1Md75qXD3RmByVCsVvxEd1lIWU4SOxoYe61PVSJIapSvUK6D72obVIe8V9x+vIxu192fkpof4bWptzrobuZmNfFyeczUa+mtYS7vtvQthPqwnzES2tIZHUzGk2KDfpzonzhFx+QKOqrLy3iDUVDW1se8kW9hd5GRplqIfjUI8J2KIBJ3J/tKsOqO+4jpFHCcaOG8Ef3iN+YoEmx9Hg+UtTQ5m3O3N+S8JKOJTVvkTltzNjYUBOoRZtx6UtwrWm+Bhm7HPY3C0AyK1PxviolYJPrcAzL4tNLD/YkOh+p1KtDSxbVyy43m22A4r1NpnXTxsVwFyZRdywVPoqh7htsR++Hf57XNc2QNeRD0qeYu3oNkeDzlRlG8EKklI3yK5sYV37zADsmyLisn7RHdPXi2/DMfTCR8/KOu/iAPqr2M017yIZ/ZGsyqD4FrNlVrP7OIKdWJIPSFKWAZezAszuIeYnL1U38Xwqz1F+dU+Ke3KUiqXiEtby9CfLzeymUcCXCZa8xa755h0oaED/DoeiQkod7ih2g1PtgrLlaDZBy0v7DiXUPPX+5knrEf0a5BAm4fzVcxaKpkwmPWRs6tlo6hTy8uvS8INR73yZU3x3CC6upZf9R0kpU34eCIShjU34z6I2GolVvyrMrkWWneCgnhr9DApLdeFhmTCvxnwyMAEO61xfCGUUk7CzvyNqCws1rOlveSrVVzacBYGs2YVi4DubFAX7f167rP1s2es9y2ysup6eglcMrvOODMG/M5YiWM8lV8l0UCpxgprYWcbQG+hGmKsBgMw8NX49UiOpd2FXQVFpVuGsizgWzh koonoAQn WQauAWOgK3LqLStP/EEy7+4p2XBfoIOgVcwPsg7/QYFz75oUKSG8FHufE+ok8mv9v2Eeql5wgVBn/kJzJsU8XOWBhpuppEfm/IVY0+jGFiXVdLBc2mTTLgDqGF8ahrus1CvQw+QvxSsdipZBJKaYoDcXM/ltQTku1gUA2rqN8GAftL/42ztt+UEBtQfGOf/A6pEH1aqo5K7Pxik2lJJ4ue6nkCXmfDoySZoD8aFy1OvX/Un5KJI0dDoIbhg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As noted in the discussion thread ending at [1], avoid the creep problem by collapsing to mTHPs only if max_ptes_none is zero or 511. Along with this, make mTHP collapse conditions stricter by removing scaling of max_ptes_shared and max_ptes_swap, and consider collapse only if there are no shared or swap PTEs in the range. [1] https://lore.kernel.org/all/8114d47b-b383-4d6e-ab65-a0e88b99c873@arm.com/ Signed-off-by: Dev Jain --- mm/khugepaged.c | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index d2bb008b95e7..b589f889bb5a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -417,6 +417,17 @@ static inline int hpage_collapse_test_exit_or_disable(struct mm_struct *mm) static bool thp_enabled(void) { + bool anon_pmd_enabled = (test_bit(PMD_ORDER, &huge_anon_orders_always) || + test_bit(PMD_ORDER, &huge_anon_orders_madvise) || + (test_bit(PMD_ORDER, &huge_anon_orders_inherit) && + hugepage_global_enabled())); + + /* + * If PMD_ORDER is ineligible for collapse, check if mTHP collapse policy is obeyed; + * see Documentation/admin-guide/transhuge.rst + */ + bool anon_collapse_mthp = (khugepaged_max_ptes_none == 0 || + khugepaged_max_ptes_none == HPAGE_PMD_NR - 1); /* * We cover the anon, shmem and the file-backed case here; file-backed * hugepages, when configured in, are determined by the global control. @@ -427,8 +438,9 @@ static bool thp_enabled(void) if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && hugepage_global_enabled()) return true; - if (huge_anon_orders_always || huge_anon_orders_madvise || - (huge_anon_orders_inherit && hugepage_global_enabled())) + if ((huge_anon_orders_always || huge_anon_orders_madvise || + (huge_anon_orders_inherit && hugepage_global_enabled())) && + (anon_pmd_enabled || anon_collapse_mthp)) return true; if (IS_ENABLED(CONFIG_SHMEM) && shmem_hpage_pmd_enabled()) return true; @@ -578,13 +590,16 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, pte_t *_pte; int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; bool writable = false; - unsigned int max_ptes_shared = khugepaged_max_ptes_shared >> (HPAGE_PMD_ORDER - order); + unsigned int max_ptes_shared = khugepaged_max_ptes_shared; unsigned int max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); bool all_pfns_present = true; bool all_pfns_contig = true; bool first_pfn_aligned = true; pte_t prev_pteval; + if (order != HPAGE_PMD_ORDER) + max_ptes_shared = 0; + for (_pte = pte; _pte < pte + (1UL << order); _pte++, address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); @@ -1453,11 +1468,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, order = highest_order(orders); VM_BUG_ON(address & ((PAGE_SIZE << order) - 1)); + max_ptes_none = khugepaged_max_ptes_none; + max_ptes_shared = khugepaged_max_ptes_shared; + max_ptes_swap = khugepaged_max_ptes_swap; + scan_pte_range: - max_ptes_shared = khugepaged_max_ptes_shared >> (HPAGE_PMD_ORDER - order); + if (order != HPAGE_PMD_ORDER) + max_ptes_shared = max_ptes_swap = 0; + max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); - max_ptes_swap = khugepaged_max_ptes_swap >> (HPAGE_PMD_ORDER - order); referenced = 0, shared = 0, none_or_zero = 0, unmapped = 0; all_pfns_present = true, all_pfns_contig = true, first_pfn_aligned = true; @@ -2651,6 +2671,11 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, int order; bool is_file_vma; int prev_progress = 0; + bool collapse_mthp = true; + + /* Avoid the creep problem; see Documentation/admin-guide/transhuge.rst */ + if (khugepaged_max_ptes_none && khugepaged_max_ptes_none != HPAGE_PMD_NR - 1) + collapse_mthp = false; VM_BUG_ON(!pages); lockdep_assert_held(&khugepaged_mm_lock); @@ -2710,6 +2735,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, /* select the highest possible order for the VMA */ order = highest_order(orders); while (orders) { + if (order != HPAGE_PMD_ORDER && !collapse_mthp) + goto skip; hend = round_down(vma->vm_end, PAGE_SIZE << order); if (khugepaged_scan.address <= hend) break;