From patchwork Mon Dec 16 16:50:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CFF5E77180 for ; Mon, 16 Dec 2024 16:51:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3DD86B00BB; Mon, 16 Dec 2024 11:51:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C6656B00BC; Mon, 16 Dec 2024 11:51:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83FE96B00BD; Mon, 16 Dec 2024 11:51:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 61F4F6B00BB for ; Mon, 16 Dec 2024 11:51:37 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1AA55A0CD1 for ; Mon, 16 Dec 2024 16:51:37 +0000 (UTC) X-FDA: 82901412552.24.0A095BA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 3A979C0013 for ; Mon, 16 Dec 2024 16:51:04 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367863; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=T0cRefFfZtoNtGozg1mNG5+tja4lEXjolgtx2SzWNIE=; b=MB0X9QoPenH9qzbhcLbfTtenNzR0/QMQkefp52I1qzQ46SXPcJiR1tdE5JlChcGhJHCDIT ydtwjyHRR5IiyBop++WmnKAr5D6rCV+5eAJR+2CwGkHaYrEC5LxmqNsf3StSJ4DoR563gl xKhMqlz3FApv/btsUhWXf+zhkZtWO0k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367863; a=rsa-sha256; cv=none; b=NagWBuFunpLekzMLlgh/O+iB6DM7GUrFJ4VDHh/jURaxtJ68W7lUJRw+qiCyGJp+vaLrE3 PO3wXZQTCvBVibVsJmJf9pjvMZd3sPn1jPCBWCxuNQJoGiXZbWBeMpCE9JoU+F1Bu+8inG wig/du3RtDS0Y3Fl/FD3RCf/pHJMsfk= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2BD9C106F; Mon, 16 Dec 2024 08:52:02 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 087AD3F528; Mon, 16 Dec 2024 08:51:22 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 00/12] khugepaged: Asynchronous mTHP collapse Date: Mon, 16 Dec 2024 22:20:53 +0530 Message-Id: <20241216165105.56185-1-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 X-Rspamd-Queue-Id: 3A979C0013 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: dbrnn6wnzdxsnq7jxdwwu9ioh3gziqt5 X-HE-Tag: 1734367864-632241 X-HE-Meta: U2FsdGVkX1+0VQnAMuC1C/uT1o1fzAwR8EF5n7JQXO5+Dd5Mm105WEI+5Ymh3+iWLJWeWnLOVeUZlYQpYOKyxXcd0fXiti5kXhE2NR/Y3uwBrMMHUgDbwwAgxvRnJEyVRs7QKbe2pMiSKnobwlFf6+t6+w8A68T5QMf3lFKhc+9dwO19s+amJbTgrjGwvEiaFIAPKirQckK1w8dgndxbBuid00qYQHOjc/YP07wsbC88pggzTHhor7nOFpjvhte+MUTgkSpfA7LRxYnTM9t3N9fWpC4mwCN2D5/+/2rSCHHyGMV0w9bDUMPMohLIGq2p90wB7CCSDa16YGZNM8X4VH53n1RmapmIWY5fTpRwLUqEK1biMZde5G1kYviNcTQnY1HBDwIXB8+SvOXsOSwPmIRd8ph0gCWPdSugWgss8wLtzwQng1MqH3vO+gBESLcSxuLPILGO6h51yOJKmZaO9BEXUzhLj4sav4E0HiFRFSARsjfjNRpj65E6NgYiV61MoGemNeVHRppkt20LmT+8c/Wn8FMIBnXSfwrupDRAEH302ceF+kp206tNI8HNfsjf/ib8aoGCbdvW+ma+kLuwyP4+4RzALpNgRE9wVDXnDkGl8ZwuI06gY/NZj42zYs6Mx8KReXIGHham6ezaVBnpre5FA8/SIwnK3mIBbb4Pfs0SIACJgapD2ZjwxF7tVRGdOlrlYv7Ts5yx7FhPaa8preGjlPXuqxOI+GSVQDnKVO9bTpXTD9ZjLuQGS6JcGha7qy4hq/oZTkd/Sjo32fWo/dukoOKDa1218dUSqZehrao0b6ChMAWKaJh4tGF+sUCW6fhzm6lga+e4i23598hYCY6BM821Yqu0M2Sbj2wUJKrGdLGuZCsb0Kuxtu6hSkKsfTVYya4P7xZnzQJc0pomZcesXT2wsXSiBNeA/iyjDTd11oQSl7LQZrOUcdYocLjiD5IoZMgz11D/B8u0cuT htpOOqC8 A7rgtgPiZZMcRjhTCHd16ZEDUfVc4sqYpkjZ/S2a2yE8DCOyFNhMZctwPJylJC+4Eq/LF4gKwMTt8EwjMU3V54Yk6iQrlV/4EquOXOcR7pux4BZYd0QKicHCm4IW8afSjafKQb+ttCUvFr7G1cMG8MTV1j018rEaWDeERhdkkI7e4FhOjQ9Ev62OMJ7ir+qKj1d7WJhjwtvJ5ghEfpbSNHlpP0i8hw5+q8k7uDQEPeWGa7j32FrwNFGThOrwqRqM5WT6+bU/r2v0VUmOlkBd2dSGZrQec55TU1lbLaLsTLfax07g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset extends khugepaged from collapsing only PMD-sized THPs to collapsing anonymous mTHPs. mTHPs were introduced in the kernel to improve memory management by allocating chunks of larger memory, so as to reduce number of page faults, TLB misses (due to TLB coalescing), reduce length of LRU lists, etc. However, the mTHP property is often lost due to CoW, swap-in/out, and when the kernel just cannot find enough physically contiguous memory to allocate on fault. Henceforth, there is a need to regain mTHPs in the system asynchronously. This work is an attempt in this direction, starting with anonymous folios. In the fault handler, we select the THP order in a greedy manner; the same has been used here, along with the same sysfs interface to control the order of collapse. In contrast to PMD-collapse, we (hopefully) get rid of the mmap_write_lock(). --------------------------------------------------------- Testing --------------------------------------------------------- The set has been build tested on x86_64. For Aarch64, 1. mm-selftests: No regressions. 2. Analyzing with tools/mm/thpmaps on different userspace programs mapping aligned VMAs of a large size, faulting in basepages/mTHPs (according to sysfs), and then madvise()'ing the VMA, khugepaged is able to 100% collapse the VMAs. This patchset is rebased on mm-unstable (e7e89af21ffcfd1077ca6d2188de6497db1ad84c). Some points to be noted: 1. Some stats like pages_collapsed for khugepaged have not been extended for mTHP. I'd welcome suggestions on any updation, or addition to the sysfs interface. 2. Please see patch 9 for lock handling. Dev Jain (12): khugepaged: Rename hpage_collapse_scan_pmd() -> ptes() khugepaged: Generalize alloc_charge_folio() khugepaged: Generalize hugepage_vma_revalidate() khugepaged: Generalize __collapse_huge_page_swapin() khugepaged: Generalize __collapse_huge_page_isolate() khugepaged: Generalize __collapse_huge_page_copy_failed() khugepaged: Scan PTEs order-wise khugepaged: Abstract PMD-THP collapse khugepaged: Introduce vma_collapse_anon_folio() khugepaged: Skip PTE range if a larger mTHP is already mapped khugepaged: Enable sysfs to control order of collapse selftests/mm: khugepaged: Enlighten for mTHP collapse include/linux/huge_mm.h | 2 + mm/huge_memory.c | 4 + mm/khugepaged.c | 445 +++++++++++++++++------- tools/testing/selftests/mm/khugepaged.c | 5 +- 4 files changed, 319 insertions(+), 137 deletions(-)