From patchwork Thu Apr 17 00:02:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 14054622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 547EAC369BD for ; Thu, 17 Apr 2025 00:06:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC76D6B02DC; Wed, 16 Apr 2025 20:06:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E758E6B02DD; Wed, 16 Apr 2025 20:06:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEE1928001A; Wed, 16 Apr 2025 20:06:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B114B6B02DC for ; Wed, 16 Apr 2025 20:06:12 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CB05C1A0F45 for ; Thu, 17 Apr 2025 00:06:12 +0000 (UTC) X-FDA: 83341593384.30.0B4C278 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id B520D100002 for ; Thu, 17 Apr 2025 00:06:10 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JLBEvqvD; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744848371; a=rsa-sha256; cv=none; b=q6iXjBEOgztTV5auHM4y/lI1/ewniNj+ri45bXeW8JN1VhsL/kY9a/7esz0WBewweACL4g ZcsMCWbKBpIYt5YADfU2SiIsT0p99G0Uz9BoXHMNdmw3Wj3TcGTQ7RqaZawP9oxUcLVKC7 FzBKzxZ0oUrXpmALosBeX2VaTvRBoI4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JLBEvqvD; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744848371; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=lZzURqEH2m3la60MQH0q6mI1PM9pKq3/kimYAOnNk7o=; b=Jx2CkVT+fy6QGFLR7ToE/7p/cRdnmJ964wlwvWqtWYpSrpQNkQcMfxh3v/BiPuV90vNDVF q/DQM1F/3D2z0q0e2mQ3UfjAlpruQELy1xHla3vVWbSNuvTxPz0T7iBUs9Ol4ZDFKKXZwQ 2NOsFWKZ++JajlwlLT39xJ94kFV9QYM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1744848370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=lZzURqEH2m3la60MQH0q6mI1PM9pKq3/kimYAOnNk7o=; b=JLBEvqvDyh1Hg6md2D5X2UB5Tc65yu/JO1s7krg9bkYI1TtwEFu3gy5yFdgfTchW+QtTUY 9Q9edAPj4eD6RSl/h+T+4FsB4PyvKVtlzrEsrzzGMFVFcfbDmxMn8Xn9bceMWf3IDYlfPK 9xLr3Jo7qod1kTtYSSQGpGBSz4+P/I0= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-370-gSEb2mDWOaCD-kmCEKxNWA-1; Wed, 16 Apr 2025 20:06:06 -0400 X-MC-Unique: gSEb2mDWOaCD-kmCEKxNWA-1 X-Mimecast-MFC-AGG-ID: gSEb2mDWOaCD-kmCEKxNWA_1744848360 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E45421955DC5; Thu, 17 Apr 2025 00:05:58 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.34]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F1B30180045B; Thu, 17 Apr 2025 00:05:47 +0000 (UTC) From: Nico Pache To: linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, corbet@lwn.net, rostedt@goodmis.org, mhiramat@kernel.org, mathieu.desnoyers@efficios.com, david@redhat.com, baohua@kernel.org, baolin.wang@linux.alibaba.com, ryan.roberts@arm.com, willy@infradead.org, peterx@redhat.com, ziy@nvidia.com, wangkefeng.wang@huawei.com, usamaarif642@gmail.com, sunnanyong@huawei.com, vishal.moola@gmail.com, thomas.hellstrom@linux.intel.com, yang@os.amperecomputing.com, kirill.shutemov@linux.intel.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, tiwai@suse.de, will@kernel.org, dave.hansen@linux.intel.com, jack@suse.cz, cl@gentwo.org, jglisse@google.com, surenb@google.com, zokeefe@google.com, hannes@cmpxchg.org, rientjes@google.com, mhocko@suse.com, rdunlap@infradead.org Subject: [PATCH v4 00/12] khugepaged: mTHP support Date: Wed, 16 Apr 2025 18:02:26 -0600 Message-ID: <20250417000238.74567-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B520D100002 X-Stat-Signature: bqc1sf346fixytr7j5cd61uyzdtd53wm X-HE-Tag: 1744848370-46173 X-HE-Meta: U2FsdGVkX1/4u4HjI1Dd7UU3U4BSK3xhQf1BBPluQvGbSBGpH7UYKrzqfK7fSZjERcx6gnNTiFOmYVHyAeFfPQm8QnZxlSi4LXS5AXw3Frtjb5Dwf5YrnVuMPkSKtq6N9XBM6nsXO4EO/m5Fhso1N/i9xwMAVEjr44HqwjqOSIaRrafB1cGhwO5eydIQGYcAPbGAzW6hRhY/HNRGzqCMcmp6KcjvxqPc4UXaRuupq2XcdkVGq2f6uCip/WF8wXKInxtXKlpA7BjlqUa+6/DzM2qYobA1uPVp66DxxwCoJ95oPNb3suHNVHw7BwF7OIdCyRB/7/heqr31MugibMG0I9zCL/6E2eWTM6ikLeWXU8FLUz26FrMwcPO84VwRf/XGG25E+NBO3JHUH4b2rzFQp/tsmYW7ov6Wt3KoDEXAdz+WE42HvsICiaf1zqjvp4RbjvVj3FnNtoIx01D98FnxtGclD5t4yq1fD3LsQ0Ht6G8NHZnuuH35YqaYCC4it6fIgKSbA2krbbfAhWta5AmX5srrxiDJ2Gi4f0FdDvD0x+AzazPOAISdVakQ4Hl497TRpvls+EFVDNDuGCrK6Uf2sn9Hk9L4Qdds3MKCZ/GgiQFp58c87jD4GXWnhNKO83PgUovkLsTeVCcqIfV4tS5mQBsoVupeHYDmiOXyfQGk8WV1mYFzXtiM5UfSpkNbHXock8+kGS2iMVUicZXPbzJP5A3PdnQLVVr8ghsn5za3yYrrBhMJk1G1tOz4KWWyZSRbjZB3yrSJ2rWU7vGOHoW+xyp6N2NhQoffDzHwl2+enMuoO7/9K+TEFKKdFHPGUyAoVdNpOjlVT8dS+76ppj0QsEsw7rVeY4vXVAvZ6iM1CjjxEkNooBmGDbGiXz1YQi4u2lgxMQp40fXY+T/jX/k6QwhwcPqSBXdY/j2UwyJqBVxSACfZYt8PLCNVSPemcWQSXRkf56PCyVkGXEY687b GFwn9uUt m3M3M3C2A0qkHy34Cjmdwz8Jvy4bSegihTq6Ky0MPjJzfrdENmGhc5LtYGDPgU9DIEv6LESfSuU1W6hDbj3yBPR2tqXETxRlAr2k1bkEcJO20lu0TF3gAvzRhY4Tps6QLr/UXInq6hS1cV85S3EwDsqPpORiLiUs9O6Lg4+0DiGfDNBW3cFdT0ZX1iIZNQ4Eet3KpLubsuvpqOn6Vt9D46LygNGeHUBn2NGrcOoaiwBI+dZfG3lNz6Z+hdzocCCkoZKqVLYqWmpcHMg5M3jeZiYmFUwfhixIl99RClzPbmWX018T05aIGIojRSPaKqFso8qRkcun5BvzxfkX/Ct0UWY2oDRTWTadfgfkrt2gXU6pgvpMRPBm786ytDas3hMGtO5tLKleAUfbMtmVc+WXLjdqdmz9TsSp0Z3rye/iPg/xqlLrMAPOHaPCDe3w7XvCUD3Uj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The following series provides khugepaged and madvise collapse with the capability to collapse regions to mTHPs. To achieve this we generalize the khugepaged functions to no longer depend on PMD_ORDER. Then during the PMD scan, we keep track of chunks of pages (defined by KHUGEPAGED_MTHP_MIN_ORDER) that are utilized. This info is tracked using a bitmap. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction on max_ptes_none is removed during the scan, to make sure we account for the whole PMD range. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained. max_ptes_none will be scaled by the attempted collapse order to determine how full a THP must be to be eligible. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we dont perform the collapse. With the default max_ptes_none=511, the code should keep its most of its original behavior. To exercise mTHP collapse we need to set max_ptes_none<=255. With max_ptes_none > HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. Patch 1: Some refactoring to combine madvise_collapse and khugepaged Patch 2: Refactor/rename hpage_collapse Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-9: The mTHP patches Patch 10-11: Tracing/stats Patch 12: Documentation --------- Testing --------- - Built for x86_64, aarch64, ppc64le, and s390x - selftests mm - I created a test script that I used to push khugepaged to its limits while monitoring a number of stats and tracepoints. The code is available here[1] (Run in legacy mode for these changes and set mthp sizes to inherit) The summary from my testings was that there was no significant regression noticed through this test. In some cases my changes had better collapse latencies, and was able to scan more pages in the same amount of time/work, but for the most part the results were consistent. - redis testing. I tested these changes along with my defer changes (see followup post for more details). - some basic testing on 64k page size. - lots of general use. Changes since V3: - Rebased onto mm-unstable commit 0e68b850b1d3 ("vmalloc: use atomic_long_add_return_relaxed()") - small changes to Documentation Changes since V2: - corrected legacy behavior for khugepaged and madvise_collapse - added proper mTHP stat tracking - Minor changes to prevent a nested lock on non-split-lock arches - Took Devs version of alloc_charge_folio as it has the proper stats - Skip cases were trying to collapse to a lower order would still fail - Fixed cases were the bitmap was not being updated properly - Moved Documentation update to this series instead of the defer set - Minor bugs discovered during testing and review - Minor "nit" cleanup Changes since V1 [2]: - Minor bug fixes discovered during review and testing - removed dynamic allocations for bitmaps, and made them stack based - Adjusted bitmap offset from u8 to u16 to support 64k pagesize. - Updated trace events to include collapsing order info. - Scaled max_ptes_none by order rather than scaling to a 0-100 scale. - No longer require a chunk to be fully utilized before setting the bit. Use the same max_ptes_none scaling principle to achieve this. - Skip mTHP collapse that requires swapin or shared handling. This helps prevent some of the "creep" that was discovered in v1. [1] - https://gitlab.com/npache/khugepaged_mthp_test [2] - https://lore.kernel.org/lkml/20250108233128.14484-1-npache@redhat.com/ Dev Jain (1): khugepaged: generalize alloc_charge_folio() Nico Pache (11): introduce khugepaged_collapse_single_pmd to unify khugepaged and madvise_collapse khugepaged: rename hpage_collapse_* to khugepaged_* khugepaged: generalize hugepage_vma_revalidate for mTHP support khugepaged: generalize __collapse_huge_page_* for mTHP support khugepaged: introduce khugepaged_scan_bitmap for mTHP support khugepaged: add mTHP support khugepaged: skip collapsing mTHP to smaller orders khugepaged: avoid unnecessary mTHP collapse attempts khugepaged: improve tracepoints for mTHP orders khugepaged: add per-order mTHP khugepaged stats Documentation: mm: update the admin guide for mTHP collapse Documentation/admin-guide/mm/transhuge.rst | 10 +- include/linux/huge_mm.h | 5 + include/linux/khugepaged.h | 4 + include/trace/events/huge_memory.h | 34 +- mm/huge_memory.c | 11 + mm/khugepaged.c | 457 ++++++++++++++------- 6 files changed, 369 insertions(+), 152 deletions(-)