From patchwork Tue Feb 11 00:30:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81E66C02198 for ; Tue, 11 Feb 2025 00:31:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88E7D6B007B; Mon, 10 Feb 2025 19:31:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83DA16B0085; Mon, 10 Feb 2025 19:31:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7054B6B0088; Mon, 10 Feb 2025 19:31:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 550F16B007B for ; Mon, 10 Feb 2025 19:31:16 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C7129B0DCD for ; Tue, 11 Feb 2025 00:31:15 +0000 (UTC) X-FDA: 83105784510.07.961086B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 23F31C0010 for ; Tue, 11 Feb 2025 00:31:13 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XO6EX7P+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233874; a=rsa-sha256; cv=none; b=m1sTO3Hk0g32quhBNgrULIkUVWgOYIh5QcDdnpAhkdCYIyq18ef2T+DqiqcQhED6+XxMph TNzv0WbCdVtlhTUEHTvJkQlp2UJacPQGr9kuXM2Vtng1qsqDNDMue3cmS2ONOMaE2Grt9O 8IB1buvY21jA2lzs063VRrweFmk+xTI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XO6EX7P+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233874; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=lt4B3Gv3OCrw8JiFM9sdRNcEjetfZHNbsKR6G5cmR7c=; b=MqOm319epOZPB5pIiG97EE+fC6ZPFOvYzH01IZ2wLQt/9AI4RZ0jwfqZrseld2ctwlXlzC KKgllPINK3jNF9MQnHrfCOjeDfE/8xfWwlMDhwfpNdQHOMJiq5QYmo6t6Nl2MPxmwv4MBF cbb+EBrCfdkosA9/wiKF4C+xN5+RObM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=lt4B3Gv3OCrw8JiFM9sdRNcEjetfZHNbsKR6G5cmR7c=; b=XO6EX7P+NtiVKs/5+ZXtYZ9FW85XQFrocQ2sEIx08OnMx7rihSW5xiFAg1BZACONHs9RtQ 0OlFA81lEmESAjfYJxJ1VEk7T4KjEIJorN/qsc50LCvaKVsaYQ0D6H7I+w2nkp+5QGdUQN UlR872iEOvEbnlANvQ971d5cRuadPvA= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-353-dlf1c_dsMhCtKj-fEFFg6A-1; Mon, 10 Feb 2025 19:31:10 -0500 X-MC-Unique: dlf1c_dsMhCtKj-fEFFg6A-1 X-Mimecast-MFC-AGG-ID: dlf1c_dsMhCtKj-fEFFg6A Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9C96C195608C; Tue, 11 Feb 2025 00:31:04 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EFD3B18004A7; Tue, 11 Feb 2025 00:30:52 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 0/9] khugepaged: mTHP support Date: Mon, 10 Feb 2025 17:30:19 -0700 Message-ID: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 23F31C0010 X-Stat-Signature: 56biweod6pjb7pn36ixhz1xywt11433s X-Rspam-User: X-HE-Tag: 1739233873-341879 X-HE-Meta: U2FsdGVkX18lm4NpdIAyuVVOC2cKzds1VDBkiECDWcTACX75UDj8vF40L6xlX5PFZUL4z9TexWvgvY2pe8S3suASlPvXOAlqBUnPcYrggUI5ziE9uRwYoXoWXl7x/7BBAw8TSTZaVNKKMp0OdqBrMQP0AT7yo1iq/Qym+xMG6M1TU4Mc0nO199ApBpucUkxp5m9uNnm7fJA2vtgYO5NR6y2S8jrFYUzj4DIfXjxldvAciwpuZ3WLnCLs/ODw3tYVNn15XNPtfQzp0fkXiz/mGQr4Re7106yCRq2e9mL9Yy7eIg+RoX4pqsJNz0kX9cgjQmrdUM4fnYuxcvixiDM81rRIxyb6Tvd953QbDgsh07ZEe8SN84VnXFSLyrqUqG/UNw7HwgIduxII1lTuJibhVKR+L2tofQaThdlbxa7Ar8CIJSqkrW12pb+pFPLOx8kGVOHrJTA8ExHh2/iD6eNqx/jl9LL8XPhHFOQpQFT7jmkLvBTRMpq0B0SgzapC+eBm8GOzhYqd7U6Zovre9bghb43HhprTtyON23JI1pc3UVZGRi1UmFFtIJD2ubMwe3/bEjNZSxPZY3jQXGvBP0SIHbNPkLCPjb2qXOHx12T15ozWUrs1QPvzQf8qWVYlSgq8Voh82icQithSqySgbT+mq9emNIsUKtaDyzoJmblHG/yYnmR/xiXWtNbBgynh2lAWfOibp4J+IBp5gdYuyfa/o+UOCz/rIuTpZ5WRDqMJSCExROne8zGT4aLZkGMVMSY5UcDWA+fwDxC79+u4rMVLaB7FAudmWZ4tE8KiBBDqy4XZI6mboWJFNCckWi28qCxD/9DqW7srjHMuOs56ne3wj6zkZeI1BmE2O90N1eE4yiZvhun7XydWeyE+QppXi9IArDvlaHD0883kDbF0MSmhq28Zw80KVFHgFGc4Vz1oOxY3slIHzVdRKhp9i5NnQE/AxM9iwgXjIaa8jtvNL6w 8RDfJ1Ga AyHODb3tAuO6EWExXyhUOkDLygQpnA/yWY7yAxbGHiSqnvIWJYsaiXFC4KAaE6hYqOKUiN+hBOtcksqcYsvooueCFeArxNJEiGnd0Pv+BSrCqhatYENwPf/gczvdQiN1euW+VcNGq+RLSq2RsTOpkaixpGPvxe/Pt5TLdWCDW3CaVSXV1FtBabs3dV0PjhOC04L9kL+G6DGB5SuwGO/2wowCRjubzRTcWS6hxNm/6kmggUN/Ug9oEn+pAzNQbxcUO7EhN+FywIGygAkKndUfRyg9JezMmjeSQepws8creUJ9DFZCvA67idiIF6sBVYyoL8UesvlfxwID68pDPAd8F8/eN369vxlkL3N5ZSTZdMD3LWY5plRt9RkWgJu9SGWSn2EhQLtwixfxSEhRvMXbpcbofelgpH6koxz3zlbhxDDMH643mD5xe/QCqN5+gECdVPYmP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The following series provides khugepaged and madvise collapse with the capability to collapse regions to mTHPs. To achieve this we generalize the khugepaged functions to no longer depend on PMD_ORDER. Then during the PMD scan, we keep track of chunks of pages (defined by MTHP_MIN_ORDER) that are utilized. This info is tracked using a bitmap. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction on max_ptes_none is removed during the scan, to make sure we account for the whole PMD range. max_ptes_none will be scaled by the attempted collapse order to determine how full a THP must be to be eligible. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we dont perform the collapse. With the default max_ptes_none=511, the code should keep its most of its original behavior. To exercise mTHP collapse we need to set max_ptes_none<=255. With max_ptes_none > HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. Patch 1: Some refactoring to combine madvise_collapse and khugepaged Patch 2: Refactor/rename hpage_collapse Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-9: The mTHP patches --------- Testing --------- - Built for x86_64, aarch64, ppc64le, and s390x - selftests mm - I created a test script that I used to push khugepaged to its limits while monitoring a number of stats and tracepoints. The code is available here[1] (Run in legacy mode for these changes and set mthp sizes to inherit) The summary from my testings was that there was no significant regression noticed through this test. In some cases my changes had better collapse latencies, and was able to scan more pages in the same amount of time/work, but for the most part the results were consistant. - redis testing. I tested these changes along with my defer changes (see followup post for more details). - some basic testing on 64k page size. - lots of general use. These changes have been running in my VM for some time. Changes since V1 [2]: - Minor bug fixes discovered during review and testing - removed dynamic allocations for bitmaps, and made them stack based - Adjusted bitmap offset from u8 to u16 to support 64k pagesize. - Updated trace events to include collapsing order info. - Scaled max_ptes_none by order rather than scaling to a 0-100 scale. - No longer require a chunk to be fully utilized before setting the bit. Use the same max_ptes_none scaling principle to achieve this. - Skip mTHP collapse that requires swapin or shared handling. This helps prevent some of the "creep" that was discovered in v1. [1] - https://gitlab.com/npache/khugepaged_mthp_test [2] - https://lore.kernel.org/lkml/20250108233128.14484-1-npache@redhat.com/ Nico Pache (9): introduce khugepaged_collapse_single_pmd to unify khugepaged and madvise_collapse khugepaged: rename hpage_collapse_* to khugepaged_* khugepaged: generalize hugepage_vma_revalidate for mTHP support khugepaged: generalize alloc_charge_folio for mTHP support khugepaged: generalize __collapse_huge_page_* for mTHP support khugepaged: introduce khugepaged_scan_bitmap for mTHP support khugepaged: add mTHP support khugepaged: improve tracepoints for mTHP orders khugepaged: skip collapsing mTHP to smaller orders include/linux/khugepaged.h | 4 + include/trace/events/huge_memory.h | 34 ++- mm/khugepaged.c | 422 +++++++++++++++++++---------- 3 files changed, 306 insertions(+), 154 deletions(-)