From patchwork Tue Feb 11 00:30:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29EBFC02198 for ; Tue, 11 Feb 2025 00:31:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92ABD28000A; Mon, 10 Feb 2025 19:31:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DA47280008; Mon, 10 Feb 2025 19:31:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72C9928000A; Mon, 10 Feb 2025 19:31:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 51962280008 for ; Mon, 10 Feb 2025 19:31:26 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A422043F72 for ; Tue, 11 Feb 2025 00:31:25 +0000 (UTC) X-FDA: 83105784930.28.5ABEC0A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id CDBCAC0007 for ; Tue, 11 Feb 2025 00:31:23 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=S7bQXmR6; spf=pass (imf22.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233883; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SA9DDRGRHuqABtbetLeril2muYd34wgOB7Djw56jQ5Y=; b=JCo8FIA4IffVSo8+1CE4tqYIZe3db+v3GI2LhKp4Ajc7T6vOXzSxv2cMhV45BghJVLh4pt 4Gl62qUuQ2VN9DzVwdWJ+a5KVFBlRFxj4Zf0TFZJtiX1u535H2+oCNS0uSLMeWaE64fBnq jTvMGeU9IwjaAqvMf0/yP1FyVbmSOBo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233883; a=rsa-sha256; cv=none; b=lZpQLl6FIiQx3U+J4owR6/VZsi8r+JGSOfqw34m71MBEY2kCLKOxwilhsVWF1X5d2qEQcR Fjmt6qvJZkDV5yr+p2xjghawTImH1PFxOOo9b1DQJA5rIJuxKMg+X6uni+Dku1h2qGM/fI T7XiJ8J3boLOXyYUqmowb4pPv8OYYP8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=S7bQXmR6; spf=pass (imf22.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233883; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SA9DDRGRHuqABtbetLeril2muYd34wgOB7Djw56jQ5Y=; b=S7bQXmR6D8EEmNMYivAQiMsfdjCYhaKvTF5jOzLm7N9UFV3yQAs+t8+dJGBV+hqWG8jVoC DX0pBvC43AkdKjCG8dVsgHr5hTHmlxnvYqTmvhxDl8x1dYT3j/qQjohl1kTqdzy4dN8ycP L2nj4PJ/ZUn0EnaUtvfsFmlGOrh+eII= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-646-E4IQmJcXNUSEkjimazaj8w-1; Mon, 10 Feb 2025 19:31:19 -0500 X-MC-Unique: E4IQmJcXNUSEkjimazaj8w-1 X-Mimecast-MFC-AGG-ID: E4IQmJcXNUSEkjimazaj8w Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 93D131800874; Tue, 11 Feb 2025 00:31:14 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EA485180035E; Tue, 11 Feb 2025 00:31:04 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 1/9] introduce khugepaged_collapse_single_pmd to unify khugepaged and madvise_collapse Date: Mon, 10 Feb 2025 17:30:20 -0700 Message-ID: <20250211003028.213461-2-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Queue-Id: CDBCAC0007 X-Rspamd-Server: rspam07 X-Stat-Signature: stj1baofca5r8kya8jfsx7r1t6wh9oyg X-HE-Tag: 1739233883-482581 X-HE-Meta: U2FsdGVkX198Bh5DCOqgg12TatDmoZdVl7uMntqyi5yi9Fh+x8LQxaWU7FO+6SzAl0yuaM57fgcZhv0PLmVxtFi3LSscAGsfQiDzskoGt2QBCoLJHTnFPb27IfoztpOwkeb4hfeprd+b3iPzX4CrataOGkpY80zm0eDHSWutxgO6qrmRvRKDCV7lEPCHu9DWJy37MGXsHhKoiFSz1wjV4kzAm/L6qZwCRMlwRzDFZG9N1M1/NHwX/g+eWKIol5tQFNNwGOCDC6jXjDzK8ptYUXfAXhQFqh/A07CjXM5zu/p6C1kVCsMoozxG4DnndmgZTxRk8OGs9kdhN0NHzJUdrpVHYM13CBJaQXShLptCHs02JgBE3MaO64DS64bZsGTTwn/ffrOjZPzv5uWxedFVCAarjQh9/7tspFmVVTtw3ZUI1z4AZM+Hl3BPzdWNepn0hKSukI0d1ktde63XXpACMM6f8V3fTnUJeZkFv1QXPacPX/k9RdmHUKgHrtOtQ0KxNjdElZK+jYffPZoiOfCUi8UpK9hmbhRHuoZsMv2t0vhJpXP9Mx3RkplyPMZ4U118a1iIjGyteCV5wj2q6a596tBaGE1DpSxc89c7+0QYFqKXaSr5WljEFCzGHpok8pWy2ejiNeQcYoOvr6QQ1mYUylF6lTSHUnCltZcdyBqWzrvbLtFIjxCPnJ5+OGfck3fKiZpviAlDas7JQejOwLcjbUqPrWxKIwiIr0s3DKdTaexFr9PXTXI/ZSP4OWMynWo8M33VGeBxcoW2i6vjtpC/EqW+Ql0xUqCBtPwoY1R1OPfw9sTXywcA2BqXmM2VGAZ0aPgMOoWS8VVFCu6Gtu/zrcxMy7NuaRNMPVMoz0WoGKsMNMt9p94ONR2QRCeevdkVe5Mw7S5J8MM7LnE8WwpdF0+dk78rz4mpxq4F8NkjUUNHH10vglgF0yEmHg79SnXF8q1NAtdnvZpxajfrHXt KTciOFrO gT2h08W+/WlzqelPAGKrr+KwtVPr4P0HuTC3K2TmwOPaVCgtIi93njO1SFBbOcEa3Tp7aXW3fst1JQrsrHezcClpPJfeyxudR8cqREICfVBrEVmueO96rzcU0LdAwgr/ApmsjXP9ZTFP7B5FkMHIUnjbUZfSZIL94pgzmnSyTC7rYBqPV/IPmUXNPwx5TDmmHp+74grS8/+YBkxXwgoPKB9d+w6qij1Gk3L1YRDaaKN5HP6JBZ0sERaU/VyYfzwzeV3z7Z/ND+YiP3IQyxs+riZacXaLSUyjAyZE3YEdFeZeCP45vvIGLuVuSUIyhzAthU8hCqBO8QpQ5lyl1Fxm/2lA0n2/S21wIiwevUmb9P0SAXqZeFd4hsdh+UkFThl606O8Y0aB8bJxgi8oh9ds5ivb/nh9dHDoqIe6WXd1Y/GG6JH2qKZSmrbyf3vxGIQmaxd2u X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The khugepaged daemon and madvise_collapse have two different implementations that do almost the same thing. Create khugepaged_collapse_single_pmd to increase code reuse and create an entry point for future khugepaged changes. Refactor madvise_collapse and khugepaged_scan_mm_slot to use the new khugepaged_collapse_single_pmd function. Signed-off-by: Nico Pache --- mm/khugepaged.c | 96 +++++++++++++++++++++++++------------------------ 1 file changed, 50 insertions(+), 46 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5f0be134141e..46faee67378b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2365,6 +2365,52 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } #endif +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static int khugepaged_collapse_single_pmd(unsigned long addr, struct mm_struct *mm, + struct vm_area_struct *vma, bool *mmap_locked, + struct collapse_control *cc) +{ + int result = SCAN_FAIL; + unsigned long tva_flags = cc->is_khugepaged ? TVA_ENFORCE_SYSFS : 0; + + if (!*mmap_locked) { + mmap_read_lock(mm); + *mmap_locked = true; + } + + if (thp_vma_allowable_order(vma, vma->vm_flags, + tva_flags, PMD_ORDER)) { + if (IS_ENABLED(CONFIG_SHMEM) && !vma_is_anonymous(vma)) { + struct file *file = get_file(vma->vm_file); + pgoff_t pgoff = linear_page_index(vma, addr); + + mmap_read_unlock(mm); + *mmap_locked = false; + result = hpage_collapse_scan_file(mm, addr, file, pgoff, + cc); + fput(file); + if (result == SCAN_PTE_MAPPED_HUGEPAGE) { + mmap_read_lock(mm); + if (hpage_collapse_test_exit_or_disable(mm)) + goto end; + result = collapse_pte_mapped_thp(mm, addr, + !cc->is_khugepaged); + mmap_read_unlock(mm); + } + } else { + result = hpage_collapse_scan_pmd(mm, vma, addr, + mmap_locked, cc); + } + if (result == SCAN_SUCCEED || result == SCAN_PMD_MAPPED) + ++khugepaged_pages_collapsed; + } +end: + return result; +} + static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2439,33 +2485,9 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, VM_BUG_ON(khugepaged_scan.address < hstart || khugepaged_scan.address + HPAGE_PMD_SIZE > hend); - if (IS_ENABLED(CONFIG_SHMEM) && !vma_is_anonymous(vma)) { - struct file *file = get_file(vma->vm_file); - pgoff_t pgoff = linear_page_index(vma, - khugepaged_scan.address); - mmap_read_unlock(mm); - mmap_locked = false; - *result = hpage_collapse_scan_file(mm, - khugepaged_scan.address, file, pgoff, cc); - fput(file); - if (*result == SCAN_PTE_MAPPED_HUGEPAGE) { - mmap_read_lock(mm); - if (hpage_collapse_test_exit_or_disable(mm)) - goto breakouterloop; - *result = collapse_pte_mapped_thp(mm, - khugepaged_scan.address, false); - if (*result == SCAN_PMD_MAPPED) - *result = SCAN_SUCCEED; - mmap_read_unlock(mm); - } - } else { - *result = hpage_collapse_scan_pmd(mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - - if (*result == SCAN_SUCCEED) - ++khugepaged_pages_collapsed; + *result = khugepaged_collapse_single_pmd(khugepaged_scan.address, + mm, vma, &mmap_locked, cc); /* move to next address */ khugepaged_scan.address += HPAGE_PMD_SIZE; @@ -2785,36 +2807,18 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, mmap_assert_locked(mm); memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - if (IS_ENABLED(CONFIG_SHMEM) && !vma_is_anonymous(vma)) { - struct file *file = get_file(vma->vm_file); - pgoff_t pgoff = linear_page_index(vma, addr); - mmap_read_unlock(mm); - mmap_locked = false; - result = hpage_collapse_scan_file(mm, addr, file, pgoff, - cc); - fput(file); - } else { - result = hpage_collapse_scan_pmd(mm, vma, addr, - &mmap_locked, cc); - } + result = khugepaged_collapse_single_pmd(addr, mm, vma, &mmap_locked, cc); + if (!mmap_locked) *prev = NULL; /* Tell caller we dropped mmap_lock */ -handle_result: switch (result) { case SCAN_SUCCEED: case SCAN_PMD_MAPPED: ++thps; break; case SCAN_PTE_MAPPED_HUGEPAGE: - BUG_ON(mmap_locked); - BUG_ON(*prev); - mmap_read_lock(mm); - result = collapse_pte_mapped_thp(mm, addr, true); - mmap_read_unlock(mm); - goto handle_result; - /* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL: case SCAN_PTE_NON_PRESENT: case SCAN_PTE_UFFD_WP: From patchwork Tue Feb 11 00:30:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B8CEC02198 for ; Tue, 11 Feb 2025 00:31:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF68A28000B; Mon, 10 Feb 2025 19:31:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B99D4280008; Mon, 10 Feb 2025 19:31:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A13BC28000B; Mon, 10 Feb 2025 19:31:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7FF22280008 for ; Mon, 10 Feb 2025 19:31:38 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 35E041408A8 for ; Tue, 11 Feb 2025 00:31:38 +0000 (UTC) X-FDA: 83105785476.24.CE4F314 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 6852140005 for ; Tue, 11 Feb 2025 00:31:36 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VQo73pSu; spf=pass (imf12.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sgDjeCsWqrbzoq3HtDwQTNQUeXq83w6g2bNTg/9HnMA=; b=ALlaP1W17YUoCovY4WH4hcOpJXEH0vQ8/hLl03IYau3cJ9KlaigCob0YOtWPlSTeszRUWG jvk6cZMHz9sNL+FcWPd+MoiIb7KP42MNEVvnhDD4LIIoxeU0JRzOEr7vj+cfs2cFww1ch7 Pd1QsOL1iA/onr9a1Rfe43TPbchtLpw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VQo73pSu; spf=pass (imf12.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233896; a=rsa-sha256; cv=none; b=ivhJX9RagMyQ9LqxzBohJdprk4P/RPs/8BLpCm8CB6CMxRai52pAjQq9v2Biu/W/cEibRA royfu91rOKZibaEfyrTq/aD2CvN8MDQXAfjKHNEeZrXcOCUGq8NSUbHxD9ILhHeC0pLWbQ 3+SsTSn3AzYIAw40uV7KD20lhhiTkjQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sgDjeCsWqrbzoq3HtDwQTNQUeXq83w6g2bNTg/9HnMA=; b=VQo73pSufpQPVBKJM/t/ol7V+37Rtw3GBuT9WY2buQ0C1qAXY+q0RSo8Iii+5odKx/W0m6 DQgh2WZ9SpevRtosnVxnZPw92f7LfXnOoYMmU9bTFUojaoJoU0FuSHbeK094of7+muBTS1 DJz7WLktM5Xl/LNT8uonOaTs+29azyU= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-168-J1BlQCIqOI2vd53uE_jn0w-1; Mon, 10 Feb 2025 19:31:30 -0500 X-MC-Unique: J1BlQCIqOI2vd53uE_jn0w-1 X-Mimecast-MFC-AGG-ID: J1BlQCIqOI2vd53uE_jn0w Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 020FE19560B7; Tue, 11 Feb 2025 00:31:25 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E24981800358; Tue, 11 Feb 2025 00:31:14 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 2/9] khugepaged: rename hpage_collapse_* to khugepaged_* Date: Mon, 10 Feb 2025 17:30:21 -0700 Message-ID: <20250211003028.213461-3-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 6852140005 X-Stat-Signature: doiceicezatr4qtnzez7opnf1u6pqaew X-HE-Tag: 1739233896-471979 X-HE-Meta: U2FsdGVkX1+QAdWyfka/zN9QC2BbtWbGNiCMmUQ+UJiS11iFy7OfjgOmbbxV+Z3XIGK58MV8B1atE823WlP/3N62JUC9wndL0e+TvRQ9PuYT0487Ks5DYcaYqqrmUAa9IJqQY7f5crjuJfBact5wuDJC/yzMutI+1+8jiZ6Ln2PasqOBUoedTaX5EED9NAgDNx3Ff5uGa6n7xUjmKXo5E6qT8ElazDy3eIvt0/WWrwNJSkyiRcSOyMT7+OW4Rp9/GzM/KecP2SDNVo+dj9MW/+hV4uEOpdDRjgthaXiUa3EeDbIfWvwu9Ywus6CQMMO3KtHl1gdi6qsIQCrSC2nQ+VXyWsDtoFQOxv6bfUWGd85MkH3vbXeQcI9qCm0fPGfl92fGpo26aFdXDJuFyfV7q1e4UEqpZKSaLCX/CmVAfiSa4uScoMFj/J9BCs4y7fHsg5BxJO5EXD//ql0Yf55u9Ll++A7vlUUBvFYYga+Ph0IcDwmqX0cn3zWou6mb+ZBqjODRp3h586/lsYXvgzcJ4k49XAYyGjrXR4JB7r8bhcD1PUJJF0NAxuaD/s1ZQv1OBnFOOA0c36fyJqhnaH+REQOr8zUJAaR6cIgIhQOMwoFQ7zuEjSHdvXFIl3ZHZFyMWIOhGjW8plRW0Jay7E7OB9L52YWcuWUJ727t4e/IWx+InLgcWEmY06lc+L+pvBzuvVkVdZV6lomYevfce6MmWigaltr8H5zWExrWjP23kXf1NEgLtTzF31xSENx9/49CP7cNpza8HkBLPnWFqiVRl52pUMeG5sZxk0O2GJIOpC8/YY8YGQHKuOqzkfwLxWj6QaSIFGVBJdFXpKHrP1O3aipqP0uqCrLBPBdkGoU1y58mJqww0o+1wEm+Nu/wSudbCLBMybtAAjgWwrRujh+XCE+7xteQ0G4/UDjPFtY4YtKJfxPArB0vbeaz2SZy0Qw944RyEmc6B5qOzYbGGA+ ggOTyK9S IA1Cz/Q4+mMLVOojThqBy99+opbHdMiRP7yngTpB1MHcBCDBUMcuUWxKlft2ekH9Aqpr9tvwJh8+rzwC4J7XKjMtpmxW0Nn2yAODJ4aEzm1+YwccFOpgs7mwhaRoBhuZOPoYa3Vz+/bEWa0B8Q5g61UyykiEwy1MiN4iZ7NZSLtbdMyQ0DGIf5EUKjLJihvqBvhIK3WwqCnwk0w8PC5D0n3++0BrJDuqplXaHANH+2BBmBa2UR510bB6DSO8Pfggu7jOGglFV7b0AVAruk0lxniMKXjX5Ckv1Bp//ugJqxwvObIDAh6jhabyVEeaw6J7XyTdRb/WNgPNCK07xpQaeCBrQ33Cmesmx2MekcWY6oLhP9VM9kpfKcCs1LoVvdBoVmJageq/8VK39j032ty9QaQ5SeDH8FFQlAhhIoPQitBHQT9K3soP5iVaSlieS8yYk5rp2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: functions in khugepaged.c use a mix of hpage_collapse and khugepaged as the function prefix. rename all of them to khugepaged to keep things consistent and slightly shorten the function names. Signed-off-by: Nico Pache Reviewed-by: Ryan Roberts --- mm/khugepaged.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 46faee67378b..4c88d17250f4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -402,14 +402,14 @@ void __init khugepaged_destroy(void) kmem_cache_destroy(mm_slot_cache); } -static inline int hpage_collapse_test_exit(struct mm_struct *mm) +static inline int khugepaged_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) == 0; } -static inline int hpage_collapse_test_exit_or_disable(struct mm_struct *mm) +static inline int khugepaged_test_exit_or_disable(struct mm_struct *mm) { - return hpage_collapse_test_exit(mm) || + return khugepaged_test_exit(mm) || test_bit(MMF_DISABLE_THP, &mm->flags); } @@ -444,7 +444,7 @@ void __khugepaged_enter(struct mm_struct *mm) int wakeup; /* __khugepaged_exit() must not run from under us */ - VM_BUG_ON_MM(hpage_collapse_test_exit(mm), mm); + VM_BUG_ON_MM(khugepaged_test_exit(mm), mm); if (unlikely(test_and_set_bit(MMF_VM_HUGEPAGE, &mm->flags))) return; @@ -503,7 +503,7 @@ void __khugepaged_exit(struct mm_struct *mm) } else if (mm_slot) { /* * This is required to serialize against - * hpage_collapse_test_exit() (which is guaranteed to run + * khugepaged_test_exit() (which is guaranteed to run * under mmap sem read mode). Stop here (after we return all * pagetables will be destroyed) until khugepaged has finished * working on the pagetables under the mmap_lock. @@ -606,7 +606,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); - /* See hpage_collapse_scan_pmd(). */ + /* See khugepaged_scan_pmd(). */ if (folio_likely_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && @@ -851,7 +851,7 @@ struct collapse_control khugepaged_collapse_control = { .is_khugepaged = true, }; -static bool hpage_collapse_scan_abort(int nid, struct collapse_control *cc) +static bool khugepaged_scan_abort(int nid, struct collapse_control *cc) { int i; @@ -886,7 +886,7 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) } #ifdef CONFIG_NUMA -static int hpage_collapse_find_target_node(struct collapse_control *cc) +static int khugepaged_find_target_node(struct collapse_control *cc) { int nid, target_node = 0, max_value = 0; @@ -905,7 +905,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) return target_node; } #else -static int hpage_collapse_find_target_node(struct collapse_control *cc) +static int khugepaged_find_target_node(struct collapse_control *cc) { return 0; } @@ -925,7 +925,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, struct vm_area_struct *vma; unsigned long tva_flags = cc->is_khugepaged ? TVA_ENFORCE_SYSFS : 0; - if (unlikely(hpage_collapse_test_exit_or_disable(mm))) + if (unlikely(khugepaged_test_exit_or_disable(mm))) return SCAN_ANY_PROCESS; *vmap = vma = find_vma(mm, address); @@ -992,7 +992,7 @@ static int check_pmd_still_valid(struct mm_struct *mm, /* * Bring missing pages in from swap, to complete THP collapse. - * Only done if hpage_collapse_scan_pmd believes it is worthwhile. + * Only done if khugepaged_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. @@ -1078,7 +1078,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, { gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : GFP_TRANSHUGE); - int node = hpage_collapse_find_target_node(cc); + int node = khugepaged_find_target_node(cc); struct folio *folio; folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, &cc->alloc_nmask); @@ -1264,7 +1264,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, return result; } -static int hpage_collapse_scan_pmd(struct mm_struct *mm, +static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked, struct collapse_control *cc) @@ -1380,7 +1380,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * hit record. */ node = folio_nid(folio); - if (hpage_collapse_scan_abort(node, cc)) { + if (khugepaged_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; goto out_unmap; } @@ -1449,7 +1449,7 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot) lockdep_assert_held(&khugepaged_mm_lock); - if (hpage_collapse_test_exit(mm)) { + if (khugepaged_test_exit(mm)) { /* free mm_slot */ hash_del(&slot->hash); list_del(&slot->mm_node); @@ -1744,7 +1744,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) if (find_pmd_or_thp_or_none(mm, addr, &pmd) != SCAN_SUCCEED) continue; - if (hpage_collapse_test_exit(mm)) + if (khugepaged_test_exit(mm)) continue; /* * When a vma is registered with uffd-wp, we cannot recycle @@ -2266,7 +2266,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, return result; } -static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, +static int khugepaged_scan_file(struct mm_struct *mm, unsigned long addr, struct file *file, pgoff_t start, struct collapse_control *cc) { @@ -2311,7 +2311,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } node = folio_nid(folio); - if (hpage_collapse_scan_abort(node, cc)) { + if (khugepaged_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; break; } @@ -2357,7 +2357,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, return result; } #else -static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, +static int khugepaged_scan_file(struct mm_struct *mm, unsigned long addr, struct file *file, pgoff_t start, struct collapse_control *cc) { @@ -2389,19 +2389,19 @@ static int khugepaged_collapse_single_pmd(unsigned long addr, struct mm_struct * mmap_read_unlock(mm); *mmap_locked = false; - result = hpage_collapse_scan_file(mm, addr, file, pgoff, + result = khugepaged_scan_file(mm, addr, file, pgoff, cc); fput(file); if (result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); - if (hpage_collapse_test_exit_or_disable(mm)) + if (khugepaged_test_exit_or_disable(mm)) goto end; result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged); mmap_read_unlock(mm); } } else { - result = hpage_collapse_scan_pmd(mm, vma, addr, + result = khugepaged_scan_pmd(mm, vma, addr, mmap_locked, cc); } if (result == SCAN_SUCCEED || result == SCAN_PMD_MAPPED) @@ -2449,7 +2449,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, goto breakouterloop_mmap_lock; progress++; - if (unlikely(hpage_collapse_test_exit_or_disable(mm))) + if (unlikely(khugepaged_test_exit_or_disable(mm))) goto breakouterloop; vma_iter_init(&vmi, mm, khugepaged_scan.address); @@ -2457,7 +2457,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, unsigned long hstart, hend; cond_resched(); - if (unlikely(hpage_collapse_test_exit_or_disable(mm))) { + if (unlikely(khugepaged_test_exit_or_disable(mm))) { progress++; break; } @@ -2479,7 +2479,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, bool mmap_locked = true; cond_resched(); - if (unlikely(hpage_collapse_test_exit_or_disable(mm))) + if (unlikely(khugepaged_test_exit_or_disable(mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart || @@ -2515,7 +2515,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, * Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ - if (hpage_collapse_test_exit(mm) || !vma) { + if (khugepaged_test_exit(mm) || !vma) { /* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find From patchwork Tue Feb 11 00:30:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B536C02198 for ; Tue, 11 Feb 2025 00:31:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07A2728000C; Mon, 10 Feb 2025 19:31:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 02947280008; Mon, 10 Feb 2025 19:31:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE5E428000C; Mon, 10 Feb 2025 19:31:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BFB39280008 for ; Mon, 10 Feb 2025 19:31:45 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1B9F8C08AA for ; Tue, 11 Feb 2025 00:31:45 +0000 (UTC) X-FDA: 83105785770.25.5F4476A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 3E5D540006 for ; Tue, 11 Feb 2025 00:31:43 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KQdZdTio; spf=pass (imf07.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rzjDVzZllkv1t501JGBPBZlzucpspDJXSDj68ZE8hcQ=; b=cFLT4YfZnMG59XlYRbrwht6+qJqFcs7P6shDmrBGXyfuU8R100+N+RHNx5y0LeJKuB3NyL XA2I8zn0Wp/CLS5UsOrLLadIcj4Pa2Gq63hrJXSqfAETvzSirNEDMKFxOzArQdwmtL6Zol qWhPg6Fcv/KqkYePRoo1U839chpycVs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KQdZdTio; spf=pass (imf07.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233903; a=rsa-sha256; cv=none; b=LlE23gVN3v0UnLzo2ljl44bfriHi1m+C8VDcO6aU6v4mABRzMYSJ2pb4Gep+UtITUzZa1H nQX6PwRzwokXeRAFxLQjXrgeRBX+ZxPBVjJWOnTtlR81pBXlFJZNSHcHwFfNxPBwpgTihm 48eEnoVyOCdISI9ganfDnKxOutMg0Dc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233902; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rzjDVzZllkv1t501JGBPBZlzucpspDJXSDj68ZE8hcQ=; b=KQdZdTioTInLOA5xotaReWGAL432ABFSmqxvD0YHsKEdZ6C/fxXgONgiY3zcjESvFCSF71 VticZsobbvhVEC+mH3/xWBlG4Vveh0UOxzaeCwPCwOfRvgsH6ZDgjo8kg3vcd438B12BO/ tfsb9oGM9me6xfxrqdiEwgOxUiUIGME= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-488-OM9rMKLZPIuLmOGLavR4PQ-1; Mon, 10 Feb 2025 19:31:40 -0500 X-MC-Unique: OM9rMKLZPIuLmOGLavR4PQ-1 X-Mimecast-MFC-AGG-ID: OM9rMKLZPIuLmOGLavR4PQ Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 735BC1800873; Tue, 11 Feb 2025 00:31:35 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4FABA180035E; Tue, 11 Feb 2025 00:31:25 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 3/9] khugepaged: generalize hugepage_vma_revalidate for mTHP support Date: Mon, 10 Feb 2025 17:30:22 -0700 Message-ID: <20250211003028.213461-4-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3E5D540006 X-Stat-Signature: gfqehiaae9skp1sa5nefzmdfbhrbnks8 X-Rspam-User: X-HE-Tag: 1739233903-577433 X-HE-Meta: U2FsdGVkX1+cfEduzmvhoEUuKni+0br0YXk5EmRHdOjD6oDum30zkpnFCDjHpDVQGB8mIpQKP4YNp+7V20p7vtbfQ8Nbci+KVyyD4LGoDxhvbCN+WIFxlKMvMo7Sq8uD0e/6F7IV54p67xqJQDaF2aaCnXEzYvrWmn/4mbvhglw2O9hlclmbbGxxNvIi4Fs2bZnX31Jjol7+wXedRkFj6uCEAHEwRyZSHz4ARllH9LhkkgaAhO8O/ccBAhljn5NY4RyYTZ1/klybrVGznDKYFLhtSojjNZbMM/Y4kngVpro36BO5t/401MC4kQcM6MC/H558otWk/wKnVJ/peKkd0OMwSeyCu5pTJnXJgL8UKj3rri0b+67aypj3ap8EeLqZryRt5B+HCiQ3nY+JARSBRdhH2DcL/uOGLMAvWUwWV2h5RHd4UrPVACD0K14ZBMWm8NbwWKjMKoYXwZhnPnDx0K3SzQDwGm8GGjpT2Hc0ZSik3BwetmaA74gwNup5pDL96bRa0Gn5XSARGcuGFI1v2qDP9xz0dMDry9Ktkh7JHQxpaSAOBy5qu5sMUAd0m6DjwGeQTuTWtoELNbi7S7BfP60H0k56PPGM2/RJzoiLZUHX7b5vgCDCT63Dn51NHx3yG/Wr+xe22QGcioGdYkf3XsUAnK362DpEDjq/AGGZI2fxrBpaCbYgV5nZIf0CCEMCdGq/AYRTpDkfPK+Dkm875ybAavLHaFzPbRlbLW+8gnR5aokemuy3M4PGy4dMnERf2lENve21v7p9jnGgVPSaWArh+KxVVkqhbTge6P06WXxIaFkgEULg4fUgwh4Zt3mUplepu2UwhHXMrlLN536fodOwk2KAKEkoAW4LyIRIHNF4WA31FrXdvMd6DkKsajYpKRkx8WOGRT0WDSbd+LFSHeRNmJS4azg8CCc60hYIgUBeGUD3YpllvSZZjeZSCMkuqAFphaUgMFFJ/a72T0S qsAs5Yl1 d0Z3z9rptkroD8xRdga0/0YMOq5Ad91jQQXL3wZd7rV82m7PmYBbBgDlRhV7CiAlmovcvcvdikGJ/Z3fN6V12DUThyrMnjB7IQndyi0LsJ4bR3zt9ZZ3gVQyaSlp3QjgZeWb+BYqVIe3pYiX0avGwkh59hfCrFyxzcds4gkeQcFsMNlL80odBP5tgRPKtAvLuL4zLS2LVog4rDLS4FvdPdzCcUq98ffgEQZ0klpzHseFs1WBM1i5E2T7/h0XqiJ+TzKqAQvqrLXrRwZZZPxKXXrzghxsXGykF098dFt7Gf+qdScoFN4o8N2YjJeBx2sxjiynSYIW2jD9WRbwcBT7yYXKzBJ5lw2JTt7RxCB1zWpdBqM7GdcE9t5vzJr4ixCjBEt2GBKcPC5ORs809Sl0nTm0r2Gd+Wew/HO+zc/UtKK+ybN9VKVk0fGWHUJ87pDC5KME4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For khugepaged to support different mTHP orders, we must generalize this function for arbitrary orders. No functional change in this patch. Signed-off-by: Nico Pache --- mm/khugepaged.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4c88d17250f4..c834ea842847 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -920,7 +920,7 @@ static int khugepaged_find_target_node(struct collapse_control *cc) static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, bool expect_anon, struct vm_area_struct **vmap, - struct collapse_control *cc) + struct collapse_control *cc, int order) { struct vm_area_struct *vma; unsigned long tva_flags = cc->is_khugepaged ? TVA_ENFORCE_SYSFS : 0; @@ -932,9 +932,9 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!vma) return SCAN_VMA_NULL; - if (!thp_vma_suitable_order(vma, address, PMD_ORDER)) + if (!thp_vma_suitable_order(vma, address, order)) return SCAN_ADDRESS_RANGE; - if (!thp_vma_allowable_order(vma, vma->vm_flags, tva_flags, PMD_ORDER)) + if (!thp_vma_allowable_order(vma, vma->vm_flags, tva_flags, order)) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1130,7 +1130,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock; mmap_read_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc); + result = hugepage_vma_revalidate(mm, address, true, &vma, cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) { mmap_read_unlock(mm); goto out_nolock; @@ -1164,7 +1164,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * mmap_lock. */ mmap_write_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc); + result = hugepage_vma_revalidate(mm, address, true, &vma, cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) goto out_up_write; /* check if the pmd is still valid */ @@ -2796,7 +2796,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, mmap_read_lock(mm); mmap_locked = true; result = hugepage_vma_revalidate(mm, addr, false, &vma, - cc); + cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) { last_fail = result; goto out_nolock; From patchwork Tue Feb 11 00:30:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0E86C02198 for ; Tue, 11 Feb 2025 00:31:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D38028000D; Mon, 10 Feb 2025 19:31:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3834F280008; Mon, 10 Feb 2025 19:31:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2245328000D; Mon, 10 Feb 2025 19:31:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0BBE0280008 for ; Mon, 10 Feb 2025 19:31:58 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C03631C7C12 for ; Tue, 11 Feb 2025 00:31:57 +0000 (UTC) X-FDA: 83105786274.11.469BFCC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 789C3C001D for ; Tue, 11 Feb 2025 00:31:54 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fe6uQ0NQ; spf=pass (imf10.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233914; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h0sopSeKMteOebrJDoWD56LIp+P2qMN45Do9CiZPN14=; b=Z7eQMRVMwMswG4EN0PpwuLbHu2Pibk+5Z9WaE86IcLUWdMcyhLW9DZIK+5YHd0ojXwPXcF OB/d+xtsDQdCXf96L4gc7IbA4kl5c9YFdI7LtWtIcS3cf+BFmGttYqHgZRYgWjacByCaUc h7HFCzlXrphjlXIeWSsl9WKYN61AD2g= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fe6uQ0NQ; spf=pass (imf10.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233914; a=rsa-sha256; cv=none; b=FRG9dQGBsWhyv7vB/SFDbvcxTssuNC8/reAz7QqKbqY8MS3MMzarj76KBNQomONqOcQoiC 7ed4zrUImLd9V0Q3KwnDMHU1+S6RKgGijiyK8sh9NdxV0+wCg2+3JlDUTX9nywL6Zb31Lj h1rc5b/llL4dAAEAMndfarB+k3QvO70= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h0sopSeKMteOebrJDoWD56LIp+P2qMN45Do9CiZPN14=; b=Fe6uQ0NQo7Z9YawQGvNLuga6jlclLK0hb+Ps392qs+P9eUCAxviPnKaZoF0H7MaTJbbdNz tD2Ct9uraFPPRSt1FUR3xac3mBTy8+2NLsnzR6bKmw08Ir2HeTOC8WnD+sK+x3DlWh12sc ZA2Z2fMv+oh3Ywx8QB6q9CUBV/Ponm8= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-480--IkTyaSUNSaX2JxRcwKqPA-1; Mon, 10 Feb 2025 19:31:50 -0500 X-MC-Unique: -IkTyaSUNSaX2JxRcwKqPA-1 X-Mimecast-MFC-AGG-ID: -IkTyaSUNSaX2JxRcwKqPA Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 236541800373; Tue, 11 Feb 2025 00:31:46 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C2B4718004A7; Tue, 11 Feb 2025 00:31:35 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 4/9] khugepaged: generalize alloc_charge_folio for mTHP support Date: Mon, 10 Feb 2025 17:30:23 -0700 Message-ID: <20250211003028.213461-5-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Queue-Id: 789C3C001D X-Stat-Signature: taybcojpu6bi9emp36xo1rxp8ogyte9t X-Rspamd-Server: rspam03 X-HE-Tag: 1739233914-330809 X-HE-Meta: U2FsdGVkX1+1A1hgXQLXKV16UKsqXtykAbblenSwfNWq1t1a4QYhnzIvcRLG0V1s/YyHFVxdjOAjMpSeJlgk3ZLy90Si7C5WlQc83JRqWfURXj9qWqlh1qupHs2PnhCEg+DILw0w66ejgi0AxeztybtDpkkLIzVdVm0FLQenuC92oN0cGdarmgET5dGY4seQ+ddsX+7Aeo+JvMLyDnVJugghCvgnTvR7AV2ays/U7ZlsWBtGcQcVsRQpDH8x4lzFP4/OYRfN7tOpYUWpnMj9847ucHu9ai0sLEyuTGfX74hOE10OOy23T8aO55jHJueWHnzxtUOVKTlMNxzr4LVuybqXo5w7QgjjNBbDE+950uDAwYMnoZszEjtD94VRkzf9or+02UU7azXD/+Lhm0ehjTaaMCPE4mXkl8MkbCx0UIIh1LVrrpFj8Y6gQrz3wJ0gvXiPD2ANbKN7WyYeFPcxpNu3FiuXBvk76Ob9geQQM0/QOok0YbxDhNbGmVTY6YvSXU+m78XN47lDYTOWsxPcLBqyMJ1N36D+70DEidCws5heZPQilTsALdVhrsFrYDldJOqUJfUz/QkyPMsYxnMusd/YOMuSVycXZ1Jb3An4uYUuQlru4cSmYJaoGO2X4+znrNWdDkyg66E7l7jMzrvppFVYMsQrIH2/QtS06O2vKv6GlO6E2cGMx7NsToSvGqj6+1e0QDHRlZq7vrttFIRnFDTvJzk4SLND3NibQz9ZNB4t0p2NvhmFXu/5wzf/kJbFGtHUmcaIDtKMB9Jv/dsuEYgQ6jfvkjn+E0OTV6T1FOhYxhIApnQ/Pzr02nLwoukPrKh8esO5MxljyZo8A20NJbunpv9mQ8Xhwf0I9Gp5g++9KmepXhcsoSdUAL2Fy2KGplXo7sVqHoGLh3CBoYqHu7pO5nX0y2rwVr8bTjX6ns/qAm4Ydm1Q6KWfGHeikQm3ov3g3jkTKNJDtJXgJmn VkRci+1a VmFJNesdXfbzNOO/v+pZPA1+DidYPSNBEHx197sq6NFhMeg+hByujApiXGFhpsYXCjsG40hKEfI+kdueIFeK7H6DVY8N0G3AqlfWD0zRzSXSFmbZtvTqgnJNX3rPQ4cSDbsBcOx7xPNMPaHBRhJa8Mnl11e51F9jVaQEmmMFBKOP5o4tRB9M+/8KLf/ZAkqnzs+ZquFU/Bv5boSsT68eArLMIJfjLOAuBk8ei7+sD4Hy1gcse755RmrbntLWPwUPGbnE/K9h6yo92z7cLvkOJmFma74OeABiCHqyunfU2cNkywHo8d/kyTd/LAbgyhiUYf3j7PSHVD5VTQO2y2oyYqDNXz0Sl3/KDAUluKcm6AjRfVhGqKQYrxz0RtkLTB4JJZ+eNlUCbtomdiC8mhhcUm9JzSPu2RY4Bt7zpeamElZJ3iigBU4oYmmwQwgIbz8robkvb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: alloc_charge_folio allocates the new folio for the khugepaged collapse. Generalize the order of the folio allocations to support future mTHP collapsing. No functional changes in this patch. Signed-off-by: Nico Pache --- mm/khugepaged.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c834ea842847..0cfcdc11cabd 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1074,14 +1074,14 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, } static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, - struct collapse_control *cc) + struct collapse_control *cc, int order) { gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : GFP_TRANSHUGE); int node = khugepaged_find_target_node(cc); struct folio *folio; - folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, &cc->alloc_nmask); + folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask); if (!folio) { *foliop = NULL; count_vm_event(THP_COLLAPSE_ALLOC_FAILED); @@ -1125,7 +1125,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, */ mmap_read_unlock(mm); - result = alloc_charge_folio(&folio, mm, cc); + result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) goto out_nolock; @@ -1851,7 +1851,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); VM_BUG_ON(start & (HPAGE_PMD_NR - 1)); - result = alloc_charge_folio(&new_folio, mm, cc); + result = alloc_charge_folio(&new_folio, mm, cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) goto out; From patchwork Tue Feb 11 00:30:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC0B4C0219D for ; Tue, 11 Feb 2025 00:32:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AD7028000E; Mon, 10 Feb 2025 19:32:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35CC7280008; Mon, 10 Feb 2025 19:32:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FD8F28000E; Mon, 10 Feb 2025 19:32:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 00704280008 for ; Mon, 10 Feb 2025 19:32:05 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AEDC4A081C for ; Tue, 11 Feb 2025 00:32:05 +0000 (UTC) X-FDA: 83105786610.03.627E86B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 059F4C0002 for ; Tue, 11 Feb 2025 00:32:03 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Ow4XB/WT"; spf=pass (imf22.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233924; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8DVGOzKE1Z7tgv2NOONtjQAZbYReHdoD7kuq0sx/74A=; b=7f3GAAhnHtLYrDML6eCloIb7dNX/p3EQeYa9tRSFmdseyEQHwMeRfncuSlu8Wc3/OKg6Vy 0RrP/V4JJLhoYYVTADJQ1RrxW0LAFJPgjm4DTFFlqRqM2YlufV8j3m9trpYMyloUvGUq79 kA0t360j3SV3vob22X+7Pr3OdBFTb7M= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Ow4XB/WT"; spf=pass (imf22.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233924; a=rsa-sha256; cv=none; b=BBJ4YElTOK8lfJlIk4uzYiVNDspMtrjTlxVxkT1Dbz19YyczqPaRjo97NeqtewEsPB42tL kn5InW5fP1i1akRqmQknRCbg3VMMYkZ40qM4Ai6CC8KMl9N2hpNBHOXvnalrHLyDYKzqWy wYtWzLlWSRxwtOLXPi5JmR/6e4f0Dbw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233923; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8DVGOzKE1Z7tgv2NOONtjQAZbYReHdoD7kuq0sx/74A=; b=Ow4XB/WTHtyYw7hSCMf3UmQLWjpc6kUCC0bLz5VVsIOAFHf5G46qPEvES7Bw0+cdaF+lYr vNAasUgA69SV85T0KOUWa2sl66F1KodRX17Fc/XKhS5Q/6hG5RS4aKbY8ce+BDD9khU4NH R80j5yc4bSx5zQRrEuEAEdHm72pDc5c= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-247-ry-06wylPaeLVakMhizfSw-1; Mon, 10 Feb 2025 19:31:59 -0500 X-MC-Unique: ry-06wylPaeLVakMhizfSw-1 X-Mimecast-MFC-AGG-ID: ry-06wylPaeLVakMhizfSw Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 33633195608C; Tue, 11 Feb 2025 00:31:55 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7249818004A7; Tue, 11 Feb 2025 00:31:46 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 5/9] khugepaged: generalize __collapse_huge_page_* for mTHP support Date: Mon, 10 Feb 2025 17:30:24 -0700 Message-ID: <20250211003028.213461-6-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Queue-Id: 059F4C0002 X-Stat-Signature: jbx8ajktdt81ziairisdqnj79wrjncju X-Rspamd-Server: rspam03 X-HE-Tag: 1739233923-610797 X-HE-Meta: U2FsdGVkX1/54dDIKfy2DKaTWvEaDd8O2DGAdXOGbSMVqycj4ScvgXgGhvdUgbWjMgCnQ+es6+Gw0qCSUzGkVXx5gMOyRSz1AfJhedIqLdiLXGBOT2KwfbWVuQIgUaEN80UoR9DeouRWyX8VaoWQHd+B/jje0DrEC+PBWRqBRSiXIPoH87YTVoFw0ENQbBKyCOYCt5QotnvzMW4o6IO0H7RbCYI3t9al8qSU/YByHFnnmqmIT5bUC30fs1xgd14kvGVrQQCNazhV0wweovdVV5QS2FXpOEUUfo7kYIzOqW4Y90HBVHtTN48LQkYBBrrLP07PXl8GNMIq4eZY4z49yDoZnmSyQPTSZF/pu9n4xn2nLa4IAFJhuL/2q/4keJWGMNkDZO5BeG7omZd/iH0gp8UE5Gj/HWGOcQ+2TFpRaT52lekIyUIQwYl0bM+sp8cAcMcF2DmyT5Enb3ZRr8RlbKSqjBUfWadfSByzyAFSimvbGjsfpltbUwjlZkvNs1vQetbuzpAkzSr81k03teT9DCkIjku5+n391yzTdlJJL1KcKQLzju4NL0ps8txJKkcRazZIu/jRPBssEpajWeVTFCoEgjT7KXfmqcz1PDbg5+8ca6yrroPTb82l1GNu0KNugyzroW9JDrLAZvxxX+e0oUqryQpCbsL9x4tApil7L73HGVellTXLOVWC9bGNTrKbvV3ELpvCTNS8sa4UPeftFKOUphJPbRCwVgtjtIcTaoofP5OmocmEpESqr6Jy1ySuXhORS17JUMW4nwgL3DExEvbdvnIuiFSEhK/riJLjpEcirzyqUsM3h1mL2KONq6zcHWONdWoxtWdEscCxAo1VgJRQ7WiQ2nbcD8mk6y2WZXtwXT+fTfBrkzGr7nUzA85lfIjBv9U077t0Zb7MGhlB7PQa/bSr53QKFZT/9SInXTXFlw+g8E8WL3SW5R7YXanNSXn/FkAP1t/rdUVfxve USg/v9+4 umxOl4yS1uGBIGlm7weN1/zw7/U//k8foXQ0n0vG+z4Y1isPrq/NI0TN5kosmjowZPsAzLabzeNkxMagk3Mfw1uqIy5XPlI0LWiSDRCIUeFSMK6vZ7x7ndTi742P0XXBK88bXVY4aD/jhrOjYtSNWkGT4qCcGXNoo+sUO5WdIQ4+/J/7oL7YQ+lFFABgte/C1TYpTC0s8GVhH3dShz5nMc5Xv9ph4a1GZcb7IFmPaS38xETUNUuO0bL37Drq0sj9Y3Coh2nhx70TsQrj39s2ltb633a1UsHKmsjmmWRko0FMXOI+mXcRahy8zAA/f1Q1Qz6NVBi9IXhrr0Dukgftz5EWbDHRFReZd3DVvY5IU/AQVyFrwGKzMdNRh57tCaWbmdTBrixJaaS0iNIGxN3f6ImzR/m/YrDwXk2msWaoFiJNjNvldWdP3kI+qH4bdtKg4E4Ar X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: generalize the order of the __collapse_huge_page_* functions to support future mTHP collapse. mTHP collapse can suffer from incosistant behavior, and memory waste "creep". disable swapin and shared support for mTHP collapse. No functional changes in this patch. Signed-off-by: Nico Pache --- mm/khugepaged.c | 48 ++++++++++++++++++++++++++++-------------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 0cfcdc11cabd..3776055bd477 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -565,15 +565,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, unsigned long address, pte_t *pte, struct collapse_control *cc, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + u8 order) { struct page *page = NULL; struct folio *folio = NULL; pte_t *_pte; int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; bool writable = false; + int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_pte = pte; _pte < pte + (1 << order); _pte++, address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); if (pte_none(pteval) || (pte_present(pteval) && @@ -581,7 +583,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, ++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= scaled_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -609,8 +611,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, /* See khugepaged_scan_pmd(). */ if (folio_likely_mapped_shared(folio)) { ++shared; - if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { + if (order != HPAGE_PMD_ORDER || (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared)) { result = SCAN_EXCEED_SHARED_PTE; count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); goto out; @@ -711,14 +713,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + u8 order) { struct folio *src, *tmp; pte_t *_pte; pte_t pteval; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; - _pte++, address += PAGE_SIZE) { + for (_pte = pte; _pte < pte + (1 << order); + _pte++, address += PAGE_SIZE) { pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1); @@ -764,7 +767,8 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + u8 order) { spinlock_t *pmd_ptl; @@ -781,7 +785,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, * Release both raw and compound pages isolated * in __collapse_huge_page_isolate. */ - release_pte_pages(pte, pte + HPAGE_PMD_NR, compound_pagelist); + release_pte_pages(pte, pte + (1 << order), compound_pagelist); } /* @@ -802,7 +806,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, u8 order) { unsigned int i; int result = SCAN_SUCCEED; @@ -810,7 +814,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, /* * Copying pages' contents is subject to memory poison at any iteration. */ - for (i = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0; i < (1 << order); i++) { pte_t pteval = ptep_get(pte + i); struct page *page = folio_page(folio, i); unsigned long src_addr = address + i * PAGE_SIZE; @@ -829,10 +833,10 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, if (likely(result == SCAN_SUCCEED)) __collapse_huge_page_copy_succeeded(pte, vma, address, ptl, - compound_pagelist); + compound_pagelist, order); else __collapse_huge_page_copy_failed(pte, pmd, orig_pmd, vma, - compound_pagelist); + compound_pagelist, order); return result; } @@ -1000,11 +1004,11 @@ static int check_pmd_still_valid(struct mm_struct *mm, static int __collapse_huge_page_swapin(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long haddr, pmd_t *pmd, - int referenced) + int referenced, u8 order) { int swapped_in = 0; vm_fault_t ret = 0; - unsigned long address, end = haddr + (HPAGE_PMD_NR * PAGE_SIZE); + unsigned long address, end = haddr + (PAGE_SIZE << order); int result; pte_t *pte = NULL; spinlock_t *ptl; @@ -1035,6 +1039,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, if (!is_swap_pte(vmf.orig_pte)) continue; + if (order != HPAGE_PMD_ORDER) { + result = SCAN_EXCEED_SWAP_PTE; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret = do_swap_page(&vmf); @@ -1114,7 +1123,6 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, int result = SCAN_FAIL; struct vm_area_struct *vma; struct mmu_notifier_range range; - VM_BUG_ON(address & ~HPAGE_PMD_MASK); /* @@ -1149,7 +1157,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * that case. Continuing to collapse causes inconsistency. */ result = __collapse_huge_page_swapin(mm, vma, address, pmd, - referenced); + referenced, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) goto out_nolock; } @@ -1196,7 +1204,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); if (pte) { result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist); + &compound_pagelist, HPAGE_PMD_ORDER); spin_unlock(pte_ptl); } else { result = SCAN_PMD_NULL; @@ -1226,7 +1234,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, vma, address, pte_ptl, - &compound_pagelist); + &compound_pagelist, HPAGE_PMD_ORDER); pte_unmap(pte); if (unlikely(result != SCAN_SUCCEED)) goto out_up_write; From patchwork Tue Feb 11 00:30:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C63B6C02198 for ; Tue, 11 Feb 2025 00:32:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5347128000F; Mon, 10 Feb 2025 19:32:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E3A3280008; Mon, 10 Feb 2025 19:32:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 384C628000F; Mon, 10 Feb 2025 19:32:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1683B280008 for ; Mon, 10 Feb 2025 19:32:16 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C36A01C7DD2 for ; Tue, 11 Feb 2025 00:32:15 +0000 (UTC) X-FDA: 83105787030.12.2EAC112 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id E99AE20003 for ; Tue, 11 Feb 2025 00:32:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PF4jJhrD; spf=pass (imf03.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233934; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7fUONtMIJRjZ+fpEsfBAOSqHGTOAUFKQgLDIE5UlCPg=; b=X7hozsdUEgVVvMci1wPQ5r+5KLjE145hacknJfgCZwF06hzyY1ePYotOaRAZXqIKncxWUx WU0AXvXZSVhfHXkr+rGqOQmXEIvwYb13bmPhOrME4LHFLSwO3fJQGNV2Y+Ob99yxVCBH9N D0Z3jCDg6vNbL1ev8QFG0mSjXgqZQic= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PF4jJhrD; spf=pass (imf03.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233934; a=rsa-sha256; cv=none; b=hRlMPMccK4rgRtYii+hHK0FV3YKdI9OevkogikE4x9f61KKtm9bBIB0z73gjIqgtC45ev2 3MXTDjK+5NKqarxfM7tmvP1rGbGJQ3NrMB8xAN3unNttuLGbmoHwa/KGKtUhvv8O9i/hJR ebvofogoM0R2VtQwEbLlbBxLSL4Wh/Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233933; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7fUONtMIJRjZ+fpEsfBAOSqHGTOAUFKQgLDIE5UlCPg=; b=PF4jJhrDB4yq6VoQwjBMiN6G3qaNWxyUCSEIRx5Ord7/dhfxOltYBEAsJZHBOwP0exdzUs VYJcm9aaI/KNwcymyjJSSJEwy4L8GgNDeMRergTB21YOCG0kpMK3Iqs0KUMUdzkI5F+x/j V+PoQSZ39uKwr+7CVe6u+1HTQjwIuMA= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-607-AKmiACISPX-LKpWM4wDUDg-1; Mon, 10 Feb 2025 19:32:09 -0500 X-MC-Unique: AKmiACISPX-LKpWM4wDUDg-1 X-Mimecast-MFC-AGG-ID: AKmiACISPX-LKpWM4wDUDg Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 669A519560AA; Tue, 11 Feb 2025 00:32:05 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 80DDA1800358; Tue, 11 Feb 2025 00:31:55 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 6/9] khugepaged: introduce khugepaged_scan_bitmap for mTHP support Date: Mon, 10 Feb 2025 17:30:25 -0700 Message-ID: <20250211003028.213461-7-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E99AE20003 X-Stat-Signature: 1ns16dndfbfsmwwgjjgty3cbn4fbqhre X-HE-Tag: 1739233933-235743 X-HE-Meta: U2FsdGVkX1/9gA0hY6pfdGcElYwZDzLeke9JRm5vZfj0Z6NlZ6I8wv66n0HKQciLALH8JIAUbfNbvZM2N4SI3dpYXXrx4vHRsUQYTnQl0EqyWYL44FWq3uNzoc4zXLQwf52mUKCWXA1clk5jzbB7jzfVKUsszsJGqB0cWMEckUYJNQ2oE8d0/gRDefJyHxxeZFiRjhnDszXlzWdnkxR5zadrN7Fk2CM15bRZYacNFhfoA311nzVtjg3bGzVZf1Xo8qBV+NaqhjRY1ihay4YBW68DYAg3aGxdcBKE+gYN0SQvppk2REhxHXh4OFQldZUctP6BI+IxD8UUpBbKkXiZfB05e5O3nt78E1wOXz1qhAX3AiDuOWSOxohEGqwn7ch5BGmAeAJeDCEouJRUkVnqg1Jbk5n/9JJTN47Ygqe3HCXmsQgtCjBMtiB8KJ/3yEM90jKwvNzoCnyW4XMm6AlgmtEkRoVTpN1Yc3/Y1HmeQ/djYc0LzjsU95Z2YEEwRRV40zFUZQknnBKlyqbbOqcxXNLJcYxwgbI685dr/MqCB/QrJ8vVy41QOE+POw0vbPxiMjFjuZMcpjZ7vPyYfOaFtO8S0n60izwpsP0CG4q/Mq1FeFUdQLpL9AUVb7l0EPnnwy0loYk9VINp4wjez4ZXb9R9GaPrk7kh3i+FMNxVWMikLPYApew9ilBjOiqYkRoyvIo8wirEeIVM15J0tN74EMsfpQGu1x0eewW5bwen4ajFJBfrs92tBr7XI3OZUTK8kadMBXX+BA85qIx8gsINzwaAAjK3tpTMEGWiOKF8P9icLKVLj2/xZMZrALGfC9gPltq+UGD5Wr0TiugfxbaTy4aiy4tpe60akmXVAk8PajWjB8L/qqF1dliQxosXqomisedY/vOOXHM+ZKjLRudktuMAgBwNf4TufRuC2Yn81Kdx4E2XGuZ1TJP2sj9je1e8pDkB3ABvrF8mW7rgfjb IimIyFO3 I1YQDEewXGO8fRjvXF4B0p92GOb6msZ3c9gz4V/JfJEfmWhg/MYyJnvxFYe8Z3WUj7y7d18N6c7xDhxmwMKajUcOyGZT/zZp0SdgpXlxuHZelrpRek31vzOtLIiBBBDOrDNfeMuy//+YkBti8YVTd2spPRAV6G2snZ8Zl8gSNjLCWxpd7wjYEjvueBUQl31Mey6J2gbRTQ60NPNEvADfLARxegqJSf16lpBJcN7/NPLtmzcDCrO08VhbBM66dx9cceV4PsP2fw24m6YWiBPynA4g/Jzy4KexUSFVpC8jOY9wpaQFZLUTfWD+T9GvNalo6JaDqywHcn1U5DuwxbpB62nts+1djOEf9sP4jm5KVvCpnu+36ME3L2TG2x+thMNxCJlKkiaoT8/RnKIjQcQ9od5d8sqs2zTjBta2ZM0plRom2jdD8uvCgpbIAbxcM2ZglcqeV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: khugepaged scans PMD ranges for potential collapse to a hugepage. To add mTHP support we use this scan to instead record chunks of fully utilized sections of the PMD. create a bitmap to represent a PMD in order MTHP_MIN_ORDER chunks. by default we will set this to order 3. The reasoning is that for 4K 512 PMD size this results in a 64 bit bitmap which has some optimizations. For other arches like ARM64 64K, we can set a larger order if needed. khugepaged_scan_bitmap uses a stack struct to recursively scan a bitmap that represents chunks of utilized regions. We can then determine what mTHP size fits best and in the following patch, we set this bitmap while scanning the PMD. max_ptes_none is used as a scale to determine how "full" an order must be before being considered for collapse. If a order is set to "always" lets always collapse to that order in a greedy manner. Signed-off-by: Nico Pache --- include/linux/khugepaged.h | 4 ++ mm/khugepaged.c | 89 +++++++++++++++++++++++++++++++++++--- 2 files changed, 86 insertions(+), 7 deletions(-) diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 1f46046080f5..1fe0c4fc9d37 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -1,6 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H #define _LINUX_KHUGEPAGED_H +#define MIN_MTHP_ORDER 3 +#define MIN_MTHP_NR (1<mthp_bitmap_stack[++top] = (struct scan_bit_state) + { HPAGE_PMD_ORDER - MIN_MTHP_ORDER, 0 }; + + while (top >= 0) { + state = cc->mthp_bitmap_stack[top--]; + order = state.order + MIN_MTHP_ORDER; + offset = state.offset; + num_chunks = 1 << (state.order); + // Skip mTHP orders that are not enabled + if (!test_bit(order, &enabled_orders)) + goto next; + + // copy the relavant section to a new bitmap + bitmap_shift_right(cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, + MTHP_BITMAP_SIZE); + + bits_set = bitmap_weight(cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none - 1) + >> (HPAGE_PMD_ORDER - state.order); + + //Check if the region is "almost full" based on the threshold + if (bits_set > threshold_bits + || test_bit(order, &huge_anon_orders_always)) { + ret = collapse_huge_page(mm, address, referenced, unmapped, cc, + mmap_locked, order, offset * MIN_MTHP_NR); + if (ret == SCAN_SUCCEED) { + collapsed += (1 << order); + continue; + } + } + +next: + if (state.order > 0) { + next_order = state.order - 1; + mid_offset = offset + (num_chunks / 2); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state) + { next_order, offset }; + } + } + return collapsed; +} + static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked, @@ -1440,7 +1514,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { result = collapse_huge_page(mm, address, referenced, - unmapped, cc); + unmapped, cc, mmap_locked, HPAGE_PMD_ORDER, 0); /* collapse_huge_page will return with the mmap_lock released */ *mmap_locked = false; } @@ -2856,6 +2930,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, mmdrop(mm); kfree(cc); + return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 : madvise_collapse_errno(last_fail); } From patchwork Tue Feb 11 00:30:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45BDEC02198 for ; Tue, 11 Feb 2025 00:32:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3B74280010; Mon, 10 Feb 2025 19:32:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B9AD4280008; Mon, 10 Feb 2025 19:32:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9534D280010; Mon, 10 Feb 2025 19:32:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6E29A280008 for ; Mon, 10 Feb 2025 19:32:28 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E6047120874 for ; Tue, 11 Feb 2025 00:32:27 +0000 (UTC) X-FDA: 83105787534.26.F5D918F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 3B57D100016 for ; Tue, 11 Feb 2025 00:32:26 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="HQCBAi/3"; spf=pass (imf14.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233946; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R5QV1YYqNLRVQY/ha5W4paMmIxjg/ybigGOpz+fH3QI=; b=d+8jJxbZ6DL7jdWA1bsuA8RqVkcpCl9fF5rGlyH5+Wk+8rNd0Kq3b5EUKjXkQRePvVQobH vtL6r1MroNJnzoaj6fLG2C22ATG4Oioy/amFEGUnY2EO0muCXW3pvV+sigBUg/4EJQOx9E 5n1xO6Ylza3T/UVgiU+Ptg6uav9u8vM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="HQCBAi/3"; spf=pass (imf14.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233946; a=rsa-sha256; cv=none; b=eUaHL0ZO89p61+6WOELQ5YktHZVdFDDVB2YOpMLkMr0ZkwiikyHnSzk2a3kvgnLJAwcaji QaSV/k0fgS/xduMSA2h3ZWtTcZi1ASiDlhrxV8WaREzpdTnL2Aju/wCSGTrx0jgLu1RABe PYuzleHuoXssUDNKTov2VZFit7hd5ys= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R5QV1YYqNLRVQY/ha5W4paMmIxjg/ybigGOpz+fH3QI=; b=HQCBAi/395Isg6TF3Riw7l9yXh9d5BkI+Qgy2aerERkRjQ49eGnyQxXPFNl7mC7RZmwq+P frCSBYb/aOZeRBmXhSSOLNXVs70QBWz8G91gskBp2tKSCdLezrVMxDS3FPo70Rbx7FgKZ7 +L4Xp9Sg3mxAoaI9+4TN2zFwed2PnMI= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-385-C75zRJGjOeWwYpJkNdH6-g-1; Mon, 10 Feb 2025 19:32:19 -0500 X-MC-Unique: C75zRJGjOeWwYpJkNdH6-g-1 X-Mimecast-MFC-AGG-ID: C75zRJGjOeWwYpJkNdH6-g Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 546471956088; Tue, 11 Feb 2025 00:32:14 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B53A818004A7; Tue, 11 Feb 2025 00:32:05 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 7/9] khugepaged: add mTHP support Date: Mon, 10 Feb 2025 17:30:26 -0700 Message-ID: <20250211003028.213461-8-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Stat-Signature: d3uarcy1ist8tpqtqwqe6or79w9mthnf X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3B57D100016 X-HE-Tag: 1739233946-883986 X-HE-Meta: U2FsdGVkX19djFDTF65VulZE74qBX6rM5lrvKMSQwaFSaaUYfeWFcQfNVQdStDIoOcGJGa/bRldcIPR+B80WmEB0y8g+fO0skWwxs67RC6vcyxKAs8iuqFXSmQmiVI/2tuU0TBSVDiSLuwdRHTcvaz71kC5mrXjvyhNZ8raM7edko9Ab+8/qWnsARIgcoeyv2M98NpK2Xryg1//Kbo2FQjdEbD3UbrhFMBl2tBanoox7lKeJq2LZLkmzWkS7RLpS78U0xEDrAHAEqCqa99OgBb+c7uFEVwPOPz71ovW/K4G5LSCYSL0PscLoNsMBUirTXPyQJeHvv6b1ySJh0pT4+Db6pj5RH9wElPLJmf1vysXZ2e646u/+f5dTdYyLCKaaGCm38ml9+zdi/RxQ/rHzGs+jtzYytdyPXWcTbJ8kBWMXE9OCPZ0nXdTiUZzKDDsiBWAn9b4MXxJTtCASy8vCNpOvqMvEEOLeN+EczqOLpHa0jaRKhWMSACVv734g05Wa+OfqsNWZyN2yTQzQ4qUixyGoLt7kv6IjI8+2R+tfsAaNONw6BuXDyOu/e54ytC1TP9TpQ4oT7ysRsh2WJQZGfBBbNvZb6UJWfl/QSNkWSppCPSVs1lZnFnTO36tN4yiIwfpRehNw2FNu4AYQZXFfvITIgSWdQbGc1tBfTUrHZsiR4R19HtIqXHkRdTCw2PB/jK7BMnneIBnT2INQe2Dea9XfR2yMG4AA4dthz2rtTSImtYgIk1joTdqhITXZATDCIir4Uzb83JG+vcWxonKtsRw7FPjb0GS1sPeYcyYphyGkmgWtngLntdKGCetaOehATk5TtY0iQj65ZOEhG/v8f/TpXSDUF3JW43JljNwCEZONDmeHIfj/no3xck69xaXkpHvqrRBmeXg7x56sd5oFaumpyVx6yB/Z+r1GiW5kYL7AZM3C3iQpAO1AGWRHwuHkzRHXwNZpe9lXelgJYiO LpOjSvBy wta/+i0MUhXFnj6qpsyftwMqoWhKJHbxly9+yj8odsyrxBLh7K9uuHNjAK6D1sXcyb4uhmlQIZZDWH5HCRGrae2Fhz4onwnPDKS09Gjkk0Ie1e0zaQdq5y+4xCN63bPskAF2/mEN/pnWPLqmqV3w5mhAxnR4yY17m7rlCEHDWrayPAEdY4kKfmyqZWh7pxcifXB78fUfKTdCGcHDxacYG7qCZ5iWWsiWiItanhHNhlCjOx9oT9RvIRzm1B45+XhqSQKoDP3LCJYUbutcIRZnQ7oDejNv1ZlEv8IA00UceNuCi8IJNrx9PTJ/AzODeiEEaX6xSbt401taLjhYKS+ckvMo28tuE3GNX+Z8KkHjeLnE/D7iq3BeE80QeYhgOEQCVGC4Sgbp/HY5BMCI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce the ability for khugepaged to collapse to different mTHP sizes. While scanning a PMD range for potential collapse candidates, keep track of pages in MIN_MTHP_ORDER chunks via a bitmap. Each bit represents a utilized region of order MIN_MTHP_ORDER ptes. We remove the restriction of max_ptes_none during the scan phase so we dont bailout early and miss potential mTHP candidates. After the scan is complete we will perform binary recursion on the bitmap to determine which mTHP size would be most efficient to collapse to. max_ptes_none will be scaled by the attempted collapse order to determine how full a THP must be to be eligible. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we dont perform the collapse. Signed-off-by: Nico Pache --- mm/khugepaged.c | 122 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 83 insertions(+), 39 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c8048d9ec7fb..cd310989725b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1127,13 +1127,14 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, { LIST_HEAD(compound_pagelist); pmd_t *pmd, _pmd; - pte_t *pte; + pte_t *pte, mthp_pte; pgtable_t pgtable; struct folio *folio; spinlock_t *pmd_ptl, *pte_ptl; int result = SCAN_FAIL; struct vm_area_struct *vma; struct mmu_notifier_range range; + unsigned long _address = address + offset * PAGE_SIZE; VM_BUG_ON(address & ~HPAGE_PMD_MASK); /* @@ -1148,12 +1149,13 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, *mmap_locked = false; } - result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER); + result = alloc_charge_folio(&folio, mm, cc, order); if (result != SCAN_SUCCEED) goto out_nolock; mmap_read_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc, HPAGE_PMD_ORDER); + *mmap_locked = true; + result = hugepage_vma_revalidate(mm, address, true, &vma, cc, order); if (result != SCAN_SUCCEED) { mmap_read_unlock(mm); goto out_nolock; @@ -1171,13 +1173,14 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ - result = __collapse_huge_page_swapin(mm, vma, address, pmd, - referenced, HPAGE_PMD_ORDER); + result = __collapse_huge_page_swapin(mm, vma, _address, pmd, + referenced, order); if (result != SCAN_SUCCEED) goto out_nolock; } mmap_read_unlock(mm); + *mmap_locked = false; /* * Prevent all access to pagetables with the exception of * gup_fast later handled by the ptep_clear_flush and the VM @@ -1187,7 +1190,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * mmap_lock. */ mmap_write_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc, HPAGE_PMD_ORDER); + result = hugepage_vma_revalidate(mm, address, true, &vma, cc, order); if (result != SCAN_SUCCEED) goto out_up_write; /* check if the pmd is still valid */ @@ -1198,11 +1201,12 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, - address + HPAGE_PMD_SIZE); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, _address, + _address + (PAGE_SIZE << order)); mmu_notifier_invalidate_range_start(&range); pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ + /* * This removes any huge TLB entry from the CPU so we won't allow * huge and small TLB entries for the same virtual address to @@ -1216,10 +1220,10 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, mmu_notifier_invalidate_range_end(&range); tlb_remove_table_sync_one(); - pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); + pte = pte_offset_map_lock(mm, &_pmd, _address, &pte_ptl); if (pte) { - result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist, HPAGE_PMD_ORDER); + result = __collapse_huge_page_isolate(vma, _address, pte, cc, + &compound_pagelist, order); spin_unlock(pte_ptl); } else { result = SCAN_PMD_NULL; @@ -1248,8 +1252,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, anon_vma_unlock_write(vma->anon_vma); result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, - vma, address, pte_ptl, - &compound_pagelist, HPAGE_PMD_ORDER); + vma, _address, pte_ptl, + &compound_pagelist, order); pte_unmap(pte); if (unlikely(result != SCAN_SUCCEED)) goto out_up_write; @@ -1260,20 +1264,37 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * write. */ __folio_mark_uptodate(folio); - pgtable = pmd_pgtable(_pmd); - - _pmd = mk_huge_pmd(&folio->page, vma->vm_page_prot); - _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma); - - spin_lock(pmd_ptl); - BUG_ON(!pmd_none(*pmd)); - folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); - folio_add_lru_vma(folio, vma); - pgtable_trans_huge_deposit(mm, pmd, pgtable); - set_pmd_at(mm, address, pmd, _pmd); - update_mmu_cache_pmd(vma, address, pmd); - deferred_split_folio(folio, false); - spin_unlock(pmd_ptl); + if (order == HPAGE_PMD_ORDER) { + pgtable = pmd_pgtable(_pmd); + _pmd = mk_huge_pmd(&folio->page, vma->vm_page_prot); + _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma); + + spin_lock(pmd_ptl); + BUG_ON(!pmd_none(*pmd)); + folio_add_new_anon_rmap(folio, vma, _address, RMAP_EXCLUSIVE); + folio_add_lru_vma(folio, vma); + pgtable_trans_huge_deposit(mm, pmd, pgtable); + set_pmd_at(mm, address, pmd, _pmd); + update_mmu_cache_pmd(vma, address, pmd); + deferred_split_folio(folio, false); + spin_unlock(pmd_ptl); + } else { //mTHP + mthp_pte = mk_pte(&folio->page, vma->vm_page_prot); + mthp_pte = maybe_mkwrite(pte_mkdirty(mthp_pte), vma); + + spin_lock(pmd_ptl); + folio_ref_add(folio, (1 << order) - 1); + folio_add_new_anon_rmap(folio, vma, _address, RMAP_EXCLUSIVE); + folio_add_lru_vma(folio, vma); + spin_lock(pte_ptl); + set_ptes(vma->vm_mm, _address, pte, mthp_pte, (1 << order)); + update_mmu_cache_range(NULL, vma, _address, pte, (1 << order)); + spin_unlock(pte_ptl); + smp_wmb(); /* make pte visible before pmd */ + pmd_populate(mm, pmd, pmd_pgtable(_pmd)); + deferred_split_folio(folio, false); + spin_unlock(pmd_ptl); + } folio = NULL; @@ -1353,21 +1374,27 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, { pmd_t *pmd; pte_t *pte, *_pte; + int i; int result = SCAN_FAIL, referenced = 0; int none_or_zero = 0, shared = 0; struct page *page = NULL; struct folio *folio = NULL; unsigned long _address; + unsigned long enabled_orders; spinlock_t *ptl; int node = NUMA_NO_NODE, unmapped = 0; bool writable = false; - + int chunk_none_count = 0; + int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - MIN_MTHP_ORDER); + unsigned long tva_flags = cc->is_khugepaged ? TVA_ENFORCE_SYSFS : 0; VM_BUG_ON(address & ~HPAGE_PMD_MASK); result = find_pmd_or_thp_or_none(mm, address, &pmd); if (result != SCAN_SUCCEED) goto out; + bitmap_zero(cc->mthp_bitmap, MAX_MTHP_BITMAP_SIZE); + bitmap_zero(cc->mthp_bitmap_temp, MAX_MTHP_BITMAP_SIZE); memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); pte = pte_offset_map_lock(mm, pmd, address, &ptl); @@ -1376,8 +1403,12 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out; } - for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; - _pte++, _address += PAGE_SIZE) { + for (i = 0; i < HPAGE_PMD_NR; i++) { + if (i % MIN_MTHP_NR == 0) + chunk_none_count = 0; + + _pte = pte + i; + _address = address + i * PAGE_SIZE; pte_t pteval = ptep_get(_pte); if (is_swap_pte(pteval)) { ++unmapped; @@ -1400,16 +1431,14 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, } } if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++chunk_none_count; ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { + if (userfaultfd_armed(vma)) { result = SCAN_EXCEED_NONE_PTE; count_vm_event(THP_SCAN_EXCEED_NONE_PTE); goto out_unmap; } + continue; } if (pte_uffd_wp(pteval)) { /* @@ -1500,7 +1529,16 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; + + /* + * we are reading in MIN_MTHP_NR page chunks. if there are no empty + * pages keep track of it in the bitmap for mTHP collapsing. + */ + if (chunk_none_count < scaled_none && + (i + 1) % MIN_MTHP_NR == 0) + bitmap_set(cc->mthp_bitmap, i / MIN_MTHP_NR, 1); } + if (!writable) { result = SCAN_PAGE_RO; } else if (cc->is_khugepaged && @@ -1513,10 +1551,14 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { - result = collapse_huge_page(mm, address, referenced, - unmapped, cc, mmap_locked, HPAGE_PMD_ORDER, 0); - /* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + enabled_orders = thp_vma_allowable_orders(vma, vma->vm_flags, + tva_flags, THP_ORDERS_ALL_ANON); + result = khugepaged_scan_bitmap(mm, address, referenced, unmapped, cc, + mmap_locked, enabled_orders); + if (result > 0) + result = SCAN_SUCCEED; + else + result = SCAN_FAIL; } out: trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, @@ -2476,11 +2518,13 @@ static int khugepaged_collapse_single_pmd(unsigned long addr, struct mm_struct * fput(file); if (result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); + *mmap_locked = true; if (khugepaged_test_exit_or_disable(mm)) goto end; result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged); mmap_read_unlock(mm); + *mmap_locked = false; } } else { result = khugepaged_scan_pmd(mm, vma, addr, From patchwork Tue Feb 11 00:30:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1148CC0219D for ; Tue, 11 Feb 2025 00:32:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A0F4280011; Mon, 10 Feb 2025 19:32:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 92B3D280008; Mon, 10 Feb 2025 19:32:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72E83280011; Mon, 10 Feb 2025 19:32:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4FAA3280008 for ; Mon, 10 Feb 2025 19:32:42 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 08BDBB10BE for ; Tue, 11 Feb 2025 00:32:42 +0000 (UTC) X-FDA: 83105788164.19.0FCA780 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 3B25F14000E for ; Tue, 11 Feb 2025 00:32:40 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cSjMJcf4; spf=pass (imf09.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mWdmFB3VEeA1XnEF1XTP4N/+VjRlOdLjAcJyCJVOZpY=; b=HsdK8BcAz1rxNm4pMUW1hbsXfcwX9W50KNcHOdN4vo4fxWSUc7C+kEj8ZMcgUCXs3/kfXp XKqmJbjyVUvVEJeBhvgQcrtqvZIaXN105IBoP7URvkS5jAof/810rOqNT8szPA6IWTOzEE CT4XIuvMEX9LsBVdV1sFKbRl/qAm1Vw= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cSjMJcf4; spf=pass (imf09.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233960; a=rsa-sha256; cv=none; b=qCojBg0H0QiZ7OruIHNnDpxvZmPbHiOQ0067IC4fbANDfPUeQH90Wh5MtngE0IIXZdUjLq Yze6SPNptsr8tfEJAwXnBNsBktSCdqzO6vjyBC05pdDkMKO8aPkz1kHdNhbc824PKBvlK4 ppkNptExuSNGJ6G9TEoV30WOLNm8o4Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mWdmFB3VEeA1XnEF1XTP4N/+VjRlOdLjAcJyCJVOZpY=; b=cSjMJcf4bklPEVPujHQLJhuJEjKyEeSlyNrblwO1qSfYbd318db4V8TaGscl8m7nF0v/Y5 88+WlPqaARuM/SuTVrJ5dY8ckL4AN/M+uHqQncx62Dkppzl4ntWcrra4/eRkdcNxsvNp/f KufNkbTlJLdBxcz23JwuzPjOnCCocpw= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-20-dTR64f4SM6qnszwScxMw2w-1; Mon, 10 Feb 2025 19:32:35 -0500 X-MC-Unique: dTR64f4SM6qnszwScxMw2w-1 X-Mimecast-MFC-AGG-ID: dTR64f4SM6qnszwScxMw2w Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 84DE3195608C; Tue, 11 Feb 2025 00:32:24 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A30601800876; Tue, 11 Feb 2025 00:32:14 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 8/9] khugepaged: improve tracepoints for mTHP orders Date: Mon, 10 Feb 2025 17:30:27 -0700 Message-ID: <20250211003028.213461-9-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3B25F14000E X-Stat-Signature: sn45nxxmk649fxb31gnq4qxisi4i57u4 X-Rspam-User: X-HE-Tag: 1739233960-934168 X-HE-Meta: U2FsdGVkX18aFqO3R6R5YdVcmcUuGF1KmLUyM/JypVGEHlYWvPZrD9zVTaFpU/WVQIITTsQjXL+ILwjNBOtYFdtbSYBCgVTlG6qFPnWWPXAcwysjQTFQJIrcln42Ia0LbwEuUjtjm19DjXEWtONMntTG9DwFVQAa0jLGIhBM9aMK+rRTbjRS4cR4No0f81HBxD69v5AGLExI/+n5UmfIxd2s3a7tMOG05JUb4kEVdHYeAIJVKIrOcjNiMketzRvmMtOYM4VD1f9i1bl0bi7yfeJLJJDhrsGaFEBOhER9PmyMiTVYRjSx9TccsaibDQ41XNvqA5ZPos95MV3HNB/Zc12mZnoyVGwlTozQglKZZhTKKkonmBsdl7X1sFpxFDoBqlwNuU6llsBqN2JlemYrO8tAeiUtQXdEzNJb6VxTmx0KLt+/E8L1s6ELgmE+ZT2a/1l+E6dI0INeiZpZ6scyAb7zoLPvrQOLHrfVfio3EesQIEAQ8KciB1eIOMIcfzKoBaLLc877SqVzkD+k+yGoFhDwUA6WYlV/yiEK8dtvNu49d6NwkU/idwblahZGg122FUvGHoWRqnqAsjSEsSzI8Gbk5RmjIPwjA+3OjWioR8bbsb5+YpGX/2RWi+CBPmpYTrbPSYNiOAtszz3P5VEhPDH/mgYDQBJP8Ee0JMH7ZYittcP8TOiiPmnz6gE6VzgEM4geYnsmDm3lyiFOPgIYCYSuiA/+BMiV8thorbskRqStDcixvOQXZ4f2YLKeF2PVlbQnusxpAojK/3kTLJ2hm+531KzPieSuYOrDR3GliOzITPfqk6yRQImXS7URdtn+FV24TzDPcvHALVXdJIvYYFcLn6+edbbUTqI2kLlN8EguRcG1oSG8C8zuSyoEncLa5JCqQ1qf4P8XlqOlHtD/DgNF4g8eMPi8rwqg4MgailCV782r8eG2Ic+hxXNuUb1HdEInQEef3mB+MlKA0QX 92o1hnJC pXohEFrg97qjakwUZTbn+YRIAtx3pxmlgU1cB3/wveWSzi5nwxif1TUCMEKazSUnsKrACah1xaVsp8iWt8hNc0s0RiN8T+rLSi75UfeEQuWI8e3Bjm49vnBia0QtatuBCwM5QhnPWhICETsUvSWfw9eLOwiUrVp0y+SUAO3ATQt4fBXOQ0YQkNqJqxbexJ2CA2euIvtzsydiTqAD0HoBeGuhfBjfn2V3hmoSiD8pIQTGXa7DzXueb8odZ1HuFAMcwSXoe4ybmnA2Z0Y+scwT3h/rXLd3HpoXIGxW4vOHXyxvpcuIWnf+AHAYStUL3ITje9V+wg9WPtk77ldFsRcBvvKodxyTX/GYP7PoI0ckxui+b68v1nhiDrnsPTbsmzR/U7ndnKsyt2QVIZO8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add the order to the tracepoints to give better insight into what order is being operated at for khugepaged. Signed-off-by: Nico Pache --- include/trace/events/huge_memory.h | 34 +++++++++++++++++++----------- mm/khugepaged.c | 10 +++++---- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h index 9d5c00b0285c..ea2fe20a39f5 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -92,34 +92,37 @@ TRACE_EVENT(mm_khugepaged_scan_pmd, TRACE_EVENT(mm_collapse_huge_page, - TP_PROTO(struct mm_struct *mm, int isolated, int status), + TP_PROTO(struct mm_struct *mm, int isolated, int status, int order), - TP_ARGS(mm, isolated, status), + TP_ARGS(mm, isolated, status, order), TP_STRUCT__entry( __field(struct mm_struct *, mm) __field(int, isolated) __field(int, status) + __field(int, order) ), TP_fast_assign( __entry->mm = mm; __entry->isolated = isolated; __entry->status = status; + __entry->order = order; ), - TP_printk("mm=%p, isolated=%d, status=%s", + TP_printk("mm=%p, isolated=%d, status=%s order=%d", __entry->mm, __entry->isolated, - __print_symbolic(__entry->status, SCAN_STATUS)) + __print_symbolic(__entry->status, SCAN_STATUS), + __entry->order) ); TRACE_EVENT(mm_collapse_huge_page_isolate, TP_PROTO(struct page *page, int none_or_zero, - int referenced, bool writable, int status), + int referenced, bool writable, int status, int order), - TP_ARGS(page, none_or_zero, referenced, writable, status), + TP_ARGS(page, none_or_zero, referenced, writable, status, order), TP_STRUCT__entry( __field(unsigned long, pfn) @@ -127,6 +130,7 @@ TRACE_EVENT(mm_collapse_huge_page_isolate, __field(int, referenced) __field(bool, writable) __field(int, status) + __field(int, order) ), TP_fast_assign( @@ -135,27 +139,31 @@ TRACE_EVENT(mm_collapse_huge_page_isolate, __entry->referenced = referenced; __entry->writable = writable; __entry->status = status; + __entry->order = order; ), - TP_printk("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, writable=%d, status=%s", + TP_printk("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, writable=%d, status=%s order=%d", __entry->pfn, __entry->none_or_zero, __entry->referenced, __entry->writable, - __print_symbolic(__entry->status, SCAN_STATUS)) + __print_symbolic(__entry->status, SCAN_STATUS), + __entry->order) ); TRACE_EVENT(mm_collapse_huge_page_swapin, - TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret), + TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret, + int order), - TP_ARGS(mm, swapped_in, referenced, ret), + TP_ARGS(mm, swapped_in, referenced, ret, order), TP_STRUCT__entry( __field(struct mm_struct *, mm) __field(int, swapped_in) __field(int, referenced) __field(int, ret) + __field(int, order) ), TP_fast_assign( @@ -163,13 +171,15 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, __entry->swapped_in = swapped_in; __entry->referenced = referenced; __entry->ret = ret; + __entry->order = order; ), - TP_printk("mm=%p, swapped_in=%d, referenced=%d, ret=%d", + TP_printk("mm=%p, swapped_in=%d, referenced=%d, ret=%d, order=%d", __entry->mm, __entry->swapped_in, __entry->referenced, - __entry->ret) + __entry->ret, + __entry->order) ); TRACE_EVENT(mm_khugepaged_scan_file, diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cd310989725b..e2ba18e57064 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -713,13 +713,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } else { result = SCAN_SUCCEED; trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, - referenced, writable, result); + referenced, writable, result, + order); return result; } out: release_pte_pages(pte, _pte, compound_pagelist); trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, - referenced, writable, result); + referenced, writable, result, order); return result; } @@ -1088,7 +1089,8 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, result = SCAN_SUCCEED; out: - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result); + trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result, + order); return result; } @@ -1305,7 +1307,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, *mmap_locked = false; if (folio) folio_put(folio); - trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result); + trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result, order); return result; } From patchwork Tue Feb 11 00:30:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 387C9C02198 for ; Tue, 11 Feb 2025 00:32:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B52A7280012; Mon, 10 Feb 2025 19:32:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD82B280008; Mon, 10 Feb 2025 19:32:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90448280012; Mon, 10 Feb 2025 19:32:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6AAB0280008 for ; Mon, 10 Feb 2025 19:32:44 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1F36012082E for ; Tue, 11 Feb 2025 00:32:44 +0000 (UTC) X-FDA: 83105788248.01.61C1EB6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 5B73A4000F for ; Tue, 11 Feb 2025 00:32:42 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BSQ9tysi; spf=pass (imf07.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZZxaGvzUu8D0dA0MT9NpgVy0JYjj6UualQU6CuWTo+8=; b=BnY5Ao0KPbj+HPCy6Jaxa1+w64T9XjNBD2iC0/4+ecgAZ5jH5wbrqpi+r82EWzdGpQoOnS RPS3E4XKcM7Txz/4Wwdi+PO4gpXG8xrr7Xhs2hyPEZ0epCWd6iAldD1PY4aICd0mtZycmo +odVF53UUmhszPe26VkhK5bBkf4/oe4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BSQ9tysi; spf=pass (imf07.hostedemail.com: domain of npache@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233962; a=rsa-sha256; cv=none; b=5IxBmVHMhAUuI+1rKEdOy3bdD6b+rJiyYmeVvNvpyluk3yo89VXNfuNhD2Yx1BW9Mvb/ld bSPqCmunmZm6EovJ23P6gwpgmcAojTvgrVT+mWGIB/kPP0d7u8qS7/bkmfKT1YvFDigr3R NtQ7xYxdnRPojEwV1RtCyf5ynmhfngQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZZxaGvzUu8D0dA0MT9NpgVy0JYjj6UualQU6CuWTo+8=; b=BSQ9tysiim6PTQuTD7xuhmOTAlG1LC85Ry4Wpng7DxTPT/6PeobgZJwUdg1Fwh4GIkxHVl B9epbKmLlx+0X7V+Ay1E10vo63wYw2ONNn3XFIm2IB847aCGIXY2ekv5afTqweEQGIkQm4 XGYiBaR91MSWwxcd/xdFxRxcaD7WPdo= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-608-NeNDOhnXMjSlO2wf6HHz1A-1; Mon, 10 Feb 2025 19:32:38 -0500 X-MC-Unique: NeNDOhnXMjSlO2wf6HHz1A-1 X-Mimecast-MFC-AGG-ID: NeNDOhnXMjSlO2wf6HHz1A Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 424F21956054; Tue, 11 Feb 2025 00:32:33 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D3B411800358; Tue, 11 Feb 2025 00:32:24 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 9/9] khugepaged: skip collapsing mTHP to smaller orders Date: Mon, 10 Feb 2025 17:30:28 -0700 Message-ID: <20250211003028.213461-10-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5B73A4000F X-Stat-Signature: b4gtrndq6ocmk9f41a8q15e4gkdkdgea X-HE-Tag: 1739233962-223942 X-HE-Meta: U2FsdGVkX19dHKcHYW+rJOtmK5mQTjhQYmjzc0HV1Z5aiOZZKVmXng0mtw1eWLwIcqMmrexCaSWCWVCm8PMy8SdHyrNNqasimQdHjH0baT7gUvmEpyofixYPW86F9+h8pfZTKCE5Nyw177P/UFI+8wkePmClxcMDSd2+qdFcC1+lrcTPB2LYF5uZnT1Dc4nqEVW0ZuKayirYSHqkwWt8/6T7Q1U0cKYKYqJp9DKXNuP9wrx0EFzCMdUcW5NLXuep4Nd1nhIEM5wq/cC4SgoppuP7PfrmQbAbZN8J57DaXXlQxsTGgrn2mDg5BR4J0yeqzMcJaWSsE4YSLUPy/lzDAXW4Fo8u+8hC3yVPEJ8w+fTgQBGKFbwDEbFhgxvotyIC3dllAl1nDctkRhqq8qoAWt0sXIgWT8XMDLuBwewxJ/U4WxdWEG6x0w0IA9+IYMSG2bEoGYWVh8Idq0qA+WImTi4wZl5Uc4e0d1h2HV0ivFcEtYYd1bDX8zBoWWY4ytf6lrsWkPheeiM7mDGSxu++ABLP/yQ7Pkj9bBRlu1o51Xjcs55k7SW23+WNi9piQHCAXghnjZ9cu9wbmTG3JBWOJVZ+pfd30pn5fVrv3vyWnkoEn+dof9qBCWwN+hLzwf7A4c0bGjwaS5cbMYLVLFaVnXAHIoZXANS6FGS/sHs6bS4nMjP7BAyl+N4R/U7LnEkXgIjOTGxTpSeYFllLg5iBEly3ICJC52DAQJqyELZToccoAkYNCNuM6vdxhtKucBvby6jv5zxb0ok1RRhXbYPDD5NTkFCcLZ1zcvOplWCjq3a1Lyyi08bOrm//YjFK/RyO961hMalp3z8yfjUACCJHhiJsn21sDlUCHrVr+0ohFYQiJjfOd3L3/NG5H9oYVj7CgZ7OH4KEPbVd/EltFu9Df49kyI2Lmd7zpDQJ+u5snm1IsdY9V6HjF46JdodjOm/lOoT6rPbT8ahN+jad4T1 dw5reDOl SgZJDdpx7iL8lxHhNhr78+A5bv3S1uT1MFv1YgAg3UW8L2FhYIvc6ovlq4eOt2uBjnki/FNcORnvfQ0x5qppAukRuNpuJ96PutzVj1I/d7VL93f5/5N8E3NJYEhlfyPQ8Jkewnxc46YGQRpIbvvuDyYEb2l28ChrVbqio+kVMvvsl+MM5fzuQ/uejtQFifbhFFAmMo9STNKeAGag9aP/sUMoZcTs8Hf5Yr9nFXb4kyUl5fURDUHGX/RGcqxyH9akHR6j1MgQMsRRfDusBSUzwfXdQB/FZFfHfIAdEaFn4p4CeHly1E38CRYWlTeaXV0rJ37rZNn3PuQS7WN/s9pS9l0Xm3IoZotBm98sCJnWS7MfDxEtatccq+9lozR8hiC31MDfN8O74EYSqyFJd8HHEQWuQRunfr6F2jNaHGzug6hekNZpi/sy7zsd0AiQE872sFLlvNw9+H0Yc/XAkOFEC8aYtTBHTtr+Ip7UL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: khugepaged may try to collapse a mTHP to a smaller mTHP, resulting in some pages being unmapped. Skip these cases until we have a way to check if its ok to collapse to a smaller mTHP size (like in the case of a partially mapped folio). This patch is inspired by Dev Jain's work on khugepaged mTHP support [1]. [1] https://lore.kernel.org/lkml/20241216165105.56185-11-dev.jain@arm.com/ Signed-off-by: Nico Pache --- mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index e2ba18e57064..fc30698b8e6e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -622,6 +622,11 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); + if (order != HPAGE_PMD_ORDER && folio_order(folio) >= order) { + result = SCAN_PTE_MAPPED_HUGEPAGE; + goto out; + } + /* See khugepaged_scan_pmd(). */ if (folio_likely_mapped_shared(folio)) { ++shared;