From patchwork Wed Dec 18 06:33:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ge Yang X-Patchwork-Id: 13913081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0A4AE77183 for ; Wed, 18 Dec 2024 06:33:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25C526B0082; Wed, 18 Dec 2024 01:33:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20C4F6B0083; Wed, 18 Dec 2024 01:33:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FAD46B0085; Wed, 18 Dec 2024 01:33:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E50476B0082 for ; Wed, 18 Dec 2024 01:33:26 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 68F2A80BC2 for ; Wed, 18 Dec 2024 06:33:26 +0000 (UTC) X-FDA: 82907112288.22.6257CA1 Received: from m16.mail.126.com (m16.mail.126.com [117.135.210.8]) by imf18.hostedemail.com (Postfix) with ESMTP id A6DD61C000F for ; Wed, 18 Dec 2024 06:33:09 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=126.com header.s=s110527 header.b=gQjiwuHy; spf=pass (imf18.hostedemail.com: domain of yangge1116@126.com designates 117.135.210.8 as permitted sender) smtp.mailfrom=yangge1116@126.com; dmarc=pass (policy=none) header.from=126.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734503584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references:dkim-signature; bh=B89UrKJxzsrE/qA+PJa0BPmtaZIYFDVivwLXz6i3bRo=; b=awnV8wm1YXKQfL3Dr7WrnS+Qz/1N1mF7NNTX8kt9f5DnUQe5noOwCJrSw/rymXjVlnSO6k sk6N+cfDsITqpLRdDOZx4jzANXHs3OMVf3MQYZKRT7QpMxDTTJeiroBNuU1oippxpE6vrq yaRRm4S1K4OVhQviBFThnvOO8v5Ay78= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734503584; a=rsa-sha256; cv=none; b=IUsRFNiD04ny1dzJhwSP5RZvyo1IsdrB06VIGw5IkBcuCawZtbQThWX/+ZgxDxAGOXktbI qEtHO4FgUYStthANRFxPoOKogwvajcwxpQ53Rgg5/U1vs3i3ZKFbRl0M7RUQRE4B56AxRj EW/3QzeWLIzE98+ZU/054FTnuWJmiJ0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=126.com header.s=s110527 header.b=gQjiwuHy; spf=pass (imf18.hostedemail.com: domain of yangge1116@126.com designates 117.135.210.8 as permitted sender) smtp.mailfrom=yangge1116@126.com; dmarc=pass (policy=none) header.from=126.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=From:Subject:Date:Message-Id; bh=B89UrKJxzsrE/qA+PJ a0BPmtaZIYFDVivwLXz6i3bRo=; b=gQjiwuHy9ZGXQGT7HSCOOlur7RBM/E6IIF oDMOSFyItErSjROI1Yy0yw+d1QLJ6xzpB3Ovmj1N65AkJQziOkhxMs6mjqXVRNw/ wyiXp83sUiOuSzvcZyTum35vLN8fJ4f4Q4kw0e5E1ZyLRZww/SPxxF6NgvhYFE4u GEbuptIKE= Received: from hg-OptiPlex-7040.hygon.cn (unknown [118.242.3.34]) by gzsmtp5 (Coremail) with SMTP id qSkvCgAXrFSmbGJnzL0lCg--.36335S2; Wed, 18 Dec 2024 14:33:11 +0800 (CST) From: yangge1116@126.com To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, 21cnbao@gmail.com, david@redhat.com, baolin.wang@linux.alibaba.com, muchun.song@linux.dev, liuzixing@hygon.cn, yangge Subject: [PATCH] replace free hugepage folios after migration Date: Wed, 18 Dec 2024 14:33:08 +0800 Message-Id: <1734503588-16254-1-git-send-email-yangge1116@126.com> X-Mailer: git-send-email 2.7.4 X-CM-TRANSID: qSkvCgAXrFSmbGJnzL0lCg--.36335S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxGr15GFW3Cr1DuF47JF4rGrg_yoWrtF15pF y8Grn8GrWDJr9xur4xtan8Ar1avrWkZayjyFWfJ3sxZF13tr9F9Fyqywn0v3yrAr97CF4x ZFZIqFWkuF1UAaDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0zRoGQDUUUUU= X-Originating-IP: [118.242.3.34] X-CM-SenderInfo: 51dqwwjhrrila6rslhhfrp/1tbiWQS5G2diZLmDAAAAsk X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A6DD61C000F X-Stat-Signature: b61y886mmuw4txs5tygez63zzzx8hgm7 X-Rspam-User: X-HE-Tag: 1734503589-784130 X-HE-Meta: U2FsdGVkX180hQD3ygGlCCRtQANzWLLF5dQMaY00yjLfWgBIuQrxNq576VoK3KlNjtXm/lTMARO2hfxcBl8RXlARzxHxDU7zE+zTfzt44RUoYVeqJgiK4Szj7Cpshvn/x34WROMxt6dOVLWoHGmJn7G+G24JlJyher5B3fG8oPi/WSJu1XD4EkTQ8iP8waFWxyrS/H1BqQRlFyMDnXfqOPX7MAH45WsyhFf0Q2Z7Mh97um5us/OnVAwKdWfSNNw0bPDglErHXcoG+CfBl7fYe8g1zZ4IO06ZDqYi1OfwKy6uo0aw+6dKV+dcnVdXKOIZ+I4UCvPda2fRC8JdE6VKbGcbmRHZ3JaYUAYKQLdt5Zm0HF6iJ94i3rEKl5f6FPCmThXeqnGR6/m6tQHkg/Oc6lXLg3H0jRIkfR7E7d7Kv/yecZPYqFLpyvPtv4/yP/oJh8MsasvL0r9VOMtJ9nRFNRu4GGx9k7PUt7pa5T+YEaauHqJMCMWas8roIZvznG7J6MtB0HqnO8E6mLdMRmpMCZT9DRrqj+4O1eeLi4sNQzhvwVbgtQFS6uAwCcEJYvrEFaQlZVTg7LfhXLaJF46KNYeTNVjsRit2MblW9KlARIX6Diu3MoNzPNrmaPQTKjUxw+3017Ci3hvTdv2vTcJhRYEIDPbFmmb8cwaa1vXeS0v3Tp9Uacpf+Qvk18EpsO73glt4Vf4jMpsjNKe///TP10Jnq/jZuqPkyaowRInAq3xP5wTvxHEpck+vlphPxMlCh9mI6DN+dp4smZDvAnXfqmzsrk47wks2thmj4KCFv+zbPGlR/mchDrxep7thEawAsTmS1/9l2h7hLOCFjd8pItkUH1GUASLBHG+te+mtgL+WhuJQwEp3lE5DSlx2zqjIGaegErZJ/2NX8nC1jgNgeaDx8ozShtZSf1c+F1QPMEbYGmMlJTs5U+n9A6wFCJ8TMbBbnRTQ0Ukp+iU98Od 30d3ukib bGHHu9gL3+26OTL4B/S26cy4zlgAhTMQA94D6QSPPq+jta0jA9rUee0kWS4X/kzuDRVrq7BcDGBo6Hjktur11Xz87LIW2AOOqxiIXqsgu/BAX9X0DdBiAjpe4L4hr+Lcjp7J1yuP0+EFlvKXKcWXhcrAoXLC0zUJc0rlFblFJBGeoqinCcWkUIIn9hHujVpiic1nIAvp3vIieFuG2nLUgjJhpWImWLJjp1aDIy1nRhryleXTxoN/1i30mdNSOBvhSyo47uoq/UfVjE/3d10Cj3dm2vH/pCOoIlbLxHsoo5OSxeugGrFqZIKDyRQ0g+FI3hiBy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: yangge My machine has 4 NUMA nodes, each equipped with 32GB of memory. I have configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb pages. The allocation of contiguous memory via the cma_alloc() function can fail probabilistically. The cma_alloc() function may fail if it sees an in-use hugetlb page within the allocation range, even if that page has already been migrated. When in-use hugetlb pages are migrated, they may simply be released back into the free hugepage pool instead of being returned to the buddy system. This can cause the test_pages_isolated() function check to fail, ultimately leading to the failure of the cma_alloc() function: cma_alloc() __alloc_contig_migrate_range() // migrate in-use hugepage test_pages_isolated() __test_page_isolated_in_pageblock() PageBuddy(page) // check if the page is in buddy To address this issue, we will add a function named replace_free_hugepage_folios(). This function will replace the hugepage in the free hugepage pool with a new one and release the old one to the buddy system. After the migration of in-use hugetlb pages is completed, we will invoke the replace_free_hugepage_folios() function to ensure that these hugepages are properly released to the buddy system. Following this step, when the test_pages_isolated() function is executed for inspection, it will successfully pass. Signed-off-by: yangge --- include/linux/hugetlb.h | 6 ++++++ mm/hugetlb.c | 37 +++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 13 ++++++++++++- 3 files changed, 55 insertions(+), 1 deletion(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ae4fe86..7d36ac8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -681,6 +681,7 @@ struct huge_bootmem_page { }; int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list); +int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn); struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, @@ -1059,6 +1060,11 @@ static inline int isolate_or_dissolve_huge_page(struct page *page, return -ENOMEM; } +int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn) +{ + return 0; +} + static inline struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8e1db80..a099c54 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2975,6 +2975,43 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) return ret; } +/* + * replace_free_hugepage_folios - Replace free hugepage folios in a given pfn + * range with new folios. + * @stat_pfn: start pfn of the given pfn range + * @end_pfn: end pfn of the given pfn range + * Returns 0 on success, otherwise negated error. + */ +int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn) +{ + struct hstate *h; + struct folio *folio; + int ret = 0; + + LIST_HEAD(isolate_list); + + while (start_pfn < end_pfn) { + folio = pfn_folio(start_pfn); + if (folio_test_hugetlb(folio)) { + h = folio_hstate(folio); + } else { + start_pfn++; + continue; + } + + if (!folio_ref_count(folio)) { + ret = alloc_and_dissolve_hugetlb_folio(h, folio, &isolate_list); + if (ret) + break; + + putback_movable_pages(&isolate_list); + } + start_pfn++; + } + + return ret; +} + struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index dde19db..1dcea28 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6504,7 +6504,18 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, ret = __alloc_contig_migrate_range(&cc, start, end, migratetype); if (ret && ret != -EBUSY) goto done; - ret = 0; + + /* + * When in-use hugetlb pages are migrated, they may simply be + * released back into the free hugepage pool instead of being + * returned to the buddy system. After the migration of in-use + * huge pages is completed, we will invoke the + * replace_free_hugepage_folios() function to ensure that + * these hugepages are properly released to the buddy system. + */ + ret = replace_free_hugepage_folios(start, end); + if (ret) + goto done; /* * Pages from [start, end) are within a pageblock_nr_pages