| Message ID | 20210219104954.67390-5-songmuchun@bytedance.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Free some vmemmap pages of HugeTLB page | expand |
On Fri 19-02-21 18:49:49, Muchun Song wrote: > When we free a HugeTLB page to the buddy allocator, we should allocate > the vmemmap pages associated with it. But we may cannot allocate vmemmap > pages when the system is under memory pressure, in this case, we just > refuse to free the HugeTLB page instead of looping forever trying to > allocate the pages. This changes some behavior (list below) on some > corner cases. > > 1) Failing to free a huge page triggered by the user (decrease nr_pages). > > Need try again later by the user. > > 2) Failing to free a surplus huge page when freed by the application. > > Try again later when freeing a huge page next time. This means that surplus pages can accumulate right? This should be rather unlikely because one released huge page could then be reused for normal allocations - including vmemmap. Unlucky timing might still end up in the accumulation though. Not something critical though. > 3) Failing to dissolve a free huge page on ZONE_MOVABLE via > offline_pages(). > > This is a bit unfortunate if we have plenty of ZONE_MOVABLE memory > but are low on kernel memory. For example, migration of huge pages > would still work, however, dissolving the free page does not work. > This is a corner cases. When the system is that much under memory > pressure, offlining/unplug can be expected to fail. Please mention that this is unfortunate because it prevents from the memory offlining which shouldn't happen for movable zones. People depending on the memory hotplug and movable zone should carefuly consider whether savings on unmovable memory are worth losing their hotplug functionality in some situations. > 4) Failing to dissolve a huge page on CMA/ZONE_MOVABLE via > alloc_contig_range() - once we have that handling in place. Mainly > affects CMA and virtio-mem. What about hugetlb page poisoning on HW failure (resp. soft offlining)? > > Similar to 3). virito-mem will handle migration errors gracefully. > CMA might be able to fallback on other free areas within the CMA > region. > > We do not want to use GFP_ATOMIC to allocate vmemmap pages. Because it > grants access to memory reserves and we do not think it is reasonable > to use memory reserves. We use GFP_KERNEL in alloc_huge_page_vmemmap(). This likely needs more context around. Maybe something like " Vmemmap pages are allocated from the page freeing context. In order for those allocations to be not disruptive (e.g. trigger oom killer) __GFP_NORETRY is used. hugetlb_lock is dropped for the allocation because a non sleeping allocation would be too fragile and it could fail too easily under memory pressure. GFP_ATOMIC or other modes to access memory reserves is not used because we want to prevent consuming reserves under heavy hugetlb freeing. " I haven't gone through the patch in a great detail yet, from a high level POV it looks good although the counter changes and reshuffling seems little wild. That requires a more detailed look I do not have time for right now. Mike would be much better for that anywya ;) I do not see any check for an atomic context in free_huge_page path. I have suggested to replace in_task by in_atomic check (with a gotcha that the later doesn't work without preempt_count but there is a work to address that).
On Fri, Feb 19, 2021 at 10:12 PM Michal Hocko <mhocko@suse.com> wrote: > > On Fri 19-02-21 18:49:49, Muchun Song wrote: > > When we free a HugeTLB page to the buddy allocator, we should allocate > > the vmemmap pages associated with it. But we may cannot allocate vmemmap > > pages when the system is under memory pressure, in this case, we just > > refuse to free the HugeTLB page instead of looping forever trying to > > allocate the pages. This changes some behavior (list below) on some > > corner cases. > > > > 1) Failing to free a huge page triggered by the user (decrease nr_pages). > > > > Need try again later by the user. > > > > 2) Failing to free a surplus huge page when freed by the application. > > > > Try again later when freeing a huge page next time. > > This means that surplus pages can accumulate right? This should be > rather unlikely because one released huge page could then be reused for > normal allocations - including vmemmap. Unlucky timing might still end > up in the accumulation though. Not something critical though. Agree. > > > 3) Failing to dissolve a free huge page on ZONE_MOVABLE via > > offline_pages(). > > > > This is a bit unfortunate if we have plenty of ZONE_MOVABLE memory > > but are low on kernel memory. For example, migration of huge pages > > would still work, however, dissolving the free page does not work. > > This is a corner cases. When the system is that much under memory > > pressure, offlining/unplug can be expected to fail. > > Please mention that this is unfortunate because it prevents from the > memory offlining which shouldn't happen for movable zones. People > depending on the memory hotplug and movable zone should carefuly > consider whether savings on unmovable memory are worth losing their > hotplug functionality in some situations. Make sense. I will mention this in the change log. Thanks. > > > 4) Failing to dissolve a huge page on CMA/ZONE_MOVABLE via > > alloc_contig_range() - once we have that handling in place. Mainly > > affects CMA and virtio-mem. > > What about hugetlb page poisoning on HW failure (resp. soft offlining)? If the HW poisoned hugetlb page failed to be dissolved, the page will go back to the free list with PG_HWPoison set. But the page will not be used, because we will check whether the page is HW poisoned when it is dequeued from the free list. If so, we will skip this page. > > > > > Similar to 3). virito-mem will handle migration errors gracefully. > > CMA might be able to fallback on other free areas within the CMA > > region. > > > > We do not want to use GFP_ATOMIC to allocate vmemmap pages. Because it > > grants access to memory reserves and we do not think it is reasonable > > to use memory reserves. We use GFP_KERNEL in alloc_huge_page_vmemmap(). > > This likely needs more context around. Maybe something like > " > Vmemmap pages are allocated from the page freeing context. In order for > those allocations to be not disruptive (e.g. trigger oom killer) > __GFP_NORETRY is used. hugetlb_lock is dropped for the allocation > because a non sleeping allocation would be too fragile and it could fail > too easily under memory pressure. GFP_ATOMIC or other modes to access > memory reserves is not used because we want to prevent consuming > reserves under heavy hugetlb freeing. > " Thanks. I will use this to the change log. > > I haven't gone through the patch in a great detail yet, from a high > level POV it looks good although the counter changes and reshuffling > seems little wild. That requires a more detailed look I do not have time > for right now. Mike would be much better for that anywya ;) Yeah. Hope Mike will review this (I believe he is good at this area). > > I do not see any check for an atomic context in free_huge_page path. I > have suggested to replace in_task by in_atomic check (with a gotcha that > the later doesn't work without preempt_count but there is a work to > address that). Sorry. I forgot it. I will replace in_task with in_atomic in the next version. Thanks for your suggestions. > -- > Michal Hocko > SUSE Labs
On Sat 20-02-21 12:20:36, Muchun Song wrote: > On Fri, Feb 19, 2021 at 10:12 PM Michal Hocko <mhocko@suse.com> wrote: [...] > > What about hugetlb page poisoning on HW failure (resp. soft offlining)? > > If the HW poisoned hugetlb page failed to be dissolved, the page > will go back to the free list with PG_HWPoison set. But the page > will not be used, because we will check whether the page is HW > poisoned when it is dequeued from the free list. If so, we will skip > this page. Can this lead to an underprovisioned pool then? Or is there a new hugetlb allocated to replace the poisoned one?
On Mon, Feb 22, 2021 at 5:25 PM Michal Hocko <mhocko@suse.com> wrote: > > On Sat 20-02-21 12:20:36, Muchun Song wrote: > > On Fri, Feb 19, 2021 at 10:12 PM Michal Hocko <mhocko@suse.com> wrote: > [...] > > > What about hugetlb page poisoning on HW failure (resp. soft offlining)? > > > > If the HW poisoned hugetlb page failed to be dissolved, the page > > will go back to the free list with PG_HWPoison set. But the page > > will not be used, because we will check whether the page is HW > > poisoned when it is dequeued from the free list. If so, we will skip > > this page. > > Can this lead to an under provisioned pool then? Or is there a new > hugetlb allocated to replace the poisoned one? Actually, no page will be allocated. Your concern is right. But without this patch, the result does not change. e.g. The HW poisoned page can fail to be dissolved when h->free_huge_pages is equal to h->resv_huge_pages. But no one seems to have reported this issue so far. Maybe this behavior needs improvement in the feature. > > -- > Michal Hocko > SUSE Labs
On Mon, Feb 22, 2021 at 06:31:12PM +0800, Muchun Song wrote: > On Mon, Feb 22, 2021 at 5:25 PM Michal Hocko <mhocko@suse.com> wrote: > > > > On Sat 20-02-21 12:20:36, Muchun Song wrote: > > > On Fri, Feb 19, 2021 at 10:12 PM Michal Hocko <mhocko@suse.com> wrote: > > [...] > > > > What about hugetlb page poisoning on HW failure (resp. soft offlining)? > > > > > > If the HW poisoned hugetlb page failed to be dissolved, the page > > > will go back to the free list with PG_HWPoison set. But the page > > > will not be used, because we will check whether the page is HW > > > poisoned when it is dequeued from the free list. If so, we will skip > > > this page. Not really. If the huge page is dissolved, we will take the page out of the the freelist. See take_page_off_buddy in memory_failure_hugetlb. In an ideal world, we should inspect that page in free_pages_prepare(), remove the HPWpoisoned page and process the others, without letting that page hit Buddy. And not only for hugetlb, but for any higher order page. See how memory_failure() happily disengage itself when it finds a higher order page. It does it because we have the premise that once that page hits Buddy, it will stay there as the check_new_page guards us. But this has been proofed to be quite a weak measure, as compaction does not performs such a check, and so the page can sneak in. I fixed that for soft-offline, and for memory-failure in some cases, but more needs to be done and is it in my TODO list. > > Can this lead to an under provisioned pool then? Or is there a new > > hugetlb allocated to replace the poisoned one? > > Actually, no page will be allocated. Your concern is right. But without > this patch, the result does not change. e.g. The HW poisoned page > can fail to be dissolved when h->free_huge_pages is equal to > h->resv_huge_pages. But no one seems to have reported this issue so > far. Maybe this behavior needs improvement in the feature. Yes, something to improve. I shall have a look.
On 2/19/21 2:49 AM, Muchun Song wrote: > When we free a HugeTLB page to the buddy allocator, we should allocate > the vmemmap pages associated with it. But we may cannot allocate vmemmap > pages when the system is under memory pressure, in this case, we just > refuse to free the HugeTLB page instead of looping forever trying to > allocate the pages. This changes some behavior (list below) on some > corner cases. Thank you for listing changes in behavior and possible side effects of not being able to allocate vmemmmap and free huge page to buddy! I will not repeat Michal's comment about the check for an atomic context in free_huge_page path. > > 1) Failing to free a huge page triggered by the user (decrease nr_pages). > > Need try again later by the user. > > 2) Failing to free a surplus huge page when freed by the application. > > Try again later when freeing a huge page next time. > > 3) Failing to dissolve a free huge page on ZONE_MOVABLE via > offline_pages(). > > This is a bit unfortunate if we have plenty of ZONE_MOVABLE memory > but are low on kernel memory. For example, migration of huge pages > would still work, however, dissolving the free page does not work. > This is a corner cases. When the system is that much under memory > pressure, offlining/unplug can be expected to fail. > > 4) Failing to dissolve a huge page on CMA/ZONE_MOVABLE via > alloc_contig_range() - once we have that handling in place. Mainly > affects CMA and virtio-mem. > > Similar to 3). virito-mem will handle migration errors gracefully. > CMA might be able to fallback on other free areas within the CMA > region. > > We do not want to use GFP_ATOMIC to allocate vmemmap pages. Because it > grants access to memory reserves and we do not think it is reasonable > to use memory reserves. We use GFP_KERNEL in alloc_huge_page_vmemmap(). > > Signed-off-by: Muchun Song <songmuchun@bytedance.com> > --- > Documentation/admin-guide/mm/hugetlbpage.rst | 8 +++ > include/linux/mm.h | 2 + > mm/hugetlb.c | 81 ++++++++++++++++++++-------- > mm/hugetlb_vmemmap.c | 22 ++++++++ > mm/hugetlb_vmemmap.h | 6 +++ > mm/sparse-vmemmap.c | 75 +++++++++++++++++++++++++- > 6 files changed, 171 insertions(+), 23 deletions(-) > > diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst > index f7b1c7462991..fb8f649e5635 100644 > --- a/Documentation/admin-guide/mm/hugetlbpage.rst > +++ b/Documentation/admin-guide/mm/hugetlbpage.rst > @@ -60,6 +60,10 @@ HugePages_Surp > the pool above the value in ``/proc/sys/vm/nr_hugepages``. The > maximum number of surplus huge pages is controlled by > ``/proc/sys/vm/nr_overcommit_hugepages``. > + Note: When the feature of freeing unused vmemmap pages associated > + with each hugetlb page is enabled, the number of the surplus huge Small wording change: with each hugetlb page is enabled, the number of surplus huge > + pages may be temporarily larger than the maximum number of surplus > + huge pages when the system is under memory pressure. > Hugepagesize > is the default hugepage size (in Kb). > Hugetlb > @@ -80,6 +84,10 @@ returned to the huge page pool when freed by a task. A user with root > privileges can dynamically allocate more or free some persistent huge pages > by increasing or decreasing the value of ``nr_hugepages``. > > +Note: When the feature of freeing unused vmemmap pages associated with each > +hugetlb page is enabled, we can failed to free the huge pages triggered by Small wording change: hugetlb page is enabled, we can fail to free the huge pages triggered by > +the user when ths system is under memory pressure. Please try again later. > + > Pages that are used as huge pages are reserved inside the kernel and cannot > be used for other purposes. Huge pages cannot be swapped out under > memory pressure. > diff --git a/include/linux/mm.h b/include/linux/mm.h > index d7dddf334779..33c5911afe18 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -2981,6 +2981,8 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) > > void vmemmap_remap_free(unsigned long start, unsigned long end, > unsigned long reuse); > +int vmemmap_remap_alloc(unsigned long start, unsigned long end, > + unsigned long reuse, gfp_t gfp_mask); > > void *sparse_buffer_alloc(unsigned long size); > struct page * __populate_section_memmap(unsigned long pfn, > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index 4cfca27c6d32..bcf856974c48 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -1305,37 +1305,68 @@ static inline void destroy_compound_gigantic_page(struct page *page, > unsigned int order) { } > #endif > > -static void update_and_free_page(struct hstate *h, struct page *page) > +static int update_and_free_page(struct hstate *h, struct page *page) > + __releases(&hugetlb_lock) __acquires(&hugetlb_lock) > { > int i; > + int nid = page_to_nid(page); > > if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) > - return; > + return 0; > > h->nr_huge_pages--; > - h->nr_huge_pages_node[page_to_nid(page)]--; > + h->nr_huge_pages_node[nid]--; > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > + set_page_refcounted(page); I think you added the set_page_refcounted() because the huge page will appear as just a compound page without a reference after dropping the hugetlb lock? It might be better to set the reference before modifying the destructor. Otherwise, page scanning code could find the non-hugetlb compound page with no reference. I could not find any code where this would be a problem, but I think it would be safer to set the reference first. > + spin_unlock(&hugetlb_lock); I really like the way this code is structured. It is much simpler than previous versions with retries or workqueue. There is nothing wrong with always dropping the lock here. However, I wonder if we should think about optimizing for the case where this feature is not enabled and we are not freeing a 1G huge page. I suspect this will be the most common case for some time, and there is no need to drop the lock in this case. Please do not change the code based on my comment. I just wanted to bring this up for thought. Is it as simple as checking? if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) spin_unlock(&hugetlb_lock); /* before return */ if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) spin_lock(&hugetlb_lock); > + > + if (alloc_huge_page_vmemmap(h, page)) { > + int zeroed; > + > + spin_lock(&hugetlb_lock); > + INIT_LIST_HEAD(&page->lru); > + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); > + h->nr_huge_pages++; > + h->nr_huge_pages_node[nid]++; > + > + /* > + * If we cannot allocate vmemmap pages, just refuse to free the > + * page and put the page back on the hugetlb free list and treat > + * as a surplus page. > + */ > + h->surplus_huge_pages++; > + h->surplus_huge_pages_node[nid]++; > + > + /* > + * This page is now managed by the hugetlb allocator and has > + * no users -- drop the last reference. > + */ > + zeroed = put_page_testzero(page); > + VM_BUG_ON_PAGE(!zeroed, page); > + arch_clear_hugepage_flags(page); > + enqueue_huge_page(h, page); > + > + return -ENOMEM; > + } > + > for (i = 0; i < pages_per_huge_page(h); i++) { > page[i].flags &= ~(1 << PG_locked | 1 << PG_error | > 1 << PG_referenced | 1 << PG_dirty | > 1 << PG_active | 1 << PG_private | > 1 << PG_writeback); > } > - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > - set_page_refcounted(page); > if (hstate_is_gigantic(h)) { > - /* > - * Temporarily drop the hugetlb_lock, because > - * we might block in free_gigantic_page(). > - */ > - spin_unlock(&hugetlb_lock); > destroy_compound_gigantic_page(page, huge_page_order(h)); > free_gigantic_page(page, huge_page_order(h)); > - spin_lock(&hugetlb_lock); > } else { > __free_pages(page, huge_page_order(h)); > } > + > + spin_lock(&hugetlb_lock); > + > + return 0; > } > > struct hstate *size_to_hstate(unsigned long size) > @@ -1403,9 +1434,9 @@ static void __free_huge_page(struct page *page) > } else if (h->surplus_huge_pages_node[nid]) { > /* remove the page from active list */ > list_del(&page->lru); > - update_and_free_page(h, page); > h->surplus_huge_pages--; > h->surplus_huge_pages_node[nid]--; > + update_and_free_page(h, page); > } else { > arch_clear_hugepage_flags(page); > enqueue_huge_page(h, page); > @@ -1693,6 +1724,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, > struct page *page = > list_entry(h->hugepage_freelists[node].next, > struct page, lru); > + ClearHPageFreed(page); Quick question. Is this change directly related to the vmemmap changes, or is it a cleanup that you noticed? > list_del(&page->lru); > h->free_huge_pages--; > h->free_huge_pages_node[node]--; > @@ -1700,8 +1732,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, > h->surplus_huge_pages--; > h->surplus_huge_pages_node[node]--; > } > - update_and_free_page(h, page); > - ret = 1; > + ret = !update_and_free_page(h, page); > break; > } > } > @@ -1714,10 +1745,14 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, > * nothing for in-use hugepages and non-hugepages. > * This function returns values like below: > * > - * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use > - * (allocated or reserved.) > - * 0: successfully dissolved free hugepages or the page is not a > - * hugepage (considered as already dissolved) > + * -ENOMEM: failed to allocate vmemmap pages to free the freed hugepages > + * when the system is under memory pressure and the feature of > + * freeing unused vmemmap pages associated with each hugetlb page > + * is enabled. > + * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use > + * (allocated or reserved.) > + * 0: successfully dissolved free hugepages or the page is not a > + * hugepage (considered as already dissolved) > */ > int dissolve_free_huge_page(struct page *page) > { > @@ -1768,12 +1803,14 @@ int dissolve_free_huge_page(struct page *page) > SetPageHWPoison(page); > ClearPageHWPoison(head); > } > + ClearHPageFreed(page); > list_del(&head->lru); > h->free_huge_pages--; > h->free_huge_pages_node[nid]--; > h->max_huge_pages--; > - update_and_free_page(h, head); > - rc = 0; > + rc = update_and_free_page(h, head); > + if (rc) > + h->max_huge_pages++; Since update_and_free_page failed, the number of surplus pages was incremented. Surplus pages are the number of pages greater than max_huge_pages. Since we are incrementing max_huge_pages, we should decrement (undo) the addition to surplus_huge_pages and surplus_huge_pages_node[nid]. So, I think we want h->surplus_huge_pages--; h->surplus_huge_pages_node[nid]--; here as well. > } > out: > spin_unlock(&hugetlb_lock); In previous version of this patch series, we discussed and refined the vmemmap manipulation routines below. They still look good to me. In general, I like the approach taken in this patch. Hopefully, others will comment and we can move the series forward.
On Tue, Feb 23, 2021 at 8:01 AM Mike Kravetz <mike.kravetz@oracle.com> wrote: > > On 2/19/21 2:49 AM, Muchun Song wrote: > > When we free a HugeTLB page to the buddy allocator, we should allocate > > the vmemmap pages associated with it. But we may cannot allocate vmemmap > > pages when the system is under memory pressure, in this case, we just > > refuse to free the HugeTLB page instead of looping forever trying to > > allocate the pages. This changes some behavior (list below) on some > > corner cases. > > Thank you for listing changes in behavior and possible side effects of > not being able to allocate vmemmmap and free huge page to buddy! > > I will not repeat Michal's comment about the check for an atomic context > in free_huge_page path. > > > > > 1) Failing to free a huge page triggered by the user (decrease nr_pages). > > > > Need try again later by the user. > > > > 2) Failing to free a surplus huge page when freed by the application. > > > > Try again later when freeing a huge page next time. > > > > 3) Failing to dissolve a free huge page on ZONE_MOVABLE via > > offline_pages(). > > > > This is a bit unfortunate if we have plenty of ZONE_MOVABLE memory > > but are low on kernel memory. For example, migration of huge pages > > would still work, however, dissolving the free page does not work. > > This is a corner cases. When the system is that much under memory > > pressure, offlining/unplug can be expected to fail. > > > > 4) Failing to dissolve a huge page on CMA/ZONE_MOVABLE via > > alloc_contig_range() - once we have that handling in place. Mainly > > affects CMA and virtio-mem. > > > > Similar to 3). virito-mem will handle migration errors gracefully. > > CMA might be able to fallback on other free areas within the CMA > > region. > > > > We do not want to use GFP_ATOMIC to allocate vmemmap pages. Because it > > grants access to memory reserves and we do not think it is reasonable > > to use memory reserves. We use GFP_KERNEL in alloc_huge_page_vmemmap(). > > > > Signed-off-by: Muchun Song <songmuchun@bytedance.com> > > --- > > Documentation/admin-guide/mm/hugetlbpage.rst | 8 +++ > > include/linux/mm.h | 2 + > > mm/hugetlb.c | 81 ++++++++++++++++++++-------- > > mm/hugetlb_vmemmap.c | 22 ++++++++ > > mm/hugetlb_vmemmap.h | 6 +++ > > mm/sparse-vmemmap.c | 75 +++++++++++++++++++++++++- > > 6 files changed, 171 insertions(+), 23 deletions(-) > > > > diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst > > index f7b1c7462991..fb8f649e5635 100644 > > --- a/Documentation/admin-guide/mm/hugetlbpage.rst > > +++ b/Documentation/admin-guide/mm/hugetlbpage.rst > > @@ -60,6 +60,10 @@ HugePages_Surp > > the pool above the value in ``/proc/sys/vm/nr_hugepages``. The > > maximum number of surplus huge pages is controlled by > > ``/proc/sys/vm/nr_overcommit_hugepages``. > > + Note: When the feature of freeing unused vmemmap pages associated > > + with each hugetlb page is enabled, the number of the surplus huge > > Small wording change: > > with each hugetlb page is enabled, the number of surplus huge Thanks. I will update this. > > > + pages may be temporarily larger than the maximum number of surplus > > + huge pages when the system is under memory pressure. > > Hugepagesize > > is the default hugepage size (in Kb). > > Hugetlb > > @@ -80,6 +84,10 @@ returned to the huge page pool when freed by a task. A user with root > > privileges can dynamically allocate more or free some persistent huge pages > > by increasing or decreasing the value of ``nr_hugepages``. > > > > +Note: When the feature of freeing unused vmemmap pages associated with each > > +hugetlb page is enabled, we can failed to free the huge pages triggered by > > Small wording change: > > hugetlb page is enabled, we can fail to free the huge pages triggered by Thanks. I will update this. > > > +the user when ths system is under memory pressure. Please try again later. > > + > > Pages that are used as huge pages are reserved inside the kernel and cannot > > be used for other purposes. Huge pages cannot be swapped out under > > memory pressure. > > diff --git a/include/linux/mm.h b/include/linux/mm.h > > index d7dddf334779..33c5911afe18 100644 > > --- a/include/linux/mm.h > > +++ b/include/linux/mm.h > > @@ -2981,6 +2981,8 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) > > > > void vmemmap_remap_free(unsigned long start, unsigned long end, > > unsigned long reuse); > > +int vmemmap_remap_alloc(unsigned long start, unsigned long end, > > + unsigned long reuse, gfp_t gfp_mask); > > > > void *sparse_buffer_alloc(unsigned long size); > > struct page * __populate_section_memmap(unsigned long pfn, > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > > index 4cfca27c6d32..bcf856974c48 100644 > > --- a/mm/hugetlb.c > > +++ b/mm/hugetlb.c > > @@ -1305,37 +1305,68 @@ static inline void destroy_compound_gigantic_page(struct page *page, > > unsigned int order) { } > > #endif > > > > -static void update_and_free_page(struct hstate *h, struct page *page) > > +static int update_and_free_page(struct hstate *h, struct page *page) > > + __releases(&hugetlb_lock) __acquires(&hugetlb_lock) > > { > > int i; > > + int nid = page_to_nid(page); > > > > if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) > > - return; > > + return 0; > > > > h->nr_huge_pages--; > > - h->nr_huge_pages_node[page_to_nid(page)]--; > > + h->nr_huge_pages_node[nid]--; > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > > + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > > + set_page_refcounted(page); > > I think you added the set_page_refcounted() because the huge page will > appear as just a compound page without a reference after dropping the > hugetlb lock? Right. > It might be better to set the reference before modifying > the destructor. Otherwise, page scanning code could find the non-hugetlb > compound page with no reference. I could not find any code where this > would be a problem, but I think it would be safer to set the reference > first. Make sense to me. It is better to set the refcount first. > > > + spin_unlock(&hugetlb_lock); > > I really like the way this code is structured. It is much simpler than > previous versions with retries or workqueue. There is nothing wrong with > always dropping the lock here. However, I wonder if we should think about > optimizing for the case where this feature is not enabled and we are not > freeing a 1G huge page. I suspect this will be the most common case for > some time, and there is no need to drop the lock in this case. > > Please do not change the code based on my comment. I just wanted to bring > this up for thought. At least make sense to me. It may take a long time to free a 1G huge page. Dropping the lock may be a good choice. But I also want to listen to Oscar and Michal's opinion on this. > > Is it as simple as checking? > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > spin_unlock(&hugetlb_lock); > > /* before return */ > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > spin_lock(&hugetlb_lock); > > > + > > + if (alloc_huge_page_vmemmap(h, page)) { > > + int zeroed; > > + > > + spin_lock(&hugetlb_lock); > > + INIT_LIST_HEAD(&page->lru); > > + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); > > + h->nr_huge_pages++; > > + h->nr_huge_pages_node[nid]++; > > + > > + /* > > + * If we cannot allocate vmemmap pages, just refuse to free the > > + * page and put the page back on the hugetlb free list and treat > > + * as a surplus page. > > + */ > > + h->surplus_huge_pages++; > > + h->surplus_huge_pages_node[nid]++; > > + > > + /* > > + * This page is now managed by the hugetlb allocator and has > > + * no users -- drop the last reference. > > + */ > > + zeroed = put_page_testzero(page); > > + VM_BUG_ON_PAGE(!zeroed, page); > > + arch_clear_hugepage_flags(page); > > + enqueue_huge_page(h, page); > > + > > + return -ENOMEM; > > + } > > + > > for (i = 0; i < pages_per_huge_page(h); i++) { > > page[i].flags &= ~(1 << PG_locked | 1 << PG_error | > > 1 << PG_referenced | 1 << PG_dirty | > > 1 << PG_active | 1 << PG_private | > > 1 << PG_writeback); > > } > > - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > > - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > > - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > > - set_page_refcounted(page); > > if (hstate_is_gigantic(h)) { > > - /* > > - * Temporarily drop the hugetlb_lock, because > > - * we might block in free_gigantic_page(). > > - */ > > - spin_unlock(&hugetlb_lock); > > destroy_compound_gigantic_page(page, huge_page_order(h)); > > free_gigantic_page(page, huge_page_order(h)); > > - spin_lock(&hugetlb_lock); > > } else { > > __free_pages(page, huge_page_order(h)); > > } > > + > > + spin_lock(&hugetlb_lock); > > + > > + return 0; > > } > > > > struct hstate *size_to_hstate(unsigned long size) > > @@ -1403,9 +1434,9 @@ static void __free_huge_page(struct page *page) > > } else if (h->surplus_huge_pages_node[nid]) { > > /* remove the page from active list */ > > list_del(&page->lru); > > - update_and_free_page(h, page); > > h->surplus_huge_pages--; > > h->surplus_huge_pages_node[nid]--; > > + update_and_free_page(h, page); > > } else { > > arch_clear_hugepage_flags(page); > > enqueue_huge_page(h, page); > > @@ -1693,6 +1724,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, > > struct page *page = > > list_entry(h->hugepage_freelists[node].next, > > struct page, lru); > > + ClearHPageFreed(page); > > Quick question. Is this change directly related to the vmemmap changes, > or is it a cleanup that you noticed? Just a cleanup. Maybe there should be a separate patch for this. > > > list_del(&page->lru); > > h->free_huge_pages--; > > h->free_huge_pages_node[node]--; > > @@ -1700,8 +1732,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, > > h->surplus_huge_pages--; > > h->surplus_huge_pages_node[node]--; > > } > > - update_and_free_page(h, page); > > - ret = 1; > > + ret = !update_and_free_page(h, page); > > break; > > } > > } > > @@ -1714,10 +1745,14 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, > > * nothing for in-use hugepages and non-hugepages. > > * This function returns values like below: > > * > > - * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use > > - * (allocated or reserved.) > > - * 0: successfully dissolved free hugepages or the page is not a > > - * hugepage (considered as already dissolved) > > + * -ENOMEM: failed to allocate vmemmap pages to free the freed hugepages > > + * when the system is under memory pressure and the feature of > > + * freeing unused vmemmap pages associated with each hugetlb page > > + * is enabled. > > + * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use > > + * (allocated or reserved.) > > + * 0: successfully dissolved free hugepages or the page is not a > > + * hugepage (considered as already dissolved) > > */ > > int dissolve_free_huge_page(struct page *page) > > { > > @@ -1768,12 +1803,14 @@ int dissolve_free_huge_page(struct page *page) > > SetPageHWPoison(page); > > ClearPageHWPoison(head); > > } > > + ClearHPageFreed(page); > > list_del(&head->lru); > > h->free_huge_pages--; > > h->free_huge_pages_node[nid]--; > > h->max_huge_pages--; > > - update_and_free_page(h, head); > > - rc = 0; > > + rc = update_and_free_page(h, head); > > + if (rc) > > + h->max_huge_pages++; > > Since update_and_free_page failed, the number of surplus pages was > incremented. Surplus pages are the number of pages greater than > max_huge_pages. Since we are incrementing max_huge_pages, we should > decrement (undo) the addition to surplus_huge_pages and > surplus_huge_pages_node[nid]. So, I think we want > h->surplus_huge_pages--; > h->surplus_huge_pages_node[nid]--; > here as well. You are right. Thanks for reminding me of this. > > > } > > out: > > spin_unlock(&hugetlb_lock); > > In previous version of this patch series, we discussed and refined the > vmemmap manipulation routines below. They still look good to me. > > In general, I like the approach taken in this patch. Hopefully, others > will comment and we can move the series forward. > -- > Mike Kravetz > > > diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c > > index 0209b736e0b4..29a3380f3b20 100644 > > --- a/mm/hugetlb_vmemmap.c > > +++ b/mm/hugetlb_vmemmap.c > > @@ -198,6 +198,28 @@ static inline unsigned long free_vmemmap_pages_size_per_hpage(struct hstate *h) > > return (unsigned long)free_vmemmap_pages_per_hpage(h) << PAGE_SHIFT; > > } > > > > +int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) > > +{ > > + unsigned long vmemmap_addr = (unsigned long)head; > > + unsigned long vmemmap_end, vmemmap_reuse; > > + > > + if (!free_vmemmap_pages_per_hpage(h)) > > + return 0; > > + > > + vmemmap_addr += RESERVE_VMEMMAP_SIZE; > > + vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); > > + vmemmap_reuse = vmemmap_addr - PAGE_SIZE; > > + /* > > + * The pages which the vmemmap virtual address range [@vmemmap_addr, > > + * @vmemmap_end) are mapped to are freed to the buddy allocator, and > > + * the range is mapped to the page which @vmemmap_reuse is mapped to. > > + * When a HugeTLB page is freed to the buddy allocator, previously > > + * discarded vmemmap pages must be allocated and remapping. > > + */ > > + return vmemmap_remap_alloc(vmemmap_addr, vmemmap_end, vmemmap_reuse, > > + GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); > > +} > > + > > void free_huge_page_vmemmap(struct hstate *h, struct page *head) > > { > > unsigned long vmemmap_addr = (unsigned long)head; > > diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h > > index 6923f03534d5..e5547d53b9f5 100644 > > --- a/mm/hugetlb_vmemmap.h > > +++ b/mm/hugetlb_vmemmap.h > > @@ -11,8 +11,14 @@ > > #include <linux/hugetlb.h> > > > > #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP > > +int alloc_huge_page_vmemmap(struct hstate *h, struct page *head); > > void free_huge_page_vmemmap(struct hstate *h, struct page *head); > > #else > > +static inline int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) > > +{ > > + return 0; > > +} > > + > > static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) > > { > > } > > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c > > index d3076a7a3783..60fc6cd6cd23 100644 > > --- a/mm/sparse-vmemmap.c > > +++ b/mm/sparse-vmemmap.c > > @@ -40,7 +40,8 @@ > > * @remap_pte: called for each lowest-level entry (PTE). > > * @reuse_page: the page which is reused for the tail vmemmap pages. > > * @reuse_addr: the virtual address of the @reuse_page page. > > - * @vmemmap_pages: the list head of the vmemmap pages that can be freed. > > + * @vmemmap_pages: the list head of the vmemmap pages that can be freed > > + * or is mapped from. > > */ > > struct vmemmap_remap_walk { > > void (*remap_pte)(pte_t *pte, unsigned long addr, > > @@ -237,6 +238,78 @@ void vmemmap_remap_free(unsigned long start, unsigned long end, > > free_vmemmap_page_list(&vmemmap_pages); > > } > > > > +static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, > > + struct vmemmap_remap_walk *walk) > > +{ > > + pgprot_t pgprot = PAGE_KERNEL; > > + struct page *page; > > + void *to; > > + > > + BUG_ON(pte_page(*pte) != walk->reuse_page); > > + > > + page = list_first_entry(walk->vmemmap_pages, struct page, lru); > > + list_del(&page->lru); > > + to = page_to_virt(page); > > + copy_page(to, (void *)walk->reuse_addr); > > + > > + set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); > > +} > > + > > +static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, > > + gfp_t gfp_mask, struct list_head *list) > > +{ > > + unsigned long nr_pages = (end - start) >> PAGE_SHIFT; > > + int nid = page_to_nid((struct page *)start); > > + struct page *page, *next; > > + > > + while (nr_pages--) { > > + page = alloc_pages_node(nid, gfp_mask, 0); > > + if (!page) > > + goto out; > > + list_add_tail(&page->lru, list); > > + } > > + > > + return 0; > > +out: > > + list_for_each_entry_safe(page, next, list, lru) > > + __free_pages(page, 0); > > + return -ENOMEM; > > +} > > + > > +/** > > + * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end) > > + * to the page which is from the @vmemmap_pages > > + * respectively. > > + * @start: start address of the vmemmap virtual address range that we want > > + * to remap. > > + * @end: end address of the vmemmap virtual address range that we want to > > + * remap. > > + * @reuse: reuse address. > > + * @gpf_mask: GFP flag for allocating vmemmap pages. > > + */ > > +int vmemmap_remap_alloc(unsigned long start, unsigned long end, > > + unsigned long reuse, gfp_t gfp_mask) > > +{ > > + LIST_HEAD(vmemmap_pages); > > + struct vmemmap_remap_walk walk = { > > + .remap_pte = vmemmap_restore_pte, > > + .reuse_addr = reuse, > > + .vmemmap_pages = &vmemmap_pages, > > + }; > > + > > + /* See the comment in the vmemmap_remap_free(). */ > > + BUG_ON(start - reuse != PAGE_SIZE); > > + > > + might_sleep_if(gfpflags_allow_blocking(gfp_mask)); > > + > > + if (alloc_vmemmap_page_list(start, end, gfp_mask, &vmemmap_pages)) > > + return -ENOMEM; > > + > > + vmemmap_remap_range(reuse, end, &walk); > > + > > + return 0; > > +} > > + > > /* > > * Allocate a block of memory to be used to back the virtual memory map > > * or to back the page tables that are used to create the mapping. > >
On Mon, Feb 22, 2021 at 04:00:27PM -0800, Mike Kravetz wrote: > > -static void update_and_free_page(struct hstate *h, struct page *page) > > +static int update_and_free_page(struct hstate *h, struct page *page) > > + __releases(&hugetlb_lock) __acquires(&hugetlb_lock) > > { > > int i; > > + int nid = page_to_nid(page); > > > > if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) > > - return; > > + return 0; > > > > h->nr_huge_pages--; > > - h->nr_huge_pages_node[page_to_nid(page)]--; > > + h->nr_huge_pages_node[nid]--; > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > > + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > > + set_page_refcounted(page); > > I think you added the set_page_refcounted() because the huge page will > appear as just a compound page without a reference after dropping the > hugetlb lock? It might be better to set the reference before modifying > the destructor. Otherwise, page scanning code could find the non-hugetlb > compound page with no reference. I could not find any code where this > would be a problem, but I think it would be safer to set the reference > first. But we already had set_page_refcounted() before this patchset there. Are the worries only because we drop the lock? AFAICS, the "page-scanning" problem could have happened before as well? Although, what does page scanning mean in this context? I am not opposed to move it above, but I would like to understand the concern here. > > > + spin_unlock(&hugetlb_lock); > > I really like the way this code is structured. It is much simpler than > previous versions with retries or workqueue. There is nothing wrong with > always dropping the lock here. However, I wonder if we should think about > optimizing for the case where this feature is not enabled and we are not > freeing a 1G huge page. I suspect this will be the most common case for > some time, and there is no need to drop the lock in this case. > > Please do not change the code based on my comment. I just wanted to bring > this up for thought. > > Is it as simple as checking? > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > spin_unlock(&hugetlb_lock); > > /* before return */ > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > spin_lock(&hugetlb_lock); AFAIK, we at least need the hstate_is_gigantic? Comment below says that free_gigantic_page might block, so we need to drop the lock. And I am fine with the change overall. Unless I am missing something, we should not need to drop the lock unless we need to allocate vmemmap pages (apart from gigantic pages). > > > + > > + if (alloc_huge_page_vmemmap(h, page)) { > > + int zeroed; > > + > > + spin_lock(&hugetlb_lock); > > + INIT_LIST_HEAD(&page->lru); > > + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); > > + h->nr_huge_pages++; > > + h->nr_huge_pages_node[nid]++; I think prep_new_huge_page() does this for us? > > + > > + /* > > + * If we cannot allocate vmemmap pages, just refuse to free the > > + * page and put the page back on the hugetlb free list and treat > > + * as a surplus page. > > + */ > > + h->surplus_huge_pages++; > > + h->surplus_huge_pages_node[nid]++; > > + > > + /* > > + * This page is now managed by the hugetlb allocator and has > > + * no users -- drop the last reference. > > + */ > > + zeroed = put_page_testzero(page); > > + VM_BUG_ON_PAGE(!zeroed, page); Can this actually happen? AFAIK, page landed in update_and_free_page should be zero refcounted, then we increase the reference, and I cannot see how the reference might have changed in the meantime. I am all for catching corner cases, but not sure how realistic this is. Moreover, if we __ever__ get there, things can get nasty. We basically will have an in-use page in the free hugetlb pool, so corruption will happen. At that point, a plain BUG_ON might be better. But as I said, I do not think we need that. I yet need to look further, but what I have seen so far looks good.
On Tue, Feb 23, 2021 at 5:28 PM Oscar Salvador <osalvador@suse.de> wrote: > > On Mon, Feb 22, 2021 at 04:00:27PM -0800, Mike Kravetz wrote: > > > -static void update_and_free_page(struct hstate *h, struct page *page) > > > +static int update_and_free_page(struct hstate *h, struct page *page) > > > + __releases(&hugetlb_lock) __acquires(&hugetlb_lock) > > > { > > > int i; > > > + int nid = page_to_nid(page); > > > > > > if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) > > > - return; > > > + return 0; > > > > > > h->nr_huge_pages--; > > > - h->nr_huge_pages_node[page_to_nid(page)]--; > > > + h->nr_huge_pages_node[nid]--; > > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > > > + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > > > + set_page_refcounted(page); > > > > I think you added the set_page_refcounted() because the huge page will > > appear as just a compound page without a reference after dropping the > > hugetlb lock? It might be better to set the reference before modifying > > the destructor. Otherwise, page scanning code could find the non-hugetlb > > compound page with no reference. I could not find any code where this > > would be a problem, but I think it would be safer to set the reference > > first. > > But we already had set_page_refcounted() before this patchset there. > Are the worries only because we drop the lock? AFAICS, the "page-scanning" > problem could have happened before as well? > Although, what does page scanning mean in this context? > > I am not opposed to move it above, but I would like to understand the concern > here. > > > > > > + spin_unlock(&hugetlb_lock); > > > > I really like the way this code is structured. It is much simpler than > > previous versions with retries or workqueue. There is nothing wrong with > > always dropping the lock here. However, I wonder if we should think about > > optimizing for the case where this feature is not enabled and we are not > > freeing a 1G huge page. I suspect this will be the most common case for > > some time, and there is no need to drop the lock in this case. > > > > Please do not change the code based on my comment. I just wanted to bring > > this up for thought. > > > > Is it as simple as checking? > > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > > spin_unlock(&hugetlb_lock); > > > > /* before return */ > > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > > spin_lock(&hugetlb_lock); > > AFAIK, we at least need the hstate_is_gigantic? Comment below says that > free_gigantic_page might block, so we need to drop the lock. > And I am fine with the change overall. > > Unless I am missing something, we should not need to drop the lock unless > we need to allocate vmemmap pages (apart from gigantic pages). > > > > > > + > > > + if (alloc_huge_page_vmemmap(h, page)) { > > > + int zeroed; > > > + > > > + spin_lock(&hugetlb_lock); > > > + INIT_LIST_HEAD(&page->lru); > > > + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); > > > + h->nr_huge_pages++; > > > + h->nr_huge_pages_node[nid]++; > > I think prep_new_huge_page() does this for us? Actually, there are some differences. e.g. prep_new_huge_page() will reset hugetlb cgroup and ClearHPageFreed, but we do not need them here. And prep_new_huge_page will acquire and release the hugetlb_lock. But here we also need hold the lock to update the surplus counter and enqueue the page to the free list. So I do not think reuse prep_new_huge_page is a good idea. > > > > + > > > + /* > > > + * If we cannot allocate vmemmap pages, just refuse to free the > > > + * page and put the page back on the hugetlb free list and treat > > > + * as a surplus page. > > > + */ > > > + h->surplus_huge_pages++; > > > + h->surplus_huge_pages_node[nid]++; > > > + > > > + /* > > > + * This page is now managed by the hugetlb allocator and has > > > + * no users -- drop the last reference. > > > + */ > > > + zeroed = put_page_testzero(page); > > > + VM_BUG_ON_PAGE(!zeroed, page); > > Can this actually happen? AFAIK, page landed in update_and_free_page should be > zero refcounted, then we increase the reference, and I cannot see how the > reference might have changed in the meantime. I am not sure whether other modules get the page and then put the page. I see gather_surplus_pages does the same thing. So I copied from there. I try to look at the memory_failure routine. CPU0: CPU1: set_compound_page_dtor(HUGETLB_PAGE_DTOR); memory_failure_hugetlb get_hwpoison_page __get_hwpoison_page get_page_unless_zero put_page_testzero() Maybe this can happen. But it is a very corner case. If we want to deal with this. We can put_page_testzero() first and then set_compound_page_dtor(HUGETLB_PAGE_DTOR). > > I am all for catching corner cases, but not sure how realistic this is. > Moreover, if we __ever__ get there, things can get nasty. > > We basically will have an in-use page in the free hugetlb pool, so corruption > will happen. At that point, a plain BUG_ON might be better. > > But as I said, I do not think we need that. > > I yet need to look further, but what I have seen so far looks good. > > -- > Oscar Salvador > SUSE L3
On Tue, Feb 23, 2021 at 06:27:07PM +0800, Muchun Song wrote: > > > > + > > > > + if (alloc_huge_page_vmemmap(h, page)) { > > > > + int zeroed; > > > > + > > > > + spin_lock(&hugetlb_lock); > > > > + INIT_LIST_HEAD(&page->lru); > > > > + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); > > > > + h->nr_huge_pages++; > > > > + h->nr_huge_pages_node[nid]++; > > > > I think prep_new_huge_page() does this for us? > > Actually, there are some differences. e.g. prep_new_huge_page() > will reset hugetlb cgroup and ClearHPageFreed, but we do not need > them here. And prep_new_huge_page will acquire and release > the hugetlb_lock. But here we also need hold the lock to update > the surplus counter and enqueue the page to the free list. > So I do not think reuse prep_new_huge_page is a good idea. I see, I missed that. > > Can this actually happen? AFAIK, page landed in update_and_free_page should be > > zero refcounted, then we increase the reference, and I cannot see how the > > reference might have changed in the meantime. > > I am not sure whether other modules get the page and then put the > page. I see gather_surplus_pages does the same thing. So I copied > from there. I try to look at the memory_failure routine. > > > CPU0: CPU1: > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > memory_failure_hugetlb > get_hwpoison_page > __get_hwpoison_page > get_page_unless_zero > put_page_testzero() > > Maybe this can happen. But it is a very corner case. If we want to > deal with this. We can put_page_testzero() first and then > set_compound_page_dtor(HUGETLB_PAGE_DTOR). I have to check further, but it looks like this could actually happen. Handling this with VM_BUG_ON is wrong, because memory_failure/soft_offline are entitled to increase the refcount of the page. AFAICS, CPU0: CPU1: set_compound_page_dtor(HUGETLB_PAGE_DTOR); memory_failure_hugetlb get_hwpoison_page __get_hwpoison_page get_page_unless_zero put_page_testzero() identify_page_state me_huge_page I think we can reach me_huge_page with either refcount = 1 or refcount =2, depending whether put_page_testzero has been issued. For now, I would not re-enqueue the page if put_page_testzero == false. I have to see how this can be handled gracefully.
On Tue, Feb 23, 2021 at 11:50:05AM +0100, Oscar Salvador wrote: > > CPU0: CPU1: > > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > > memory_failure_hugetlb > > get_hwpoison_page > > __get_hwpoison_page > > get_page_unless_zero > > put_page_testzero() > > > > Maybe this can happen. But it is a very corner case. If we want to > > deal with this. We can put_page_testzero() first and then > > set_compound_page_dtor(HUGETLB_PAGE_DTOR). > > I have to check further, but it looks like this could actually happen. > Handling this with VM_BUG_ON is wrong, because memory_failure/soft_offline are > entitled to increase the refcount of the page. > > AFAICS, > > CPU0: CPU1: > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > memory_failure_hugetlb > get_hwpoison_page > __get_hwpoison_page > get_page_unless_zero > put_page_testzero() > identify_page_state > me_huge_page > > I think we can reach me_huge_page with either refcount = 1 or refcount =2, > depending whether put_page_testzero has been issued. > > For now, I would not re-enqueue the page if put_page_testzero == false. > I have to see how this can be handled gracefully. I took a brief look. It is not really your patch fault. Hugetlb <-> memory-failure synchronization is a bit odd, it definitely needs improvment. The thing is, we can have different scenarios here. E.g: by the time we return from put_page_testzero, we might have refcount == 0 and PageHWPoison, or refcount == 1 PageHWPoison. The former will let a user get a page from the pool and get a sigbus when it faults in the page, and the latter will be even more odd as we will have a self-refcounted page in the free pool (and hwpoisoned). As I said, it is not this patchset fault. I just made me realize this problem. I have to think some more about this.
On Tue, Feb 23, 2021 at 04:41:28PM +0100, Oscar Salvador wrote: > On Tue, Feb 23, 2021 at 11:50:05AM +0100, Oscar Salvador wrote: > > > CPU0: CPU1: > > > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > > > memory_failure_hugetlb > > > get_hwpoison_page > > > __get_hwpoison_page > > > get_page_unless_zero > > > put_page_testzero() > > > > > > Maybe this can happen. But it is a very corner case. If we want to > > > deal with this. We can put_page_testzero() first and then > > > set_compound_page_dtor(HUGETLB_PAGE_DTOR). > > > > I have to check further, but it looks like this could actually happen. > > Handling this with VM_BUG_ON is wrong, because memory_failure/soft_offline are > > entitled to increase the refcount of the page. > > > > AFAICS, > > > > CPU0: CPU1: > > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > > memory_failure_hugetlb > > get_hwpoison_page > > __get_hwpoison_page > > get_page_unless_zero > > put_page_testzero() > > identify_page_state > > me_huge_page > > > > I think we can reach me_huge_page with either refcount = 1 or refcount =2, > > depending whether put_page_testzero has been issued. > > > > For now, I would not re-enqueue the page if put_page_testzero == false. > > I have to see how this can be handled gracefully. > > I took a brief look. > It is not really your patch fault. Hugetlb <-> memory-failure synchronization is > a bit odd, it definitely needs improvment. > > The thing is, we can have different scenarios here. > E.g: by the time we return from put_page_testzero, we might have refcount == > 0 and PageHWPoison, or refcount == 1 PageHWPoison. > > The former will let a user get a page from the pool and get a sigbus > when it faults in the page, and the latter will be even more odd as we > will have a self-refcounted page in the free pool (and hwpoisoned). > > As I said, it is not this patchset fault. I just made me realize this > problem. > > I have to think some more about this. I have been thinking more about this. memory failure events can occur at any time, and we might not be in a position where we can handle gracefully the error, meaning that the page might end up in non desirable state. E.g: we could flag the page right before enqueing it. I still think that VM_BUG_ON should go, as the refcount can be perfectly increased by memory-failure/soft_offline handlers, so BUGing there does not make much sense. One think we could do is to check the state of the page we want to retrieve from the free hugepage pool. We should discard any HWpoisoned ones, and dissolve them. The thing is, memory-failure/soft_offline should allocate a new hugepage for the free pool, so keep the pool stable. Something like [1]. Anyway, this is orthogonal to this patch, and something I will work on soon. [1] https://lore.kernel.org/linux-mm/20210222135137.25717-2-osalvador@suse.de/T/#u
On Wed, Feb 24, 2021 at 6:32 AM Oscar Salvador <osalvador@suse.de> wrote: > > On Tue, Feb 23, 2021 at 04:41:28PM +0100, Oscar Salvador wrote: > > On Tue, Feb 23, 2021 at 11:50:05AM +0100, Oscar Salvador wrote: > > > > CPU0: CPU1: > > > > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > > > > memory_failure_hugetlb > > > > get_hwpoison_page > > > > __get_hwpoison_page > > > > get_page_unless_zero > > > > put_page_testzero() > > > > > > > > Maybe this can happen. But it is a very corner case. If we want to > > > > deal with this. We can put_page_testzero() first and then > > > > set_compound_page_dtor(HUGETLB_PAGE_DTOR). > > > > > > I have to check further, but it looks like this could actually happen. > > > Handling this with VM_BUG_ON is wrong, because memory_failure/soft_offline are > > > entitled to increase the refcount of the page. > > > > > > AFAICS, > > > > > > CPU0: CPU1: > > > set_compound_page_dtor(HUGETLB_PAGE_DTOR); > > > memory_failure_hugetlb > > > get_hwpoison_page > > > __get_hwpoison_page > > > get_page_unless_zero > > > put_page_testzero() > > > identify_page_state > > > me_huge_page > > > > > > I think we can reach me_huge_page with either refcount = 1 or refcount =2, > > > depending whether put_page_testzero has been issued. > > > > > > For now, I would not re-enqueue the page if put_page_testzero == false. > > > I have to see how this can be handled gracefully. > > > > I took a brief look. > > It is not really your patch fault. Hugetlb <-> memory-failure synchronization is > > a bit odd, it definitely needs improvment. > > > > The thing is, we can have different scenarios here. > > E.g: by the time we return from put_page_testzero, we might have refcount == > > 0 and PageHWPoison, or refcount == 1 PageHWPoison. > > > > The former will let a user get a page from the pool and get a sigbus > > when it faults in the page, and the latter will be even more odd as we > > will have a self-refcounted page in the free pool (and hwpoisoned). I have been looking at the dequeue_huge_page_node_exact(). If a PageHWPoison huge page is in the free pool list, the page will not be allocated to the user. The PageHWPoison huge page will be skip in the dequeue_huge_page_node_exact(). > > > > As I said, it is not this patchset fault. I just made me realize this > > problem. > > > > I have to think some more about this. > > I have been thinking more about this. > memory failure events can occur at any time, and we might not be in a > position where we can handle gracefully the error, meaning that the page > might end up in non desirable state. > > E.g: we could flag the page right before enqueing it. > > I still think that VM_BUG_ON should go, as the refcount can be perfectly > increased by memory-failure/soft_offline handlers, so BUGing there does > not make much sense. Make sense. I will remove the VM_BUG_ON. > > One think we could do is to check the state of the page we want to > retrieve from the free hugepage pool. > We should discard any HWpoisoned ones, and dissolve them. > > The thing is, memory-failure/soft_offline should allocate a new hugepage > for the free pool, so keep the pool stable. > Something like [1]. > > Anyway, this is orthogonal to this patch, and something I will work on > soon. > > [1] https://lore.kernel.org/linux-mm/20210222135137.25717-2-osalvador@suse.de/T/#u Thanks for your efforts on this. > > -- > Oscar Salvador > SUSE L3
On Wed, Feb 24, 2021 at 11:47:49AM +0800, Muchun Song wrote: > I have been looking at the dequeue_huge_page_node_exact(). > If a PageHWPoison huge page is in the free pool list, the page will > not be allocated to the user. The PageHWPoison huge page > will be skip in the dequeue_huge_page_node_exact(). Yes, now I see where the problem lies. hugetlb_no_page()->..->dequeue_huge_page_node_exact() will fail if the only page in the pool is hwpoisoned, as expected. Then alloc_buddy_huge_page_with_mpol() will be tried, but since surplus_huge_pages counter is stale, we will fail there. That relates to the problem Mike pointed out, that we should decrease again the surplus_huge_pages. I think hwpoisoned pages should not be in the free pool though. Probably we want to take them off when we notice we have one: e.g: dequeue_huge_page_node_exact could place the page in another list and place it back in case it was unpoisoned. But anyway, that has nothing to do with this (apart from the surplus problem).
diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index f7b1c7462991..fb8f649e5635 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -60,6 +60,10 @@ HugePages_Surp the pool above the value in ``/proc/sys/vm/nr_hugepages``. The maximum number of surplus huge pages is controlled by ``/proc/sys/vm/nr_overcommit_hugepages``. + Note: When the feature of freeing unused vmemmap pages associated + with each hugetlb page is enabled, the number of the surplus huge + pages may be temporarily larger than the maximum number of surplus + huge pages when the system is under memory pressure. Hugepagesize is the default hugepage size (in Kb). Hugetlb @@ -80,6 +84,10 @@ returned to the huge page pool when freed by a task. A user with root privileges can dynamically allocate more or free some persistent huge pages by increasing or decreasing the value of ``nr_hugepages``. +Note: When the feature of freeing unused vmemmap pages associated with each +hugetlb page is enabled, we can failed to free the huge pages triggered by +the user when ths system is under memory pressure. Please try again later. + Pages that are used as huge pages are reserved inside the kernel and cannot be used for other purposes. Huge pages cannot be swapped out under memory pressure. diff --git a/include/linux/mm.h b/include/linux/mm.h index d7dddf334779..33c5911afe18 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2981,6 +2981,8 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) void vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse); +int vmemmap_remap_alloc(unsigned long start, unsigned long end, + unsigned long reuse, gfp_t gfp_mask); void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4cfca27c6d32..bcf856974c48 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1305,37 +1305,68 @@ static inline void destroy_compound_gigantic_page(struct page *page, unsigned int order) { } #endif -static void update_and_free_page(struct hstate *h, struct page *page) +static int update_and_free_page(struct hstate *h, struct page *page) + __releases(&hugetlb_lock) __acquires(&hugetlb_lock) { int i; + int nid = page_to_nid(page); if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) - return; + return 0; h->nr_huge_pages--; - h->nr_huge_pages_node[page_to_nid(page)]--; + h->nr_huge_pages_node[nid]--; + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + set_page_refcounted(page); + spin_unlock(&hugetlb_lock); + + if (alloc_huge_page_vmemmap(h, page)) { + int zeroed; + + spin_lock(&hugetlb_lock); + INIT_LIST_HEAD(&page->lru); + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); + h->nr_huge_pages++; + h->nr_huge_pages_node[nid]++; + + /* + * If we cannot allocate vmemmap pages, just refuse to free the + * page and put the page back on the hugetlb free list and treat + * as a surplus page. + */ + h->surplus_huge_pages++; + h->surplus_huge_pages_node[nid]++; + + /* + * This page is now managed by the hugetlb allocator and has + * no users -- drop the last reference. + */ + zeroed = put_page_testzero(page); + VM_BUG_ON_PAGE(!zeroed, page); + arch_clear_hugepage_flags(page); + enqueue_huge_page(h, page); + + return -ENOMEM; + } + for (i = 0; i < pages_per_huge_page(h); i++) { page[i].flags &= ~(1 << PG_locked | 1 << PG_error | 1 << PG_referenced | 1 << PG_dirty | 1 << PG_active | 1 << PG_private | 1 << PG_writeback); } - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); - set_page_refcounted(page); if (hstate_is_gigantic(h)) { - /* - * Temporarily drop the hugetlb_lock, because - * we might block in free_gigantic_page(). - */ - spin_unlock(&hugetlb_lock); destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); - spin_lock(&hugetlb_lock); } else { __free_pages(page, huge_page_order(h)); } + + spin_lock(&hugetlb_lock); + + return 0; } struct hstate *size_to_hstate(unsigned long size) @@ -1403,9 +1434,9 @@ static void __free_huge_page(struct page *page) } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ list_del(&page->lru); - update_and_free_page(h, page); h->surplus_huge_pages--; h->surplus_huge_pages_node[nid]--; + update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -1693,6 +1724,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, struct page *page = list_entry(h->hugepage_freelists[node].next, struct page, lru); + ClearHPageFreed(page); list_del(&page->lru); h->free_huge_pages--; h->free_huge_pages_node[node]--; @@ -1700,8 +1732,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, h->surplus_huge_pages--; h->surplus_huge_pages_node[node]--; } - update_and_free_page(h, page); - ret = 1; + ret = !update_and_free_page(h, page); break; } } @@ -1714,10 +1745,14 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, * nothing for in-use hugepages and non-hugepages. * This function returns values like below: * - * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use - * (allocated or reserved.) - * 0: successfully dissolved free hugepages or the page is not a - * hugepage (considered as already dissolved) + * -ENOMEM: failed to allocate vmemmap pages to free the freed hugepages + * when the system is under memory pressure and the feature of + * freeing unused vmemmap pages associated with each hugetlb page + * is enabled. + * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use + * (allocated or reserved.) + * 0: successfully dissolved free hugepages or the page is not a + * hugepage (considered as already dissolved) */ int dissolve_free_huge_page(struct page *page) { @@ -1768,12 +1803,14 @@ int dissolve_free_huge_page(struct page *page) SetPageHWPoison(page); ClearPageHWPoison(head); } + ClearHPageFreed(page); list_del(&head->lru); h->free_huge_pages--; h->free_huge_pages_node[nid]--; h->max_huge_pages--; - update_and_free_page(h, head); - rc = 0; + rc = update_and_free_page(h, head); + if (rc) + h->max_huge_pages++; } out: spin_unlock(&hugetlb_lock); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 0209b736e0b4..29a3380f3b20 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -198,6 +198,28 @@ static inline unsigned long free_vmemmap_pages_size_per_hpage(struct hstate *h) return (unsigned long)free_vmemmap_pages_per_hpage(h) << PAGE_SHIFT; } +int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + unsigned long vmemmap_addr = (unsigned long)head; + unsigned long vmemmap_end, vmemmap_reuse; + + if (!free_vmemmap_pages_per_hpage(h)) + return 0; + + vmemmap_addr += RESERVE_VMEMMAP_SIZE; + vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); + vmemmap_reuse = vmemmap_addr - PAGE_SIZE; + /* + * The pages which the vmemmap virtual address range [@vmemmap_addr, + * @vmemmap_end) are mapped to are freed to the buddy allocator, and + * the range is mapped to the page which @vmemmap_reuse is mapped to. + * When a HugeTLB page is freed to the buddy allocator, previously + * discarded vmemmap pages must be allocated and remapping. + */ + return vmemmap_remap_alloc(vmemmap_addr, vmemmap_end, vmemmap_reuse, + GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); +} + void free_huge_page_vmemmap(struct hstate *h, struct page *head) { unsigned long vmemmap_addr = (unsigned long)head; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 6923f03534d5..e5547d53b9f5 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -11,8 +11,14 @@ #include <linux/hugetlb.h> #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP +int alloc_huge_page_vmemmap(struct hstate *h, struct page *head); void free_huge_page_vmemmap(struct hstate *h, struct page *head); #else +static inline int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + return 0; +} + static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) { } diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index d3076a7a3783..60fc6cd6cd23 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -40,7 +40,8 @@ * @remap_pte: called for each lowest-level entry (PTE). * @reuse_page: the page which is reused for the tail vmemmap pages. * @reuse_addr: the virtual address of the @reuse_page page. - * @vmemmap_pages: the list head of the vmemmap pages that can be freed. + * @vmemmap_pages: the list head of the vmemmap pages that can be freed + * or is mapped from. */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -237,6 +238,78 @@ void vmemmap_remap_free(unsigned long start, unsigned long end, free_vmemmap_page_list(&vmemmap_pages); } +static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, + struct vmemmap_remap_walk *walk) +{ + pgprot_t pgprot = PAGE_KERNEL; + struct page *page; + void *to; + + BUG_ON(pte_page(*pte) != walk->reuse_page); + + page = list_first_entry(walk->vmemmap_pages, struct page, lru); + list_del(&page->lru); + to = page_to_virt(page); + copy_page(to, (void *)walk->reuse_addr); + + set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); +} + +static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, + gfp_t gfp_mask, struct list_head *list) +{ + unsigned long nr_pages = (end - start) >> PAGE_SHIFT; + int nid = page_to_nid((struct page *)start); + struct page *page, *next; + + while (nr_pages--) { + page = alloc_pages_node(nid, gfp_mask, 0); + if (!page) + goto out; + list_add_tail(&page->lru, list); + } + + return 0; +out: + list_for_each_entry_safe(page, next, list, lru) + __free_pages(page, 0); + return -ENOMEM; +} + +/** + * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end) + * to the page which is from the @vmemmap_pages + * respectively. + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * @gpf_mask: GFP flag for allocating vmemmap pages. + */ +int vmemmap_remap_alloc(unsigned long start, unsigned long end, + unsigned long reuse, gfp_t gfp_mask) +{ + LIST_HEAD(vmemmap_pages); + struct vmemmap_remap_walk walk = { + .remap_pte = vmemmap_restore_pte, + .reuse_addr = reuse, + .vmemmap_pages = &vmemmap_pages, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + might_sleep_if(gfpflags_allow_blocking(gfp_mask)); + + if (alloc_vmemmap_page_list(start, end, gfp_mask, &vmemmap_pages)) + return -ENOMEM; + + vmemmap_remap_range(reuse, end, &walk); + + return 0; +} + /* * Allocate a block of memory to be used to back the virtual memory map * or to back the page tables that are used to create the mapping.
When we free a HugeTLB page to the buddy allocator, we should allocate the vmemmap pages associated with it. But we may cannot allocate vmemmap pages when the system is under memory pressure, in this case, we just refuse to free the HugeTLB page instead of looping forever trying to allocate the pages. This changes some behavior (list below) on some corner cases. 1) Failing to free a huge page triggered by the user (decrease nr_pages). Need try again later by the user. 2) Failing to free a surplus huge page when freed by the application. Try again later when freeing a huge page next time. 3) Failing to dissolve a free huge page on ZONE_MOVABLE via offline_pages(). This is a bit unfortunate if we have plenty of ZONE_MOVABLE memory but are low on kernel memory. For example, migration of huge pages would still work, however, dissolving the free page does not work. This is a corner cases. When the system is that much under memory pressure, offlining/unplug can be expected to fail. 4) Failing to dissolve a huge page on CMA/ZONE_MOVABLE via alloc_contig_range() - once we have that handling in place. Mainly affects CMA and virtio-mem. Similar to 3). virito-mem will handle migration errors gracefully. CMA might be able to fallback on other free areas within the CMA region. We do not want to use GFP_ATOMIC to allocate vmemmap pages. Because it grants access to memory reserves and we do not think it is reasonable to use memory reserves. We use GFP_KERNEL in alloc_huge_page_vmemmap(). Signed-off-by: Muchun Song <songmuchun@bytedance.com> --- Documentation/admin-guide/mm/hugetlbpage.rst | 8 +++ include/linux/mm.h | 2 + mm/hugetlb.c | 81 ++++++++++++++++++++-------- mm/hugetlb_vmemmap.c | 22 ++++++++ mm/hugetlb_vmemmap.h | 6 +++ mm/sparse-vmemmap.c | 75 +++++++++++++++++++++++++- 6 files changed, 171 insertions(+), 23 deletions(-)