From patchwork Thu Feb 18 12:00:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 12093471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D3CBC433E0 for ; Thu, 18 Feb 2021 12:00:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9399461493 for ; Thu, 18 Feb 2021 12:00:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9399461493 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BC63B6B0006; Thu, 18 Feb 2021 07:00:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B517E6B0070; Thu, 18 Feb 2021 07:00:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F1766B0006; Thu, 18 Feb 2021 07:00:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0167.hostedemail.com [216.40.44.167]) by kanga.kvack.org (Postfix) with ESMTP id 6CE326B0006 for ; Thu, 18 Feb 2021 07:00:41 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 316C98249980 for ; Thu, 18 Feb 2021 12:00:41 +0000 (UTC) X-FDA: 77831246682.27.C98B281 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf27.hostedemail.com (Postfix) with ESMTP id 03AD0801A802 for ; Thu, 18 Feb 2021 12:00:36 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 99A7AAF1B; Thu, 18 Feb 2021 12:00:39 +0000 (UTC) From: Oscar Salvador To: Andrew Morton Cc: Mike Kravetz , David Hildenbrand , Michal Hocko , Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador Subject: [PATCH v2 1/2] mm: Make alloc_contig_range handle free hugetlb pages Date: Thu, 18 Feb 2021 13:00:27 +0100 Message-Id: <20210218120028.6499-2-osalvador@suse.de> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20210218120028.6499-1-osalvador@suse.de> References: <20210218120028.6499-1-osalvador@suse.de> MIME-Version: 1.0 X-Stat-Signature: msemb9reu75nqg9rxi7db9stt77dibsp X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 03AD0801A802 Received-SPF: none (suse.de>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1613649636-891976 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range will fail if it ever sees a HugeTLB page within the range we are trying to allocate, even when that page is free and can be easily reallocated. This has proofed to be problematic for some users of alloc_contic_range, e.g: CMA and virtio-mem, where those would fail the call even when those pages lay in ZONE_MOVABLE and are free. We can do better by trying to dissolve such pages. Free hugepages are tricky to handle so as to no userspace application notices disruption, we need to replace the current free hugepage with a new one. In order to do that, a new function called alloc_and_dissolve_huge_page is introduced. This function will first try to get a new fresh hugepage, and if it succeeds, it will dissolve the old one. If the old hugepage cannot be be dissolved, we have to dissolve the new hugepage we just got. Should that fail as well, we count is as a surplus, so the pool will be re-balanced when a hugepage gets free instead of enqueues again. With regard to the allocation, we restrict it to the node the page belongs to with __GFP_THISNODE, meaning we do not fallback on other node's zones. Note that gigantic hugetlb pages are fenced off since there is a cyclic dependency between them and alloc_contig_range. Signed-off-by: Oscar Salvador --- include/linux/hugetlb.h | 6 ++++ mm/compaction.c | 12 ++++++++ mm/hugetlb.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 93 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index b5807f23caf8..72352d718829 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -505,6 +505,7 @@ struct huge_bootmem_page { struct hstate *hstate; }; +bool isolate_or_dissolve_huge_page(struct page *page); struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid, @@ -775,6 +776,11 @@ void set_page_huge_active(struct page *page); #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; +static inline bool isolate_or_dissolve_huge_page(struct page *page) +{ + return false; +} + static inline struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) diff --git a/mm/compaction.c b/mm/compaction.c index 190ccdaa6c19..d52506ed9db7 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -905,6 +905,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, valid_page = page; } + if (PageHuge(page) && cc->alloc_contig) { + if (!isolate_or_dissolve_huge_page(page)) + goto isolate_fail; + + /* + * Ok, the hugepage was dissolved. Now these pages are + * Buddy and cannot be re-allocated because they are + * isolated. Fall-through as the check below handles + * Buddy pages. + */ + } + /* * Skip if free. We read page order here without zone lock * which is generally unsafe, but the race window is small and diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4bdb58ab14cb..a4fbbe924a55 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2294,6 +2294,81 @@ static void restore_reserve_on_error(struct hstate *h, } } +static bool alloc_and_dissolve_huge_page(struct hstate *h, struct page *page) +{ + gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nid = page_to_nid(page); + struct page *new_page; + bool ret = false; + + /* + * Before dissolving the page, we need to allocate a new one, + * so the pool remains stable. + */ + new_page = alloc_fresh_huge_page(h, gfp_mask, nid, NULL, NULL); + if (new_page) { + /* + * Free it into the hugepage allocator + */ + put_page(new_page); + + /* + * Ok, we got a new free hugepage to replace this one. Try to + * dissolve the old page. + */ + if (!dissolve_free_huge_page(page)) { + ret = true; + } else if (dissolve_free_huge_page(new_page)) { + /* + * Seems the old page could not be dissolved, so try to + * dissolve the freshly allocated page. If that fails + * too, let us count the new page as a surplus. Doing so + * allows the pool to be re-balanced when pages are freed + * instead of enqueued again. + */ + spin_lock(&hugetlb_lock); + h->surplus_huge_pages++; + h->surplus_huge_pages_node[nid]++; + spin_unlock(&hugetlb_lock); + } + } + + return ret; +} + +bool isolate_or_dissolve_huge_page(struct page *page) +{ + struct hstate *h = NULL; + struct page *head; + bool ret = false; + + spin_lock(&hugetlb_lock); + if (PageHuge(page)) { + head = compound_head(page); + h = page_hstate(head); + } + spin_unlock(&hugetlb_lock); + + /* + * The page might have been dissolved from under our feet. + * If that is the case, return success as if we dissolved it ourselves. + */ + if (!h) + return true; + + /* + * Fence off gigantic pages as there is a cyclic dependency + * between alloc_contig_range and them. + */ + if (hstate_is_gigantic(h)) + return ret; + + if(!page_count(head) && alloc_and_dissolve_huge_page(h, head)) + ret = true; + + return ret; +} + struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) { From patchwork Thu Feb 18 12:00:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 12093473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F91DC433E6 for ; Thu, 18 Feb 2021 12:00:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E9BDE61493 for ; Thu, 18 Feb 2021 12:00:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9BDE61493 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 906736B006E; Thu, 18 Feb 2021 07:00:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 891088D0002; Thu, 18 Feb 2021 07:00:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 670098D0001; Thu, 18 Feb 2021 07:00:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0155.hostedemail.com [216.40.44.155]) by kanga.kvack.org (Postfix) with ESMTP id 45A6F6B006E for ; Thu, 18 Feb 2021 07:00:42 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E9F5452D6 for ; Thu, 18 Feb 2021 12:00:41 +0000 (UTC) X-FDA: 77831246682.24.53FDC47 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf02.hostedemail.com (Postfix) with ESMTP id 10D9840B8CD5 for ; Thu, 18 Feb 2021 12:00:32 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 4576CAEBE; Thu, 18 Feb 2021 12:00:40 +0000 (UTC) From: Oscar Salvador To: Andrew Morton Cc: Mike Kravetz , David Hildenbrand , Michal Hocko , Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador Subject: [PATCH v2 2/2] mm: Make alloc_contig_range handle in-use hugetlb pages Date: Thu, 18 Feb 2021 13:00:28 +0100 Message-Id: <20210218120028.6499-3-osalvador@suse.de> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20210218120028.6499-1-osalvador@suse.de> References: <20210218120028.6499-1-osalvador@suse.de> MIME-Version: 1.0 X-Stat-Signature: b5e7t69u8tdhyp9ukajbnb7uw83j4pts X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 10D9840B8CD5 Received-SPF: none (suse.de>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1613649632-475980 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range() will fail miserably if it finds a HugeTLB page within the range, without a chance to handle them. Since HugeTLB pages can be migrated as any other page (LRU and Movable), it does not make sense to bail out without trying. Enable the interface to recognize in-use HugeTLB pages so we can migrate them, and have much better chances to succeed the call. Signed-off-by: Oscar Salvador --- include/linux/hugetlb.h | 5 +++-- mm/compaction.c | 12 +++++++++++- mm/hugetlb.c | 6 ++++-- mm/vmscan.c | 5 +++-- 4 files changed, 21 insertions(+), 7 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 72352d718829..8c17d0dbc87c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -505,7 +505,7 @@ struct huge_bootmem_page { struct hstate *hstate; }; -bool isolate_or_dissolve_huge_page(struct page *page); +bool isolate_or_dissolve_huge_page(struct page *page, struct list_head *list); struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid, @@ -776,7 +776,8 @@ void set_page_huge_active(struct page *page); #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; -static inline bool isolate_or_dissolve_huge_page(struct page *page) +static inline bool isolate_or_dissolve_huge_page(struct page *page, + struct list_head *list) { return false; } diff --git a/mm/compaction.c b/mm/compaction.c index d52506ed9db7..3394ab385915 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -906,9 +906,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } if (PageHuge(page) && cc->alloc_contig) { - if (!isolate_or_dissolve_huge_page(page)) + if (!isolate_or_dissolve_huge_page(page, &cc->migratepages)) goto isolate_fail; + if (PageHuge(page)) { + /* + * Hugepage was succesfully isolated and placed + * on the cc->migratepages list. + */ + low_pfn += compound_nr(page) - 1; + goto isolate_success_no_list; + } + /* * Ok, the hugepage was dissolved. Now these pages are * Buddy and cannot be re-allocated because they are @@ -1053,6 +1062,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, isolate_success: list_add(&page->lru, &cc->migratepages); +isolate_success_no_list: cc->nr_migratepages += compound_nr(page); nr_isolated += compound_nr(page); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a4fbbe924a55..1208b5f278b0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2336,7 +2336,7 @@ static bool alloc_and_dissolve_huge_page(struct hstate *h, struct page *page) return ret; } -bool isolate_or_dissolve_huge_page(struct page *page) +bool isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) { struct hstate *h = NULL; struct page *head; @@ -2363,7 +2363,9 @@ bool isolate_or_dissolve_huge_page(struct page *page) if (hstate_is_gigantic(h)) return ret; - if(!page_count(head) && alloc_and_dissolve_huge_page(h, head)) + if (page_count(head) && isolate_huge_page(head, list)) + ret = true; + else if(!page_count(head) && alloc_and_dissolve_huge_page(h, head)) ret = true; return ret; diff --git a/mm/vmscan.c b/mm/vmscan.c index b1b574ad199d..0803adca4469 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1506,8 +1506,9 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, LIST_HEAD(clean_pages); list_for_each_entry_safe(page, next, page_list, lru) { - if (page_is_file_lru(page) && !PageDirty(page) && - !__PageMovable(page) && !PageUnevictable(page)) { + if (!PageHuge(page) && page_is_file_lru(page) && + !PageDirty(page) && !__PageMovable(page) && + !PageUnevictable(page)) { ClearPageActive(page); list_move(&page->lru, &clean_pages); }