From patchwork Mon Apr 25 14:31:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12825847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8E28C4332F for ; Mon, 25 Apr 2022 14:31:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C71D66B007D; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C45306B0080; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 857946B007E; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 4E1E96B0078 for ; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 887F8278B4 for ; Mon, 25 Apr 2022 14:31:29 +0000 (UTC) X-FDA: 79395639498.23.2D4946B Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf05.hostedemail.com (Postfix) with ESMTP id 5FC4E10004E for ; Mon, 25 Apr 2022 14:31:23 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id AE6735C0190; Mon, 25 Apr 2022 10:31:28 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 25 Apr 2022 10:31:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm1; t= 1650897088; x=1650983488; bh=b9zZIyD7Mszj2Y0eHN99UaRE3mmqG7wYgAJ fo/E5H3M=; b=RDwY4ZjqzJbLnM0fzrKzWU+DFKRsrjzIVYmGv5XVueXa6XEm4U/ z4XICaDTG3VZCYqHEOasJszuQB1WqZH0yZuBVNC2kfjFAxBjX0KxT5QkMKh1sfRg bqLeV5JIqGTO0UcnZZ01f/BqnnB4Q/v/7OH1S0+TG/TEbPnV5hMF253JfmZubQi+ OWisKV0spmEUy/S296X71mfw6VUVkPo27Q6dE2MEhpjm6UXLiiqP8iQocUHzjxcR XnEOvVY15m1SDCk/8IknJfXbNptO+0ct1DXdlYWQ78MO0GZ+OminKg1sWBx//iAF 9YydTOMJNLaZjLd7fAsZpmf+a55OhBi5P1w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1650897088; x=1650983488; bh=b 9zZIyD7Mszj2Y0eHN99UaRE3mmqG7wYgAJfo/E5H3M=; b=lYdIcN6YTbrrV7aFp XcwEQ36wQv+Lebxgku5HURiahyEMG4WO+yZAv9ZWym9ssdJkzI5WWPYua0o2D4g2 3NKIv8qQ0sOPCRWiMfsMENpvJ6/+trFPZIqYJONtt5p8nF65cHpTp5q9E22fTlAk yngrOP+jdHsSBVWDC7n5wNIJdAyum/Aofl7y0cuwXXUkmKH3+SbTajB+8aKC/7HC 7ZC4iaCCW7TJ+PZiFNcKn8w6C14gleGtTkBxR6c0LKoc69RuxeorG7tavPJBb5HI +GWJeBs+gs4DGiJdRo8dyFCAE5GZ5rCTjyhuwKoDdUbEJyRRwJsl5/6wYRH14Uy0 0Nu3g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddugdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhggtgfgsehtqhertdertdejnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeevhe evleehgfduhffhudefteehhfegjeeiudevheevffetuedttdfhkeeuleehudenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdrhigrnhessh gvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Apr 2022 10:31:27 -0400 (EDT) From: Zi Yan To: David Hildenbrand , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Vlastimil Babka , Mel Gorman , Eric Ren , Mike Rapoport , Oscar Salvador , Christophe Leroy , Andrew Morton , Zi Yan , Mike Rapoport Subject: [PATCH v11 1/6] mm: page_isolation: move has_unmovable_pages() to mm/page_isolation.c Date: Mon, 25 Apr 2022 10:31:13 -0400 Message-Id: <20220425143118.2850746-2-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220425143118.2850746-1-zi.yan@sent.com> References: <20220425143118.2850746-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 5FC4E10004E X-Stat-Signature: y8aiw7egyunty4zuz8jz7igo6mmday3a X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b=RDwY4Zjq; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=lYdIcN6Y; spf=pass (imf05.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com X-HE-Tag: 1650897083-507454 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan has_unmovable_pages() is only used in mm/page_isolation.c. Move it from mm/page_alloc.c and make it static. Signed-off-by: Zi Yan Reviewed-by: Oscar Salvador Reviewed-by: Mike Rapoport Acked-by: David Hildenbrand --- include/linux/page-isolation.h | 2 - mm/page_alloc.c | 119 --------------------------------- mm/page_isolation.c | 119 +++++++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+), 121 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 572458016331..e14eddf6741a 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -33,8 +33,6 @@ static inline bool is_migrate_isolate(int migratetype) #define MEMORY_OFFLINE 0x1 #define REPORT_FAILURE 0x2 -struct page *has_unmovable_pages(struct zone *zone, struct page *page, - int migratetype, int flags); void set_pageblock_migratetype(struct page *page, int migratetype); int move_freepages_block(struct zone *zone, struct page *page, int migratetype, int *num_movable); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d77e8d15523d..ce23ac8ad085 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8886,125 +8886,6 @@ void *__init alloc_large_system_hash(const char *tablename, return table; } -/* - * This function checks whether pageblock includes unmovable pages or not. - * - * PageLRU check without isolation or lru_lock could race so that - * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable - * check without lock_page also may miss some movable non-lru pages at - * race condition. So you can't expect this function should be exact. - * - * Returns a page without holding a reference. If the caller wants to - * dereference that page (e.g., dumping), it has to make sure that it - * cannot get removed (e.g., via memory unplug) concurrently. - * - */ -struct page *has_unmovable_pages(struct zone *zone, struct page *page, - int migratetype, int flags) -{ - unsigned long iter = 0; - unsigned long pfn = page_to_pfn(page); - unsigned long offset = pfn % pageblock_nr_pages; - - if (is_migrate_cma_page(page)) { - /* - * CMA allocations (alloc_contig_range) really need to mark - * isolate CMA pageblocks even when they are not movable in fact - * so consider them movable here. - */ - if (is_migrate_cma(migratetype)) - return NULL; - - return page; - } - - for (; iter < pageblock_nr_pages - offset; iter++) { - page = pfn_to_page(pfn + iter); - - /* - * Both, bootmem allocations and memory holes are marked - * PG_reserved and are unmovable. We can even have unmovable - * allocations inside ZONE_MOVABLE, for example when - * specifying "movablecore". - */ - if (PageReserved(page)) - return page; - - /* - * If the zone is movable and we have ruled out all reserved - * pages then it should be reasonably safe to assume the rest - * is movable. - */ - if (zone_idx(zone) == ZONE_MOVABLE) - continue; - - /* - * Hugepages are not in LRU lists, but they're movable. - * THPs are on the LRU, but need to be counted as #small pages. - * We need not scan over tail pages because we don't - * handle each tail page individually in migration. - */ - if (PageHuge(page) || PageTransCompound(page)) { - struct page *head = compound_head(page); - unsigned int skip_pages; - - if (PageHuge(page)) { - if (!hugepage_migration_supported(page_hstate(head))) - return page; - } else if (!PageLRU(head) && !__PageMovable(head)) { - return page; - } - - skip_pages = compound_nr(head) - (page - head); - iter += skip_pages - 1; - continue; - } - - /* - * We can't use page_count without pin a page - * because another CPU can free compound page. - * This check already skips compound tails of THP - * because their page->_refcount is zero at all time. - */ - if (!page_ref_count(page)) { - if (PageBuddy(page)) - iter += (1 << buddy_order(page)) - 1; - continue; - } - - /* - * The HWPoisoned page may be not in buddy system, and - * page_count() is not 0. - */ - if ((flags & MEMORY_OFFLINE) && PageHWPoison(page)) - continue; - - /* - * We treat all PageOffline() pages as movable when offlining - * to give drivers a chance to decrement their reference count - * in MEM_GOING_OFFLINE in order to indicate that these pages - * can be offlined as there are no direct references anymore. - * For actually unmovable PageOffline() where the driver does - * not support this, we will fail later when trying to actually - * move these pages that still have a reference count > 0. - * (false negatives in this function only) - */ - if ((flags & MEMORY_OFFLINE) && PageOffline(page)) - continue; - - if (__PageMovable(page) || PageLRU(page)) - continue; - - /* - * If there are RECLAIMABLE pages, we need to check - * it. But now, memory offline itself doesn't call - * shrink_node_slabs() and it still to be fixed. - */ - return page; - } - return NULL; -} - #ifdef CONFIG_CONTIG_ALLOC static unsigned long pfn_max_align_down(unsigned long pfn) { diff --git a/mm/page_isolation.c b/mm/page_isolation.c index ff0ea6308299..df49f86a6ed1 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -15,6 +15,125 @@ #define CREATE_TRACE_POINTS #include +/* + * This function checks whether pageblock includes unmovable pages or not. + * + * PageLRU check without isolation or lru_lock could race so that + * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable + * check without lock_page also may miss some movable non-lru pages at + * race condition. So you can't expect this function should be exact. + * + * Returns a page without holding a reference. If the caller wants to + * dereference that page (e.g., dumping), it has to make sure that it + * cannot get removed (e.g., via memory unplug) concurrently. + * + */ +static struct page *has_unmovable_pages(struct zone *zone, struct page *page, + int migratetype, int flags) +{ + unsigned long iter = 0; + unsigned long pfn = page_to_pfn(page); + unsigned long offset = pfn % pageblock_nr_pages; + + if (is_migrate_cma_page(page)) { + /* + * CMA allocations (alloc_contig_range) really need to mark + * isolate CMA pageblocks even when they are not movable in fact + * so consider them movable here. + */ + if (is_migrate_cma(migratetype)) + return NULL; + + return page; + } + + for (; iter < pageblock_nr_pages - offset; iter++) { + page = pfn_to_page(pfn + iter); + + /* + * Both, bootmem allocations and memory holes are marked + * PG_reserved and are unmovable. We can even have unmovable + * allocations inside ZONE_MOVABLE, for example when + * specifying "movablecore". + */ + if (PageReserved(page)) + return page; + + /* + * If the zone is movable and we have ruled out all reserved + * pages then it should be reasonably safe to assume the rest + * is movable. + */ + if (zone_idx(zone) == ZONE_MOVABLE) + continue; + + /* + * Hugepages are not in LRU lists, but they're movable. + * THPs are on the LRU, but need to be counted as #small pages. + * We need not scan over tail pages because we don't + * handle each tail page individually in migration. + */ + if (PageHuge(page) || PageTransCompound(page)) { + struct page *head = compound_head(page); + unsigned int skip_pages; + + if (PageHuge(page)) { + if (!hugepage_migration_supported(page_hstate(head))) + return page; + } else if (!PageLRU(head) && !__PageMovable(head)) { + return page; + } + + skip_pages = compound_nr(head) - (page - head); + iter += skip_pages - 1; + continue; + } + + /* + * We can't use page_count without pin a page + * because another CPU can free compound page. + * This check already skips compound tails of THP + * because their page->_refcount is zero at all time. + */ + if (!page_ref_count(page)) { + if (PageBuddy(page)) + iter += (1 << buddy_order(page)) - 1; + continue; + } + + /* + * The HWPoisoned page may be not in buddy system, and + * page_count() is not 0. + */ + if ((flags & MEMORY_OFFLINE) && PageHWPoison(page)) + continue; + + /* + * We treat all PageOffline() pages as movable when offlining + * to give drivers a chance to decrement their reference count + * in MEM_GOING_OFFLINE in order to indicate that these pages + * can be offlined as there are no direct references anymore. + * For actually unmovable PageOffline() where the driver does + * not support this, we will fail later when trying to actually + * move these pages that still have a reference count > 0. + * (false negatives in this function only) + */ + if ((flags & MEMORY_OFFLINE) && PageOffline(page)) + continue; + + if (__PageMovable(page) || PageLRU(page)) + continue; + + /* + * If there are RECLAIMABLE pages, we need to check + * it. But now, memory offline itself doesn't call + * shrink_node_slabs() and it still to be fixed. + */ + return page; + } + return NULL; +} + static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags) { struct zone *zone = page_zone(page); From patchwork Mon Apr 25 14:31:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12825845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA154C433EF for ; Mon, 25 Apr 2022 14:31:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 658466B0074; Mon, 25 Apr 2022 10:31:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6062E6B0075; Mon, 25 Apr 2022 10:31:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 458556B0078; Mon, 25 Apr 2022 10:31:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 315236B0074 for ; Mon, 25 Apr 2022 10:31:32 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 7646C81BC1 for ; Mon, 25 Apr 2022 14:31:30 +0000 (UTC) X-FDA: 79395639540.18.3536E57 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf28.hostedemail.com (Postfix) with ESMTP id AE019C0043 for ; Mon, 25 Apr 2022 14:31:23 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 655155C0165; Mon, 25 Apr 2022 10:31:29 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 25 Apr 2022 10:31:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm1; t= 1650897089; x=1650983489; bh=OeQFxiIFvToaCLpTa2plz66iPIth4OqYoQ1 7JpnzViQ=; b=A9b3waUUGp3F2UWeLiSYPfFrFcvP+AopJc6K0snGGxr9P40nHHo EIoODQMsQjv0W6y8i2bYfegJq0o9QJgpx/pcL/vmu1ZfX87ybZYI44mMea8YhNa/ N39Ba+44iuTydKxdvHlTwn3oWeXJmk7zgEsxvKouV4GswEQZE4OyV10tjlE/W0On 0mCmoW7i+4K0TKA0Pg52p6M7xMIy1Sgl6e4GpkrTDPFGvUAA7zegvbqAebNYDlhe zD7kCKskjDgdQABpnmOOuzux09fEh3E/q6YpCCXqABPto1TkNp9BNskU6NI7vTpE HLsvtyXguNt4SClWV4nyoEUaxApG+61awxw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1650897089; x=1650983489; bh=O eQFxiIFvToaCLpTa2plz66iPIth4OqYoQ17JpnzViQ=; b=oYsissR3PHWqDArA4 HfFAmFfENAfSER9HKJGmrMOrOCpfxw0HustA7k6EVvUhGGUryXMusRnyuDwV+8ub iZ0PxQRtPkk9ABrQhiqeenaony75e23HvlqmH/+p9KUnttwYyY6IPtP1Z7PE005K d/EippddZZJnHM5pkdbZtal+8YW3oghi3t4c1ceARr9xQV9nePP3NK7nvjL/gugm fY2nY+XvzV5GPRCPiV0a5o7wDXQ+ZaF8zvVRgDvfefckuPHJVuAZTa/fghuv2N2a pXk+5Dw1CKTqX/s9dVdc/PbCWl0MtoKo0mjqzU95VedAD5PNwG/MDtPnaflvR4DG 0kbDg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddugdejhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhggtgfgsehtqhertdertdejnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeevhe evleehgfduhffhudefteehhfegjeeiudevheevffetuedttdfhkeeuleehudenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdrhigrnhessh gvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Apr 2022 10:31:28 -0400 (EDT) From: Zi Yan To: David Hildenbrand , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Vlastimil Babka , Mel Gorman , Eric Ren , Mike Rapoport , Oscar Salvador , Christophe Leroy , Andrew Morton , Zi Yan Subject: [PATCH v11 2/6] mm: page_isolation: check specified range for unmovable pages Date: Mon, 25 Apr 2022 10:31:14 -0400 Message-Id: <20220425143118.2850746-3-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220425143118.2850746-1-zi.yan@sent.com> References: <20220425143118.2850746-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: AE019C0043 X-Stat-Signature: enugpp1n4emr6bd83nybdmtxomcfk6ko Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b=A9b3waUU; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=oYsissR3; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf28.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com X-Rspam-User: X-HE-Tag: 1650897083-78785 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan Enable set_migratetype_isolate() to check specified range for unmovable pages during isolation to prepare arbitrary range page isolation. The functionality will take effect in upcoming commits by adjusting the callers of start_isolate_page_range(), which uses set_migratetype_isolate(). For example, alloc_contig_range(), which calls start_isolate_page_range(), accepts unaligned ranges, but because page isolation is currently done at MAX_ORDER_NR_PAEGS granularity, pages that are out of the specified range but withint MAX_ORDER_NR_PAEGS alignment might be attempted for isolation and the failure of isolating these unrelated pages fails the whole operation undesirably. Signed-off-by: Zi Yan --- mm/page_isolation.c | 47 ++++++++++++++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/mm/page_isolation.c b/mm/page_isolation.c index df49f86a6ed1..c2f7a8bb634d 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -16,7 +16,9 @@ #include /* - * This function checks whether pageblock includes unmovable pages or not. + * This function checks whether the range [start_pfn, end_pfn) includes + * unmovable pages or not. The range must fall into a single pageblock and + * consequently belong to a single zone. * * PageLRU check without isolation or lru_lock could race so that * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable @@ -28,12 +30,15 @@ * cannot get removed (e.g., via memory unplug) concurrently. * */ -static struct page *has_unmovable_pages(struct zone *zone, struct page *page, - int migratetype, int flags) +static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long end_pfn, + int migratetype, int flags) { - unsigned long iter = 0; - unsigned long pfn = page_to_pfn(page); - unsigned long offset = pfn % pageblock_nr_pages; + struct page *page = pfn_to_page(start_pfn); + struct zone *zone = page_zone(page); + unsigned long pfn; + + VM_BUG_ON(ALIGN_DOWN(start_pfn, pageblock_nr_pages) != + ALIGN_DOWN(end_pfn - 1, pageblock_nr_pages)); if (is_migrate_cma_page(page)) { /* @@ -47,8 +52,8 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, return page; } - for (; iter < pageblock_nr_pages - offset; iter++) { - page = pfn_to_page(pfn + iter); + for (pfn = start_pfn; pfn < end_pfn; pfn++) { + page = pfn_to_page(pfn); /* * Both, bootmem allocations and memory holes are marked @@ -85,7 +90,7 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, } skip_pages = compound_nr(head) - (page - head); - iter += skip_pages - 1; + pfn += skip_pages - 1; continue; } @@ -97,7 +102,7 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, */ if (!page_ref_count(page)) { if (PageBuddy(page)) - iter += (1 << buddy_order(page)) - 1; + pfn += (1 << buddy_order(page)) - 1; continue; } @@ -134,11 +139,18 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, return NULL; } -static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags) +/* + * This function set pageblock migratetype to isolate if no unmovable page is + * present in [start_pfn, end_pfn). The pageblock must intersect with + * [start_pfn, end_pfn). + */ +static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags, + unsigned long start_pfn, unsigned long end_pfn) { struct zone *zone = page_zone(page); struct page *unmovable; unsigned long flags; + unsigned long check_unmovable_start, check_unmovable_end; spin_lock_irqsave(&zone->lock, flags); @@ -155,8 +167,16 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ /* * FIXME: Now, memory hotplug doesn't call shrink_slab() by itself. * We just check MOVABLE pages. + * + * Pass the intersection of [start_pfn, end_pfn) and the page's pageblock + * to avoid redundant checks. */ - unmovable = has_unmovable_pages(zone, page, migratetype, isol_flags); + check_unmovable_start = max(page_to_pfn(page), start_pfn); + check_unmovable_end = min(ALIGN(page_to_pfn(page) + 1, pageblock_nr_pages), + end_pfn); + + unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end, + migratetype, isol_flags); if (!unmovable) { unsigned long nr_pages; int mt = get_pageblock_migratetype(page); @@ -313,7 +333,8 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, pfn < end_pfn; pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); - if (page && set_migratetype_isolate(page, migratetype, flags)) { + if (page && set_migratetype_isolate(page, migratetype, flags, + start_pfn, end_pfn)) { undo_isolate_page_range(start_pfn, pfn, migratetype); return -EBUSY; } From patchwork Mon Apr 25 14:31:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12825848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79489C433FE for ; Mon, 25 Apr 2022 14:31:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03A466B007E; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F06E76B0083; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B44D96B007E; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 87DA06B007B for ; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4532821D19 for ; Mon, 25 Apr 2022 14:31:31 +0000 (UTC) X-FDA: 79395639582.25.5EE154B Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf24.hostedemail.com (Postfix) with ESMTP id 82A2918005C for ; Mon, 25 Apr 2022 14:31:27 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 156D95C0187; Mon, 25 Apr 2022 10:31:30 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 25 Apr 2022 10:31:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm1; t= 1650897090; x=1650983490; bh=JZpLw6fnQnPze5Yn7EzGGyNnuGr18Ikq05h 0cvuKnMk=; b=L+aYrOlZyn6hJXPRoCjqNZGoPXyNB9odmV363tpeLatzp5DmdSL L7ECgOcM6iN32gPONENDNP1Lv+WsNcJJgp2AfsLYbdWWD4KOLcugYseQr+6wBAhc yUjiiy6RBUM10vfCC9Bk3vnVTtKIP4yxlYNo1iMhZrnb30hxtCWJR+xvpfJUEJEN IKadMY8UXbYk39GKYwATGZPyeiM4d4+jEaT7bMI1QiZ9M/R9rn6nsTDnn5Ve41LU FphBwSPc/ibVenxv23HVEv0BO1Kz2HJgGDN7n84sO+o2UyakbcoVqGPZqxdbAnG0 IQ8zFuj8FOOCW+XT+s8R74/69eauD5F6WkA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1650897090; x=1650983490; bh=J ZpLw6fnQnPze5Yn7EzGGyNnuGr18Ikq05h0cvuKnMk=; b=HAhaV4bOixAyT7g3T qFEHTkn1xj+sxuUk7b3R0/kFU4VcP72rki+DB32tAlRZLA/gEu3L4ZDBY83uw3gv 6PM5I35cYdpv5eJ91MEJG9n0D9Q3SMcBp0g8G4Th795zxeSxz5f9CJaeUM0YeKeQ aTQxPvxkOXxeD4oBCUxxSABsHF1P2aCRJnhSb6oMP+jDKmOKiWNIvVCEBQ+xC3ig 6jdBmZptNh5xLc30vbAdb42QBliz40+ftVAwPcON2GCrwr2aDipbJ4W8165d3Gg9 L79flY6+3axCskKhEa8wFY8sunc5spyRbnAN185/Tx7kh4igUop7cj2kKJbqrY4N X4oZg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddugdejhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhggtgfgsehtqhertdertdejnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeevhe evleehgfduhffhudefteehhfegjeeiudevheevffetuedttdfhkeeuleehudenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdrhigrnhessh gvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Apr 2022 10:31:29 -0400 (EDT) From: Zi Yan To: David Hildenbrand , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Vlastimil Babka , Mel Gorman , Eric Ren , Mike Rapoport , Oscar Salvador , Christophe Leroy , Andrew Morton , Zi Yan , kernel test robot Subject: [PATCH v11 3/6] mm: make alloc_contig_range work at pageblock granularity Date: Mon, 25 Apr 2022 10:31:15 -0400 Message-Id: <20220425143118.2850746-4-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220425143118.2850746-1-zi.yan@sent.com> References: <20220425143118.2850746-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Stat-Signature: 8hpbddk5ih1dp6oi8ye8dfedjnxf93kp Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b=L+aYrOlZ; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=HAhaV4bO; spf=pass (imf24.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 82A2918005C X-HE-Tag: 1650897087-460513 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan alloc_contig_range() worked at MAX_ORDER_NR_PAGES granularity to avoid merging pageblocks with different migratetypes. It might unnecessarily convert extra pageblocks at the beginning and at the end of the range. Change alloc_contig_range() to work at pageblock granularity. Special handling is needed for free pages and in-use pages across the boundaries of the range specified by alloc_contig_range(). Because these partially isolated pages causes free page accounting issues. The free pages will be split and freed into separate migratetype lists; the in-use pages will be migrated then the freed pages will be handled in the aforementioned way. Reported-by: kernel test robot Signed-off-by: Zi Yan Signed-off-by: Zi Yan Reported-by: kernel test robot Signed-off-by: Andrew Morton --- include/linux/page-isolation.h | 4 +- mm/internal.h | 6 ++ mm/memory_hotplug.c | 3 +- mm/page_alloc.c | 54 ++++++++-- mm/page_isolation.c | 184 ++++++++++++++++++++++++++++++++- 5 files changed, 233 insertions(+), 18 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index e14eddf6741a..5456b7be38ae 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -42,7 +42,7 @@ int move_freepages_block(struct zone *zone, struct page *page, */ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, - unsigned migratetype, int flags); + int migratetype, int flags, gfp_t gfp_flags); /* * Changes MIGRATE_ISOLATE to MIGRATE_MOVABLE. @@ -50,7 +50,7 @@ start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, */ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, - unsigned migratetype); + int migratetype); /* * Test all pages in [start_pfn, end_pfn) are isolated or not. diff --git a/mm/internal.h b/mm/internal.h index 919fa07e1031..0667abd57634 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -359,6 +359,9 @@ extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, phys_addr_t min_addr, int nid, bool exact_nid); +void split_free_page(struct page *free_page, + int order, unsigned long split_pfn_offset); + #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* @@ -422,6 +425,9 @@ isolate_freepages_range(struct compact_control *cc, int isolate_migratepages_range(struct compact_control *cc, unsigned long low_pfn, unsigned long end_pfn); + +int __alloc_contig_migrate_range(struct compact_control *cc, + unsigned long start, unsigned long end); #endif int find_suitable_fallback(struct free_area *area, unsigned int order, int migratetype, bool only_stealable, bool *can_steal); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 4c6065e5d274..9f8ae4cb77ee 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1845,7 +1845,8 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, /* set above range as isolated */ ret = start_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - MEMORY_OFFLINE | REPORT_FAILURE); + MEMORY_OFFLINE | REPORT_FAILURE, + GFP_USER | __GFP_MOVABLE | __GFP_RETRY_MAYFAIL); if (ret) { reason = "failure to isolate range"; goto failed_removal_pcplists_disabled; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ce23ac8ad085..70ddd9a0bcf3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1094,6 +1094,43 @@ static inline void __free_one_page(struct page *page, page_reporting_notify_free(order); } +/** + * split_free_page() -- split a free page at split_pfn_offset + * @free_page: the original free page + * @order: the order of the page + * @split_pfn_offset: split offset within the page + * + * It is used when the free page crosses two pageblocks with different migratetypes + * at split_pfn_offset within the page. The split free page will be put into + * separate migratetype lists afterwards. Otherwise, the function achieves + * nothing. + */ +void split_free_page(struct page *free_page, + int order, unsigned long split_pfn_offset) +{ + struct zone *zone = page_zone(free_page); + unsigned long free_page_pfn = page_to_pfn(free_page); + unsigned long pfn; + unsigned long flags; + int free_page_order; + + spin_lock_irqsave(&zone->lock, flags); + del_page_from_free_list(free_page, zone, order); + for (pfn = free_page_pfn; + pfn < free_page_pfn + (1UL << order);) { + int mt = get_pfnblock_migratetype(pfn_to_page(pfn), pfn); + + free_page_order = ffs(split_pfn_offset) - 1; + __free_one_page(pfn_to_page(pfn), pfn, zone, free_page_order, + mt, FPI_NONE); + pfn += 1UL << free_page_order; + split_pfn_offset -= (1UL << free_page_order); + /* we have done the first part, now switch to second part */ + if (split_pfn_offset == 0) + split_pfn_offset = (1UL << order) - (pfn - free_page_pfn); + } + spin_unlock_irqrestore(&zone->lock, flags); +} /* * A bad page could be due to a number of fields. Instead of multiple branches, * try and check multiple fields with one check. The caller must do a detailed @@ -8919,7 +8956,7 @@ static inline void alloc_contig_dump_pages(struct list_head *page_list) #endif /* [start, end) must belong to a single zone. */ -static int __alloc_contig_migrate_range(struct compact_control *cc, +int __alloc_contig_migrate_range(struct compact_control *cc, unsigned long start, unsigned long end) { /* This function is based on compact_zone() from compaction.c. */ @@ -9002,7 +9039,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; - unsigned int order; + int order; int ret = 0; struct compact_control cc = { @@ -9021,14 +9058,11 @@ int alloc_contig_range(unsigned long start, unsigned long end, * What we do here is we mark all pageblocks in range as * MIGRATE_ISOLATE. Because pageblock and max order pages may * have different sizes, and due to the way page allocator - * work, we align the range to biggest of the two pages so - * that page allocator won't try to merge buddies from - * different pageblocks and change MIGRATE_ISOLATE to some - * other migration type. + * work, start_isolate_page_range() has special handlings for this. * * Once the pageblocks are marked as MIGRATE_ISOLATE, we * migrate the pages from an unaligned range (ie. pages that - * we are interested in). This will put all the pages in + * we are interested in). This will put all the pages in * range back to page allocator as MIGRATE_ISOLATE. * * When this is done, we take the pages in range from page @@ -9042,9 +9076,9 @@ int alloc_contig_range(unsigned long start, unsigned long end, */ ret = start_isolate_page_range(pfn_max_align_down(start), - pfn_max_align_up(end), migratetype, 0); + pfn_max_align_up(end), migratetype, 0, gfp_mask); if (ret) - return ret; + goto done; drain_all_pages(cc.zone); @@ -9064,7 +9098,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, ret = 0; /* - * Pages from [start, end) are within a MAX_ORDER_NR_PAGES + * Pages from [start, end) are within a pageblock_nr_pages * aligned blocks that are marked as MIGRATE_ISOLATE. What's * more, all pages in [start, end) are free in page allocator. * What we are going to do is to allocate all pages from diff --git a/mm/page_isolation.c b/mm/page_isolation.c index c2f7a8bb634d..94b3467e5ba2 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -203,7 +203,7 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ return -EBUSY; } -static void unset_migratetype_isolate(struct page *page, unsigned migratetype) +static void unset_migratetype_isolate(struct page *page, int migratetype) { struct zone *zone; unsigned long flags, nr_pages; @@ -279,6 +279,157 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) return NULL; } +/** + * isolate_single_pageblock() -- tries to isolate a pageblock that might be + * within a free or in-use page. + * @boundary_pfn: pageblock-aligned pfn that a page might cross + * @gfp_flags: GFP flags used for migrating pages + * @isolate_before: isolate the pageblock before the boundary_pfn + * + * Free and in-use pages can be as big as MAX_ORDER-1 and contain more than one + * pageblock. When not all pageblocks within a page are isolated at the same + * time, free page accounting can go wrong. For example, in the case of + * MAX_ORDER-1 = pageblock_order + 1, a MAX_ORDER-1 page has two pagelbocks. + * [ MAX_ORDER-1 ] + * [ pageblock0 | pageblock1 ] + * When either pageblock is isolated, if it is a free page, the page is not + * split into separate migratetype lists, which is supposed to; if it is an + * in-use page and freed later, __free_one_page() does not split the free page + * either. The function handles this by splitting the free page or migrating + * the in-use page then splitting the free page. + */ +static int isolate_single_pageblock(unsigned long boundary_pfn, gfp_t gfp_flags, + bool isolate_before) +{ + unsigned char saved_mt; + unsigned long start_pfn; + unsigned long isolate_pageblock; + unsigned long pfn; + struct zone *zone; + + VM_BUG_ON(!IS_ALIGNED(boundary_pfn, pageblock_nr_pages)); + + if (isolate_before) + isolate_pageblock = boundary_pfn - pageblock_nr_pages; + else + isolate_pageblock = boundary_pfn; + + /* + * scan at the beginning of MAX_ORDER_NR_PAGES aligned range to avoid + * only isolating a subset of pageblocks from a bigger than pageblock + * free or in-use page. Also make sure all to-be-isolated pageblocks + * are within the same zone. + */ + zone = page_zone(pfn_to_page(isolate_pageblock)); + start_pfn = max(ALIGN_DOWN(isolate_pageblock, MAX_ORDER_NR_PAGES), + zone->zone_start_pfn); + + saved_mt = get_pageblock_migratetype(pfn_to_page(isolate_pageblock)); + set_pageblock_migratetype(pfn_to_page(isolate_pageblock), MIGRATE_ISOLATE); + + /* + * Bail out early when the to-be-isolated pageblock does not form + * a free or in-use page across boundary_pfn: + * + * 1. isolate before boundary_pfn: the page after is not online + * 2. isolate after boundary_pfn: the page before is not online + * + * This also ensures correctness. Without it, when isolate after + * boundary_pfn and [start_pfn, boundary_pfn) are not online, + * __first_valid_page() will return unexpected NULL in the for loop + * below. + */ + if (isolate_before) { + if (!pfn_to_online_page(boundary_pfn)) + return 0; + } else { + if (!pfn_to_online_page(boundary_pfn - 1)) + return 0; + } + + for (pfn = start_pfn; pfn < boundary_pfn;) { + struct page *page = __first_valid_page(pfn, boundary_pfn - pfn); + + VM_BUG_ON(!page); + pfn = page_to_pfn(page); + /* + * start_pfn is MAX_ORDER_NR_PAGES aligned, if there is any + * free pages in [start_pfn, boundary_pfn), its head page will + * always be in the range. + */ + if (PageBuddy(page)) { + int order = buddy_order(page); + + if (pfn + (1UL << order) > boundary_pfn) + split_free_page(page, order, boundary_pfn - pfn); + pfn += (1UL << order); + continue; + } + /* + * migrate compound pages then let the free page handling code + * above do the rest. If migration is not enabled, just fail. + */ + if (PageHuge(page) || PageTransCompound(page)) { +#if defined CONFIG_COMPACTION || defined CONFIG_CMA + unsigned long nr_pages = compound_nr(page); + int order = compound_order(page); + struct page *head = compound_head(page); + unsigned long head_pfn = page_to_pfn(head); + int ret; + struct compact_control cc = { + .nr_migratepages = 0, + .order = -1, + .zone = page_zone(pfn_to_page(head_pfn)), + .mode = MIGRATE_SYNC, + .ignore_skip_hint = true, + .no_set_skip_hint = true, + .gfp_mask = gfp_flags, + .alloc_contig = true, + }; + INIT_LIST_HEAD(&cc.migratepages); + + if (head_pfn + nr_pages < boundary_pfn) { + pfn += nr_pages; + continue; + } + + ret = __alloc_contig_migrate_range(&cc, head_pfn, + head_pfn + nr_pages); + + if (ret) + goto failed; + /* + * reset pfn, let the free page handling code above + * split the free page to the right migratetype list. + * + * head_pfn is not used here as a hugetlb page order + * can be bigger than MAX_ORDER-1, but after it is + * freed, the free page order is not. Use pfn within + * the range to find the head of the free page and + * reset order to 0 if a hugetlb page with + * >MAX_ORDER-1 order is encountered. + */ + if (order > MAX_ORDER-1) + order = 0; + while (!PageBuddy(pfn_to_page(pfn))) { + order++; + pfn &= ~0UL << order; + } + continue; +#else + goto failed; +#endif + } + + pfn++; + } + return 0; +failed: + /* restore the original migratetype */ + set_pageblock_migratetype(pfn_to_page(isolate_pageblock), saved_mt); + return -EBUSY; +} + /** * start_isolate_page_range() - make page-allocation-type of range of pages to * be MIGRATE_ISOLATE. @@ -293,6 +444,8 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * and PageOffline() pages. * REPORT_FAILURE - report details about the failure to * isolate the range + * @gfp_flags: GFP flags used for migrating pages that sit across the + * range boundaries. * * Making page-allocation-type to be MIGRATE_ISOLATE means free pages in * the range will never be allocated. Any free pages and pages freed in the @@ -301,6 +454,10 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * pages in the range finally, the caller have to free all pages in the range. * test_page_isolated() can be used for test it. * + * The function first tries to isolate the pageblocks at the beginning and end + * of the range, since there might be pages across the range boundaries. + * Afterwards, it isolates the rest of the range. + * * There is no high level synchronization mechanism that prevents two threads * from trying to isolate overlapping ranges. If this happens, one thread * will notice pageblocks in the overlapping range already set to isolate. @@ -321,21 +478,38 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * Return: 0 on success and -EBUSY if any part of range cannot be isolated. */ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, - unsigned migratetype, int flags) + int migratetype, int flags, gfp_t gfp_flags) { unsigned long pfn; struct page *page; + int ret; BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); - for (pfn = start_pfn; - pfn < end_pfn; + /* isolate [start_pfn, start_pfn + pageblock_nr_pages) pageblock */ + ret = isolate_single_pageblock(start_pfn, gfp_flags, false); + if (ret) + return ret; + + /* isolate [end_pfn - pageblock_nr_pages, end_pfn) pageblock */ + ret = isolate_single_pageblock(end_pfn, gfp_flags, true); + if (ret) { + unset_migratetype_isolate(pfn_to_page(start_pfn), migratetype); + return ret; + } + + /* skip isolated pageblocks at the beginning and end */ + for (pfn = start_pfn + pageblock_nr_pages; + pfn < end_pfn - pageblock_nr_pages; pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (page && set_migratetype_isolate(page, migratetype, flags, start_pfn, end_pfn)) { undo_isolate_page_range(start_pfn, pfn, migratetype); + unset_migratetype_isolate( + pfn_to_page(end_pfn - pageblock_nr_pages), + migratetype); return -EBUSY; } } @@ -346,7 +520,7 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, * Make isolated pages available again. */ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, - unsigned migratetype) + int migratetype) { unsigned long pfn; struct page *page; From patchwork Mon Apr 25 14:31:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12825850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41F3CC433F5 for ; Mon, 25 Apr 2022 14:31:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C7BE6B0080; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DC796B0085; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E94D16B007B; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id A8D1B6B0081 for ; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 239BD27E03 for ; Mon, 25 Apr 2022 14:31:32 +0000 (UTC) X-FDA: 79395639624.26.A728891 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf31.hostedemail.com (Postfix) with ESMTP id AAD682004B for ; Mon, 25 Apr 2022 14:31:24 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 2C26B5C0185; Mon, 25 Apr 2022 10:31:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 25 Apr 2022 10:31:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm1; t= 1650897091; x=1650983491; bh=FGuSjq7Qo3xUzFw7F3hZNaEuHq5HI+jj6I+ FwTpeURQ=; b=PMgvB8euRccq21wM7F4HnHrBKTiTgoZEpN3WVqB5zlpFZxbrB48 qhfl/a7b8yXeFFi3Xo0fNpxjbOzr998/Bcq1/6FxoX7iCxu4bMMuBfOY8faEBNCM exGviOuRiFZXq9CnnfR6DyJ/HpUzPMVYYv6f8k8BQCcVMFZAvOoxU9jO3eZkdD6A SEej2hXqTRrTnVmKtGKdnMVcxyRwVxcVkpLzfML5EE6l2xCwsJsnk4QhXKukybla PmYZcwpo3flYMI2amf9m5hD/2RgjBdV4hzHRtMQFod1gXz5Bn9Q/JANUgYmeio/2 J2CAAPclPa0iEjDu/Mo0aG6DvfxYyXtXdbA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1650897091; x=1650983491; bh=F GuSjq7Qo3xUzFw7F3hZNaEuHq5HI+jj6I+FwTpeURQ=; b=qF6TRLPv4pfzD9D9/ Va14A9OhcHzxGGmwchLkfogHJk8Kw/w3urkm17g1zS7m5zLGfrkHltP491iRgIlL SjVWPErSQ6CHljQY1LzsKrHF8LMxSbfXjVEv0RODooUTNTHkqi15kT/KqCjSKVe1 RZ4tR2vxCLxmJjSf5Ik2WhoxTTR0XqWmkPyjTUKLJWbvM0SShmVESvM6vo2xG+x5 w5cKHyDn8FBsn22O2i/GNF2hbLAq/X8yx9bOlsYlpwxEMJT9UB4fx54Je+Bw5hF0 KvV5szFX8WUati0aY0SlUcEGqhdbY9VcXGEDQy2ce5BvdG9hiNbfUBPOaqEh2J1d ZH1Qw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddugdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhggtgfgsehtqhertdertdejnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeevhe evleehgfduhffhudefteehhfegjeeiudevheevffetuedttdfhkeeuleehudenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdrhigrnhessh gvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Apr 2022 10:31:30 -0400 (EDT) From: Zi Yan To: David Hildenbrand , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Vlastimil Babka , Mel Gorman , Eric Ren , Mike Rapoport , Oscar Salvador , Christophe Leroy , Andrew Morton , Zi Yan Subject: [PATCH v11 4/6] mm: page_isolation: enable arbitrary range page isolation. Date: Mon, 25 Apr 2022 10:31:16 -0400 Message-Id: <20220425143118.2850746-5-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220425143118.2850746-1-zi.yan@sent.com> References: <20220425143118.2850746-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Queue-Id: AAD682004B X-Stat-Signature: m8oa85gja18ntckbymqg5n4596rjxkuj Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b=PMgvB8eu; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=qF6TRLPv; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf31.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1650897084-886774 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan Now start_isolate_page_range() is ready to handle arbitrary range isolation, so move the alignment check/adjustment into the function body. Do the same for its counterpart undo_isolate_page_range(). alloc_contig_range(), its caller, can pass an arbitrary range instead of a MAX_ORDER_NR_PAGES aligned one. Signed-off-by: Zi Yan Signed-off-by: Zi Yan Signed-off-by: Andrew Morton --- mm/page_alloc.c | 16 ++-------------- mm/page_isolation.c | 33 ++++++++++++++++----------------- 2 files changed, 18 insertions(+), 31 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 70ddd9a0bcf3..a002cf12eb6c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8924,16 +8924,6 @@ void *__init alloc_large_system_hash(const char *tablename, } #ifdef CONFIG_CONTIG_ALLOC -static unsigned long pfn_max_align_down(unsigned long pfn) -{ - return ALIGN_DOWN(pfn, MAX_ORDER_NR_PAGES); -} - -static unsigned long pfn_max_align_up(unsigned long pfn) -{ - return ALIGN(pfn, MAX_ORDER_NR_PAGES); -} - #if defined(CONFIG_DYNAMIC_DEBUG) || \ (defined(CONFIG_DYNAMIC_DEBUG_CORE) && defined(DYNAMIC_DEBUG_MODULE)) /* Usage: See admin-guide/dynamic-debug-howto.rst */ @@ -9075,8 +9065,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, * put back to page allocator so that buddy can use them. */ - ret = start_isolate_page_range(pfn_max_align_down(start), - pfn_max_align_up(end), migratetype, 0, gfp_mask); + ret = start_isolate_page_range(start, end, migratetype, 0, gfp_mask); if (ret) goto done; @@ -9157,8 +9146,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, free_contig_range(end, outer_end - end); done: - undo_isolate_page_range(pfn_max_align_down(start), - pfn_max_align_up(end), migratetype); + undo_isolate_page_range(start, end, migratetype); return ret; } EXPORT_SYMBOL(alloc_contig_range); diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 94b3467e5ba2..75e454f5cf45 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -435,7 +435,6 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, gfp_t gfp_flags, * be MIGRATE_ISOLATE. * @start_pfn: The lower PFN of the range to be isolated. * @end_pfn: The upper PFN of the range to be isolated. - * start_pfn/end_pfn must be aligned to pageblock_order. * @migratetype: Migrate type to set in error recovery. * @flags: The following flags are allowed (they can be combined in * a bit mask) @@ -482,33 +481,33 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, { unsigned long pfn; struct page *page; + /* isolation is done at page block granularity */ + unsigned long isolate_start = ALIGN_DOWN(start_pfn, pageblock_nr_pages); + unsigned long isolate_end = ALIGN(end_pfn, pageblock_nr_pages); int ret; - BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); - BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); - - /* isolate [start_pfn, start_pfn + pageblock_nr_pages) pageblock */ - ret = isolate_single_pageblock(start_pfn, gfp_flags, false); + /* isolate [isolate_start, isolate_start + pageblock_nr_pages) pageblock */ + ret = isolate_single_pageblock(isolate_start, gfp_flags, false); if (ret) return ret; - /* isolate [end_pfn - pageblock_nr_pages, end_pfn) pageblock */ - ret = isolate_single_pageblock(end_pfn, gfp_flags, true); + /* isolate [isolate_end - pageblock_nr_pages, isolate_end) pageblock */ + ret = isolate_single_pageblock(isolate_end, gfp_flags, true); if (ret) { - unset_migratetype_isolate(pfn_to_page(start_pfn), migratetype); + unset_migratetype_isolate(pfn_to_page(isolate_start), migratetype); return ret; } /* skip isolated pageblocks at the beginning and end */ - for (pfn = start_pfn + pageblock_nr_pages; - pfn < end_pfn - pageblock_nr_pages; + for (pfn = isolate_start + pageblock_nr_pages; + pfn < isolate_end - pageblock_nr_pages; pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (page && set_migratetype_isolate(page, migratetype, flags, start_pfn, end_pfn)) { - undo_isolate_page_range(start_pfn, pfn, migratetype); + undo_isolate_page_range(isolate_start, pfn, migratetype); unset_migratetype_isolate( - pfn_to_page(end_pfn - pageblock_nr_pages), + pfn_to_page(isolate_end - pageblock_nr_pages), migratetype); return -EBUSY; } @@ -524,12 +523,12 @@ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, { unsigned long pfn; struct page *page; + unsigned long isolate_start = ALIGN_DOWN(start_pfn, pageblock_nr_pages); + unsigned long isolate_end = ALIGN(end_pfn, pageblock_nr_pages); - BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); - BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); - for (pfn = start_pfn; - pfn < end_pfn; + for (pfn = isolate_start; + pfn < isolate_end; pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (!page || !is_migrate_isolate_page(page)) From patchwork Mon Apr 25 14:31:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12825849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 566DAC4332F for ; Mon, 25 Apr 2022 14:31:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5533F6B0078; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 463226B0082; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6EF76B0078; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 932826B0078 for ; Mon, 25 Apr 2022 10:31:33 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C5EA727E08 for ; Mon, 25 Apr 2022 14:31:32 +0000 (UTC) X-FDA: 79395639624.14.EB6A9F6 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf29.hostedemail.com (Postfix) with ESMTP id 0D83412003D for ; Mon, 25 Apr 2022 14:31:29 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id DE5505C015E; Mon, 25 Apr 2022 10:31:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 25 Apr 2022 10:31:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm1; t= 1650897091; x=1650983491; bh=JX3ps6oeKyuY4MRNJ2GnR5dfK9tpHY2lYtV KJFK+Xqk=; b=TDLUN33aq/3MMOOBQQuus8Fk/UJeHWVmjSrGGfM1mfa8bAK7vJ5 wBgjeOJBfDbCb1I+PWErV8urwm1X8HsvklEN2dNwEBFu94cLiujXIAvbXoffx4gk l2kjZo7ki3kbSZyrX5cNTWKwN64199owIRtTiDNJgbbg8OHol/1LlxZuobIz0GVg InGz1TautCYF+GhvEtdFDS4AkX0OenV93TdDt+1GDmoLvymu8FlJ6s+1hBmj+Mkk SZar04DgOqU5FQqIekQsE4D+SpcP3Mg2/7TC2gu6nKcBytgL90mZ/VmMEprwPZ7U shukvJcGtjcIIeMfMU5hz29FVM+3qGS/iew== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1650897091; x=1650983491; bh=J X3ps6oeKyuY4MRNJ2GnR5dfK9tpHY2lYtVKJFK+Xqk=; b=kPGxsDlj958gfs7of XLw5DbJmBrF786oXF8yS51MeyrHP/OkunAM2eAuyNQdZi0QGdoccKuDxO1XUYst8 yiFIaBmFTDJ7CWWYlvSwpqV9LLqzLRcXPmnXmFk5ZaBjX1kiCHnT/nE6xylRvB+j lT+TQJnJ3Y9RGorphVoO7+m056go8ZZopyTT2x7vMqUeNb5Z0C2yyRiep1NW0t1v VqqTDedauWXYj1AsuuOJNYVa2NZDx/MF2dpkqgz0/cdLt52NSLD2OPWd0sjE4xMX I0KJdFXQD/ThxZwdIYcZPbxwittjpaGNQi/XUJtLaQ68tM/UAUUvv/sZi1WwMNE5 a++TA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddugdejhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhggtgfgsehtqhertdertdejnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeevhe evleehgfduhffhudefteehhfegjeeiudevheevffetuedttdfhkeeuleehudenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdrhigrnhessh gvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Apr 2022 10:31:31 -0400 (EDT) From: Zi Yan To: David Hildenbrand , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Vlastimil Babka , Mel Gorman , Eric Ren , Mike Rapoport , Oscar Salvador , Christophe Leroy , Andrew Morton , Zi Yan Subject: [PATCH v11 5/6] mm: cma: use pageblock_order as the single alignment Date: Mon, 25 Apr 2022 10:31:17 -0400 Message-Id: <20220425143118.2850746-6-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220425143118.2850746-1-zi.yan@sent.com> References: <20220425143118.2850746-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0D83412003D X-Stat-Signature: w333xu84er3nebgqwyqgr5sqy78dgcjr X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b=TDLUN33a; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=kPGxsDlj; spf=pass (imf29.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com X-HE-Tag: 1650897089-11238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan Now alloc_contig_range() works at pageblock granularity. Change CMA allocation, which uses alloc_contig_range(), to use pageblock_nr_pages alignment. Signed-off-by: Zi Yan --- include/linux/cma.h | 4 ++-- include/linux/mmzone.h | 5 +---- mm/page_alloc.c | 4 ++-- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index a6f637342740..63873b93deaa 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -17,11 +17,11 @@ #define CMA_MAX_NAME 64 /* - * TODO: once the buddy -- especially pageblock merging and alloc_contig_range() + * the buddy -- especially pageblock merging and alloc_contig_range() * -- can deal with only some pageblocks of a higher-order page being * MIGRATE_CMA, we can use pageblock_nr_pages. */ -#define CMA_MIN_ALIGNMENT_PAGES MAX_ORDER_NR_PAGES +#define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages #define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES) struct cma; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 46ffab808f03..aab70355d64f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -54,10 +54,7 @@ enum migratetype { * * The way to use it is to change migratetype of a range of * pageblocks to MIGRATE_CMA which can be done by - * __free_pageblock_cma() function. What is important though - * is that a range of pageblocks must be aligned to - * MAX_ORDER_NR_PAGES should biggest page be bigger than - * a single pageblock. + * __free_pageblock_cma() function. */ MIGRATE_CMA, #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a002cf12eb6c..bc9e129ab3d1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9014,8 +9014,8 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * be either of the two. * @gfp_mask: GFP mask to use during compaction * - * The PFN range does not have to be pageblock or MAX_ORDER_NR_PAGES - * aligned. The PFN range must belong to a single zone. + * The PFN range does not have to be pageblock aligned. The PFN range must + * belong to a single zone. * * The first thing this routine does is attempt to MIGRATE_ISOLATE all * pageblocks in the range. Once isolated, the pageblocks should not From patchwork Mon Apr 25 14:31:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12825851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A441CC433EF for ; Mon, 25 Apr 2022 14:31:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BAAE36B007B; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC3BC6B0082; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C4B76B007B; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 378076B0081 for ; Mon, 25 Apr 2022 10:31:34 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5B79A1F7C for ; Mon, 25 Apr 2022 14:31:33 +0000 (UTC) X-FDA: 79395639666.18.A61FB18 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf16.hostedemail.com (Postfix) with ESMTP id 02260180050 for ; Mon, 25 Apr 2022 14:31:29 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 8D32C5C0164; Mon, 25 Apr 2022 10:31:32 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 25 Apr 2022 10:31:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm1; t= 1650897092; x=1650983492; bh=RypWuAcjXYhvuMrAXCozdcgxcIlBzpaT89Q 2nmHZmv8=; b=D5uhdhKfZsWZJIiNwrYTpLyeejy7tifoZrD6Jw6grTvCevtOo4+ pukYeJ21+jwXCD+HCsI55c35QRp5sWWgm8OyZ3kQfjZsEoRYTUvvREu9c6MIRD30 6SnY7hWWIriRowoJTvWDWeVPvDtruV/xjyY/Xeyk/I3IOonR++GQFuVlN4/BTbWt ULB7LF6lYWoMY1ci0hLEYjwQSOBkqxNPjowExCj5PePqS8iNfs0K528zvw1bG5SU CrmXHo5p4mf8H6sni0AW8DRIzFjTuxWuc/eJq7cFDGPNz6zEduG/HIlnJvCFO8ij VoVxTsqYjmWNmTNB1r6p9sMbKjMJUOfIk6Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1650897092; x=1650983492; bh=R ypWuAcjXYhvuMrAXCozdcgxcIlBzpaT89Q2nmHZmv8=; b=vw2kQpwrp/ECD/KAm wvBZK1OZK+1cr/Lx5BBB62qILIkfjHbsGZlNwfwXg+R6orfq1W9q9vB9nx6rSTE/ 0SyywO0ZXE+iYjvHkK9JP2uwtJCB4XIWHjlzq/CaEpiKXuZ0Yi73Cflu7T72sFvR 6g1aauAs+mT6R47rpLO6hn4qm+WH4dIU4rhCONK6zvOFtSkk9P2QfjxI81ygOeRk U2qwEIznwcMRQjYI6MnsoxuU0C4Zec77ilUjwOs5qsLFU9a5JLmDOfRveD+j9XAB pdZ+PaJlfmn1www5IS6myrcgId2ng4tdyFWWm+V13tFb7uZ+tsMzUnBFooIhO549 aAZ9Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddugdejhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhggtgfgsehtqhertdertdejnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeevhe evleehgfduhffhudefteehhfegjeeiudevheevffetuedttdfhkeeuleehudenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdrhigrnhessh gvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Apr 2022 10:31:31 -0400 (EDT) From: Zi Yan To: David Hildenbrand , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Vlastimil Babka , Mel Gorman , Eric Ren , Mike Rapoport , Oscar Salvador , Christophe Leroy , Andrew Morton , Zi Yan Subject: [PATCH v11 6/6] drivers: virtio_mem: use pageblock size as the minimum virtio_mem size. Date: Mon, 25 Apr 2022 10:31:18 -0400 Message-Id: <20220425143118.2850746-7-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220425143118.2850746-1-zi.yan@sent.com> References: <20220425143118.2850746-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 02260180050 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b=D5uhdhKf; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=vw2kQpwr; spf=pass (imf16.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com X-Rspam-User: X-Stat-Signature: yro6a6bwzorfg33hbhpzn8eojzdipgom X-HE-Tag: 1650897089-301324 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan alloc_contig_range() now only needs to be aligned to pageblock_nr_pages, drop virtio_mem size requirement that it needs to be MAX_ORDER_NR_PAGES. Signed-off-by: Zi Yan --- drivers/virtio/virtio_mem.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index e7d6b679596d..e07486f01999 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -2476,10 +2476,10 @@ static int virtio_mem_init_hotplug(struct virtio_mem *vm) VIRTIO_MEM_DEFAULT_OFFLINE_THRESHOLD); /* - * TODO: once alloc_contig_range() works reliably with pageblock - * granularity on ZONE_NORMAL, use pageblock_nr_pages instead. + * alloc_contig_range() works reliably with pageblock + * granularity on ZONE_NORMAL, use pageblock_nr_pages. */ - sb_size = PAGE_SIZE * MAX_ORDER_NR_PAGES; + sb_size = PAGE_SIZE * pageblock_nr_pages; sb_size = max_t(uint64_t, vm->device_block_size, sb_size); if (sb_size < memory_block_size_bytes() && !force_bbm) {