From patchwork Thu May 6 15:26:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12242445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8503C433B4 for ; Thu, 6 May 2021 15:27:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7822961942 for ; Thu, 6 May 2021 15:27:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7822961942 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 09D0E6B0072; Thu, 6 May 2021 11:27:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0727E6B0073; Thu, 6 May 2021 11:27:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E087C6B0074; Thu, 6 May 2021 11:27:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id C04006B0072 for ; Thu, 6 May 2021 11:27:09 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5EBD58150 for ; Thu, 6 May 2021 15:27:09 +0000 (UTC) X-FDA: 78111184578.25.852BFEE Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com [66.111.4.230]) by imf21.hostedemail.com (Postfix) with ESMTP id 85C83E000120 for ; Thu, 6 May 2021 15:27:04 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.nyi.internal (Postfix) with ESMTP id 98937580B93; Thu, 6 May 2021 11:27:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 06 May 2021 11:27:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm3; bh=FbRZXE7bST17Y nSdlcP8bsWaivyCC5qvTsiLA/pmPMU=; b=g19R6QCBByhaWnAudNOcOPg0FC8GF BYLgrbe2GpFbn8aiFMbJfkjfAP7I+idIgloc1puvSRm5QXSPjXc1FVYCRsWuxy3N 4+RdTjNwWmB4CDQ9BruYX948D0TJQ7ABUpYoUhhNDJmpRabw6UHFIoR/yztt4C1x jpAA2fru8FeUtiUKkrUzkALe/+j95KJ8I5jQTIkECmLIobzung+2zJOXPFXI4iBG 2gwK0c3pZYRSF4+H5QqwYqBPWxNVngm+qGjsDU7emPnuw3TXZCFsu2JQDIJN3tAa sUqKNHPtEjsUl+OLVjPjY46yYWmnCFI6q0K3OHn4Xh3LcGb3ffvyvUOSA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=FbRZXE7bST17YnSdlcP8bsWaivyCC5qvTsiLA/pmPMU=; b=FjUx3ehA KBXkLVqnp+CZgmPRqcpQ+a4y2H1eLHkJBnxQMiHDRcENbL7f27xoAS6B5espCQYU NF2xKMFW65O1OIeUY877OfVjH8emx+SzWdRNe1gDdnYe/C8nSvrjkfvw29kGE8/o DY/Zm2uJyCRGCzGWOrC0W26OtOhpNOA4Kw30vahuL77+MWAKa9v9Cfr2QWt1rsoJ 0ieOXX+7dnt17NcaXTVN0OqsKkVng1XtD/8uGsOpgbsIwKxus4ibj5pUEgOuZh+K 2/5Yh68QoW2EWy933c+61oDgf4cgy5Zf9oXSEeXzueyp65r7quse/nt1Rm+wU1yT /S++4wnJppjsRg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdegtddgkeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppeef hedrudeigedrvdegiedrfeegnecuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from Threadripper.local (ec2-35-164-246-34.us-west-2.compute.amazonaws.com [35.164.246.34]) by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 6 May 2021 11:27:04 -0400 (EDT) From: Zi Yan To: David Hildenbrand , Oscar Salvador Cc: Michael Ellerman , Benjamin Herrenschmidt , Thomas Gleixner , x86@kernel.org, Andy Lutomirski , "Rafael J . Wysocki" , Andrew Morton , Mike Rapoport , Anshuman Khandual , Michal Hocko , Dan Williams , Wei Yang , linux-ia64@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Zi Yan Subject: [RFC PATCH 4/7] mm: pageblock: allow set/unset migratetype for partial pageblock Date: Thu, 6 May 2021 11:26:20 -0400 Message-Id: <20210506152623.178731-5-zi.yan@sent.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210506152623.178731-1-zi.yan@sent.com> References: <20210506152623.178731-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=g19R6QCB; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=FjUx3ehA; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf21.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.230 as permitted sender) smtp.mailfrom=zi.yan@sent.com X-Stat-Signature: 99iihf53axj5xci7c789w4z6fenf6kdg X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 85C83E000120 Received-SPF: none (sent.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=new4-smtp.messagingengine.com; client-ip=66.111.4.230 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620314824-117101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan When we online subsections and pageblock size is larger than a subsection, we should be able to change migratetype for partial pageblocks. We also change the assumption that all pageblocks are in a whole. Signed-off-by: Zi Yan --- include/linux/page-isolation.h | 8 ++++++-- mm/page_alloc.c | 27 ++++++++++++++++++--------- mm/page_isolation.c | 26 ++++++++++++++------------ 3 files changed, 38 insertions(+), 23 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 572458016331..308b540865b7 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -33,11 +33,15 @@ static inline bool is_migrate_isolate(int migratetype) #define MEMORY_OFFLINE 0x1 #define REPORT_FAILURE 0x2 -struct page *has_unmovable_pages(struct zone *zone, struct page *page, - int migratetype, int flags); +struct page *has_unmovable_pages(struct zone *zone, unsigned long start_pfn, + unsigned long end_pfn, int migratetype, + int flags); void set_pageblock_migratetype(struct page *page, int migratetype); int move_freepages_block(struct zone *zone, struct page *page, int migratetype, int *num_movable); +int move_freepages(struct zone *zone, + unsigned long start_pfn, unsigned long end_pfn, + int migratetype, int *num_movable); /* * Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE. diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 72bb4a300e03..bc410f45c355 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2433,7 +2433,7 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, * Note that start_page and end_pages are not aligned on a pageblock * boundary. If alignment is required, use move_freepages_block() */ -static int move_freepages(struct zone *zone, +int move_freepages(struct zone *zone, unsigned long start_pfn, unsigned long end_pfn, int migratetype, int *num_movable) { @@ -6328,6 +6328,7 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone { unsigned long pfn, end_pfn = start_pfn + size; struct page *page; + bool set_migratetype = false; if (highest_memmap_pfn < end_pfn - 1) highest_memmap_pfn = end_pfn - 1; @@ -6374,10 +6375,16 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone */ if (IS_ALIGNED(pfn, pageblock_nr_pages)) { set_pageblock_migratetype(page, migratetype); + set_migratetype = true; cond_resched(); } pfn++; } + /* in case the range is smaller than a pageblock */ + if (!set_migratetype && context == MEMINIT_HOTPLUG) { + page = pfn_to_page(start_pfn); + set_pageblock_migratetype(page, migratetype); + } } #ifdef CONFIG_ZONE_DEVICE @@ -8524,12 +8531,14 @@ void *__init alloc_large_system_hash(const char *tablename, * cannot get removed (e.g., via memory unplug) concurrently. * */ -struct page *has_unmovable_pages(struct zone *zone, struct page *page, - int migratetype, int flags) +struct page *has_unmovable_pages(struct zone *zone, unsigned long start_pfn, + unsigned long end_pfn, int migratetype, + int flags) { unsigned long iter = 0; - unsigned long pfn = page_to_pfn(page); - unsigned long offset = pfn % pageblock_nr_pages; + unsigned long pfn = start_pfn; + struct page *page = pfn_to_page(pfn); + if (is_migrate_cma_page(page)) { /* @@ -8543,11 +8552,11 @@ struct page *has_unmovable_pages(struct zone *zone, struct page *page, return page; } - for (; iter < pageblock_nr_pages - offset; iter++) { - if (!pfn_valid_within(pfn + iter)) + for (pfn = start_pfn; pfn < end_pfn; pfn++) { + if (!pfn_valid_within(pfn)) continue; - page = pfn_to_page(pfn + iter); + page = pfn_to_page(pfn); /* * Both, bootmem allocations and memory holes are marked @@ -8596,7 +8605,7 @@ struct page *has_unmovable_pages(struct zone *zone, struct page *page, */ if (!page_ref_count(page)) { if (PageBuddy(page)) - iter += (1 << buddy_order(page)) - 1; + pfn += (1 << buddy_order(page)) - 1; continue; } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index bddf788f45bf..c1b9b8848382 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -15,8 +15,10 @@ #define CREATE_TRACE_POINTS #include -static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags) +static int set_migratetype_isolate(unsigned long start_pfn, unsigned long end_pfn, + int migratetype, int isol_flags) { + struct page *page = pfn_to_page(start_pfn); struct zone *zone = page_zone(page); struct page *unmovable; unsigned long flags; @@ -37,15 +39,14 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ * FIXME: Now, memory hotplug doesn't call shrink_slab() by itself. * We just check MOVABLE pages. */ - unmovable = has_unmovable_pages(zone, page, migratetype, isol_flags); + unmovable = has_unmovable_pages(zone, start_pfn, end_pfn, migratetype, isol_flags); if (!unmovable) { unsigned long nr_pages; int mt = get_pageblock_migratetype(page); set_pageblock_migratetype(page, MIGRATE_ISOLATE); zone->nr_isolate_pageblock++; - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, - NULL); + nr_pages = move_freepages(zone, start_pfn, end_pfn, MIGRATE_ISOLATE, NULL); __mod_zone_freepage_state(zone, -nr_pages, mt); spin_unlock_irqrestore(&zone->lock, flags); @@ -64,7 +65,8 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ return -EBUSY; } -static void unset_migratetype_isolate(struct page *page, unsigned migratetype) +static void unset_migratetype_isolate(unsigned long start_pfn, unsigned long end_pfn, + unsigned migratetype) { struct zone *zone; unsigned long flags, nr_pages; @@ -72,6 +74,7 @@ static void unset_migratetype_isolate(struct page *page, unsigned migratetype) unsigned int order; unsigned long pfn, buddy_pfn; struct page *buddy; + struct page *page = pfn_to_page(start_pfn); zone = page_zone(page); spin_lock_irqsave(&zone->lock, flags); @@ -112,7 +115,7 @@ static void unset_migratetype_isolate(struct page *page, unsigned migratetype) * allocation. */ if (!isolated_page) { - nr_pages = move_freepages_block(zone, page, migratetype, NULL); + nr_pages = move_freepages(zone, start_pfn, end_pfn, migratetype, NULL); __mod_zone_freepage_state(zone, nr_pages, migratetype); } set_pageblock_migratetype(page, migratetype); @@ -195,7 +198,8 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (page) { - if (set_migratetype_isolate(page, migratetype, flags)) { + if (set_migratetype_isolate(pfn, min(end_pfn, pfn + pageblock_nr_pages), + migratetype, flags)) { undo_pfn = pfn; goto undo; } @@ -209,7 +213,8 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, struct page *page = pfn_to_online_page(pfn); if (!page) continue; - unset_migratetype_isolate(page, migratetype); + unset_migratetype_isolate(pfn, min(pfn + pageblock_nr_pages, undo_pfn), + migratetype); } return -EBUSY; @@ -224,16 +229,13 @@ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, unsigned long pfn; struct page *page; - BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); - BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); - for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (!page || !is_migrate_isolate_page(page)) continue; - unset_migratetype_isolate(page, migratetype); + unset_migratetype_isolate(pfn, min(pfn + pageblock_nr_pages, end_pfn), migratetype); } } /*