From patchwork Wed Mar 6 04:08:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04ED6C54E49 for ; Wed, 6 Mar 2024 04:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 291CE6B007E; Tue, 5 Mar 2024 23:16:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 241FA6B0080; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06E516B0081; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DF1A36B007E for ; Tue, 5 Mar 2024 23:16:23 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B9EA140261 for ; Wed, 6 Mar 2024 04:16:23 +0000 (UTC) X-FDA: 81865302246.07.31FCF0C Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf15.hostedemail.com (Postfix) with ESMTP id E68FCA0005 for ; Wed, 6 Mar 2024 04:16:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Af1ZDlZE; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf15.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698580; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TANN7sLLWedgpgoKB84Ck0djJTV8XiwLLoov3tglmKE=; b=iWZ4/QVrw+NJ78+qA9wwR5ofnfk8aiaTkLu0gBg1/2HY7J9FdfUvMeaS52bkgkO2ETcjYO mYquPIq8ijzTUCUal/R+6xl3XbxwMYatbJWXYR2uNK1OxH+t9teVUQ44L5afJuVCFQ5Kck 3SqzS5AuuOsz0dgJkCEEkSckqWKb3Lo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Af1ZDlZE; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf15.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698580; a=rsa-sha256; cv=none; b=sKsOEyWvxi8rGaZL5dLdl/1V9AwNsLZebtflCyUngtR6qsVAZ4HuAAcQL1khYFTF9HB8/5 WeB7SJjRLAuDICGQ7FWQFzEHZ9dz0wNSys8+kNb/FwUWZF1HNlzP1bnkBb2hTMPLAHJkBe 6A+S715j3byMrHLNM7p46iYPv0wgp2I= Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7882b02ed2dso122304285a.3 for ; Tue, 05 Mar 2024 20:16:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698580; x=1710303380; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TANN7sLLWedgpgoKB84Ck0djJTV8XiwLLoov3tglmKE=; b=Af1ZDlZEQJu+do5e/LsQsAV4N66IqT38wUTtchclzn79MdWkrB1XsjuOQ4X4F01aUl 8bp84ifc+8c2smHyxo17s+X2uyPTSVHUp3/ahz9TTB2WOtaaMAn0UVrtCOAZCTkAQ3pf Bth1tlO8AxMUiGOtAkSA1FGKT56pxJS6KHkjaPUItF5qCdyfFyPNQkO02fLmPGr0+y7k OScr3dwmSbOYkgTDHgdpdHoVVFzT5g50XY/B2EcCGUhw0jk3GydMuuY9M3zVFazNpQqk u2EGhDy3H37bWZzXjhtqa+tkxfopvDT4+g5KwI9RhUtwv45x9Dg4Bkoojf9uA/mb6t0T yEUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698580; x=1710303380; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TANN7sLLWedgpgoKB84Ck0djJTV8XiwLLoov3tglmKE=; b=w/hsZnx2bplH/PLoDSVI4pNstpG+lKIybLANmYEkmWiAq1tB//q9/0oN8brcec5XVF XH4NpC40g/e17t6FQzUfhAwWY+3wl8h5gPbuQTtTvgGZbxF4B7Mc0C2RuCYDOg96MFq3 FK/ivmu45Mm8+0Eo1ERqT+kX3JmtUFLnnI53RB1rOl1HsEHZX2zMr8qpYxccx/oAr93d 83setqWZ4fvbgzHL/NXuR8ahrBsiROuIUfdgNY6sVt7f9Qp/25ZC74MZtX1O+8Cj2/te 4RXre4pHkMPsUG3fypVZYMam1Q4oy92YScbmvCu2enjUhPhzjSW4HCwtzDqkQIXLx4BC MC4w== X-Forwarded-Encrypted: i=1; AJvYcCXJOGWYLCbdif3dIXly/X6oSfs978OWpu/r9rDHUq34572yL/j3fZemmumv1tITBgb0cHGwhxvPrLiXvrwBY4A+YlQ= X-Gm-Message-State: AOJu0YyUQXcVon0TBIaZMDykN+arKtWEMDAfvPCIdKJtkt91SIT/iyDF xlA1slc3Gn1H4xSL1A9QEl2glyi4gzNBPVkIg7IuknEKRIsc+DdNRqfhG+9lBWw= X-Google-Smtp-Source: AGHT+IHwq/V0fU9mVZstG8iv3aIs4lRIY1sCdiOu2ll0CLJKWGqdfEQ3Aa7QbBpFir2jp2PM7CxgKg== X-Received: by 2002:a05:620a:22b4:b0:787:effe:472 with SMTP id p20-20020a05620a22b400b00787effe0472mr3409431qkh.64.1709698579913; Tue, 05 Mar 2024 20:16:19 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id az17-20020a05620a171100b007882b4cecf8sm2440014qkb.57.2024.03.05.20.16.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:19 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/10] mm: page_alloc: remove pcppage migratetype caching Date: Tue, 5 Mar 2024 23:08:32 -0500 Message-ID: <20240306041526.892167-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E68FCA0005 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: rsesznxzi3wgthquixhmpnkn5igcoz4a X-HE-Tag: 1709698580-606184 X-HE-Meta: U2FsdGVkX19fpessqCvsCvPq9UvziGFiVojUp7DyVQrMo8coxKUB4GNuLYu4hrsQVDx2yEdVtDq+IhIO1v6nPkZjerGlASuMLGvaVdT3byTy1m/UHpL9INM9L2RG8Zn/yhJf70oG5spwwgv61VUukxOXu49JgeC/RWpzvO3nra8I7JOxzrk3LmX1nvwDNhfxIJCG8KWhDwL7668pghcnHg7GSK/hA0QrtTEPvphErMCY7/MqD6RNUYr8jEfHTzYZOj92NCOOnnmFZZD7SyDMaKqqsjYNoU/80IvkOUghb5E+Lo+OeF5VV2ZQ6P76TVv2c46hevmu6P7hsn61hfsKZOrhfMvL2pQ2EW4r753J/sGaww6dlp4LeDf5J2mw+1h3Dva1n47SDNRz0f3BeAhHyxTOyvljaqJJz594XeWz9K4Dwk9rXE62MJeLtxtREFWmo8+l+i+ZlbSKM3NsoUC42b+39bAzcc53Q92n5RxyINdkVNJbWRbDpkxiXuzOexPtvPx5v0oK4OZ/K3wYqKhxgi+DoP1yerqRT0esf0TEfYQ1a9ZLec507grZ3XiIdlzSzL4RcZLlbAR1nlkI1UjZ/xqjLK48mZY7UrVq1odKeIhzNnZwrNoyJVIHhqu4KikutW71hGeTg56TQcpS7NQJARxNmuEeYpRx4huec/cb6XW0EyBjICWALC4aepKU72s9cZv9HSs8z3HdEUda+l34gxU13K4aQry4vvXjHZsosoCe5XNZ9zMhk9f4WBwrNOZvVeTNywiHENbWSN1NNEQeyqQgfv7c5bf3ddIJdP86c0Mx9ulGN8/wu5D2dz9glm7PnuXx1SmuqRQvQBmxcAsKmN1UZbByUh3SinOsXD3XY3vKUk/C0RFrnmVyUfznw8DM+3tTMtFVzivDqdU8MXsU3ea48VRnKMFNk22AN6NrivHSZP6ONjOAP2sxGkX6QJg8Q/Z5GrOHk5Uh5gn49tL 5GI9LxjJ Vk6IUMRpziWkRezbvXT/G3JyFXKjz7sEE/A9/Y17HL3vBwM10iCrHMUmfWTqxYkit23tX1bpNApJqTHT9gs+UfI3cnW8QQA9niI0PL8sSOHMaRDv9KDjE2GbBFgl3B/nlb34eY7SgcI25rRuUCJzzjQDevT2EofvyrOEDruxUiDKLNEsKgCG1wMdWJWLXx9LAfGA/OYd0GXvuv0homfNL8ihgl1B+TSqP3NF/Ojw64+JKOpL2kJdkkMC2Notjc7sArTJKGmx/+7AeI3TfHW9plb4oLOTL8v1DojWiHaZP73Vky/n4SdR70jK6IuPyRWNYOrOB7cZmmNuVVDHIkSmqMYn3Xmh9bZmC6kdFWffihqN0lAIcdTWcZT5CsixKkiKPc0enTcQFX1dkXne3iLbiFsnJCA4RLf3WpDCYPYgvmyq+ydQVkX27BMAGQd36mGP0sNzmIxy6QkGocbk1BifvwelhEA+j0N0ju7+0w7e83XEhiaO5oVK2xejWbE0eHPiTzJjf1ngY2CS97ZZmAwK8Qe59H60VpzlzE/vHdhCddUCtBSLnIN7GQ7EbOeBjocwqTNgNvOUg9OQCbWx/CTRnkonG3IoumYcB6S7r X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The idea behind the cache is to save get_pageblock_migratetype() lookups during bulk freeing. A microbenchmark suggests this isn't helping, though. The pcp migratetype can get stale, which means that bulk freeing has an extra branch to check if the pageblock was isolated while on the pcp. While the variance overlaps, the cache write and the branch seem to make this a net negative. The following test allocates and frees batches of 10,000 pages (~3x the pcp high marks to trigger flushing): Before: 8,668.48 msec task-clock # 99.735 CPUs utilized ( +- 2.90% ) 19 context-switches # 4.341 /sec ( +- 3.24% ) 0 cpu-migrations # 0.000 /sec 17,440 page-faults # 3.984 K/sec ( +- 2.90% ) 41,758,692,473 cycles # 9.541 GHz ( +- 2.90% ) 126,201,294,231 instructions # 5.98 insn per cycle ( +- 2.90% ) 25,348,098,335 branches # 5.791 G/sec ( +- 2.90% ) 33,436,921 branch-misses # 0.26% of all branches ( +- 2.90% ) 0.0869148 +- 0.0000302 seconds time elapsed ( +- 0.03% ) After: 8,444.81 msec task-clock # 99.726 CPUs utilized ( +- 2.90% ) 22 context-switches # 5.160 /sec ( +- 3.23% ) 0 cpu-migrations # 0.000 /sec 17,443 page-faults # 4.091 K/sec ( +- 2.90% ) 40,616,738,355 cycles # 9.527 GHz ( +- 2.90% ) 126,383,351,792 instructions # 6.16 insn per cycle ( +- 2.90% ) 25,224,985,153 branches # 5.917 G/sec ( +- 2.90% ) 32,236,793 branch-misses # 0.25% of all branches ( +- 2.90% ) 0.0846799 +- 0.0000412 seconds time elapsed ( +- 0.05% ) A side effect is that this also ensures that pages whose pageblock gets stolen while on the pcplist end up on the right freelist and we don't perform potentially type-incompatible buddy merges (or skip merges when we shouldn't), which is likely beneficial to long-term fragmentation management, although the effects would be harder to measure. Settle for simpler and faster code as justification here. v2: - remove erroneous leftover VM_BUG_ON in pcp bulk freeing (Mike) Acked-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 66 +++++++++++-------------------------------------- 1 file changed, 14 insertions(+), 52 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 16241906a368..9665a2db3b53 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -206,24 +206,6 @@ EXPORT_SYMBOL(node_states); gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -/* - * A cached value of the page's pageblock's migratetype, used when the page is - * put on a pcplist. Used to avoid the pageblock migratetype lookup when - * freeing from pcplists in most cases, at the cost of possibly becoming stale. - * Also the migratetype set in the page does not necessarily match the pcplist - * index, e.g. page might have MIGRATE_CMA set but be on a pcplist with any - * other index - this ensures that it will be put on the correct CMA freelist. - */ -static inline int get_pcppage_migratetype(struct page *page) -{ - return page->index; -} - -static inline void set_pcppage_migratetype(struct page *page, int migratetype) -{ - page->index = migratetype; -} - #ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE unsigned int pageblock_order __read_mostly; #endif @@ -1191,7 +1173,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, { unsigned long flags; unsigned int order; - bool isolated_pageblocks; struct page *page; /* @@ -1204,7 +1185,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, pindex = pindex - 1; spin_lock_irqsave(&zone->lock, flags); - isolated_pageblocks = has_isolate_pageblock(zone); while (count > 0) { struct list_head *list; @@ -1220,23 +1200,19 @@ static void free_pcppages_bulk(struct zone *zone, int count, order = pindex_to_order(pindex); nr_pages = 1 << order; do { + unsigned long pfn; int mt; page = list_last_entry(list, struct page, pcp_list); - mt = get_pcppage_migratetype(page); + pfn = page_to_pfn(page); + mt = get_pfnblock_migratetype(page, pfn); /* must delete to avoid corrupting pcp list */ list_del(&page->pcp_list); count -= nr_pages; pcp->count -= nr_pages; - /* MIGRATE_ISOLATE page should not go to pcplists */ - VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); - /* Pageblock could have been isolated meanwhile */ - if (unlikely(isolated_pageblocks)) - mt = get_pageblock_migratetype(page); - - __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE); + __free_one_page(page, pfn, zone, order, mt, FPI_NONE); trace_mm_page_pcpu_drain(page, order, mt); } while (count > 0 && !list_empty(list)); } @@ -1575,7 +1551,6 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, continue; del_page_from_free_list(page, zone, current_order); expand(zone, page, order, current_order, migratetype); - set_pcppage_migratetype(page, migratetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, pcp_allowed_order(order) && migratetype < MIGRATE_PCPTYPES); @@ -2182,7 +2157,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - if (is_migrate_cma(get_pcppage_migratetype(page))) + if (is_migrate_cma(get_pageblock_migratetype(page))) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(1 << order)); } @@ -2375,19 +2350,6 @@ void drain_all_pages(struct zone *zone) __drain_all_pages(zone, false); } -static bool free_unref_page_prepare(struct page *page, unsigned long pfn, - unsigned int order) -{ - int migratetype; - - if (!free_pages_prepare(page, order)) - return false; - - migratetype = get_pfnblock_migratetype(page, pfn); - set_pcppage_migratetype(page, migratetype); - return true; -} - static int nr_pcp_free(struct per_cpu_pages *pcp, int batch, int high, bool free_high) { int min_nr_free, max_nr_free; @@ -2520,7 +2482,7 @@ void free_unref_page(struct page *page, unsigned int order) unsigned long pfn = page_to_pfn(page); int migratetype, pcpmigratetype; - if (!free_unref_page_prepare(page, pfn, order)) + if (!free_pages_prepare(page, order)) return; /* @@ -2530,7 +2492,7 @@ void free_unref_page(struct page *page, unsigned int order) * get those areas back if necessary. Otherwise, we may have to free * excessively into the page allocator */ - migratetype = pcpmigratetype = get_pcppage_migratetype(page); + migratetype = pcpmigratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); @@ -2569,14 +2531,14 @@ void free_unref_folios(struct folio_batch *folios) if (order > 0 && folio_test_large_rmappable(folio)) folio_undo_large_rmappable(folio); - if (!free_unref_page_prepare(&folio->page, pfn, order)) + if (!free_pages_prepare(&folio->page, order)) continue; /* * Free isolated folios and orders not handled on the PCP * directly to the allocator, see comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(&folio->page); + migratetype = get_pfnblock_migratetype(&folio->page, pfn); if (!pcp_allowed_order(order) || is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, @@ -2593,10 +2555,11 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; folio->private = NULL; - migratetype = get_pcppage_migratetype(&folio->page); + migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ if (zone != locked_zone) { @@ -2613,9 +2576,8 @@ void free_unref_folios(struct folio_batch *folios) pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, &folio->page, - folio_pfn(folio), order, - migratetype, FPI_NONE); + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2784,7 +2746,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, } } __mod_zone_freepage_state(zone, -(1 << order), - get_pcppage_migratetype(page)); + get_pageblock_migratetype(page)); spin_unlock_irqrestore(&zone->lock, flags); } while (check_new_pages(page, order)); From patchwork Wed Mar 6 04:08:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D9A9C54E55 for ; Wed, 6 Mar 2024 04:16:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74B8C6B0080; Tue, 5 Mar 2024 23:16:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB266B0081; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C2DB6B0082; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4FC8A6B0080 for ; Tue, 5 Mar 2024 23:16:24 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EE337160368 for ; Wed, 6 Mar 2024 04:16:23 +0000 (UTC) X-FDA: 81865302246.07.076D8B4 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf05.hostedemail.com (Postfix) with ESMTP id 52668100015 for ; Wed, 6 Mar 2024 04:16:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=VxBW8bOt; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.169 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698582; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xpqItcyH2EDAXtPSFuVCpgEvxVG9s6bJRth9b0kyPMA=; b=H/qXeLBFUsamkEbXIMOtMhrSEtghCZOiiftwI1E4agZf8RYA0EP/MZd7TTTMLWhyqWKCuy rglpQmGvqAT7H8+L+7xSwQUBakd3bodYngkmhT9TQnEW/zN2DTluSI5UFXY+J+pvJidE2+ ddZcid7qLsWzOZDvBNxrX/Thkm0Ez3c= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=VxBW8bOt; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.169 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698582; a=rsa-sha256; cv=none; b=jBYluGks+GBMSKlLvI26j3UbEmOSeCTmAJJ/XJ14fhSgvniPzNJ5b9o3Q+TmQb/FziVDS7 r3eFLmo1xPBkNN2OwH9krRU6pMATHrohJ2k9BNqX6rcQEyhOAf3gSabBpallgHCoWJFuc6 be6foVEgF0tYgugOKNnXW2QxXEaOCuY= Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-42f0d09f567so1372491cf.1 for ; Tue, 05 Mar 2024 20:16:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698581; x=1710303381; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xpqItcyH2EDAXtPSFuVCpgEvxVG9s6bJRth9b0kyPMA=; b=VxBW8bOtgcF6pXXw21YnmKsp/O1njeQfURVaXXNttLt843f0sKF0zoHLrsptz8FmhM /5mGYXV2wi1U1e9PPYincARtZLs1yb6PWEYoEs71s6c6vWnzyYjpWYVm2AgRmqI5h+vb Gn44GoRsB91VewMqJ95ake1RQwcFnV6gUqL0dk6w0Sr2eHqUTv/wC5oANZPT8NdKqsOY sIjf84z4cwCtKrqXvg2DD3FVgMW9oLxUsXStmiqIN9QzmLzG36hKtju6IhUo1SjqtSwm LDqlCdmBnG6L0tWOza4lnXMgg3XnJXZV9qcqenTnqKTRMZ3KYHnrBWnYztt469BQslfA tDIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698581; x=1710303381; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xpqItcyH2EDAXtPSFuVCpgEvxVG9s6bJRth9b0kyPMA=; b=Q/TU2Juj4IaRFLXoZFcVSi5RUPIHNcQCBTZO+bOITPZV6Ojkic8LwnncNcIFExnc9E jh2ON0eYFxnpY5r/Skl7VB3S9bjikgwqy9QLA3FZE+Ww/zS2TlHe78uHcvGnnnI7oiVL N1prGECWzwQ3McrLYfj8Xc1xaHrd5Gaj+SzYSie7kyeqwuYawi/CmQsAWf9+g6oY0bC3 hDlRaCWoJJ24joa7fxC0wEj7ZHe6ZMVzxx3axf7Tj0U8ibqLdg0ayJFPWH0n/HOLK1t8 wFasdlsNAwwWGgyYrdEXIrr3rQpFDjJ0toLqA+HSQ4SvZyI1OW3f7oERgOqji2t8ZjsJ 4kgw== X-Forwarded-Encrypted: i=1; AJvYcCX5mF+YgVMnzP/54OJV2bYX6NjsA/FnsYcedRsAWoFUouXcZyItKpKs4QUq8EQqIDbEC/DJrSovoHnNBhKLtjO3g0Q= X-Gm-Message-State: AOJu0Yw04ehrhwwanc/D7miNt2dSjIL2C5oD4WFSMWRGChenjfokHB2o mu2GdbTERXMG7t+sZvp2HraeaUJqPn6aXBtFJQQ8/P6nST3n6g2I3eptOoCQBcY= X-Google-Smtp-Source: AGHT+IFlnJve3Rb16fSRkg2cnwF59nsIv8qKr5z8db/VPIGVmj2pSzSHnn6osAw7nOgwMuODvBzfPg== X-Received: by 2002:a05:622a:110b:b0:42e:f56b:91d1 with SMTP id e11-20020a05622a110b00b0042ef56b91d1mr4214660qty.32.1709698581469; Tue, 05 Mar 2024 20:16:21 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id k6-20020ac80746000000b0042ee243ca95sm3509250qth.50.2024.03.05.20.16.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:20 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/10] mm: page_alloc: optimize free_unref_page_list() Date: Tue, 5 Mar 2024 23:08:33 -0500 Message-ID: <20240306041526.892167-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 52668100015 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: c1sist7skg5ubrmcy7gu9ha7oz9txi6q X-HE-Tag: 1709698582-590586 X-HE-Meta: U2FsdGVkX1/y83N09Y4ot6JRLM55s1FmLal00FrRrXw7Vr1ZiYB4fXTGFCC10tgbakxt7V1VLReh8iNAIF5tH6/aB3uA7IzXkFZDvSWoCF2+03Hm29PTmDqa0A8zfi5BZUcRa0PiAKAqwLLlA8viKE/TbfiJW6j9z2hW1EK95crID/dzCFNTgeTB+E1MHiEdJB0BqY6ZvvRemeC3e7GHhfZmARnwgFTOrzSG6x4pbb6aI+5vjd0s7ftJ4wtIe4H3P3BX8HWUhrdYZFGjmfyZ5oWaKPUykSSk0rDnWrSjbJjXRf+bTz/CQGfoISBuvoR0Gn6WEHWn+K29NNrV8YO/WgyDsqZEh1+vrE02/hfxO6WVYYybgJB5ARCt5XaQ9BL9Qbb/NpbR4oMjD4nz+adAnSMr6gPXdsn0LmNrJehe/5ldJdf4aYY2F9NXiWm00fOtz1kgIB3E3dMSkWKZq3ct2YqIdzcjDHmTuHu/OKtjYn/cA9vs66tH+oA5VlIQ2DT7amFzznberpVlYsA0cGFRewEBGx52UbrDy2oMZ6K9i9XKa5LpxNPU7YkwP4FBr8qP/VIdy2WIlha9p8SRq2Gf36iPfHFf63hCFXdn3GuYqFv83LQdxt6SkLA7mtFnnvR0tEhcT0Rg5wEBr2nlqSuzJ2YwxvzROL3CdfaqydqzrMy/M9FxtU4vkviB26gxb0S5KpcwjkS124mMIx7X64ZbWieC91gu8K/5Bn5t6FuC3HVJPfOvcSRYrRxBPAeOI7V+nyehaef6IuYd6qpAzD6EBneOoW79yGu8ksvxkLZNcON73T+fbD7G5iMVmQ6krjZxsDVoahJLTqG82WsDsBafTlTrir/k0vBzFd3lV5t6Wgtv4wLtP4310cCg7PJGmNy/z6QDOpZW5P1dFYweGszVw9Kb+He0EfzUKyVbU4RFSpZRdy7g7gO+ynT3b3doyNf+mEre/d6U5jE3IxXoZ04 z5ZNO8Em Tj8THEAl0WDNEeXizryNtLD9d+WOw+UMDBalJ4WAbzTeXEHT8KKYrF/8+zkuQXVFzYz7AHUKcM7rBiH+NEdqrra/QaBn/G2JJO13119OWwLir3PfakJc0bhRZTpBjyoKND335M6nKq38qvVYdHy2seAmbrUmuJ+Qk5nARfEXE2k+CHaSlbOOUIhjdPyBFlAwc+m7CdowJw2REA+/eSzQJjBgSOld437aD6WOe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move direct freeing of isolated pages to the lock-breaking block in the second loop. This saves an unnecessary migratetype reassessment. Minor comment and local variable scoping cleanups. Suggested-by: Vlastimil Babka Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9665a2db3b53..88f4e08726ee 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2521,30 +2521,17 @@ void free_unref_folios(struct folio_batch *folios) unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int i, j, migratetype; + int i, j; /* Prepare folios for freeing */ for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; - unsigned long pfn = folio_pfn(folio); unsigned int order = folio_order(folio); if (order > 0 && folio_test_large_rmappable(folio)) folio_undo_large_rmappable(folio); if (!free_pages_prepare(&folio->page, order)) continue; - - /* - * Free isolated folios and orders not handled on the PCP - * directly to the allocator, see comment in free_unref_page. - */ - migratetype = get_pfnblock_migratetype(&folio->page, pfn); - if (!pcp_allowed_order(order) || - is_migrate_isolate(migratetype)) { - free_one_page(folio_zone(folio), &folio->page, pfn, - order, migratetype, FPI_NONE); - continue; - } folio->private = (void *)(unsigned long)order; if (j != i) folios->folios[j] = folio; @@ -2557,15 +2544,29 @@ void free_unref_folios(struct folio_batch *folios) struct zone *zone = folio_zone(folio); unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; + int migratetype; folio->private = NULL; migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ - if (zone != locked_zone) { + if (zone != locked_zone || + is_migrate_isolate(migratetype)) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); + locked_zone = NULL; + pcp = NULL; + } + + /* + * Free isolated pages directly to the + * allocator, see comment in free_unref_page. + */ + if (is_migrate_isolate(migratetype)) { + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); + continue; } /* @@ -2578,7 +2579,6 @@ void free_unref_folios(struct folio_batch *folios) pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, pfn, order, migratetype, FPI_NONE); - locked_zone = NULL; continue; } locked_zone = zone; From patchwork Wed Mar 6 04:08:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B325DC5475B for ; Wed, 6 Mar 2024 04:16:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C071E6B0082; Tue, 5 Mar 2024 23:16:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BBA646B0083; Tue, 5 Mar 2024 23:16:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A30856B0085; Tue, 5 Mar 2024 23:16:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8F3DA6B0082 for ; Tue, 5 Mar 2024 23:16:26 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5CA8EA0757 for ; Wed, 6 Mar 2024 04:16:26 +0000 (UTC) X-FDA: 81865302372.22.641325A Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf30.hostedemail.com (Postfix) with ESMTP id 916DE8000B for ; Wed, 6 Mar 2024 04:16:24 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Bzt0QzLr; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/m+XHRukptduyyFcyZXFpaIgdTql7GFreCc0MXKSlXc=; b=eY9ybyGcVSQIgmhEByoTPK3rhXH1KLE5RC9PJN1KsPnoVUVxKw//vlXtczITc+uQpPXeJo L0nhnbZ0NgPkZu+soLit7u2ajZqErx53ge9jlIUV8zMDxMl/ABaOo3Fln+Dc/FsRg0UTK7 57VbZIC6Gjtsf90pNaMd41lEBZfxwOE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698584; a=rsa-sha256; cv=none; b=PWj/OylpmpQ8Xd/U9OX10Ow0J3lpKVNyL+wZmpUjazPUh57ri4MWL5Gd2X+XrEikZhPzLr 5fI8OHBdYV3Tl+pl04Ch+f22wQ9SuHQ+05gk7jp+pi0jp8+inFxHEpfnj6mKzmknEwGmJn w4mUZNTdo9AZGyPLWSHxipeAUYt8WpI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Bzt0QzLr; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-788237c4536so24002685a.2 for ; Tue, 05 Mar 2024 20:16:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698584; x=1710303384; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/m+XHRukptduyyFcyZXFpaIgdTql7GFreCc0MXKSlXc=; b=Bzt0QzLr88FMXNbrfHYFAvMyaCRklToxAqQGm507wyBS7mcWqduB8DRusLOGqdy4fC 6TcieHqn/LFuB5JCfKZeydJiib0DoVBbBTD+OgqehvYesBneIWv8VCUS3ZnG7ynkgw9S knFZBwf72xtYXvBE8z32P2xe4wrZ8X8ou/G7tC5yKOth9w59Wv+xqXHGXY66hcVv2749 74szddC9saeprc6/dqy1dt5i7C6Rh20gyoPtSH2Mk9Yah3zAvqVfjcKkKdnp0wc8hAwB RVyxN+11nnWFAe2ariR5hi2rq1bIjcXLUVQq+DQlWzRBJbLCJBiBqzdIxS4emz7Hwflo KrzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698584; x=1710303384; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/m+XHRukptduyyFcyZXFpaIgdTql7GFreCc0MXKSlXc=; b=qJMJfj8t6ydmVCd+PJAvNCyrCiACwm7nmsr8PY6VgS864YBLhEhL22B1Cabgmr6pbN wmI4TSyXzqACQThkCimaUiwO9d/SYiFl+YsdwAm1lEkZUI6xJG1aHgyY8kuFzGcarq4j H0yVWCWSfFc+mg1uhZ9nR0dAx/QoU6X38RR9DwkwBwqNrM7svG0L2dMP20NBRvouuIOH mXWxmpjllpnWanPrzNiE5mdBNZzlhSkUH8OZ9aUgFf+BIXit3UhPqcSMC221mH9z5nLO J9SF1atMRc8WKVqj6jbGhDsR+zhl52kW5qqKjwY+QIuiHRtX6GsTmFiBnU/6tkYKbJjF S1pg== X-Forwarded-Encrypted: i=1; AJvYcCVcNhk1aSlUwJ+j1gHkKd+ku7188IB/timq2mnW59eJJyR1PsMwArRT19nVXGz77ISEq6tcTH4PLUBabp+Ei4vRlNE= X-Gm-Message-State: AOJu0YyOXUe4uAh0qkjhbbNcoIBKbXzuExkCHJZ4+3z0elRqmyq2dLP9 5nGIfISvvvO6Z6VZK0Upt6ke4IJi4nBQHdhbNGqTZ07BVuNxrBIXZ/RVP5/5uTs= X-Google-Smtp-Source: AGHT+IEXUAmf9zcOiQ21viaXa9n/2vz9wsETICwKDGogRNNcrHNmI77n49+NVvK4GXUrVLm+lFphsA== X-Received: by 2002:a05:620a:1272:b0:787:ed4f:a091 with SMTP id b18-20020a05620a127200b00787ed4fa091mr3769084qkl.72.1709698583788; Tue, 05 Mar 2024 20:16:23 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id e14-20020a05620a208e00b00788323a0c14sm1716137qka.39.2024.03.05.20.16.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:22 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/10] mm: page_alloc: fix up block types when merging compatible blocks Date: Tue, 5 Mar 2024 23:08:34 -0500 Message-ID: <20240306041526.892167-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 916DE8000B X-Rspam-User: X-Stat-Signature: mrgk44tyk5mm4ezf78nrcm1u1bzroeqm X-Rspamd-Server: rspam03 X-HE-Tag: 1709698584-589029 X-HE-Meta: U2FsdGVkX18Qc6Y1ehpQZNVOn4vnIaGWA0mtV362sABx+zydaBJo5d8MODmJ9aXX+4aThXF+594EZT6v1up1DgRwNj+W9k7qKWYYWOcBWKKQfdpIJKiyLLAk03nVBrycAgpazelnjOApuNv/7JD6OLHP8caQc1ugq7BgV5eLT6mHxq8DK8Y7xy2+NZPZrJ4HCO9cl9Tv9FUZo1q5RV2N6ShegRLeEtrOWOqArLbrsoXpjx5mlILbcz8nvk4O86suJevTnSnkuNOntrOowLUK3oxtiv2DpOaQ+NW19RnTvhAZ1Ol0kdNJz5sPLClVOwcIzLMc/YfSWEc7nAkpsUBWHruz5vIWsqtGrdx+ew5LhWng8ht9ZmxdDUEaWtK2wnJ8QH23Sh9VDKmKWhjerdh3GTCILevma2MtHluKkeNUiZ35nRTJXDuFf9/1zFD/Ur7RFbbtYvv7rfGJ22+HxG1xNnHZk5+ff16vwy13E6Wf9yFdy1Op/eQIQ9xrKO2iRxvDGFH5bQSZXRvj87dbEzsJWzXdOWYcGbiJACvW8Q0+7SiEv5yre7GnUuohUaFRoTFAcG32OqbhErWGe189+LJhuhiI/hYUITbW2wAIdz69Eymr9GFttmgFmzFfG7+c+8k8IXdi04ydXuNXqd8kwuyW69co1oPq7YZSa+n3nyYSIZHZwfgzgBDBQAJxL6Cs5hGY6o/43oroGj86zqvBZjtj3fUQXVEnivxGWMvIPFR9sCK2wiQZ74ez2XceI9ZU9tyllKn5RfOZxFosFbS7H/92m17NhsHcJt/BsM3ni55fVm1chCaoWvHUhApBoicNV41xtzJ1fRiuj8EWBMBtyA73WAc0pagbAAXc0AqkTeX4XJAjxb0vgBfhYtmxXVBhIH6+m2ZFI6sY2Gnvy3CPUk0sGbejzPCeCH/MVuakqysw+sstwqcOQBrX6QiSXYifGLJCFKpmZURMWQORGYauCJn KTjdStIh vGxPutqDCu/WM7T1ANgulDmGQbRz9DVRcHvbc2PSp1AM1GSXAOHu3FFnoQXhtlKeBRqI+OTm6ZOggfycQvl2to6sVQ10mBv3tGUYpFIwJJ2kIf6lLWZhB4YzpMB4oiBuLMuXSAIvgjlwfJzWsJqwClV5a+CN95V7RRy8A36f7wTLu8lrP5yn7niiWIE6EWQbNX/JKEtcQU7C1Avyn2TS00E9WE2TCP/Okbs6Cj2U9USFn1AkPxkZTd9GrWiENceDHT4s8kVs+4LrAx80AHkCWW8zpYPPJvVKXcDdP+Dmc6xKlOZSct2indTFpvmo4D7z3XhnRqMY1g5rB6LqkEPE7HfPeC/fYjBp5qIqE795q6E2L3RBp9s1MNFVDgCRVhOtxct4ZbFIrYVSFyLOdu10zOaZXXMb3KAE68HbCDV+cuhyN5eZ3hYXzFz9t1SRtrOvtiqt6COmP9cMSznBo2qZRWKpcEBYPAMb+5ALzyit+orYw+69B6Pa6cU66aLUl/1C3qvW2HpSYtp47U0ywBxgFdUeIRCYmFIVXp0SCzC4JoB0Q0SSma/EEFfgxjWjFr8ZdtuNhTeBfYFaJr6zoS9s23IkYcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The buddy allocator coalesces compatible blocks during freeing, but it doesn't update the types of the subblocks to match. When an allocation later breaks the chunk down again, its pieces will be put on freelists of the wrong type. This encourages incompatible page mixing (ask for one type, get another), and thus long-term fragmentation. Update the subblocks when merging a larger chunk, such that a later expand() will maintain freelist type hygiene. v2: - remove spurious change_pageblock_range() move (Zi Yan) Reviewed-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 88f4e08726ee..3f9ea4c392d5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -785,10 +785,17 @@ static inline void __free_one_page(struct page *page, */ int buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); - if (migratetype != buddy_mt - && (!migratetype_is_mergeable(migratetype) || - !migratetype_is_mergeable(buddy_mt))) - goto done_merging; + if (migratetype != buddy_mt) { + if (!migratetype_is_mergeable(migratetype) || + !migratetype_is_mergeable(buddy_mt)) + goto done_merging; + /* + * Match buddy type. This ensures that + * an expand() down the line puts the + * sub-blocks on the right freelists. + */ + set_pageblock_migratetype(buddy, migratetype); + } } /* From patchwork Wed Mar 6 04:08:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCC4AC54E55 for ; Wed, 6 Mar 2024 04:16:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F16DA6B0085; Tue, 5 Mar 2024 23:16:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E9DF96B0088; Tue, 5 Mar 2024 23:16:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCAEF6B0089; Tue, 5 Mar 2024 23:16:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B2EE16B0085 for ; Tue, 5 Mar 2024 23:16:28 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id ABB60C04C0 for ; Wed, 6 Mar 2024 04:16:27 +0000 (UTC) X-FDA: 81865302414.16.DD9E42B Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf23.hostedemail.com (Postfix) with ESMTP id 1AFDD140004 for ; Wed, 6 Mar 2024 04:16:25 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=ZcxPekAq; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf23.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lMSYidS0AoPvbN6qA3im2YqpbK/tNkyrqR1A6eq5TWE=; b=MOihuVzE1PIhJ1Lk8O6SwIIL8uq97wCfELCePHogDht7E1yecUzUd0nkrLN6qkCIcC1o2U ZlzJuZY5jGBYwGhfz2ep/4+u7hmLl9O5Ca5nVGTwzuN163/WPFKBg6Lao8oTTwPru/mtxU ppuFFbRlL55IIEjuB+BkIxb5soGXJp0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=ZcxPekAq; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf23.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698586; a=rsa-sha256; cv=none; b=ypdbyoIYVtnvVq4m2uXTjYkVi+L8VJ0YIWWk+DmwahsSDEuplpL19IgOjjWoU/LcMrabxs iwHmP8DTdY0Kd+jOGAuTEHtSQsWlannBJI3uLjyZUo22vSJEK4IG74GAGa5TCH4sphLGYr glipdF/DplHrP6eEn6gYa+oAycfV96c= Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-781753f52afso447909585a.2 for ; Tue, 05 Mar 2024 20:16:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698585; x=1710303385; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lMSYidS0AoPvbN6qA3im2YqpbK/tNkyrqR1A6eq5TWE=; b=ZcxPekAqhcV1mbkjaeYK06WAy2HQSFqVYSzNFItRnSB70MIFRbFpYC4UrBl/oRT5gk 0ywGIauiY6Ys4mZYQYp7XLqFpERbY8NFSVdE4SLhjbvFQSHNb47Vm6KqvEIRSDC1ZyM/ 8iQT1xq3MJpz6TsHtbsE3XeTDCBY7kNe1wyv6ZmUqbNdiB78gs9SI3cxGvnfMQ3/qH+G lBqcAaFP/Bh56XfWyRbmhYu9qoB+M98hwUC5o3+kI2+Yy4GaKtn26dVp6V9IOAbs9C2t 4ymTHi+OLiHR24UazCHDzA9CvKWDyE3o5XE/zCwQzIVIAoLzak/mGVMpwSxORl/3Qzt4 DUag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698585; x=1710303385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lMSYidS0AoPvbN6qA3im2YqpbK/tNkyrqR1A6eq5TWE=; b=HD7pVUMsxjnN1maO29lPCXusPdwtrmdKSdInN7+6xF08UcvQ+hSusdR4u1py/coGsw CD9gQOPvtgUGVEA/eNOij1T1TbyNVQn1Hk93AaC5TTWXq9bbPoFbDKYZAWDHQRn5hiok o0fdpjNw6QmofshiQY0voQnEmLMDr9GyU0GjZHNhDcmQG+WUidAMUFrKUIX7OFBSlaOG DazRDVAmHGRKQQvWRaOMujlOQgrlzian/vcV1UuqKSTyYh2pgtmQml7I/w+RwgjOOyho U215DAoOBhBwvqopdhIOR0Nd7HGo4n4F8ccGhhhJill/hNUFBSc4HWNcbubJt1kYfWyE sQRA== X-Forwarded-Encrypted: i=1; AJvYcCUitTEd+Oy5kwjrrCTm8E62noSpFI5yZf6YN61cU5BZtuOnMHxmZ6vHWjv7zv+fG0g+rFvZDFys0A8bGJvFC0oTNbw= X-Gm-Message-State: AOJu0YxHnNIYmXpq73KCDk0irurtzmO9kgouvL4WQyJiC7Qdie3R/Pzw a3FpNpuJjzP/i/Ryx40A/MFMtM2GYG6lznf0+KTzvuP98oRbf1uVRThTgJd1Z3A= X-Google-Smtp-Source: AGHT+IG6XvS6MFHFJ6jCjiu2kUNS7WFyQEOYG64hXNNhRZh6uxR9igo5+MITru6+zIGSD4f9tY7ZWQ== X-Received: by 2002:a05:620a:389b:b0:788:1f51:5822 with SMTP id qp27-20020a05620a389b00b007881f515822mr3895540qkn.40.1709698585227; Tue, 05 Mar 2024 20:16:25 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id 19-20020a05620a04d300b0078838c7acbfsm984096qks.42.2024.03.05.20.16.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:24 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/10] mm: page_alloc: move free pages when converting block during isolation Date: Tue, 5 Mar 2024 23:08:35 -0500 Message-ID: <20240306041526.892167-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1AFDD140004 X-Stat-Signature: 95zgt161c3fjeww7u5b3f8rhoriwhr9d X-HE-Tag: 1709698585-339609 X-HE-Meta: U2FsdGVkX1+IW1D7f+KaohvsiXKn/RzHrRMc5uhfk2x6hC7NRFDrQxgZ9DmgndCWk49y0LkitMXPUtQm+786r0CCBTq6e4E7Nq+nwowkD/C+PTFD5MGJZvMv8MBa8/YQAP1/LBN5QWdMj5kvYwc5RyMUqgJXb4Vz5L+6LojPT7oy/Uf9W7ttmkjMKI2dhyWlGMjQ16tL3qaX0Amdba9mj+pjXEykl7hR+CBGCZ7gXovoSmczclfLz3Acwz9e3WYzoArkUsnNO7jwmsQyndKUMdV/mjuVnyCzE3TDcA7MD/O5NlpCXYaGb8IFw8cnXMTLZYXWcB4oRQjmKs7+eHJDuKeLKYpCJxYwoYEtDuqVA1Ba61WPKyjV8uFJ7dGPctOOMWyOGRlcpcn88K733x/8qFRn/VtdoN5R2Pycb9s1/Owg9NBa+NL0KaeOMxsL7KPdVMwsGIaj2LV2HATF4pRP2z+CT8OeKgETM0xCVKKx6rCySYMBWqorXUMEp82wM/xdaRoeczYJAmC5H8kS9y2UQd6+WtccRFyqmFvI2zvmRHPZ4P614J5o2GWcH4arOxxPB5hc4b6KypqGaKDj1CK3frnfYsXKGMzjYypccMp+xywjAdtz4XtoPIi+nQAVrne34dGdCZFCAjGe2CYlWlp6o9Q5dU88hXdX333HyqOYB9RUUc1cBRZwL2Qlv3C3V0qrFWS0k3+MwuiVsJnylz/P3TPXT4tyoBXFjOKMJYOmUQ4kuEinhY23tn5a97gP/L44O4TjjSBFo7CBwA8K4G86GjZOpOAb09CTYQ6EJ1wOhTIWEmXKvai09YbesFDLpAkrkgb17DtnZhw4W5WIJDbwoCXo2HEBqZaJ9yGu6CTNtELGgqXgZuZcrkJqxxwrDKrJVUaiym6ESzSIZYtbFxaCHx2dK5Fgntb2mMX7kDxFqP/bQ6P5iDMDjb4DRSpBrKd1FpJk/g3qq0mv29johR3 Xm36zXiI roZ+sTdnHI/oN9hH05iyF7WMrfRIxHgQbQQsJeahGIyPuzZqm+GwgOSMu2lOZ367dsz9Hpebi0ttsEH73mgccHFaOufpVQd3arkLp47pwIlYDuxwl1i4f2rUJrnAZDhqr2ZA2fDMoRLopn7rdTb3AnPxGn/LCmN00b+MTb+ABgtochs+tDX/r02iRSyTbheblzCqnmD6iCaGs3zL9VeyruC+OZEtaLU0K2ZP5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When claiming a block during compaction isolation, move any remaining free pages to the correct freelists as well, instead of stranding them on the wrong list. Otherwise, this encourages incompatible page mixing down the line, and thus long-term fragmentation. Reviewed-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3f9ea4c392d5..0f385531e130 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2666,9 +2666,12 @@ int __isolate_free_page(struct page *page, unsigned int order) * Only change normal pageblocks (i.e., they can merge * with others) */ - if (migratetype_is_mergeable(mt)) + if (migratetype_is_mergeable(mt)) { set_pageblock_migratetype(page, MIGRATE_MOVABLE); + move_freepages_block(zone, page, + MIGRATE_MOVABLE, NULL); + } } } From patchwork Wed Mar 6 04:08:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A384C54E58 for ; Wed, 6 Mar 2024 04:16:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E88586B0088; Tue, 5 Mar 2024 23:16:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E27A26B008C; Tue, 5 Mar 2024 23:16:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C05E66B008A; Tue, 5 Mar 2024 23:16:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AC9136B0088 for ; Tue, 5 Mar 2024 23:16:29 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 60E04160366 for ; Wed, 6 Mar 2024 04:16:29 +0000 (UTC) X-FDA: 81865302498.09.E240D68 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf22.hostedemail.com (Postfix) with ESMTP id B48E3C0016 for ; Wed, 6 Mar 2024 04:16:27 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=NfvqadLq; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf22.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PHzhRKVaAXnUD3rwBUEd7H3L3SS0VEeXzGBInhtmHvo=; b=114jRgNHK7OQD3YxhcmjQ55mV67FDj/DrW6i50HPoeuq+oWceGKHzqnvQ0CFwmJqkKI3A7 FXmCWx1u0WFlUFsaK0CyX1bPhiVA9hckxFGfznHGkZA10Q3DvsaziWjhWPw0YN5txg/s9j IKXQe6eioqUlp6Gi/bw+0GNDuIA98+Q= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=NfvqadLq; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf22.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698587; a=rsa-sha256; cv=none; b=rHxf/eneuP2DvWh43id93ZRYWqcy66OZyFTyR+4+uy5B24SrXLO/af0r3m5m9JyVnffJMP XAwI8qStlE1KoCsz5Tt5npmVehSmD/B8/m4wHIOi1uUtq+K+Pb4dfkLAouAdgqwML00c6f AKI27rq8n55Qsf3wHG079zRiVEz13go= Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-69096fe743dso298496d6.3 for ; Tue, 05 Mar 2024 20:16:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698586; x=1710303386; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PHzhRKVaAXnUD3rwBUEd7H3L3SS0VEeXzGBInhtmHvo=; b=NfvqadLq/boO+sdZXcgiTiisrO3QjedtQrvC12WauA1Mk35Ep6jq89AaiPFUt4JG6z MMZcqyuAXGtVteyVRuszRzKjzv52Il6kzvKWUtBTSNYGxouE13A4ADUjh6Cup8hxXPXi CL18vNUaPz+D0U4fPbOZn13VtS1o5sZbPreIIod6XkQZchyTy9poX7vCnOALVUB+1/v3 MOxfbkUi0y9cWhs05COZE3PJl8aBZdiMMBHOT0w4jlLF9UdcAnww1dewbczGzV4UJ+S2 BaCc9wUU8nkyoIpUdToAmh0pe19RzjcxpiqyPCBpRxzRlhv6RselOKC33qKR52Sf5BR+ wIZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698586; x=1710303386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PHzhRKVaAXnUD3rwBUEd7H3L3SS0VEeXzGBInhtmHvo=; b=djkT5eV7sxTRaVXj/MNog6Nm50qVOgv8rC7SPoMUzrJ3HPya/qz62M/svGb/g+NQPu aO67WTK15yiRzkVQ8y6mkNQ/yrfBkei3YPHgNjGMfDYxlmSZb8ZOVpkHkwAvzy144nHx L3+hpNSx/pPwFdhlG9CFyFcPzod+e+RtZO0UqD6ZfB7Qro4L44hJ/vuVaC09I9nf7iPc OqfEJtmSitjsj3R33o7HPMRbqpCQ51/mttuDe3ENOvEJfIscYqgw4ztAlKAR7oEMTyHd ca+l3fFh4NaUddINis56HDUMXmDH4fw5+ySNAHhIpnARYw2FHVjzCXznx8X4ir7ppMl3 A/eg== X-Forwarded-Encrypted: i=1; AJvYcCXSxaa9hVFMVzQ14RDalkPGpn9GTbYtu0g8npEjXmv7P8h/lz8IRVLLkUGWaV4VIlITz1XxGQ6pbBUKrXplWHF1JHo= X-Gm-Message-State: AOJu0YzDsoqok9G/DN8a5gyX/QbQNTpVfk3CVoFQQqDmvL2lhFfbGIUP hoqSzeQqCa0iEYrdjNXn42fxvAWK6norJZsHQ2efC4T7olH0kGl68xxshts5uTc= X-Google-Smtp-Source: AGHT+IH8smcjvgVjtxikEQ23VUlaKW5EeZbBSaDCdCJJHUd5ddTW2ftWOlTEg3i010CDIhz/uidLPA== X-Received: by 2002:a0c:e9c4:0:b0:690:8aee:60ee with SMTP id q4-20020a0ce9c4000000b006908aee60eemr790372qvo.46.1709698586714; Tue, 05 Mar 2024 20:16:26 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id ph7-20020a0562144a4700b0068f985ceb11sm6973369qvb.70.2024.03.05.20.16.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:26 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/10] mm: page_alloc: fix move_freepages_block() range error Date: Tue, 5 Mar 2024 23:08:36 -0500 Message-ID: <20240306041526.892167-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B48E3C0016 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1f6wtnce95y3tgqxyekpmoorjn3i38ni X-HE-Tag: 1709698587-696884 X-HE-Meta: U2FsdGVkX18tOG01HQNP7rviJbdwgV/kPpbNpAmflHlzfGI3neGu/3uxEi27VJVrHkZqakA1cBIUVvsHFj2HtviDilG1mk5XVNrFEhlyiKIuQM2qD8vVaF+vnVse9tuxJVDr34BBbsmsr0NAiiw5iydQQJ/PCzTI9j1ckSafyDcdVSnn68nwKvg7PHKhBpUy1C0U5KubIAuxcBAvfSWCk+HpZftN8xPZGaBrv052ZTWbFVNkWi8AotmWxiaKr2ooNy4LskuIqgaqfYKf5Q0mWH3oaXBAsFIvrw3+km9lQuueUfLX1f/Go5CW+j53aaACnQEXHB/dH/K1UoDS0Qedec8AlN/Aj0jAWBipq1Vg/1G1t90h7RnHqhrzud1GX+R9mfHo38CVhz/m3sHsNsKRwHUCnvnZ/RNhuAQfRPL/e/nfInM1vbWSsBpUxROqYK+aE9JXNCAk4FST/+SszNFXH7yc1HVQ3bQPLOvm1hTAhK2YBax9tpXMXO1Srmjjapbv9soXzTQrnYxLxe/z0a1nVugxeyA/znaBSzwjrQQSfAf63j+O0O2KVzccQWR8M8e2AF7/jvvhwYRquJzcKg6q978h9ZPdPozMs07LhQJ8UhnoKd9Dyw4280noJLmAdnfTZWAzNRzR5dbH56p8XLoyAZ6laNQoqC2hvaP/WqzUmkIl4H5WRW6xA+4oYcfykwu+9sB9c9sxKPvuOri12J/jf0feQuS9WcYvUJujEsR6+ZiNsugouFCWL/U/nOA1LXHia2J8+HMdl6YjxbSCmGoRz5RCIgDTkz3AQHAwbl6iZT/W8DlaiHajjj9DvpnifHFFgIXWzzHqjM0S5/WA8HcaJj4jythQCb5C/Tg04TxLolWBiQJQT8jYDo7V0jj1xmImsULf3OaTyDMQLKDvhJr5Cid35T1SrZ/vCrHanqhBJKAFstnsOyPEQ+vvWQ6wo06YmyU4+q7ocaCX0ig9XKN ax1+78OT t4fXjcxSuBmkbm2OFStNUIylJzib0Nl2tJQM5kxw+ntH1Ii6Aj+AnwoWDK/Y+3U8cQOkb3OLNOK8qCW2mPs7XCvwy6purAt4rqrhAdIgh9p31vzjN9VXVv0HicueVx4TUyW2mFrjSH1dW3aauE4qgW9gVp5QsLpoyeVq70xrW6I68CTS8o2CJByO8hqaoqdVyo8rF6focy/9mZoyjM2yLE3BnyfoSXMaUeyOw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a block is partially outside the zone of the cursor page, the function cuts the range to the pivot page instead of the zone start. This can leave large parts of the block behind, which encourages incompatible page mixing down the line (ask for one type, get another), and thus long-term fragmentation. This triggers reliably on the first block in the DMA zone, whose start_pfn is 1. The block is stolen, but everything before the pivot page (which was often hundreds of pages) is left on the old list. Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0f385531e130..787ae3f0ac06 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1645,9 +1645,15 @@ int move_freepages_block(struct zone *zone, struct page *page, start_pfn = pageblock_start_pfn(pfn); end_pfn = pageblock_end_pfn(pfn) - 1; - /* Do not cross zone boundaries */ + /* + * The caller only has the lock for @zone, don't touch ranges + * that straddle into other zones. While we could move part of + * the range that's inside the zone, this call is usually + * accompanied by other operations such as migratetype updates + * which also should be locked. + */ if (!zone_spans_pfn(zone, start_pfn)) - start_pfn = pfn; + return 0; if (!zone_spans_pfn(zone, end_pfn)) return 0; From patchwork Wed Mar 6 04:08:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AAA6C54E55 for ; Wed, 6 Mar 2024 04:16:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DDBD6B008A; Tue, 5 Mar 2024 23:16:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48DCD6B008C; Tue, 5 Mar 2024 23:16:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3080C6B0092; Tue, 5 Mar 2024 23:16:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1272D6B008A for ; Tue, 5 Mar 2024 23:16:31 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E7CCD40F60 for ; Wed, 6 Mar 2024 04:16:30 +0000 (UTC) X-FDA: 81865302540.07.47908F9 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by imf08.hostedemail.com (Postfix) with ESMTP id 22FE4160003 for ; Wed, 6 Mar 2024 04:16:28 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b="FV2r/P30"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.182 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mOxJcdsW+fkh1fTmaf28JkWR8jDRw9yVS7tVyo6ygLc=; b=1JSn7uZG13W8sOrTnWjxyrOGbDln1fbhrp3iEiBK9Fw7VYp2qqqPiP4HrP9SYi4U5IpeBf h6g9T8+7ryW5aKDPc5VP76HTSBB1HLFB/u8BtjOH+PBmaefMCPwtGu4Vv3HaYtxG5c2bvd E/R0xFRzKSF944XKxtQy1uR3cH0gUpQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b="FV2r/P30"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.182 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698589; a=rsa-sha256; cv=none; b=eY873ftevdXs2Wg2AoPXRwuRNhqKXHcyC2RfUJuO/tBCkI6O4gg6G7BSN985ZlBpiJd+rr z38g+CdZSLaAZsJT0CvPHmSZnDe2yxnnsDhb4Dpd2NjLbG3yN12PS5QuYYsgGCut8KGaml s73qTGNocGxRgInAcJM7D4BKP8ps3wQ= Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-42a029c8e62so50049551cf.1 for ; Tue, 05 Mar 2024 20:16:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698588; x=1710303388; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mOxJcdsW+fkh1fTmaf28JkWR8jDRw9yVS7tVyo6ygLc=; b=FV2r/P30TCxhwxXdqEj6LDSMWUU3oSzcHWcWibxJ1s8FPbjmtqu7YG0YvUF2BPhBjn yLoEzUiUaj5vpsy22LTmUTq50Y0vvalcnzwS+bzUb/AZ0xlBaej4uJEhXxKs4vVXq54f yjpZHqnfScf2rCDttYi5kLj2pglY/J7JPJWKZY9yBKLY3kNMjThaxGxhZjqNxFjHoqAk HkBjJhE3T4DU5B+cJ3SBtSCt3cZGcEwb2v8anMR2He5xePI6wJ7fNxaNzCKC/NCVMsJH bN67F64B7tPqUv6FJIoqmZc/cuhW5r/nrodDrNpXv2KWizMImtkLAH/BabUVHWLJdj+P 49Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698588; x=1710303388; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mOxJcdsW+fkh1fTmaf28JkWR8jDRw9yVS7tVyo6ygLc=; b=MQvQ8V5bzzbwXwXEKUrtjiHWOUt201jhJgwfayPLDy1y3trEWVRd8Jr7kcpe0vP7c2 O9/2HhuVP+l0a5OdScfp24NiKfJQ6N/x1QFwYobI1pkiIghUUlC9Nl7Tk9IrV/9opIHW uvItXGIpo5ALH3nR3LzBX6d0DZb8KY5YnJGi1iHOi1UaBaei1LK5hTk9BGRN/wOwcYcU nh4VtO1ICPniVKEt4IUs56KOpZf0l9ezFZuyfYzAtpBfrYJ6pLOzb+BtigWMEDxTk4rI SBOmq22y9gFVmKEavk3taB0BV+V78MfzxKht7qjyqGN8XbVZlyi9qr7y9m6PnL46aRZp cdNw== X-Forwarded-Encrypted: i=1; AJvYcCVLi44LKgSaQhtA9RHs9/f9jsw+aY6RFi8rJzrO0lmZvnDo8iNmxFXvzL+b/LhMVtZKImtRq9ys4YMi2X6D8YnQUqI= X-Gm-Message-State: AOJu0Yzr8WPj7gSqgNSxEzng8Uv6OkNR/8fdHpGIxYPimPvp/2DEomaj MVdFpFoDdIg9w7gVlcZ8nHARckZ2wEbz0z2CUlKIx7w+OLZgnUNsIJPriN2lf9A= X-Google-Smtp-Source: AGHT+IFJvT3s0a3iT7I+hclpblwvoPOqaKROp/uBLu39M5KRQh7DpEKmnGy5yCCJ2YIyvBlrTk3AJw== X-Received: by 2002:a05:622a:653:b0:42f:10a9:e8df with SMTP id a19-20020a05622a065300b0042f10a9e8dfmr324741qtb.11.1709698588166; Tue, 05 Mar 2024 20:16:28 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id l2-20020ac80782000000b0042eef746ef7sm2795888qth.9.2024.03.05.20.16.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:27 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/10] mm: page_alloc: fix freelist movement during block conversion Date: Tue, 5 Mar 2024 23:08:37 -0500 Message-ID: <20240306041526.892167-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 22FE4160003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5hkhbmemijnhkugyqhinwntowhwurgzk X-HE-Tag: 1709698588-443996 X-HE-Meta: U2FsdGVkX1+HiwBbpl6UM7X4EWecHWa+orpV4qaCxZKR1u9XHktRV8260dpJ5+sMbTw5DhkLefVqn0dOMB1XGUUvNIU03AHHsmm06zGyNSAPPH/BQy6jNmJgwlwHoIjK6QNdQ7QcJj+pUSZpyAw19DFpNU7E2HhupxCekZqU0Ljdu2LTEKyqQAGL8nRjoOjL1kNyPdF3mefy6JYJfw4a+qBj0wjcM48FPCFPDy6Ejxux3/L0JHLEm88NeP2izbkh5DQp8Pnfeo+bbD0KhvjE5ZMhrsCjY5Oh0UuHsBlWB5RXNKiy0QSl4q/kjrW3LDtsz53aGN0XbNCrlhPF8uwcl/ZXSDLRUt80oRZysrd2DyPfsCruNIPNB6r7RGkyAKf/gBkO0SQJ/4slIs+mHuN7Sj/9Y2ux4KUVWgAQXsg8oOMhCKdLZTBhpJ51yqsPcqyswE6Pm89ZElFkk+zZCy26ZnGAoo/xRcCSS/Icao/RCI/dvOesHYhLxYovwpHAQPiAltJ++9nIQsto9AO3LqWj1pKdV+7XD8dJtAW0xz27m9mwUKvet4FkgfLAcE+GtmtRkFFvP0+7UQM+ap4rW9kFpt1cRKbuPw6F3yPrG+vkZNq/31l9lM696Ap5XgdQdDh+9XZQeWD9cYvcSHV010Tf1PYGM7S8z4BZYBGOB1qSGIxN0xa+gC3BMcTGTEB+AOCSDvQ9NgUcJZNZYpH/4axEKBEmaq3EVp/fQoSEOwlVCeuhULw+ghEpl2H+MFuZqLpesCUOeEukJEbGANKcPLHDJVC3Flofrq39NMCFKei/58mmylO1Nd+9iF8IK9zmVtnWIria0qCponL8aS20u/3DKy4yf+Nmokgjr5jpchE0unEuVZQOojLfo8Ga5IL+JRJ82AWdI4iMjA7+5HTFpCKbA5RVrvG8dbTfbUaMMtzbhDBZRyrFm7JJDRyMdyEB7B9yTWmnEZiIpYTKOCSyhI2 182cvdWp Ffht8p8X0jtN+B0Ayzi6mSNT1XlVfJkTBl/V2d+7MYhk4qNgCwNcLJyriypGPn7pPXqMhnT8RI9fT7xYTfbLnENry5+Gq/R9tDAvcXaCbsDNg/kVZ90Fl7uJ4V7eQ/gYhkYpE/FbgxNjD6bKhAbrw+hwWpQcS8A6xC4mqS6xqcD3xdOPkRf6ksDjv/OL/TETk5iiMFeCJwBlFpmJcPPJFhFyP+kKPustSQQc5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, page block type conversion during fallbacks, atomic reservations and isolation can strand various amounts of free pages on incorrect freelists. For example, fallback stealing moves free pages in the block to the new type's freelists, but then may not actually claim the block for that type if there aren't enough compatible pages already allocated. In all cases, free page moving might fail if the block straddles more than one zone, in which case no free pages are moved at all, but the block type is changed anyway. This is detrimental to type hygiene on the freelists. It encourages incompatible page mixing down the line (ask for one type, get another) and thus contributes to long-term fragmentation. Split the process into a proper transaction: check first if conversion will happen, then try to move the free pages, and only if that was successful convert the block to the new type. Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- include/linux/page-isolation.h | 3 +- mm/page_alloc.c | 175 ++++++++++++++++++++------------- mm/page_isolation.c | 22 +++-- 3 files changed, 121 insertions(+), 79 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 4ac34392823a..8550b3c91480 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -34,8 +34,7 @@ static inline bool is_migrate_isolate(int migratetype) #define REPORT_FAILURE 0x2 void set_pageblock_migratetype(struct page *page, int migratetype); -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype, int *num_movable); +int move_freepages_block(struct zone *zone, struct page *page, int migratetype); int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype, int flags, gfp_t gfp_flags); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 787ae3f0ac06..9cf7ed0c4cd6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1596,9 +1596,8 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, * Note that start_page and end_pages are not aligned on a pageblock * boundary. If alignment is required, use move_freepages_block() */ -static int move_freepages(struct zone *zone, - unsigned long start_pfn, unsigned long end_pfn, - int migratetype, int *num_movable) +static int move_freepages(struct zone *zone, unsigned long start_pfn, + unsigned long end_pfn, int migratetype) { struct page *page; unsigned long pfn; @@ -1608,14 +1607,6 @@ static int move_freepages(struct zone *zone, for (pfn = start_pfn; pfn <= end_pfn;) { page = pfn_to_page(pfn); if (!PageBuddy(page)) { - /* - * We assume that pages that could be isolated for - * migration are movable. But we don't actually try - * isolating, as that would be expensive. - */ - if (num_movable && - (PageLRU(page) || __PageMovable(page))) - (*num_movable)++; pfn++; continue; } @@ -1633,17 +1624,16 @@ static int move_freepages(struct zone *zone, return pages_moved; } -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype, int *num_movable) +static bool prep_move_freepages_block(struct zone *zone, struct page *page, + unsigned long *start_pfn, + unsigned long *end_pfn, + int *num_free, int *num_movable) { - unsigned long start_pfn, end_pfn, pfn; - - if (num_movable) - *num_movable = 0; + unsigned long pfn, start, end; pfn = page_to_pfn(page); - start_pfn = pageblock_start_pfn(pfn); - end_pfn = pageblock_end_pfn(pfn) - 1; + start = pageblock_start_pfn(pfn); + end = pageblock_end_pfn(pfn) - 1; /* * The caller only has the lock for @zone, don't touch ranges @@ -1652,13 +1642,50 @@ int move_freepages_block(struct zone *zone, struct page *page, * accompanied by other operations such as migratetype updates * which also should be locked. */ - if (!zone_spans_pfn(zone, start_pfn)) - return 0; - if (!zone_spans_pfn(zone, end_pfn)) - return 0; + if (!zone_spans_pfn(zone, start)) + return false; + if (!zone_spans_pfn(zone, end)) + return false; + + *start_pfn = start; + *end_pfn = end; + + if (num_free) { + *num_free = 0; + *num_movable = 0; + for (pfn = start; pfn <= end;) { + page = pfn_to_page(pfn); + if (PageBuddy(page)) { + int nr = 1 << buddy_order(page); + + *num_free += nr; + pfn += nr; + continue; + } + /* + * We assume that pages that could be isolated for + * migration are movable. But we don't actually try + * isolating, as that would be expensive. + */ + if (PageLRU(page) || __PageMovable(page)) + (*num_movable)++; + pfn++; + } + } + + return true; +} + +int move_freepages_block(struct zone *zone, struct page *page, + int migratetype) +{ + unsigned long start_pfn, end_pfn; + + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + NULL, NULL)) + return -1; - return move_freepages(zone, start_pfn, end_pfn, migratetype, - num_movable); + return move_freepages(zone, start_pfn, end_pfn, migratetype); } static void change_pageblock_range(struct page *pageblock_page, @@ -1743,33 +1770,37 @@ static inline bool boost_watermark(struct zone *zone) } /* - * This function implements actual steal behaviour. If order is large enough, - * we can steal whole pageblock. If not, we first move freepages in this - * pageblock to our migratetype and determine how many already-allocated pages - * are there in the pageblock with a compatible migratetype. If at least half - * of pages are free or compatible, we can change migratetype of the pageblock - * itself, so pages freed in the future will be put on the correct free list. + * This function implements actual steal behaviour. If order is large enough, we + * can claim the whole pageblock for the requested migratetype. If not, we check + * the pageblock for constituent pages; if at least half of the pages are free + * or compatible, we can still claim the whole block, so pages freed in the + * future will be put on the correct free list. Otherwise, we isolate exactly + * the order we need from the fallback block and leave its migratetype alone. */ -static void steal_suitable_fallback(struct zone *zone, struct page *page, - unsigned int alloc_flags, int start_type, bool whole_block) +static struct page * +steal_suitable_fallback(struct zone *zone, struct page *page, + int current_order, int order, int start_type, + unsigned int alloc_flags, bool whole_block) { - unsigned int current_order = buddy_order(page); int free_pages, movable_pages, alike_pages; - int old_block_type; + unsigned long start_pfn, end_pfn; + int block_type; - old_block_type = get_pageblock_migratetype(page); + block_type = get_pageblock_migratetype(page); /* * This can happen due to races and we want to prevent broken * highatomic accounting. */ - if (is_migrate_highatomic(old_block_type)) + if (is_migrate_highatomic(block_type)) goto single_page; /* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) { + del_page_from_free_list(page, zone, current_order); change_pageblock_range(page, current_order, start_type); - goto single_page; + expand(zone, page, order, current_order, start_type); + return page; } /* @@ -1784,10 +1815,9 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, if (!whole_block) goto single_page; - free_pages = move_freepages_block(zone, page, start_type, - &movable_pages); /* moving whole block can fail due to zone boundary conditions */ - if (!free_pages) + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + &free_pages, &movable_pages)) goto single_page; /* @@ -1805,7 +1835,7 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, * vice versa, be conservative since we can't distinguish the * exact migratetype of non-movable pages. */ - if (old_block_type == MIGRATE_MOVABLE) + if (block_type == MIGRATE_MOVABLE) alike_pages = pageblock_nr_pages - (free_pages + movable_pages); else @@ -1816,13 +1846,16 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, * compatible migratability as our allocation, claim the whole block. */ if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || - page_group_by_mobility_disabled) + page_group_by_mobility_disabled) { + move_freepages(zone, start_pfn, end_pfn, start_type); set_pageblock_migratetype(page, start_type); - - return; + return __rmqueue_smallest(zone, order, start_type); + } single_page: - move_to_free_list(page, zone, current_order, start_type); + del_page_from_free_list(page, zone, current_order); + expand(zone, page, order, current_order, block_type); + return page; } /* @@ -1890,9 +1923,10 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone) mt = get_pageblock_migratetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ if (migratetype_is_mergeable(mt)) { - zone->nr_reserved_highatomic += pageblock_nr_pages; - set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); - move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL); + if (move_freepages_block(zone, page, MIGRATE_HIGHATOMIC) != -1) { + set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); + zone->nr_reserved_highatomic += pageblock_nr_pages; + } } out_unlock: @@ -1917,7 +1951,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, struct zone *zone; struct page *page; int order; - bool ret; + int ret; for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx, ac->nodemask) { @@ -1966,10 +2000,14 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * of pageblocks that cannot be completely freed * may increase. */ + ret = move_freepages_block(zone, page, ac->migratetype); + /* + * Reserving this block already succeeded, so this should + * not fail on zone boundaries. + */ + WARN_ON_ONCE(ret == -1); set_pageblock_migratetype(page, ac->migratetype); - ret = move_freepages_block(zone, page, ac->migratetype, - NULL); - if (ret) { + if (ret > 0) { spin_unlock_irqrestore(&zone->lock, flags); return ret; } @@ -1990,7 +2028,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * deviation from the rest of this file, to make the for loop * condition simpler. */ -static __always_inline bool +static __always_inline struct page * __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, unsigned int alloc_flags) { @@ -2037,7 +2075,7 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, goto do_steal; } - return false; + return NULL; find_smallest: for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) { @@ -2057,14 +2095,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, do_steal: page = get_page_from_free_area(area, fallback_mt); - steal_suitable_fallback(zone, page, alloc_flags, start_migratetype, - can_steal); + /* take off list, maybe claim block, expand remainder */ + page = steal_suitable_fallback(zone, page, current_order, order, + start_migratetype, alloc_flags, can_steal); trace_mm_page_alloc_extfrag(page, order, current_order, start_migratetype, fallback_mt); - return true; - + return page; } #ifdef CONFIG_CMA @@ -2127,15 +2165,14 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype, return page; } } -retry: + page = __rmqueue_smallest(zone, order, migratetype); if (unlikely(!page)) { if (alloc_flags & ALLOC_CMA) page = __rmqueue_cma_fallback(zone, order); - - if (!page && __rmqueue_fallback(zone, order, migratetype, - alloc_flags)) - goto retry; + else + page = __rmqueue_fallback(zone, order, migratetype, + alloc_flags); } return page; } @@ -2672,12 +2709,10 @@ int __isolate_free_page(struct page *page, unsigned int order) * Only change normal pageblocks (i.e., they can merge * with others) */ - if (migratetype_is_mergeable(mt)) { - set_pageblock_migratetype(page, - MIGRATE_MOVABLE); - move_freepages_block(zone, page, - MIGRATE_MOVABLE, NULL); - } + if (migratetype_is_mergeable(mt) && + move_freepages_block(zone, page, + MIGRATE_MOVABLE) != -1) + set_pageblock_migratetype(page, MIGRATE_MOVABLE); } } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index a5c8fa4c2a75..71539d7b96cf 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -178,15 +178,18 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end, migratetype, isol_flags); if (!unmovable) { - unsigned long nr_pages; + int nr_pages; int mt = get_pageblock_migratetype(page); + nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE); + /* Block spans zone boundaries? */ + if (nr_pages == -1) { + spin_unlock_irqrestore(&zone->lock, flags); + return -EBUSY; + } + __mod_zone_freepage_state(zone, -nr_pages, mt); set_pageblock_migratetype(page, MIGRATE_ISOLATE); zone->nr_isolate_pageblock++; - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, - NULL); - - __mod_zone_freepage_state(zone, -nr_pages, mt); spin_unlock_irqrestore(&zone->lock, flags); return 0; } @@ -206,7 +209,7 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ static void unset_migratetype_isolate(struct page *page, int migratetype) { struct zone *zone; - unsigned long flags, nr_pages; + unsigned long flags; bool isolated_page = false; unsigned int order; struct page *buddy; @@ -252,7 +255,12 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * allocation. */ if (!isolated_page) { - nr_pages = move_freepages_block(zone, page, migratetype, NULL); + int nr_pages = move_freepages_block(zone, page, migratetype); + /* + * Isolating this block already succeeded, so this + * should not fail on zone boundaries. + */ + WARN_ON_ONCE(nr_pages == -1); __mod_zone_freepage_state(zone, nr_pages, migratetype); } set_pageblock_migratetype(page, migratetype); From patchwork Wed Mar 6 04:08:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 910A4C54E49 for ; Wed, 6 Mar 2024 04:16:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 616066B0092; Tue, 5 Mar 2024 23:16:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C5496B0093; Tue, 5 Mar 2024 23:16:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F1606B0095; Tue, 5 Mar 2024 23:16:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2034C6B0092 for ; Tue, 5 Mar 2024 23:16:33 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CAD3A1C0BC7 for ; Wed, 6 Mar 2024 04:16:32 +0000 (UTC) X-FDA: 81865302624.25.D64F98F Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf02.hostedemail.com (Postfix) with ESMTP id 2A23080009 for ; Wed, 6 Mar 2024 04:16:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b="R8/jonVh"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf02.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=36xbR356rfIzWtgB9qLDyiEGey6MMsGLn5MqndsELSg=; b=NDN2hgUj2pR31dz7e57nOXRuqJAQ0LUYb//KjNsI95jxZ8SHL4ElwALbMsX0eSKBdHD/p8 Bj9IgQ9wwWWfBCaRmJocodMWUmMz+7NnQipah5q/5yMDQsNgvKxNquKltqsiS/eE/WzolK /fBdRsTV9OBBbzii0wJlsY3dDzMvt3g= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b="R8/jonVh"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf02.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698591; a=rsa-sha256; cv=none; b=SezpY6OkZTg4/suWNbeBiIbrvRj0D5NN574dTXYyCa/dhJgjvmzeExyvvF2bDPDjYyo7JU ufq6SFj4KEhI1JmR3xKzsXllY0QAXsbZXDRsH6Ch2cMT8nmSl/PjqfdlHY/E84Mr9mLB64 1/O2idkp8D+1RW1mzBxvCgDgImY2OYk= Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-78822adc835so169180885a.3 for ; Tue, 05 Mar 2024 20:16:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698590; x=1710303390; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=36xbR356rfIzWtgB9qLDyiEGey6MMsGLn5MqndsELSg=; b=R8/jonVhnarpcr6M0jLp41uJoZ+q/1sl6CmSUKbljXaqimj5CyhvfRF0QaSQedv28A RVDtkcF2eIUAT4T14fQoz2tueaqlI59GXdXuoJQ6eYs4dCz5JV2yWevNfGvuTBxIWS59 8HiSazn4nNKZ3pszsHkpU/nWDH0Qq6sEnSV+ZAQEsBq82ONucYN2OeEMoJq9fnKV197F +HblZzYeJH4g0UkIDVlGmAyrwfVgG5QRA0mC5qCSxPvjMYtuvIA9Ub7rwf9Hbu3siWX8 8L2MDNP5RKQ/ugVtBdPWVzJP6Vk3Fjs1SiWc8q8vEpJ1BaLToY100DFZGxWSBKQBb1qe DY8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698590; x=1710303390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=36xbR356rfIzWtgB9qLDyiEGey6MMsGLn5MqndsELSg=; b=ENStCra0L2he4lYsx9pD1awk8jiMkQEDxLkjmDB6n3QT5YA5+kPQJWEOz79Q5QjCtp stdXAmQ4ffISPzyy8YIAUXmTcs2tMrKpd+V41xeWPPX7N6NaXSzvkGlCDrJSmQMi1n4P qjZDK6yAC1XnO6c/jjLb57yiwpV5iINw5lnEv18K0uSxOE/32FuBEPwzED6KJZGG/+/2 Oz/REV5brTNj8PY+aajDCpJGSDjWJEMZaojL4NR7S1JUbRx2Xd6FObFKTwV+N9e8ZzT+ uArF+wg5pSelRNUz/iu+Jt8GjgmjLy8ow01nVDSf+V8ehl9tQ3oVbbQoUC8G+RFcW0ak 5m9w== X-Forwarded-Encrypted: i=1; AJvYcCUa5B+gMaXXQYTBUcBWz7s8z0zhaU4+kV7V2kymC6RU9kSJumCKlxec5/oFQH1DsxiznmQ8D0l8td7IcCZ1JlvVzyY= X-Gm-Message-State: AOJu0YzkG4xkcAeUrgEqkmxo9nmdmgF+Rv2/MRtyzGclFB0nle1ew47N vmuTLOvCBf446VqIKJ0LKGUgInERZxAE9l/omK/BLGgqMlVOZDYed7+oEPENMIs= X-Google-Smtp-Source: AGHT+IHjBLTEjXsYssAa7PuQ/BMacfQ6VruPXkruqLKHw1gogijgd2A4F8i/4YwGrtqS5m5epciAkw== X-Received: by 2002:a05:620a:24c8:b0:788:3f44:f2bf with SMTP id m8-20020a05620a24c800b007883f44f2bfmr488280qkn.43.1709698590368; Tue, 05 Mar 2024 20:16:30 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id v13-20020a05620a122d00b00788357d6759sm1351474qkj.11.2024.03.05.20.16.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:29 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/10] mm: page_alloc: close migratetype race between freeing and stealing Date: Tue, 5 Mar 2024 23:08:38 -0500 Message-ID: <20240306041526.892167-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2A23080009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: b91pwpy8ytnj1mb7744nje66f34acua8 X-HE-Tag: 1709698590-257578 X-HE-Meta: U2FsdGVkX1/abQf5F6UC0olEFw9ghhOIr32Fp4lpleql21MT3nqoQHqHq0/snkRiYM+0MseGT2sAS6MJbkZfBA7x9Omj2C50eJnSbdQDfvbTGjTlp14vb2SAdfLG9c0C6jeAu/wqrb9ANudUrb7Bf+VZ1hpYmrIp0Wa5+XIXua4sZ5mgfEutNrjjQcWEuSiOvdYq8BZw6SUBhTdTgskklvcOcSMeeikQm1m1H9Qn1j7xbYCD9Yyh/8djoRvT7Tg1qipKyLGLT+dEYBPA3Qh/fvoYMVdEyNZvPoKjWbj1tB7bm7JmZawZk4m0VYuq5bn/vi+cnI99Lk1jdZBDWu6bvyIwFxCPUFK+wrRkqESA7f/nJLnbjYIJMm/e1AXnl1d8z5kCsnAHCmObqFPwa/md2dace95zr507ydj/iPoflXUDBl3BXVuHrXy05AAKLwhGu01X3gpyMwaNXpOii+3Ue8hkPyjcZdIzYyly8+9b3m1+z8KT82VuNAN1yN+uHn8fZOnjZrYV9NVgKYvuuaywRztp+z5zbL6UrEsBoAfYOTlbJ5/xTFzz3JLT+kabV5VfDQSSAAOOya+5cGqjj4reBtp815mxbdF+4qgNHa59X4thzZjFrlU7PnqW/+k9P/YGiYedZRBhwF2mQvuIXmBjNOTIlYmXPflEzr7tGWrYa00z42E/2QD4ykpa3CRa7XGAzmX7Xqz2Q4YXbN3eAQZ7CkErfjGdg7PXHi9XZIDu66xpz8VsvOCYZL3obDT+zX46YwZe9F3dEXBLb/ApjsEvgcVYATmK0IcQdyMk5fPDS0jpHXKfb5TzhHH4OIQlp5DUY/OXVdbDaayB6I7z/irrEsmV87U81HNu+cdAZVLaTHdAqS7imB9wxrgbLMUmwdjH4ISY5WcLhuBZfGAUnxYHElYtaQGXjIJDXyJaS6bvOLmB8nU/totqrceeBFfWLo6AN5CeIFNCwVHC+FsCPIo s2gZHAha O92AkaflqWeRImO4tBqiiiluOe1kSkNnI1ucvjKJEdfKC3ZYzkEVdyeQBRU5Mh1T0UjsYpyC6C6XZolPZeMaKoMTPSP4DuIfJyedy0RuXMCymA1ZkgWlntm8xYjtCbC6VpX7FNJWBg3Y1S2kqmjehcXGF0sZ97D95j3caCK/lJdIMES8brbjcZP38Sq671zRBepHt8MpJq5OHCAyYTWPm/x3TFCZ4zh+QdJZA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are three freeing paths that read the page's migratetype optimistically before grabbing the zone lock. When this races with block stealing, those pages go on the wrong freelist. The paths in question are: - when freeing >costly orders that aren't THP - when freeing pages to the buddy upon pcp lock contention - when freeing pages that are isolated - when freeing pages initially during boot - when freeing the remainder in alloc_pages_exact() - when "accepting" unaccepted VM host memory before first use - when freeing pages during unpoisoning None of these are so hot that they would need this optimization at the cost of hampering defrag efforts. Especially when contrasted with the fact that the most common buddy freeing path - free_pcppages_bulk - is checking the migratetype under the zone->lock just fine. In addition, isolated pages need to look up the migratetype under the lock anyway, which adds branches to the locked section, and results in a double lookup when the pages are in fact isolated. Move the lookups into the lock. Reported-by: Vlastimil Babka Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 44 +++++++++++++++++--------------------------- 1 file changed, 17 insertions(+), 27 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9cf7ed0c4cd6..82e6c4068647 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1227,18 +1227,15 @@ static void free_pcppages_bulk(struct zone *zone, int count, spin_unlock_irqrestore(&zone->lock, flags); } -static void free_one_page(struct zone *zone, - struct page *page, unsigned long pfn, - unsigned int order, - int migratetype, fpi_t fpi_flags) +static void free_one_page(struct zone *zone, struct page *page, + unsigned long pfn, unsigned int order, + fpi_t fpi_flags) { unsigned long flags; + int migratetype; spin_lock_irqsave(&zone->lock, flags); - if (unlikely(has_isolate_pageblock(zone) || - is_migrate_isolate(migratetype))) { - migratetype = get_pfnblock_migratetype(page, pfn); - } + migratetype = get_pfnblock_migratetype(page, pfn); __free_one_page(page, pfn, zone, order, migratetype, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); } @@ -1246,21 +1243,13 @@ static void free_one_page(struct zone *zone, static void __free_pages_ok(struct page *page, unsigned int order, fpi_t fpi_flags) { - int migratetype; unsigned long pfn = page_to_pfn(page); struct zone *zone = page_zone(page); if (!free_pages_prepare(page, order)) return; - /* - * Calling get_pfnblock_migratetype() without spin_lock_irqsave() here - * is used to avoid calling get_pfnblock_migratetype() under the lock. - * This will reduce the lock holding time. - */ - migratetype = get_pfnblock_migratetype(page, pfn); - - free_one_page(zone, page, pfn, order, migratetype, fpi_flags); + free_one_page(zone, page, pfn, order, fpi_flags); __count_vm_events(PGFREE, 1 << order); } @@ -2530,7 +2519,7 @@ void free_unref_page(struct page *page, unsigned int order) struct per_cpu_pages *pcp; struct zone *zone; unsigned long pfn = page_to_pfn(page); - int migratetype, pcpmigratetype; + int migratetype; if (!free_pages_prepare(page, order)) return; @@ -2542,23 +2531,23 @@ void free_unref_page(struct page *page, unsigned int order) * get those areas back if necessary. Otherwise, we may have to free * excessively into the page allocator */ - migratetype = pcpmigratetype = get_pfnblock_migratetype(page, pfn); + migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, FPI_NONE); return; } - pcpmigratetype = MIGRATE_MOVABLE; + migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, pcpmigratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, migratetype, FPI_NONE); + free_one_page(zone, page, pfn, order, FPI_NONE); } pcp_trylock_finish(UP_flags); } @@ -2615,7 +2604,7 @@ void free_unref_folios(struct folio_batch *folios) */ if (is_migrate_isolate(migratetype)) { free_one_page(zone, &folio->page, pfn, - order, migratetype, FPI_NONE); + order, FPI_NONE); continue; } @@ -2628,7 +2617,7 @@ void free_unref_folios(struct folio_batch *folios) if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, pfn, - order, migratetype, FPI_NONE); + order, FPI_NONE); continue; } locked_zone = zone; @@ -6796,13 +6785,14 @@ bool take_page_off_buddy(struct page *page) bool put_page_back_buddy(struct page *page) { struct zone *zone = page_zone(page); - unsigned long pfn = page_to_pfn(page); unsigned long flags; - int migratetype = get_pfnblock_migratetype(page, pfn); bool ret = false; spin_lock_irqsave(&zone->lock, flags); if (put_page_testzero(page)) { + unsigned long pfn = page_to_pfn(page); + int migratetype = get_pfnblock_migratetype(page, pfn); + ClearPageHWPoisonTakenOff(page); __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE); if (TestClearPageHWPoison(page)) { From patchwork Wed Mar 6 04:08:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE5CDC5475B for ; Wed, 6 Mar 2024 04:16:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA7E56B0098; Tue, 5 Mar 2024 23:16:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A5BAE6B0096; Tue, 5 Mar 2024 23:16:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AB756B0098; Tue, 5 Mar 2024 23:16:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 741036B0095 for ; Tue, 5 Mar 2024 23:16:34 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4B6A0120A59 for ; Wed, 6 Mar 2024 04:16:34 +0000 (UTC) X-FDA: 81865302708.21.1950B60 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf01.hostedemail.com (Postfix) with ESMTP id A3FD040005 for ; Wed, 6 Mar 2024 04:16:32 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=lHU5s+e8; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698592; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m/ZzL/TH/R4lBA3ZDd+dHH0WpYTGgWIE5ifE2nwqWl8=; b=OeqzewUpp1+n8szMFihrG7n534vqoPyHAJscCfeZ80sCwLTfbeFMDk8O8hZBjlvs7wD5Wu UUhDdU6FJQ627RTiXdcsG1PJwrCDJD/Wi9DSfGs20OYmm92Q2Z+XjEd3Et9RE//MH6Sgm6 AhoCNmTRSFdH8ISzE54/Ww+lpgBCSIg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=lHU5s+e8; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698592; a=rsa-sha256; cv=none; b=mWQ4lnp87ItjL1w/02uUaGv5S1qH7CB/PxP8qhXKDL359dyf/DSDzIiAD4m8uLhXSO/887 Ie/yIcbccuREdV0Mk+GOPKrSkfjMKf6/XMxC4kOEEpEAL29JEQE47pOMOG2ZQnMirvQNB1 odSPA9IZN2//Za/peaXWRgzunBqqQ+0= Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-6902947c507so8763046d6.3 for ; Tue, 05 Mar 2024 20:16:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698592; x=1710303392; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m/ZzL/TH/R4lBA3ZDd+dHH0WpYTGgWIE5ifE2nwqWl8=; b=lHU5s+e8URHvQUxdPvjXJIEYrFoO4XLJyY7N3yu8V1jTk98j+rqKqZk/tKRbrF8OO2 wRZAu08vs4nCnGn7FJOninP4m6h9qUF5fR5FkpIrLCb/ErpXp2YxJBhM/8ofT75tC9Dz mqQEqnZjcr8Sqfk4So1Mjfacv8C0Utvww2XMFSBzQHIoCHQrzipzCUfEf7Ubft6bG5vc 0C+W+cULZtSbHoAz3UQLiu+DfgTwK9BleN39Rjjlo9XxElKkkU/NsirqjS0zOM/7xNoi WVRonbzkPnwdBdVpAEVXvoxrlg3w4RbHwJMlh0iFCtDtK/q/w4wQTLeJ+ZXEtdqoLWEr vLaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698592; x=1710303392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m/ZzL/TH/R4lBA3ZDd+dHH0WpYTGgWIE5ifE2nwqWl8=; b=HJzb18iP466SSvOA53XnmPXTvPBHygW9J5b4pvxOiJfQkUhVhvz+t+07GMogb7Banh 9DUY591/l9kENXRVqdnQ8fAAi1GiyN+254rfrYi1BGKk8Q+1sq6eYoV09T9VWb74j0lu 12yyYDG6IRUXNUSBZFFGdtNZSpck5Ggv3POaEe4j3Lv3liF9Gg5Zq+bLJ0+BKiHabtJ7 vwu22W0lFd3HgcBGEU/0igD+ZamDkd14GtB9UrHFK1Gp0EJTIlB5uSTbW4mSycicDfeY hvSHhrOzTFCOomBrYCkuH0hoDC/bE2Vj0rY7OEXcHrV0vtHurBnyUrls7/SjMZQzTPe0 XHyQ== X-Forwarded-Encrypted: i=1; AJvYcCWTQCgtuOEnNWwH66oOPI6R1fMPT/r+HYs0lW7cH+CDuVuZmdneg4CvOR5akCtNiLH5taih+dsOgAkcKh5aSfRXSXM= X-Gm-Message-State: AOJu0YxksOK09RXAMjcYAIdr3RPpG4W8bHWz3weaWTXHzoUAh5nhF5Ls W+qJ/Gi5aF2SOZDCWqymSUqBcputWBks1/dfnYicY/eS8LdU1efgGNZBg441gjY= X-Google-Smtp-Source: AGHT+IEhLmA5Z/jbA3zn2+GD6DrmyUDwK6kcHB1ddo4FwuhIkm61lmFRuGfA57LnzXsN6BEZH/RY/w== X-Received: by 2002:ad4:4d06:0:b0:690:6ce7:432e with SMTP id l6-20020ad44d06000000b006906ce7432emr4033712qvl.9.1709698591864; Tue, 05 Mar 2024 20:16:31 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id pd6-20020a056214490600b006907801a000sm2714835qvb.26.2024.03.05.20.16.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:31 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/10] mm: page_alloc: set migratetype inside move_freepages() Date: Tue, 5 Mar 2024 23:08:39 -0500 Message-ID: <20240306041526.892167-9-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: jogkpy9fqynjxnird799afnmbbrm6j8a X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A3FD040005 X-HE-Tag: 1709698592-504796 X-HE-Meta: U2FsdGVkX180dEwVFidXYVAsKTugm8rXV9Fjdpxng9zQIqqlEiKqIkmMdxw7XCzaeSAekSonsb7tTVaA5f9KSxRxebacCHu9IDoKoRCE5cqdGFXnhqxELHkHPt3RhCzmZwHpxjFY3gxfM7gHJ0v1zC+mN4gY7gKhVxIVCuULuWryi5iE9p2YbSV1ZySeaPejcf/2aDcHpGcS3NnXLrJIC2UrN095GfWajOf8xGJybugpcUCkvDkf6b6IHvrjB/6hHkMdrdeVNbVChzGGqCxxzWKBnccY8rEBlIj9wZYdjnOSMFmbZjye1YMXz4ZNFiM5Up5m5fgm+ypAo/rZdJ0b4aY9Iwg+oanlrzAEKlevc5ejDLK4cwcop4mzfl5Ky+bTmAFKf5l228P98Uv2KhaFzISBqXogWMaylX61dZIilurItxSDG/+m2FK2v2ns++S8HxmkMhMR5/xtDlI1iX0AmGOJurkvjXxyseLf9IkUKI4wgq2B6Vsc01WiFbKpRbdOUu5+UQv6QesXZH+HoBuTrMDMLAH8XKc/BaHTTh5n7cwhIwuPlkvVGVA+fcZs2m2quLwZoYgeRSemXVRJUDRmQ8ZvarDWtXZT9zeXtO33YKPwvbrivRWS3CSgm8t3mLNVu6H2IECpkqzFVD860CaLTk9krzOD+s+Dw4Bp2o27I+klA4Besq+uN2QfZn/IWwV1BdV73JDc0Iy0AwSPOLssVPgU0AGqfwRTcpnJOioH5ovZ2VOD/boqPk8vuW1ds44SGptIGEhSPmcw7MhiFyeXRjHaICsU5DCe/wX/OaE7wTQxhlkmm5jD6rHbwf//9y2FgN/amArc0BAnUEaSnv1BnVX5e2zNrHYSfW5ySQZP+Et6lfpARp/fCBj+7AL/hu03RbAZMUsiGyxFiHQ++9CVkkBz4GfetJe6BKDe+t3KiZXYCnfz789o6M8+7nENzI/H++v7YA1pfniCS93nvDX fJDgGqV1 OOqmI1W808Oe0Wumjfvo0z49V3BznP2U2mSInHUQGSn0IiY7KFuap5Beno6Y9r9pCAaW3ci9FDKeqBPKj56VxcEeuelDjv5y+Kf6hxIDcfgqmfXvqlRdXp4JofV6VQuIwTJco4QbMAMWNA6BQK+JiDXjzmx1zKh+Q/cUkmiXtSUqlvd/NnXUM34lD7YobpAqvQg6rXVzdFRuFeus7szezRvNpSUkXUtwJ065O X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan This avoids changing migratetype after move_freepages() or move_freepages_block(), which is error prone. It also prepares for upcoming changes to fix move_freepages() not moving free pages partially in the range. Signed-off-by: Zi Yan Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 27 +++++++++++++-------------- mm/page_isolation.c | 7 +++---- 2 files changed, 16 insertions(+), 18 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 82e6c4068647..a057b82c4f1d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1581,9 +1581,8 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, #endif /* - * Move the free pages in a range to the freelist tail of the requested type. - * Note that start_page and end_pages are not aligned on a pageblock - * boundary. If alignment is required, use move_freepages_block() + * Change the type of a block and move all its free pages to that + * type's freelist. */ static int move_freepages(struct zone *zone, unsigned long start_pfn, unsigned long end_pfn, int migratetype) @@ -1593,6 +1592,9 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, unsigned int order; int pages_moved = 0; + VM_WARN_ON(start_pfn & (pageblock_nr_pages - 1)); + VM_WARN_ON(start_pfn + pageblock_nr_pages - 1 != end_pfn); + for (pfn = start_pfn; pfn <= end_pfn;) { page = pfn_to_page(pfn); if (!PageBuddy(page)) { @@ -1610,6 +1612,8 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, pages_moved += 1 << order; } + set_pageblock_migratetype(pfn_to_page(start_pfn), migratetype); + return pages_moved; } @@ -1837,7 +1841,6 @@ steal_suitable_fallback(struct zone *zone, struct page *page, if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || page_group_by_mobility_disabled) { move_freepages(zone, start_pfn, end_pfn, start_type); - set_pageblock_migratetype(page, start_type); return __rmqueue_smallest(zone, order, start_type); } @@ -1911,12 +1914,10 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone) /* Yoink! */ mt = get_pageblock_migratetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ - if (migratetype_is_mergeable(mt)) { - if (move_freepages_block(zone, page, MIGRATE_HIGHATOMIC) != -1) { - set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); + if (migratetype_is_mergeable(mt)) + if (move_freepages_block(zone, page, + MIGRATE_HIGHATOMIC) != -1) zone->nr_reserved_highatomic += pageblock_nr_pages; - } - } out_unlock: spin_unlock_irqrestore(&zone->lock, flags); @@ -1995,7 +1996,6 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * not fail on zone boundaries. */ WARN_ON_ONCE(ret == -1); - set_pageblock_migratetype(page, ac->migratetype); if (ret > 0) { spin_unlock_irqrestore(&zone->lock, flags); return ret; @@ -2698,10 +2698,9 @@ int __isolate_free_page(struct page *page, unsigned int order) * Only change normal pageblocks (i.e., they can merge * with others) */ - if (migratetype_is_mergeable(mt) && - move_freepages_block(zone, page, - MIGRATE_MOVABLE) != -1) - set_pageblock_migratetype(page, MIGRATE_MOVABLE); + if (migratetype_is_mergeable(mt)) + move_freepages_block(zone, page, + MIGRATE_MOVABLE); } } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 71539d7b96cf..f84f0981b2df 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -188,7 +188,6 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ return -EBUSY; } __mod_zone_freepage_state(zone, -nr_pages, mt); - set_pageblock_migratetype(page, MIGRATE_ISOLATE); zone->nr_isolate_pageblock++; spin_unlock_irqrestore(&zone->lock, flags); return 0; @@ -262,10 +261,10 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) */ WARN_ON_ONCE(nr_pages == -1); __mod_zone_freepage_state(zone, nr_pages, migratetype); - } - set_pageblock_migratetype(page, migratetype); - if (isolated_page) + } else { + set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype); + } zone->nr_isolate_pageblock--; out: spin_unlock_irqrestore(&zone->lock, flags); From patchwork Wed Mar 6 04:08:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 108C4C54E55 for ; Wed, 6 Mar 2024 04:16:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63C2F6B0095; Tue, 5 Mar 2024 23:16:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EAE76B0096; Tue, 5 Mar 2024 23:16:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4417D6B0099; Tue, 5 Mar 2024 23:16:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2B49D6B0095 for ; Tue, 5 Mar 2024 23:16:36 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0962EA07DA for ; Wed, 6 Mar 2024 04:16:36 +0000 (UTC) X-FDA: 81865302792.11.4D4FC80 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf14.hostedemail.com (Postfix) with ESMTP id 308A6100008 for ; Wed, 6 Mar 2024 04:16:34 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=V3G3bzaU; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698594; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FPEgY8XXzPX2/R/ApQUsBC8lGy3We3T1q8AN7Sb/Hqs=; b=LlL6ExV2fRIKJit8ZnO+j8Fq2tKM4laWAEYYt90GXgaLeHiCKIMZtXuNuV64wLahHx6Hgs mVGXmSRKzPCBSY9bQoP42z5Wlg06BFMwOxvJvfaoHUxYkFIRS4zAQ8SIWm9nOOjDhTWwnl ZqTP/KH5JBx9B1O15ZE+70ePm0EDwzA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=V3G3bzaU; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698594; a=rsa-sha256; cv=none; b=o4MgjOeNyvbupzailYXGjKs0UFXwgaLhbfxvMgsemBAJFBjwrnSgga6fAxHe5wmLBb2a3A pLwNZgYyLPj6ECe9guEjR6aqcO7R3qilOeNr2D6/ejotDREofb1Mc2u1plOrB7Bkpu2Lin h2/poJ7R7RcSm3FZ6/P3TytT64zHeU8= Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7810827e54eso486968485a.2 for ; Tue, 05 Mar 2024 20:16:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698593; x=1710303393; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FPEgY8XXzPX2/R/ApQUsBC8lGy3We3T1q8AN7Sb/Hqs=; b=V3G3bzaUN+/eyqWfEWkSeiBcHkd+d5M3nHEb2k8dH1pXIzKwAa1aSduqQ39UpOCn3f F9xXQg40oJ+eX2/xv1/22C3Lfj0NXyxikeP4a5JFPA/7wyvU2TSbZnt8KBYfvPeaoWcH oux+sOb6O64ZBhS3LIzz/LX5wPRvvK/YIscJKZ2Fz8NJRGJXq1+//JKGSdIS+NwmmWEA w4q8W90y5lPBAeJqFXOFqZbBym9kgBPKR4PPiqbPM7fjSor45fZ2Iqz4tD8sVbhQQswL H8X5NHg0oPTAVXQDFPffT9c3ea88Gv60Du1+Mtyn96hdf3NRIL4P8UoGIeZlxNpjJSIn hoQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698593; x=1710303393; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FPEgY8XXzPX2/R/ApQUsBC8lGy3We3T1q8AN7Sb/Hqs=; b=jsHHEVgKFur3qc3a+UtZgkbGUGoIOjEs+LvhKJa81EYv3z1r2QSXfRpVbnMftUSFqr T3manI7W5lEci+Xfb6Ya5R+eVrhNhpwhDUQAKTb5giw2RF/uBnOibZ/yj/6vHV2TmwaI 63BgoaBodmexjHpEYogWoF+f//CJUElaw8QqO09D7qaVY5iktLFEKVdywFPlJ+hiu2ua orIs2fijRgMH9YmfDFMxOI6S3NAz/spuwd42m7FGxX4va2dEZHMfK/yIdXKFQEsiMFb9 6b7UeEa/TP1vUtgUXh+5ylF4HhLXc5SozpZGn1c0wDVFQdCsrsFcSysBnFUXThtOtCCe Vpbg== X-Forwarded-Encrypted: i=1; AJvYcCWWjK8ZIzAjXnLhl5NjSthcOGcWwcFZCemzxyDp/SHw/CGltdkXpYtIDRmQcn9n3o3QCz+MXBTYMmiiEYV3jO4+UGA= X-Gm-Message-State: AOJu0Yx3Gk2IkgzLWL+JA7eHOztZoEouEDVOUXkL+0XSUP5Z9ZzbVpMF wj/MJUsFHRTPgDGbYOOyKdJwfyFi2ww8artaBQjQGXT9yHYPyak9rDSaSPJZots= X-Google-Smtp-Source: AGHT+IE5l8tTVSCecMuIYsWM28jFo6JMp6MN53REOmUEHQisewZzr1b+7MBDcSBV6IFI9ga7MVtHMg== X-Received: by 2002:a05:620a:10aa:b0:788:2682:9f5f with SMTP id h10-20020a05620a10aa00b0078826829f5fmr4308776qkk.76.1709698593306; Tue, 05 Mar 2024 20:16:33 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id or10-20020a05620a618a00b007882c35b349sm2426238qkn.91.2024.03.05.20.16.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:32 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/10] mm: page_isolation: prepare for hygienic freelists Date: Tue, 5 Mar 2024 23:08:40 -0500 Message-ID: <20240306041526.892167-10-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 308A6100008 X-Stat-Signature: oui9r75su74har1qdxmhgrxfpb7bnp9e X-Rspam-User: X-HE-Tag: 1709698594-315631 X-HE-Meta: U2FsdGVkX18jYSK4XGYBCSj+dvEI8Fsy1HksVEJaS7M3EsEfBwsZEQ2xzrxrhaqXy3x+ycM8z4jUSKI1+rXSqP363pJVTP6JaTqM53/GzaVZ68Q3S7TujxjQJijDaPGYmLVH0lUWY2139uV4uYjvnf8ONR4wr3mggoTVGjMyYth1tflc+83HwJnPd5Hkc4o3imuiFZZxyStNIM6DNeoBVWdi8hJjC6uNWwRIMkd08gLjTZwYbV8F13Z687E5DPsizcxWrcvRv5kPfItM033MoPR/yCUQMr22aTwxGFhTrTfs7PuKV2ew3+OBOjqNTt4G8EjSylVVOcLq2srE6fXmjkd2JoQb1I6IFB9img01d10kvPyyycVm27WYMoDoZqcBFcip9gZq64FlspSXyJ877W2ApLN02oA4yLAt5UJ1nDycivkS9hNhFvXWZ4ADEJrLnBQBr62F+UTtjK9jCzw0bp1Zf/eK3OhsBfWx2mn+wz00yic1sIlv/u1FDR0A12L84HNAh+esGKLhU4bxLrbDrSTiLqhvCBgbBO9Gx2/hZrdGtgHM+seAdsfOoGk3VgfcMxSypEpptBpk7/bcWJ+Gwseexv3iB44gr2CP0EpsM1zzMyKC/NTUGDw8KVf3BNU4lStsfkhEkkG2E1qix6KDJwJX0xVHevmHzbrFT3701FbIj4XNWkCGnzizpIdpiWuowQihUiRMKcLXVt4CsEMQ9kJxgq7nnqTdHumKArktl6uv32oO5JEAOhQugn/+G9MFBoGqBYn0PeXZ93VZWtrkc6PGkP+Ag1rkBClStyew9TxFgcWhraYHuX0/2EMJp4FK5P9+MGsui8bxJRvL/sKXBafxN12OxcZZPd4Q71VeHKCnFD1r2FA6s44tuw0WtgfX2t9fwTLz2D5HNurUxQwl3nv0gvmCkwtunZshMZvUTDM+U3jRgll4ZGxGjLy+Gj690D2w+HeorOV6qNpWPpz ZCgC/iSH FGuDpcBP7QhOjd8ard8MF9YuNRwBuTll05mpFHQ0SVq6LC4NifgW6EBpY7nBMKX+vzvXY2meIxzkuNdwZo9tBNY08RLiBXFla1li2TbbMqfvz4o+wir/r6I/dGmFHgJ/UB9KbPe7MSF6MtCJnsToRUKaYGJIb0la0ZFdrPWybtKVRre8SQMsNMfvG2Of3B5NliATHZrHp5LnKtQ8CksGjfdHWhoAhVBZoxt/3ksG0ROMkuL4dld04rIApYmrgSJJfybHoSPHywx/w9mtY2OVgi16nPnZmGwhQUT5k1QyuRDnviFvHSZi1LMAa+c2i7uc/xZDU3uiB99LLK3VPltQOROc4GcuIwFmuSWDeKs7OD2zyy5/NYk+SXR6zseis9EsGa7O0UFDs0/D0TfokirOrqvw9lld66wuUIRgrlGRJ+O63HBwupQQve8Ra5hUBIVhRsgd9E68aAEgch3DQwf7r5pgKrPw57/Rfq7yIU0gaT3yNPL8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Page isolation currently sets MIGRATE_ISOLATE on a block, then drops zone->lock and scans the block for straddling buddies to split up. Because this happens non-atomically wrt the page allocator, it's possible for allocations to get a buddy whose first block is a regular pcp migratetype but whose tail is isolated. This means that in certain cases memory can still be allocated after isolation. It will also trigger the freelist type hygiene warnings in subsequent patches. start_isolate_page_range() isolate_single_pageblock() set_migratetype_isolate(tail) lock zone->lock move_freepages_block(tail) // nop set_pageblock_migratetype(tail) unlock zone->lock __rmqueue_smallest() del_page_from_freelist(head) expand(head, head_mt) WARN(head_mt != tail_mt) start_pfn = ALIGN_DOWN(MAX_ORDER_NR_PAGES) for (pfn = start_pfn, pfn < end_pfn) if (PageBuddy()) split_free_page(head) Introduce a variant of move_freepages_block() provided by the allocator specifically for page isolation; it moves free pages, converts the block, and handles the splitting of straddling buddies while holding zone->lock. The allocator knows that pageblocks and buddies are always naturally aligned, which means that buddies can only straddle blocks if they're actually >pageblock_order. This means the search-and-split part can be simplified compared to what page isolation used to do. Also tighten up the page isolation code around the expectations of which pages can be large, and how they are freed. Based on extensive discussions with and invaluable input from Zi Yan. Signed-off-by: Johannes Weiner --- include/linux/page-isolation.h | 4 +- mm/internal.h | 4 - mm/page_alloc.c | 200 +++++++++++++++++++-------------- mm/page_isolation.c | 106 ++++++----------- 4 files changed, 151 insertions(+), 163 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 8550b3c91480..c16db0067090 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -34,7 +34,9 @@ static inline bool is_migrate_isolate(int migratetype) #define REPORT_FAILURE 0x2 void set_pageblock_migratetype(struct page *page, int migratetype); -int move_freepages_block(struct zone *zone, struct page *page, int migratetype); + +bool move_freepages_block_isolate(struct zone *zone, struct page *page, + int migratetype); int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype, int flags, gfp_t gfp_flags); diff --git a/mm/internal.h b/mm/internal.h index d1c69119b24f..ccf5a90a3ac8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -559,10 +559,6 @@ extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, void memmap_init_range(unsigned long, int, unsigned long, unsigned long, unsigned long, enum meminit_context, struct vmem_altmap *, int); - -int split_free_page(struct page *free_page, - unsigned int order, unsigned long split_pfn_offset); - #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a057b82c4f1d..862f508835b8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -832,64 +832,6 @@ static inline void __free_one_page(struct page *page, page_reporting_notify_free(order); } -/** - * split_free_page() -- split a free page at split_pfn_offset - * @free_page: the original free page - * @order: the order of the page - * @split_pfn_offset: split offset within the page - * - * Return -ENOENT if the free page is changed, otherwise 0 - * - * It is used when the free page crosses two pageblocks with different migratetypes - * at split_pfn_offset within the page. The split free page will be put into - * separate migratetype lists afterwards. Otherwise, the function achieves - * nothing. - */ -int split_free_page(struct page *free_page, - unsigned int order, unsigned long split_pfn_offset) -{ - struct zone *zone = page_zone(free_page); - unsigned long free_page_pfn = page_to_pfn(free_page); - unsigned long pfn; - unsigned long flags; - int free_page_order; - int mt; - int ret = 0; - - if (split_pfn_offset == 0) - return ret; - - spin_lock_irqsave(&zone->lock, flags); - - if (!PageBuddy(free_page) || buddy_order(free_page) != order) { - ret = -ENOENT; - goto out; - } - - mt = get_pfnblock_migratetype(free_page, free_page_pfn); - if (likely(!is_migrate_isolate(mt))) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - - del_page_from_free_list(free_page, zone, order); - for (pfn = free_page_pfn; - pfn < free_page_pfn + (1UL << order);) { - int mt = get_pfnblock_migratetype(pfn_to_page(pfn), pfn); - - free_page_order = min_t(unsigned int, - pfn ? __ffs(pfn) : order, - __fls(split_pfn_offset)); - __free_one_page(pfn_to_page(pfn), pfn, zone, free_page_order, - mt, FPI_NONE); - pfn += 1UL << free_page_order; - split_pfn_offset -= (1UL << free_page_order); - /* we have done the first part, now switch to second part */ - if (split_pfn_offset == 0) - split_pfn_offset = (1UL << order) - (pfn - free_page_pfn); - } -out: - spin_unlock_irqrestore(&zone->lock, flags); - return ret; -} /* * A bad page could be due to a number of fields. Instead of multiple branches, * try and check multiple fields with one check. The caller must do a detailed @@ -1669,8 +1611,8 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, return true; } -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype) +static int move_freepages_block(struct zone *zone, struct page *page, + int migratetype) { unsigned long start_pfn, end_pfn; @@ -1681,6 +1623,119 @@ int move_freepages_block(struct zone *zone, struct page *page, return move_freepages(zone, start_pfn, end_pfn, migratetype); } +#ifdef CONFIG_MEMORY_ISOLATION +/* Look for a buddy that straddles start_pfn */ +static unsigned long find_large_buddy(unsigned long start_pfn) +{ + int order = 0; + struct page *page; + unsigned long pfn = start_pfn; + + while (!PageBuddy(page = pfn_to_page(pfn))) { + /* Nothing found */ + if (++order > MAX_PAGE_ORDER) + return start_pfn; + pfn &= ~0UL << order; + } + + /* + * Found a preceding buddy, but does it straddle? + */ + if (pfn + (1 << buddy_order(page)) > start_pfn) + return pfn; + + /* Nothing found */ + return start_pfn; +} + +/* Split a multi-block free page into its individual pageblocks */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order) +{ + unsigned long end_pfn = pfn + (1 << order); + + VM_WARN_ON_ONCE(order <= pageblock_order); + VM_WARN_ON_ONCE(pfn & (pageblock_nr_pages - 1)); + + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + while (pfn != end_pfn) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, pageblock_order, mt, FPI_NONE); + pfn += pageblock_nr_pages; + page = pfn_to_page(pfn); + } +} + +/** + * move_freepages_block_isolate - move free pages in block for page isolation + * @zone: the zone + * @page: the pageblock page + * @migratetype: migratetype to set on the pageblock + * + * This is similar to move_freepages_block(), but handles the special + * case encountered in page isolation, where the block of interest + * might be part of a larger buddy spanning multiple pageblocks. + * + * Unlike the regular page allocator path, which moves pages while + * stealing buddies off the freelist, page isolation is interested in + * arbitrary pfn ranges that may have overlapping buddies on both ends. + * + * This function handles that. Straddling buddies are split into + * individual pageblocks. Only the block of interest is moved. + * + * Returns %true if pages could be moved, %false otherwise. + */ +bool move_freepages_block_isolate(struct zone *zone, struct page *page, + int migratetype) +{ + unsigned long start_pfn, end_pfn, pfn; + int nr_moved, mt; + + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + NULL, NULL)) + return false; + + /* We're a tail block in a larger buddy */ + pfn = find_large_buddy(start_pfn); + if (pfn != start_pfn) { + struct page *buddy = pfn_to_page(pfn); + int order = buddy_order(buddy); + int mt = get_pfnblock_migratetype(buddy, pfn); + + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(buddy, zone, order); + set_pageblock_migratetype(page, migratetype); + split_large_buddy(zone, buddy, pfn, order); + return true; + } + + /* We're the starting block of a larger buddy */ + if (PageBuddy(page) && buddy_order(page) > pageblock_order) { + int mt = get_pfnblock_migratetype(page, pfn); + int order = buddy_order(page); + + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(page, zone, order); + set_pageblock_migratetype(page, migratetype); + split_large_buddy(zone, page, pfn, order); + return true; + } + + mt = get_pfnblock_migratetype(page, start_pfn); + nr_moved = move_freepages(zone, start_pfn, end_pfn, migratetype); + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -nr_moved, mt); + else if (!is_migrate_isolate(migratetype)) + __mod_zone_freepage_state(zone, nr_moved, migratetype); + return true; +} +#endif /* CONFIG_MEMORY_ISOLATION */ + static void change_pageblock_range(struct page *pageblock_page, int start_order, int migratetype) { @@ -6367,7 +6422,6 @@ int alloc_contig_range(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; - int order; int ret = 0; struct compact_control cc = { @@ -6440,29 +6494,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, * We don't have to hold zone->lock here because the pages are * isolated thus they won't get removed from buddy. */ - - order = 0; - outer_start = start; - while (!PageBuddy(pfn_to_page(outer_start))) { - if (++order > MAX_PAGE_ORDER) { - outer_start = start; - break; - } - outer_start &= ~0UL << order; - } - - if (outer_start != start) { - order = buddy_order(pfn_to_page(outer_start)); - - /* - * outer_start page could be small order buddy page and - * it doesn't include start page. Adjust outer_start - * in this case to report failed page properly - * on tracepoint in test_pages_isolated() - */ - if (outer_start + (1UL << order) <= start) - outer_start = start; - } + outer_start = find_large_buddy(start); /* Make sure the range is really isolated. */ if (test_pages_isolated(outer_start, end, 0)) { diff --git a/mm/page_isolation.c b/mm/page_isolation.c index f84f0981b2df..042937d5abe4 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -178,16 +178,10 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end, migratetype, isol_flags); if (!unmovable) { - int nr_pages; - int mt = get_pageblock_migratetype(page); - - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE); - /* Block spans zone boundaries? */ - if (nr_pages == -1) { + if (!move_freepages_block_isolate(zone, page, MIGRATE_ISOLATE)) { spin_unlock_irqrestore(&zone->lock, flags); return -EBUSY; } - __mod_zone_freepage_state(zone, -nr_pages, mt); zone->nr_isolate_pageblock++; spin_unlock_irqrestore(&zone->lock, flags); return 0; @@ -254,13 +248,11 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * allocation. */ if (!isolated_page) { - int nr_pages = move_freepages_block(zone, page, migratetype); /* * Isolating this block already succeeded, so this * should not fail on zone boundaries. */ - WARN_ON_ONCE(nr_pages == -1); - __mod_zone_freepage_state(zone, nr_pages, migratetype); + WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, migratetype)); } else { set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype); @@ -374,26 +366,29 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, VM_BUG_ON(!page); pfn = page_to_pfn(page); - /* - * start_pfn is MAX_ORDER_NR_PAGES aligned, if there is any - * free pages in [start_pfn, boundary_pfn), its head page will - * always be in the range. - */ + if (PageBuddy(page)) { int order = buddy_order(page); - if (pfn + (1UL << order) > boundary_pfn) { - /* free page changed before split, check it again */ - if (split_free_page(page, order, boundary_pfn - pfn)) - continue; - } + /* move_freepages_block_isolate() handled this */ + VM_WARN_ON_ONCE(pfn + (1 << order) > boundary_pfn); pfn += 1UL << order; continue; } + /* - * migrate compound pages then let the free page handling code - * above do the rest. If migration is not possible, just fail. + * If a compound page is straddling our block, attempt + * to migrate it out of the way. + * + * We don't have to worry about this creating a large + * free page that straddles into our block: gigantic + * pages are freed as order-0 chunks, and LRU pages + * (currently) do not exceed pageblock_order. + * + * The block of interest has already been marked + * MIGRATE_ISOLATE above, so when migration is done it + * will free its pages onto the correct freelists. */ if (PageCompound(page)) { struct page *head = compound_head(page); @@ -404,16 +399,10 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = head_pfn + nr_pages; continue; } + #if defined CONFIG_COMPACTION || defined CONFIG_CMA - /* - * hugetlb, lru compound (THP), and movable compound pages - * can be migrated. Otherwise, fail the isolation. - */ - if (PageHuge(page) || PageLRU(page) || __PageMovable(page)) { - int order; - unsigned long outer_pfn; + if (PageHuge(page)) { int page_mt = get_pageblock_migratetype(page); - bool isolate_page = !is_migrate_isolate_page(page); struct compact_control cc = { .nr_migratepages = 0, .order = -1, @@ -426,56 +415,25 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, }; INIT_LIST_HEAD(&cc.migratepages); - /* - * XXX: mark the page as MIGRATE_ISOLATE so that - * no one else can grab the freed page after migration. - * Ideally, the page should be freed as two separate - * pages to be added into separate migratetype free - * lists. - */ - if (isolate_page) { - ret = set_migratetype_isolate(page, page_mt, - flags, head_pfn, head_pfn + nr_pages); - if (ret) - goto failed; - } - ret = __alloc_contig_migrate_range(&cc, head_pfn, head_pfn + nr_pages, page_mt); - - /* - * restore the page's migratetype so that it can - * be split into separate migratetype free lists - * later. - */ - if (isolate_page) - unset_migratetype_isolate(page, page_mt); - if (ret) goto failed; - /* - * reset pfn to the head of the free page, so - * that the free page handling code above can split - * the free page to the right migratetype list. - * - * head_pfn is not used here as a hugetlb page order - * can be bigger than MAX_PAGE_ORDER, but after it is - * freed, the free page order is not. Use pfn within - * the range to find the head of the free page. - */ - order = 0; - outer_pfn = pfn; - while (!PageBuddy(pfn_to_page(outer_pfn))) { - /* stop if we cannot find the free page */ - if (++order > MAX_PAGE_ORDER) - goto failed; - outer_pfn &= ~0UL << order; - } - pfn = outer_pfn; + pfn = head_pfn + nr_pages; continue; - } else + } + + /* + * These pages are movable too, but they're + * not expected to exceed pageblock_order. + * + * Let us know when they do, so we can add + * proper free and split handling for them. + */ + VM_WARN_ON_ONCE_PAGE(PageLRU(page), page); + VM_WARN_ON_ONCE_PAGE(__PageMovable(page), page); #endif - goto failed; + goto failed; } pfn++; From patchwork Wed Mar 6 04:08:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AE91C5475B for ; Wed, 6 Mar 2024 04:16:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB6DF6B0096; Tue, 5 Mar 2024 23:16:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D42666B0099; Tue, 5 Mar 2024 23:16:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD30D6B009A; Tue, 5 Mar 2024 23:16:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8E0D86B0096 for ; Tue, 5 Mar 2024 23:16:38 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5C9E7A0B9F for ; Wed, 6 Mar 2024 04:16:38 +0000 (UTC) X-FDA: 81865302876.04.76181F4 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf10.hostedemail.com (Postfix) with ESMTP id 9351EC0011 for ; Wed, 6 Mar 2024 04:16:36 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=y1stRujB; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.180 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698596; a=rsa-sha256; cv=none; b=gxGmwuutFuj0HhDkHlPSMKMI0QLr6zwiLnJdXtQPQ+CBG4iQI/27o4E/ifefSJxJ9YV/4s I6Ao1ZdFB1MS8v9JinEeZvNxTc+T0dFf1xEhi5Q4Nu8tBjdU7YQpAHgWgkPqtmgM/ZZ2pd IvF4q/18jOYn/V9e+dR0UstPuKV9njg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=y1stRujB; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.180 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698596; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A7m8LJlBz+flm+pO5mYYHnLahd++TtrakxZ+tQ6DjjM=; b=l2iKAXEOD6U1mOrEv1gprAJBJSOSrOd6ai8ySgWqwE/Q4cU/2s30K5bA9uAiq7s99Mo7JY ALI1QWin0duGzKxrSjRT0ftdswkWHCm2bT8iS6ZO4KWXSXGUcbfOlJe5OF2bFzn4avuR5I 8613yDzueBfI8YI3Ascaxf3/mPgQNB4= Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-42f0aa61b67so3330021cf.1 for ; Tue, 05 Mar 2024 20:16:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698595; x=1710303395; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A7m8LJlBz+flm+pO5mYYHnLahd++TtrakxZ+tQ6DjjM=; b=y1stRujB0oFRSkjtKXKI78IuSR07Cl5JshjHXjllyfPfbVZbSkv0qVWa0aK/tDzDZ2 MnbFtUFMz435mRj5Ti57+FzcolV0KwrpRr/SuJMykDKOESe6KxUbyzESltFpZt3kLLiD su7SOg84rtqff5GAtxZi3QH5R+hjZhekA+S/zW+BhUtbB83CvXhNEWpkfMcovzLSdzL/ 59O70JWKy9LPizW68WJ74kddMGuGcHqQBkmjK7eihpcmwbmPmJvHjuEoYSKOqRt2bjHR NsWQdSTHQg9EdS8YdO44G7fUiSto9mdSPLu6ItWNylobmhxG5QmHfTrp4dltOUVfOo2g dKzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698595; x=1710303395; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A7m8LJlBz+flm+pO5mYYHnLahd++TtrakxZ+tQ6DjjM=; b=DnnCC250UqBTnAycNwuLjfh5pqD9zy2QboHYEQCfCWhr+HhZLmXyz19SLeRU7Ldvnb Q8kVuqXl/MUNrb8OKJ4DyJGwFgEpBGF3TSnhWX2YoYL1ztXMakTr/pwU53yGZwNyUgCt 9zyNs/YxRVXpxhWHAQLkgXsYsVUaMzZdwZctNK90HlzPQ7346XBv8W+YTZ96QcuyoaWN XzWsUx2Ct88czc/BFsHVc8PjycDjWS4N/Tqsg3kcATlNVRptap6rDoeDaB/8jNQ+1nqE k3onoSKuOYPDEgcYn1CLOaqJcPfZ0oxJKWTThbSiGKtJrt8qEHNilz4CeEweaEICCxBq eklA== X-Forwarded-Encrypted: i=1; AJvYcCVLG82Uc8oSez7a4XPgEfux8hxzncNy7+UvUnz/VQuS6MsXN/HgqysIEclQck/w0E35MY33NWx7kpbiIYCCX+HjmaU= X-Gm-Message-State: AOJu0YyNQiqUL5zExJbXwnjPcAZU+fvfVi8pgIzIWwQiq/cjX4yyir7b ZDUBZlb/Vvh0LD+EzUfRCU0S2/DRw4oYfDs0ckWcPikJV+ba/kth/DlJIYSojiA= X-Google-Smtp-Source: AGHT+IElD6HXrosddbzbwxwGc5nPYCAJzhLw1oR8smCTkqpfDZjE4HikDkKfp5RvAXFm63ZVnuZsnw== X-Received: by 2002:a0c:eb4b:0:b0:68f:f768:350a with SMTP id c11-20020a0ceb4b000000b0068ff768350amr4290406qvq.45.1709698595626; Tue, 05 Mar 2024 20:16:35 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id n4-20020a0cbe84000000b0069074aa1cbfsm2995768qvi.142.2024.03.05.20.16.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:35 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/10] mm: page_alloc: consolidate free page accounting Date: Tue, 5 Mar 2024 23:08:41 -0500 Message-ID: <20240306041526.892167-11-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9351EC0011 X-Stat-Signature: 9fu7gkdmu9xs434p4cayacq83iw9ozdh X-Rspam-User: X-HE-Tag: 1709698596-605246 X-HE-Meta: U2FsdGVkX18ybgvsAcX8FV4DzVsOxMLNTMOqg1vnmluZoRLA1kWEBbJhlANqSVf2vYoS7bpSgzJ/YoWQ60utp8ZaOauHOod6XpJmCPvWmIdFZqT7YL3oO8k5yQBIDrBr2OCSfMQlzkHGkZQY5/UEVYrNjvBCdCi/anzMNYT4Cl53ODm+NwMFkUWgmpfXit/83bchrBJHG8BqrhjFnSvin0OCR/HM1FS6eZ9e6QORb6lg1h4BnetAwOpWjv7FOsGl4RllhLtqOBJBZs3qqXz/9ubVbZyrRQgfkVQUWGCq1KCY2bpX3ikK+aSZChg8UWYiYzYMytxG/GO0C3pfUQERHJ5Ny8W/dLBifaucdECQ+rH9OA5XzsLNvYS9q7oPY1vCgiqJYPhQOq7u8z2BWKBrn1RHFsr5jqFmchpOPtZPujMh2ITSUSWLcRHf0PsEDmNWap2LOA/fgEG+k1b/B9Hp89gsrwN5YHv61klp/2vrfemzE2THuwCWy7XhVM9JRMWED+WgOKcbCbDpTliHLFcajUt1Tj8R+kqN5904AWHhO11/OKrLaXEOAJ9TcKbScCjYV7lT8Fz4haCiURPd70HyZiiXkjIROcYpFEErYnidoQIgkpIZJp+C670OJu+1h9+tZe8ynx1oSWXUavbO5YzchJgHWZV9F3ofVUb5cUs5wZoUcg7VUd4/h0FUkdh3leAN8EVsUfVVoDij00+6+bDbqOu/khHXhBEgHprkGq4PijY/2PUp5xwqd8+Gd1VwoYW+fjt6WpbolyNQJOUy0Td11GTtzjK3kP9POZBtyT3ungzSQWlQ8bMU+NOFMbqxbfs9R9KaxKrDZZPShseVyyV+oT+I/aEi57LqKPinAWp97nQEdGmQz56+VmD74YqzIbdiRszVzBCCSiREd6GDWt53r1aKcn75rs5taEXFS7X9GRf8StPBLAgg4iBgP3ud8kExZTwp0FhSCEHn/jPZyWC oyr5py14 KPBIVNGW5GDQMpbDSSeOPPyxD3qS0bhz/BzUaiu6e1T1VOab+M4SaNOGrZz2gxEASmQ45BZZGVX2PTKJeHLE1k18nt2NLl8GYcKmPZ+3qNoBv05IM/QMHpDDnHPXwf4lGFqD36u2EDQWa8KvyBiN+gKqgdE2FWN4m2sfMVxZ63JckjCX9faZPZf3G3NmLBkYbFn2jYpIA5S7yFv1YRAYdmtDX/A3bSJTy0k33 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Free page accounting currently happens a bit too high up the call stack, where it has to deal with guard pages, compaction capturing, block stealing and even page isolation. This is subtle and fragile, and makes it difficult to hack on the code. Now that type violations on the freelists have been fixed, push the accounting down to where pages enter and leave the freelist. Signed-off-by: Johannes Weiner --- include/linux/mm.h | 18 ++-- include/linux/vmstat.h | 8 -- mm/debug_page_alloc.c | 12 +-- mm/internal.h | 5 -- mm/page_alloc.c | 194 +++++++++++++++++++++++------------------ mm/page_isolation.c | 3 +- 6 files changed, 120 insertions(+), 120 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 699e850d143c..54954df107ba 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3771,24 +3771,22 @@ static inline bool page_is_guard(struct page *page) return PageGuard(page); } -bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype); +bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order); static inline bool set_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) + unsigned int order) { if (!debug_guardpage_enabled()) return false; - return __set_page_guard(zone, page, order, migratetype); + return __set_page_guard(zone, page, order); } -void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype); +void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order); static inline void clear_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) + unsigned int order) { if (!debug_guardpage_enabled()) return; - __clear_page_guard(zone, page, order, migratetype); + __clear_page_guard(zone, page, order); } #else /* CONFIG_DEBUG_PAGEALLOC */ @@ -3798,9 +3796,9 @@ static inline unsigned int debug_guardpage_minorder(void) { return 0; } static inline bool debug_guardpage_enabled(void) { return false; } static inline bool page_is_guard(struct page *page) { return false; } static inline bool set_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) { return false; } + unsigned int order) { return false; } static inline void clear_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) {} + unsigned int order) {} #endif /* CONFIG_DEBUG_PAGEALLOC */ #ifdef __HAVE_ARCH_GATE_AREA diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 343906a98d6e..735eae6e272c 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -487,14 +487,6 @@ static inline void node_stat_sub_folio(struct folio *folio, mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); } -static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, - int migratetype) -{ - __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); - if (is_migrate_cma(migratetype)) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); -} - extern const char * const vmstat_text[]; static inline const char *zone_stat_name(enum zone_stat_item item) diff --git a/mm/debug_page_alloc.c b/mm/debug_page_alloc.c index 6755f0c9d4a3..d46acf989dde 100644 --- a/mm/debug_page_alloc.c +++ b/mm/debug_page_alloc.c @@ -32,8 +32,7 @@ static int __init debug_guardpage_minorder_setup(char *buf) } early_param("debug_guardpage_minorder", debug_guardpage_minorder_setup); -bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype) +bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order) { if (order >= debug_guardpage_minorder()) return false; @@ -41,19 +40,12 @@ bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order, __SetPageGuard(page); INIT_LIST_HEAD(&page->buddy_list); set_page_private(page, order); - /* Guard pages are not available for any usage */ - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, -(1 << order), migratetype); return true; } -void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype) +void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order) { __ClearPageGuard(page); - set_page_private(page, 0); - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, (1 << order), migratetype); } diff --git a/mm/internal.h b/mm/internal.h index ccf5a90a3ac8..5e59a19f7f8e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1037,11 +1037,6 @@ static inline bool is_migrate_highatomic(enum migratetype migratetype) return migratetype == MIGRATE_HIGHATOMIC; } -static inline bool is_migrate_highatomic_page(struct page *page) -{ - return get_pageblock_migratetype(page) == MIGRATE_HIGHATOMIC; -} - void setup_zone_pageset(struct zone *zone); struct migration_target_control { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 862f508835b8..422093707a3d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -642,42 +642,72 @@ compaction_capture(struct capture_control *capc, struct page *page, } #endif /* CONFIG_COMPACTION */ -/* Used for pages not on another list */ -static inline void add_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype) +static inline void account_freepages(struct page *page, struct zone *zone, + int nr_pages, int migratetype) { - struct free_area *area = &zone->free_area[order]; + if (is_migrate_isolate(migratetype)) + return; - list_add(&page->buddy_list, &area->free_list[migratetype]); - area->nr_free++; + __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); + + if (is_migrate_cma(migratetype)) + __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); } /* Used for pages not on another list */ -static inline void add_to_free_list_tail(struct page *page, struct zone *zone, - unsigned int order, int migratetype) +static inline void __add_to_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype, + bool tail) { struct free_area *area = &zone->free_area[order]; - list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), migratetype, 1 << order); + + if (tail) + list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + else + list_add(&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; } +static inline void add_to_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype, + bool tail) +{ + __add_to_free_list(page, zone, order, migratetype, tail); + account_freepages(page, zone, 1 << order, migratetype); +} + /* * Used for pages which are on another list. Move the pages to the tail * of the list - so the moved pages won't immediately be considered for * allocation again (e.g., optimization for memory onlining). */ static inline void move_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype) + unsigned int order, int old_mt, int new_mt) { struct free_area *area = &zone->free_area[order]; - list_move_tail(&page->buddy_list, &area->free_list[migratetype]); + /* Free page moving can fail, so it happens before the type update */ + VM_WARN_ONCE(get_pageblock_migratetype(page) != old_mt, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), old_mt, 1 << order); + + list_move_tail(&page->buddy_list, &area->free_list[new_mt]); + + account_freepages(page, zone, -(1 << order), old_mt); + account_freepages(page, zone, 1 << order, new_mt); } -static inline void del_page_from_free_list(struct page *page, struct zone *zone, - unsigned int order) +static inline void __del_page_from_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype) { + VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), migratetype, 1 << order); + /* clear reported state and update reported page count */ if (page_reported(page)) __ClearPageReported(page); @@ -688,6 +718,13 @@ static inline void del_page_from_free_list(struct page *page, struct zone *zone, zone->free_area[order].nr_free--; } +static inline void del_page_from_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype) +{ + __del_page_from_free_list(page, zone, order, migratetype); + account_freepages(page, zone, -(1 << order), migratetype); +} + static inline struct page *get_page_from_free_area(struct free_area *area, int migratetype) { @@ -759,18 +796,16 @@ static inline void __free_one_page(struct page *page, VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); VM_BUG_ON(migratetype == -1); - if (likely(!is_migrate_isolate(migratetype))) - __mod_zone_freepage_state(zone, 1 << order, migratetype); - VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); VM_BUG_ON_PAGE(bad_range(zone, page), page); + account_freepages(page, zone, 1 << order, migratetype); + while (order < MAX_PAGE_ORDER) { - if (compaction_capture(capc, page, order, migratetype)) { - __mod_zone_freepage_state(zone, -(1 << order), - migratetype); + int buddy_mt = migratetype; + + if (compaction_capture(capc, page, order, migratetype)) return; - } buddy = find_buddy_page_pfn(page, pfn, order, &buddy_pfn); if (!buddy) @@ -783,19 +818,12 @@ static inline void __free_one_page(struct page *page, * pageblock isolation could cause incorrect freepage or CMA * accounting or HIGHATOMIC accounting. */ - int buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); + buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); - if (migratetype != buddy_mt) { - if (!migratetype_is_mergeable(migratetype) || - !migratetype_is_mergeable(buddy_mt)) - goto done_merging; - /* - * Match buddy type. This ensures that - * an expand() down the line puts the - * sub-blocks on the right freelists. - */ - set_pageblock_migratetype(buddy, migratetype); - } + if (migratetype != buddy_mt && + (!migratetype_is_mergeable(migratetype) || + !migratetype_is_mergeable(buddy_mt))) + goto done_merging; } /* @@ -803,9 +831,19 @@ static inline void __free_one_page(struct page *page, * merge with it and move up one order. */ if (page_is_guard(buddy)) - clear_page_guard(zone, buddy, order, migratetype); + clear_page_guard(zone, buddy, order); else - del_page_from_free_list(buddy, zone, order); + __del_page_from_free_list(buddy, zone, order, buddy_mt); + + if (unlikely(buddy_mt != migratetype)) { + /* + * Match buddy type. This ensures that an + * expand() down the line puts the sub-blocks + * on the right freelists. + */ + set_pageblock_migratetype(buddy, migratetype); + } + combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn; @@ -822,10 +860,7 @@ static inline void __free_one_page(struct page *page, else to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); - if (to_tail) - add_to_free_list_tail(page, zone, order, migratetype); - else - add_to_free_list(page, zone, order, migratetype); + __add_to_free_list(page, zone, order, migratetype, to_tail); /* Notify page reporting subsystem of freed page */ if (!(fpi_flags & FPI_SKIP_REPORT_NOTIFY)) @@ -1314,10 +1349,10 @@ static inline void expand(struct zone *zone, struct page *page, * Corresponding page table entries will not be touched, * pages will stay not present in virtual address space */ - if (set_page_guard(zone, &page[size], high, migratetype)) + if (set_page_guard(zone, &page[size], high)) continue; - add_to_free_list(&page[size], zone, high, migratetype); + add_to_free_list(&page[size], zone, high, migratetype, false); set_buddy_order(&page[size], high); } } @@ -1487,7 +1522,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, page = get_page_from_free_area(area, migratetype); if (!page) continue; - del_page_from_free_list(page, zone, current_order); + del_page_from_free_list(page, zone, current_order, migratetype); expand(zone, page, order, current_order, migratetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, pcp_allowed_order(order) && @@ -1527,7 +1562,7 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, * type's freelist. */ static int move_freepages(struct zone *zone, unsigned long start_pfn, - unsigned long end_pfn, int migratetype) + unsigned long end_pfn, int old_mt, int new_mt) { struct page *page; unsigned long pfn; @@ -1549,12 +1584,14 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, VM_BUG_ON_PAGE(page_zone(page) != zone, page); order = buddy_order(page); - move_to_free_list(page, zone, order, migratetype); + + move_to_free_list(page, zone, order, old_mt, new_mt); + pfn += 1 << order; pages_moved += 1 << order; } - set_pageblock_migratetype(pfn_to_page(start_pfn), migratetype); + set_pageblock_migratetype(pfn_to_page(start_pfn), new_mt); return pages_moved; } @@ -1612,7 +1649,7 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, } static int move_freepages_block(struct zone *zone, struct page *page, - int migratetype) + int old_mt, int new_mt) { unsigned long start_pfn, end_pfn; @@ -1620,7 +1657,7 @@ static int move_freepages_block(struct zone *zone, struct page *page, NULL, NULL)) return -1; - return move_freepages(zone, start_pfn, end_pfn, migratetype); + return move_freepages(zone, start_pfn, end_pfn, old_mt, new_mt); } #ifdef CONFIG_MEMORY_ISOLATION @@ -1692,7 +1729,6 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, int migratetype) { unsigned long start_pfn, end_pfn, pfn; - int nr_moved, mt; if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, NULL, NULL)) @@ -1703,11 +1739,9 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, if (pfn != start_pfn) { struct page *buddy = pfn_to_page(pfn); int order = buddy_order(buddy); - int mt = get_pfnblock_migratetype(buddy, pfn); - if (!is_migrate_isolate(mt)) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - del_page_from_free_list(buddy, zone, order); + del_page_from_free_list(buddy, zone, order, + get_pfnblock_migratetype(buddy, pfn)); set_pageblock_migratetype(page, migratetype); split_large_buddy(zone, buddy, pfn, order); return true; @@ -1715,23 +1749,17 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, /* We're the starting block of a larger buddy */ if (PageBuddy(page) && buddy_order(page) > pageblock_order) { - int mt = get_pfnblock_migratetype(page, pfn); int order = buddy_order(page); - if (!is_migrate_isolate(mt)) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, + get_pfnblock_migratetype(page, pfn)); set_pageblock_migratetype(page, migratetype); split_large_buddy(zone, page, pfn, order); return true; } - mt = get_pfnblock_migratetype(page, start_pfn); - nr_moved = move_freepages(zone, start_pfn, end_pfn, migratetype); - if (!is_migrate_isolate(mt)) - __mod_zone_freepage_state(zone, -nr_moved, mt); - else if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, nr_moved, migratetype); + move_freepages(zone, start_pfn, end_pfn, + get_pfnblock_migratetype(page, start_pfn), migratetype); return true; } #endif /* CONFIG_MEMORY_ISOLATION */ @@ -1845,7 +1873,7 @@ steal_suitable_fallback(struct zone *zone, struct page *page, /* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) { - del_page_from_free_list(page, zone, current_order); + del_page_from_free_list(page, zone, current_order, block_type); change_pageblock_range(page, current_order, start_type); expand(zone, page, order, current_order, start_type); return page; @@ -1895,12 +1923,12 @@ steal_suitable_fallback(struct zone *zone, struct page *page, */ if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || page_group_by_mobility_disabled) { - move_freepages(zone, start_pfn, end_pfn, start_type); + move_freepages(zone, start_pfn, end_pfn, block_type, start_type); return __rmqueue_smallest(zone, order, start_type); } single_page: - del_page_from_free_list(page, zone, current_order); + del_page_from_free_list(page, zone, current_order, block_type); expand(zone, page, order, current_order, block_type); return page; } @@ -1970,7 +1998,7 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone) mt = get_pageblock_migratetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ if (migratetype_is_mergeable(mt)) - if (move_freepages_block(zone, page, + if (move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC) != -1) zone->nr_reserved_highatomic += pageblock_nr_pages; @@ -2011,11 +2039,13 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, spin_lock_irqsave(&zone->lock, flags); for (order = 0; order < NR_PAGE_ORDERS; order++) { struct free_area *area = &(zone->free_area[order]); + int mt; page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC); if (!page) continue; + mt = get_pageblock_migratetype(page); /* * In page freeing path, migratetype change is racy so * we can counter several free pages in a pageblock @@ -2023,7 +2053,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * from highatomic to ac->migratetype. So we should * adjust the count once. */ - if (is_migrate_highatomic_page(page)) { + if (is_migrate_highatomic(mt)) { /* * It should never happen but changes to * locking could inadvertently allow a per-cpu @@ -2045,7 +2075,8 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * of pageblocks that cannot be completely freed * may increase. */ - ret = move_freepages_block(zone, page, ac->migratetype); + ret = move_freepages_block(zone, page, mt, + ac->migratetype); /* * Reserving this block already succeeded, so this should * not fail on zone boundaries. @@ -2251,12 +2282,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - if (is_migrate_cma(get_pageblock_migratetype(page))) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, - -(1 << order)); } - - __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); spin_unlock_irqrestore(&zone->lock, flags); return i; @@ -2735,11 +2761,9 @@ int __isolate_free_page(struct page *page, unsigned int order) watermark = zone->_watermark[WMARK_MIN] + (1UL << order); if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) return 0; - - __mod_zone_freepage_state(zone, -(1UL << order), mt); } - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, mt); /* * Set the pageblock if the isolated page is at least half of a @@ -2754,7 +2778,7 @@ int __isolate_free_page(struct page *page, unsigned int order) * with others) */ if (migratetype_is_mergeable(mt)) - move_freepages_block(zone, page, + move_freepages_block(zone, page, mt, MIGRATE_MOVABLE); } } @@ -2839,8 +2863,6 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, return NULL; } } - __mod_zone_freepage_state(zone, -(1 << order), - get_pageblock_migratetype(page)); spin_unlock_irqrestore(&zone->lock, flags); } while (check_new_pages(page, order)); @@ -6714,8 +6736,9 @@ void __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn) BUG_ON(page_count(page)); BUG_ON(!PageBuddy(page)); + VM_WARN_ON(get_pageblock_migratetype(page) != MIGRATE_ISOLATE); order = buddy_order(page); - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, MIGRATE_ISOLATE); pfn += (1 << order); } spin_unlock_irqrestore(&zone->lock, flags); @@ -6765,10 +6788,10 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, current_buddy = page + size; } - if (set_page_guard(zone, current_buddy, high, migratetype)) + if (set_page_guard(zone, current_buddy, high)) continue; - add_to_free_list(current_buddy, zone, high, migratetype); + add_to_free_list(current_buddy, zone, high, migratetype, false); set_buddy_order(current_buddy, high); } } @@ -6794,12 +6817,11 @@ bool take_page_off_buddy(struct page *page) int migratetype = get_pfnblock_migratetype(page_head, pfn_head); - del_page_from_free_list(page_head, zone, page_order); + del_page_from_free_list(page_head, zone, page_order, + migratetype); break_down_buddy_pages(zone, page_head, page, 0, page_order, migratetype); SetPageHWPoisonTakenOff(page); - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, -1, migratetype); ret = true; break; } @@ -6907,7 +6929,7 @@ static bool try_to_accept_memory_one(struct zone *zone) list_del(&page->lru); last = list_empty(&zone->unaccepted_pages); - __mod_zone_freepage_state(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + account_freepages(page, zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); spin_unlock_irqrestore(&zone->lock, flags); @@ -6959,7 +6981,7 @@ static bool __free_unaccepted(struct page *page) spin_lock_irqsave(&zone->lock, flags); first = list_empty(&zone->unaccepted_pages); list_add_tail(&page->lru, &zone->unaccepted_pages); - __mod_zone_freepage_state(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + account_freepages(page, zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES); spin_unlock_irqrestore(&zone->lock, flags); diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 042937d5abe4..914a71c580d8 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -252,7 +252,8 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * Isolating this block already succeeded, so this * should not fail on zone boundaries. */ - WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, migratetype)); + WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, + migratetype)); } else { set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype);