From patchwork Wed Mar 6 04:08:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04ED6C54E49 for ; Wed, 6 Mar 2024 04:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 291CE6B007E; Tue, 5 Mar 2024 23:16:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 241FA6B0080; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06E516B0081; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DF1A36B007E for ; Tue, 5 Mar 2024 23:16:23 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B9EA140261 for ; Wed, 6 Mar 2024 04:16:23 +0000 (UTC) X-FDA: 81865302246.07.31FCF0C Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf15.hostedemail.com (Postfix) with ESMTP id E68FCA0005 for ; Wed, 6 Mar 2024 04:16:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Af1ZDlZE; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf15.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698580; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TANN7sLLWedgpgoKB84Ck0djJTV8XiwLLoov3tglmKE=; b=iWZ4/QVrw+NJ78+qA9wwR5ofnfk8aiaTkLu0gBg1/2HY7J9FdfUvMeaS52bkgkO2ETcjYO mYquPIq8ijzTUCUal/R+6xl3XbxwMYatbJWXYR2uNK1OxH+t9teVUQ44L5afJuVCFQ5Kck 3SqzS5AuuOsz0dgJkCEEkSckqWKb3Lo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Af1ZDlZE; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf15.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698580; a=rsa-sha256; cv=none; b=sKsOEyWvxi8rGaZL5dLdl/1V9AwNsLZebtflCyUngtR6qsVAZ4HuAAcQL1khYFTF9HB8/5 WeB7SJjRLAuDICGQ7FWQFzEHZ9dz0wNSys8+kNb/FwUWZF1HNlzP1bnkBb2hTMPLAHJkBe 6A+S715j3byMrHLNM7p46iYPv0wgp2I= Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7882b02ed2dso122304285a.3 for ; Tue, 05 Mar 2024 20:16:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698580; x=1710303380; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TANN7sLLWedgpgoKB84Ck0djJTV8XiwLLoov3tglmKE=; b=Af1ZDlZEQJu+do5e/LsQsAV4N66IqT38wUTtchclzn79MdWkrB1XsjuOQ4X4F01aUl 8bp84ifc+8c2smHyxo17s+X2uyPTSVHUp3/ahz9TTB2WOtaaMAn0UVrtCOAZCTkAQ3pf Bth1tlO8AxMUiGOtAkSA1FGKT56pxJS6KHkjaPUItF5qCdyfFyPNQkO02fLmPGr0+y7k OScr3dwmSbOYkgTDHgdpdHoVVFzT5g50XY/B2EcCGUhw0jk3GydMuuY9M3zVFazNpQqk u2EGhDy3H37bWZzXjhtqa+tkxfopvDT4+g5KwI9RhUtwv45x9Dg4Bkoojf9uA/mb6t0T yEUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698580; x=1710303380; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TANN7sLLWedgpgoKB84Ck0djJTV8XiwLLoov3tglmKE=; b=w/hsZnx2bplH/PLoDSVI4pNstpG+lKIybLANmYEkmWiAq1tB//q9/0oN8brcec5XVF XH4NpC40g/e17t6FQzUfhAwWY+3wl8h5gPbuQTtTvgGZbxF4B7Mc0C2RuCYDOg96MFq3 FK/ivmu45Mm8+0Eo1ERqT+kX3JmtUFLnnI53RB1rOl1HsEHZX2zMr8qpYxccx/oAr93d 83setqWZ4fvbgzHL/NXuR8ahrBsiROuIUfdgNY6sVt7f9Qp/25ZC74MZtX1O+8Cj2/te 4RXre4pHkMPsUG3fypVZYMam1Q4oy92YScbmvCu2enjUhPhzjSW4HCwtzDqkQIXLx4BC MC4w== X-Forwarded-Encrypted: i=1; AJvYcCXJOGWYLCbdif3dIXly/X6oSfs978OWpu/r9rDHUq34572yL/j3fZemmumv1tITBgb0cHGwhxvPrLiXvrwBY4A+YlQ= X-Gm-Message-State: AOJu0YyUQXcVon0TBIaZMDykN+arKtWEMDAfvPCIdKJtkt91SIT/iyDF xlA1slc3Gn1H4xSL1A9QEl2glyi4gzNBPVkIg7IuknEKRIsc+DdNRqfhG+9lBWw= X-Google-Smtp-Source: AGHT+IHwq/V0fU9mVZstG8iv3aIs4lRIY1sCdiOu2ll0CLJKWGqdfEQ3Aa7QbBpFir2jp2PM7CxgKg== X-Received: by 2002:a05:620a:22b4:b0:787:effe:472 with SMTP id p20-20020a05620a22b400b00787effe0472mr3409431qkh.64.1709698579913; Tue, 05 Mar 2024 20:16:19 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id az17-20020a05620a171100b007882b4cecf8sm2440014qkb.57.2024.03.05.20.16.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:19 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/10] mm: page_alloc: remove pcppage migratetype caching Date: Tue, 5 Mar 2024 23:08:32 -0500 Message-ID: <20240306041526.892167-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E68FCA0005 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: rsesznxzi3wgthquixhmpnkn5igcoz4a X-HE-Tag: 1709698580-606184 X-HE-Meta: U2FsdGVkX19fpessqCvsCvPq9UvziGFiVojUp7DyVQrMo8coxKUB4GNuLYu4hrsQVDx2yEdVtDq+IhIO1v6nPkZjerGlASuMLGvaVdT3byTy1m/UHpL9INM9L2RG8Zn/yhJf70oG5spwwgv61VUukxOXu49JgeC/RWpzvO3nra8I7JOxzrk3LmX1nvwDNhfxIJCG8KWhDwL7668pghcnHg7GSK/hA0QrtTEPvphErMCY7/MqD6RNUYr8jEfHTzYZOj92NCOOnnmFZZD7SyDMaKqqsjYNoU/80IvkOUghb5E+Lo+OeF5VV2ZQ6P76TVv2c46hevmu6P7hsn61hfsKZOrhfMvL2pQ2EW4r753J/sGaww6dlp4LeDf5J2mw+1h3Dva1n47SDNRz0f3BeAhHyxTOyvljaqJJz594XeWz9K4Dwk9rXE62MJeLtxtREFWmo8+l+i+ZlbSKM3NsoUC42b+39bAzcc53Q92n5RxyINdkVNJbWRbDpkxiXuzOexPtvPx5v0oK4OZ/K3wYqKhxgi+DoP1yerqRT0esf0TEfYQ1a9ZLec507grZ3XiIdlzSzL4RcZLlbAR1nlkI1UjZ/xqjLK48mZY7UrVq1odKeIhzNnZwrNoyJVIHhqu4KikutW71hGeTg56TQcpS7NQJARxNmuEeYpRx4huec/cb6XW0EyBjICWALC4aepKU72s9cZv9HSs8z3HdEUda+l34gxU13K4aQry4vvXjHZsosoCe5XNZ9zMhk9f4WBwrNOZvVeTNywiHENbWSN1NNEQeyqQgfv7c5bf3ddIJdP86c0Mx9ulGN8/wu5D2dz9glm7PnuXx1SmuqRQvQBmxcAsKmN1UZbByUh3SinOsXD3XY3vKUk/C0RFrnmVyUfznw8DM+3tTMtFVzivDqdU8MXsU3ea48VRnKMFNk22AN6NrivHSZP6ONjOAP2sxGkX6QJg8Q/Z5GrOHk5Uh5gn49tL 5GI9LxjJ Vk6IUMRpziWkRezbvXT/G3JyFXKjz7sEE/A9/Y17HL3vBwM10iCrHMUmfWTqxYkit23tX1bpNApJqTHT9gs+UfI3cnW8QQA9niI0PL8sSOHMaRDv9KDjE2GbBFgl3B/nlb34eY7SgcI25rRuUCJzzjQDevT2EofvyrOEDruxUiDKLNEsKgCG1wMdWJWLXx9LAfGA/OYd0GXvuv0homfNL8ihgl1B+TSqP3NF/Ojw64+JKOpL2kJdkkMC2Notjc7sArTJKGmx/+7AeI3TfHW9plb4oLOTL8v1DojWiHaZP73Vky/n4SdR70jK6IuPyRWNYOrOB7cZmmNuVVDHIkSmqMYn3Xmh9bZmC6kdFWffihqN0lAIcdTWcZT5CsixKkiKPc0enTcQFX1dkXne3iLbiFsnJCA4RLf3WpDCYPYgvmyq+ydQVkX27BMAGQd36mGP0sNzmIxy6QkGocbk1BifvwelhEA+j0N0ju7+0w7e83XEhiaO5oVK2xejWbE0eHPiTzJjf1ngY2CS97ZZmAwK8Qe59H60VpzlzE/vHdhCddUCtBSLnIN7GQ7EbOeBjocwqTNgNvOUg9OQCbWx/CTRnkonG3IoumYcB6S7r X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The idea behind the cache is to save get_pageblock_migratetype() lookups during bulk freeing. A microbenchmark suggests this isn't helping, though. The pcp migratetype can get stale, which means that bulk freeing has an extra branch to check if the pageblock was isolated while on the pcp. While the variance overlaps, the cache write and the branch seem to make this a net negative. The following test allocates and frees batches of 10,000 pages (~3x the pcp high marks to trigger flushing): Before: 8,668.48 msec task-clock # 99.735 CPUs utilized ( +- 2.90% ) 19 context-switches # 4.341 /sec ( +- 3.24% ) 0 cpu-migrations # 0.000 /sec 17,440 page-faults # 3.984 K/sec ( +- 2.90% ) 41,758,692,473 cycles # 9.541 GHz ( +- 2.90% ) 126,201,294,231 instructions # 5.98 insn per cycle ( +- 2.90% ) 25,348,098,335 branches # 5.791 G/sec ( +- 2.90% ) 33,436,921 branch-misses # 0.26% of all branches ( +- 2.90% ) 0.0869148 +- 0.0000302 seconds time elapsed ( +- 0.03% ) After: 8,444.81 msec task-clock # 99.726 CPUs utilized ( +- 2.90% ) 22 context-switches # 5.160 /sec ( +- 3.23% ) 0 cpu-migrations # 0.000 /sec 17,443 page-faults # 4.091 K/sec ( +- 2.90% ) 40,616,738,355 cycles # 9.527 GHz ( +- 2.90% ) 126,383,351,792 instructions # 6.16 insn per cycle ( +- 2.90% ) 25,224,985,153 branches # 5.917 G/sec ( +- 2.90% ) 32,236,793 branch-misses # 0.25% of all branches ( +- 2.90% ) 0.0846799 +- 0.0000412 seconds time elapsed ( +- 0.05% ) A side effect is that this also ensures that pages whose pageblock gets stolen while on the pcplist end up on the right freelist and we don't perform potentially type-incompatible buddy merges (or skip merges when we shouldn't), which is likely beneficial to long-term fragmentation management, although the effects would be harder to measure. Settle for simpler and faster code as justification here. v2: - remove erroneous leftover VM_BUG_ON in pcp bulk freeing (Mike) Acked-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 66 +++++++++++-------------------------------------- 1 file changed, 14 insertions(+), 52 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 16241906a368..9665a2db3b53 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -206,24 +206,6 @@ EXPORT_SYMBOL(node_states); gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -/* - * A cached value of the page's pageblock's migratetype, used when the page is - * put on a pcplist. Used to avoid the pageblock migratetype lookup when - * freeing from pcplists in most cases, at the cost of possibly becoming stale. - * Also the migratetype set in the page does not necessarily match the pcplist - * index, e.g. page might have MIGRATE_CMA set but be on a pcplist with any - * other index - this ensures that it will be put on the correct CMA freelist. - */ -static inline int get_pcppage_migratetype(struct page *page) -{ - return page->index; -} - -static inline void set_pcppage_migratetype(struct page *page, int migratetype) -{ - page->index = migratetype; -} - #ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE unsigned int pageblock_order __read_mostly; #endif @@ -1191,7 +1173,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, { unsigned long flags; unsigned int order; - bool isolated_pageblocks; struct page *page; /* @@ -1204,7 +1185,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, pindex = pindex - 1; spin_lock_irqsave(&zone->lock, flags); - isolated_pageblocks = has_isolate_pageblock(zone); while (count > 0) { struct list_head *list; @@ -1220,23 +1200,19 @@ static void free_pcppages_bulk(struct zone *zone, int count, order = pindex_to_order(pindex); nr_pages = 1 << order; do { + unsigned long pfn; int mt; page = list_last_entry(list, struct page, pcp_list); - mt = get_pcppage_migratetype(page); + pfn = page_to_pfn(page); + mt = get_pfnblock_migratetype(page, pfn); /* must delete to avoid corrupting pcp list */ list_del(&page->pcp_list); count -= nr_pages; pcp->count -= nr_pages; - /* MIGRATE_ISOLATE page should not go to pcplists */ - VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); - /* Pageblock could have been isolated meanwhile */ - if (unlikely(isolated_pageblocks)) - mt = get_pageblock_migratetype(page); - - __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE); + __free_one_page(page, pfn, zone, order, mt, FPI_NONE); trace_mm_page_pcpu_drain(page, order, mt); } while (count > 0 && !list_empty(list)); } @@ -1575,7 +1551,6 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, continue; del_page_from_free_list(page, zone, current_order); expand(zone, page, order, current_order, migratetype); - set_pcppage_migratetype(page, migratetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, pcp_allowed_order(order) && migratetype < MIGRATE_PCPTYPES); @@ -2182,7 +2157,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - if (is_migrate_cma(get_pcppage_migratetype(page))) + if (is_migrate_cma(get_pageblock_migratetype(page))) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(1 << order)); } @@ -2375,19 +2350,6 @@ void drain_all_pages(struct zone *zone) __drain_all_pages(zone, false); } -static bool free_unref_page_prepare(struct page *page, unsigned long pfn, - unsigned int order) -{ - int migratetype; - - if (!free_pages_prepare(page, order)) - return false; - - migratetype = get_pfnblock_migratetype(page, pfn); - set_pcppage_migratetype(page, migratetype); - return true; -} - static int nr_pcp_free(struct per_cpu_pages *pcp, int batch, int high, bool free_high) { int min_nr_free, max_nr_free; @@ -2520,7 +2482,7 @@ void free_unref_page(struct page *page, unsigned int order) unsigned long pfn = page_to_pfn(page); int migratetype, pcpmigratetype; - if (!free_unref_page_prepare(page, pfn, order)) + if (!free_pages_prepare(page, order)) return; /* @@ -2530,7 +2492,7 @@ void free_unref_page(struct page *page, unsigned int order) * get those areas back if necessary. Otherwise, we may have to free * excessively into the page allocator */ - migratetype = pcpmigratetype = get_pcppage_migratetype(page); + migratetype = pcpmigratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); @@ -2569,14 +2531,14 @@ void free_unref_folios(struct folio_batch *folios) if (order > 0 && folio_test_large_rmappable(folio)) folio_undo_large_rmappable(folio); - if (!free_unref_page_prepare(&folio->page, pfn, order)) + if (!free_pages_prepare(&folio->page, order)) continue; /* * Free isolated folios and orders not handled on the PCP * directly to the allocator, see comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(&folio->page); + migratetype = get_pfnblock_migratetype(&folio->page, pfn); if (!pcp_allowed_order(order) || is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, @@ -2593,10 +2555,11 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; folio->private = NULL; - migratetype = get_pcppage_migratetype(&folio->page); + migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ if (zone != locked_zone) { @@ -2613,9 +2576,8 @@ void free_unref_folios(struct folio_batch *folios) pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, &folio->page, - folio_pfn(folio), order, - migratetype, FPI_NONE); + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2784,7 +2746,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, } } __mod_zone_freepage_state(zone, -(1 << order), - get_pcppage_migratetype(page)); + get_pageblock_migratetype(page)); spin_unlock_irqrestore(&zone->lock, flags); } while (check_new_pages(page, order));