From patchwork Wed Mar 20 02:42:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kaiyang Zhao X-Patchwork-Id: 13597223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30FAEC54E68 for ; Wed, 20 Mar 2024 02:42:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A86966B008C; Tue, 19 Mar 2024 22:42:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A33FB6B0093; Tue, 19 Mar 2024 22:42:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BD306B0095; Tue, 19 Mar 2024 22:42:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7A12B6B008C for ; Tue, 19 Mar 2024 22:42:27 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4B5371A103E for ; Wed, 20 Mar 2024 02:42:27 +0000 (UTC) X-FDA: 81915868734.03.8C67320 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf14.hostedemail.com (Postfix) with ESMTP id 76E0D10000B for ; Wed, 20 Mar 2024 02:42:25 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=bmxKdpLW; spf=pass (imf14.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.45 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710902545; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Yh8JatHXgZIgb1eyxT1DhEdr2aiRbmcRrkpZMUB4hHw=; b=epbBr9VJ4xDSBEayAc64+Ef5gfYYyeFuT1yRX3M4kamXAdePF5E0O74W9vSUY7dukHkmhj yFMfwqkjyCvjbe/kII6nZktIY3GuyTnT082Y87nlO98K2N7M9U/BPISWHW5fQy4FLmFLGr slraGqfqpAREo1TwrDnX9+TcPx9mQLU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=bmxKdpLW; spf=pass (imf14.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.45 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710902545; a=rsa-sha256; cv=none; b=Q2Su1nuezzpCROpe7IJZW9bdq2/ppeEELRvJysWh/uvjZeg3C7F6mbRHsyIKrMYNs1/WPz 5vu4axPJgE5VbQ5GiJ+ZGT6r1oGgji+gGNtFmrd9n46YGwMQELhiPeUmRJzsMf106mlOBJ kT0gAohriU1MNAaAXBqnvxH182uzfmA= Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-6963c0c507eso5139086d6.1 for ; Tue, 19 Mar 2024 19:42:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.cmu.edu; s=google-2021; t=1710902544; x=1711507344; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Yh8JatHXgZIgb1eyxT1DhEdr2aiRbmcRrkpZMUB4hHw=; b=bmxKdpLWpsHtAhyqW7RYpRwqPFO6zU452FsDtQQ/+meP304u/kVPzXriaeUrGx51ub XOfVLtOazuXrTkEe5st0HGz828XBWET4F+2DrTjlc4v2jMzCyEYYu49b5wFG8SseOSBC MERWeZGS81dPWH3LGqsFDVf+f9VuR+z1SiTEQAWYOy+Wk294a6bpZTdm5vvyjIi+AHfk eobOyUVsJQxZ/lpvS/7dDHqx5Z/VJXJ/3OY05h2rMFN0/tqsP/SuoEtm3jzcdQg25uyb u+SpgQ2tyO9NHfG2BjJOrXKY+ZCPVeRTaBv/Xn9nQBbGvmpXOLesO+Al5kK2AodCtH94 nubg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710902544; x=1711507344; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Yh8JatHXgZIgb1eyxT1DhEdr2aiRbmcRrkpZMUB4hHw=; b=EdMyj1lpGlGBQCo/Xjjz3/PrP3aAa2EydPPDOCKcI150LzLIdc9rQcQMDMhrz/ucM5 E3nANgo2x9ca5IOfFXdvStdjURully+jYo943G6LGD+zAiS+xVpsmeyCJXPPudv4/5J7 3m4eepVhCqzi7QNFt3wcSmNk9tdquawvOOteaTxLZx80ZVBxdotTJiWk1DSi3BEoRaZ9 o9ksUKXWMPnNkkk35w1DUpqdKiyXH2pQmC9UBijZKtKTCX+4ZoYKU3tOr++Z0laFoReu SIqHzlBP7xYyvrQJ1SNAeoJTt8lioiCANdtcpvP+Qu99PWR04hvvZfRCyjTvZvYEq/eS pWcg== X-Gm-Message-State: AOJu0YyquqnZ7xHbrEAdBCboLetu5sV+8USUQSesnpg/QYbvpRPzY1cK jKg1A8lCxUiAFcNEYMXrFGaHq2WE3to+gJGB+pRE156fcamNGdLYnnbSWzf9dSBht4r+k4YJcDv 5wDwoviYaWGpOJPZyY1SMLOMOsKGbnmiVCvAeUSyjhwKcJochsP2WttLxPBdi78ctKdKYTZoteT bNcJn5N2luviOpdo3owP+NUd0pgwoqqTy8b7U= X-Google-Smtp-Source: AGHT+IGR8vrTWY1WaI5oE4IRdj4GRFNX5p3VLQAJ1dsWkq6eebPZOQskjAK08wq1ThkjZxRjS5rZPw== X-Received: by 2002:ad4:430b:0:b0:691:4d1f:6a65 with SMTP id c11-20020ad4430b000000b006914d1f6a65mr2449565qvs.27.1710902544204; Tue, 19 Mar 2024 19:42:24 -0700 (PDT) Received: from localhost (pool-74-98-221-57.pitbpa.fios.verizon.net. [74.98.221.57]) by smtp.gmail.com with UTF8SMTPSA id k12-20020ad45bec000000b006961199b96asm3707324qvc.13.2024.03.19.19.42.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 19 Mar 2024 19:42:23 -0700 (PDT) From: kaiyang2@cs.cmu.edu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Kaiyang Zhao , hannes@cmpxchg.org, ziy@nvidia.com, dskarlat@cs.cmu.edu Subject: [RFC PATCH 5/7] proactively move pages out of unmovable zones in kcompactd Date: Wed, 20 Mar 2024 02:42:16 +0000 Message-Id: <20240320024218.203491-6-kaiyang2@cs.cmu.edu> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240320024218.203491-1-kaiyang2@cs.cmu.edu> References: <20240320024218.203491-1-kaiyang2@cs.cmu.edu> Reply-To: Kaiyang Zhao MIME-Version: 1.0 X-Rspamd-Queue-Id: 76E0D10000B X-Rspam-User: X-Stat-Signature: c9o49syu5qeeot8s6sr8m5w18wga8o74 X-Rspamd-Server: rspam01 X-HE-Tag: 1710902545-554266 X-HE-Meta: U2FsdGVkX1/N58zxWFXReE6cJFayl1TArHx5twXQJenmcl5dKevBtmhJFgCypXpl1PKEnzLfagRgyW/inFg/kcDs6t1fENTA66rIcRPJiC/arFMExL74TYZgkOH8YAPFwJEXqgupbWxdgmLwDQ2OGCT7bHijYU3O110iHS7e+gCAeMe023kK+DC1t85S3+3yLq8S5G5sB4spc0Ij31MmRbDyBFFc6rLelqxgX0F7BHOGLJ9LWe8XyeuY6w60z5KWxECsS96exfNhzyn0xe9/C9BfsRIGqWWRYPWyXGZ8qtHVa9UbbIjxm+QCB95HOVOFcX1iMFTAREy9zUbQQeK9PFigvVreyLEWH7Gd4IgCNYlN5QC/sz5zMJ05K5fD1rLpNVlaluyBf9vIAq9pRYI/BMGxtdVevLRmyxbfJMshuIMc5E83TeVMpZxHOy/kOrRuNml4dezTVTlpPUP+67t8+gle0Nikfm+YCaKbzyvlmTSnL3XR5Uh0mE9g9m16PPbaJ0EPe9G67UPooSeAD2TeFYWUunuW8+K6c1H3w3VinDwbNMfE7/cyDjmrNQmJXUTME5HKQV2JgMa6eltwZlnKRdzgPgu8Lc5LgUo7gJeunF0PpL/PlAXoSH6qdT56Yt/zVASoHNyBIthg7lxJre/vOllg0bo7b06UGuT5HDBM+W8eQhZAfIfHu2E7F0bWjgr1mUCCw6w04G9dCcg6CGDWsJ8IbdhE6St0HorEV3O4sceIBdGPVmGaDQmFXAf4KxCHMTPsNs0fHolYavbCp+PQ4w+BKhmCPuVh3UqEyXpGq5LCtLafpyet5pcPpS2sUsDEy7NPFN/VeD1XS271J+qv3vxIaF5WqwFxtCGTjDH4h8nG19ezVJe7L65aU/EOCS4TfLoZd85meNlldKfkVtkxbElUVFa6gnY24F7ZV+hpEixe4IFrT94umqEHBgqVH882TL3z3w07HwkPezybpXX dObYn5se Ep5Zg4w/QUCCeYjSz/Cd8+Nx0AdJdcdeg8DOnzrGVzylNJYjeBowkEczh7K9cTXhlusvH92naoNoTkHeB+7vz0zRarDFcNF1g6dnexgCxAdXNXxWoQPU039E59ch9Mm/zNscoeLduQ/DtRMfnZekZ+LpZz7GfrxxgsnMbCFvR+dzkTX/ImetYnisFPtI5qXnArJ+MXwCgqMftxgs0ZrKxa8aFUwISyWyTaUdsxBt/AuXF94TiUTxKtluZulRdldP2ecdcA6hkQ5lPnBbUSHATFuT+bXFXzBV8RW3Db9F56gXHfreusz1kRHFh9UjCXh44OXJq4WFJX5/gddzphdX1E0jZwd0dFdQyGRoy3UuLIUtXP7F9oyLJoFSpfLrYcwfO5qqnGFaDBGjVVO3tw70dXvVhLf+mv68Rgjcl3Vfh8N1bBRqLzDnZZVMe5XXQqUT/IdMx2mv0+Zx7dbR3pb4bDNwA9O/Fwc5yA4YhKSM0ObOb8dpwdStGe0hSEJfKS/mprNYTeOVO+IuGWMkfXipwmUAoovuxq/n3HC03eDh3uI5Xmt2Uvl4J1jg/Nnj3zStK8KIN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kaiyang Zhao Proactively move pages out of unmovable zones in kcompactd Debug only: zone start and end pfn printed in vmstat Added counters for cross zone compaction start and scan Signed-off-by: Kaiyang Zhao --- include/linux/vm_event_item.h | 3 + mm/compaction.c | 101 +++++++++++++++++++++++++++++++--- mm/vmstat.c | 11 +++- 3 files changed, 104 insertions(+), 11 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index be88819085b6..c9183117c8f7 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -80,6 +80,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KCOMPACTD_WAKE, KCOMPACTD_MIGRATE_SCANNED, KCOMPACTD_FREE_SCANNED, COMPACT_CROSS_ZONE_MIGRATED, + KCOMPACTD_CROSS_ZONE_START, + COMPACT_CROSS_ZONE_MIGRATE_SCANNED, + COMPACT_CROSS_ZONE_FREE_SCANNED, #endif #ifdef CONFIG_HUGETLB_PAGE HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL, diff --git a/mm/compaction.c b/mm/compaction.c index dea10ad8ec64..94ce1282f17b 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1436,7 +1436,10 @@ fast_isolate_freepages(struct compact_control *cc) * Preferred point is in the top quarter of the scan space but take * a pfn from the top half if the search is problematic. */ - distance = (cc->free_pfn - cc->migrate_pfn); + if (cc->zone != dst_zone) + distance = (cc->free_pfn - dst_zone->zone_start_pfn) >> 1; + else + distance = (cc->free_pfn - cc->migrate_pfn); low_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 2)); min_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 1)); @@ -1602,7 +1605,10 @@ static void isolate_freepages(struct compact_control *cc) block_start_pfn = pageblock_start_pfn(isolate_start_pfn); block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); - low_pfn = pageblock_end_pfn(cc->migrate_pfn); + if (cc->dst_zone && cc->zone != cc->dst_zone) + low_pfn = pageblock_end_pfn(cc->dst_zone->zone_start_pfn); + else + low_pfn = pageblock_end_pfn(cc->migrate_pfn); stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1; /* @@ -1822,7 +1828,11 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) * within the first eighth to reduce the chances that a migration * target later becomes a source. */ - distance = (cc->free_pfn - cc->migrate_pfn) >> 1; + if (cc->dst_zone && cc->zone != cc->dst_zone) + distance = (zone_end_pfn(cc->zone) - cc->migrate_pfn) >> 1; + else + distance = (cc->free_pfn - cc->migrate_pfn) >> 1; + if (cc->migrate_pfn != cc->zone->zone_start_pfn) distance >>= 2; high_pfn = pageblock_start_pfn(cc->migrate_pfn + distance); @@ -1897,7 +1907,7 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) { unsigned long block_start_pfn; unsigned long block_end_pfn; - unsigned long low_pfn; + unsigned long low_pfn, high_pfn; struct page *page; const isolate_mode_t isolate_mode = (sysctl_compact_unevictable_allowed ? ISOLATE_UNEVICTABLE : 0) | @@ -1924,11 +1934,16 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) /* Only scan within a pageblock boundary */ block_end_pfn = pageblock_end_pfn(low_pfn); + if (cc->dst_zone && cc->zone != cc->dst_zone) + high_pfn = zone_end_pfn(cc->zone); + else + high_pfn = cc->free_pfn; + /* * Iterate over whole pageblocks until we find the first suitable. * Do not cross the free scanner. */ - for (; block_end_pfn <= cc->free_pfn; + for (; block_end_pfn <= high_pfn; fast_find_block = false, cc->migrate_pfn = low_pfn = block_end_pfn, block_start_pfn = block_end_pfn, @@ -1954,6 +1969,7 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) * before making it "skip" so other compaction instances do * not scan the same block. */ + if (pageblock_aligned(low_pfn) && !fast_find_block && !isolation_suitable(cc, page)) continue; @@ -1976,6 +1992,10 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) isolate_mode)) return ISOLATE_ABORT; + /* free_pfn may have changed. update high_pfn. */ + if (!cc->dst_zone || cc->zone == cc->dst_zone) + high_pfn = cc->free_pfn; + /* * Either we isolated something and proceed with migration. Or * we failed and compact_zone should decide if we should @@ -2141,7 +2161,9 @@ static enum compact_result __compact_finished(struct compact_control *cc) goto out; } - if (is_via_compact_memory(cc->order)) + /* Don't check if a suitable page is free if doing cross zone compaction. */ + if (is_via_compact_memory(cc->order) || + (cc->dst_zone && cc->dst_zone != cc->zone)) return COMPACT_CONTINUE; /* @@ -2224,7 +2246,8 @@ static enum compact_result __compaction_suitable(struct zone *zone, int order, * should be no need for compaction at all. */ if (zone_watermark_ok(zone, order, watermark, highest_zoneidx, - alloc_flags)) + alloc_flags) && + dst_zone == zone) return COMPACT_SUCCESS; /* @@ -2270,6 +2293,11 @@ enum compact_result compaction_suitable(struct zone *zone, int order, ret = __compaction_suitable(zone, order, alloc_flags, highest_zoneidx, zone_page_state(dst_zone, NR_FREE_PAGES), dst_zone); + + /* Allow migrating movable pages to ZONE_MOVABLE regardless of frag index */ + if (ret == COMPACT_CONTINUE && dst_zone != zone) + return ret; + /* * fragmentation index determines if allocation failures are due to * low memory or external fragmentation @@ -2841,6 +2869,14 @@ void compaction_unregister_node(struct node *node) } #endif /* CONFIG_SYSFS && CONFIG_NUMA */ +static inline bool should_compact_unmovable_zones(pg_data_t *pgdat) +{ + if (populated_zone(&pgdat->node_zones[ZONE_MOVABLE])) + return true; + else + return false; +} + static inline bool kcompactd_work_requested(pg_data_t *pgdat) { return pgdat->kcompactd_max_order > 0 || kthread_should_stop() || @@ -2942,6 +2978,48 @@ static void kcompactd_do_work(pg_data_t *pgdat) pgdat->kcompactd_highest_zoneidx = pgdat->nr_zones - 1; } +static void kcompactd_clean_unmovable_zones(pg_data_t *pgdat) +{ + int zoneid; + struct zone *zone; + struct compact_control cc = { + .order = 0, + .search_order = 0, + .highest_zoneidx = ZONE_MOVABLE, + .mode = MIGRATE_SYNC, + .ignore_skip_hint = true, + .gfp_mask = GFP_KERNEL, + .dst_zone = &pgdat->node_zones[ZONE_MOVABLE], + .whole_zone = true + }; + count_compact_event(KCOMPACTD_CROSS_ZONE_START); + + for (zoneid = 0; zoneid < ZONE_MOVABLE; zoneid++) { + int status; + + zone = &pgdat->node_zones[zoneid]; + if (!populated_zone(zone)) + continue; + + if (compaction_suitable(zone, cc.order, 0, zoneid, cc.dst_zone) != + COMPACT_CONTINUE) + continue; + + if (kthread_should_stop()) + return; + + /* Not participating in compaction defer. */ + + cc.zone = zone; + status = compact_zone(&cc, NULL); + + count_compact_events(COMPACT_CROSS_ZONE_MIGRATE_SCANNED, + cc.total_migrate_scanned); + count_compact_events(COMPACT_CROSS_ZONE_FREE_SCANNED, + cc.total_free_scanned); + } +} + void wakeup_kcompactd(pg_data_t *pgdat, int order, int highest_zoneidx) { if (!order) @@ -2994,9 +3072,10 @@ static int kcompactd(void *p) /* * Avoid the unnecessary wakeup for proactive compaction - * when it is disabled. + * and cleanup of unmovable zones + * when they are disabled. */ - if (!sysctl_compaction_proactiveness) + if (!sysctl_compaction_proactiveness && !should_compact_unmovable_zones(pgdat)) timeout = MAX_SCHEDULE_TIMEOUT; trace_mm_compaction_kcompactd_sleep(pgdat->node_id); if (wait_event_freezable_timeout(pgdat->kcompactd_wait, @@ -3017,6 +3096,10 @@ static int kcompactd(void *p) continue; } + /* Migrates movable pages out of unmovable zones if ZONE_MOVABLE exists */ + if (should_compact_unmovable_zones(pgdat)) + kcompactd_clean_unmovable_zones(pgdat); + /* * Start the proactive work with default timeout. Based * on the fragmentation score, this timeout is updated. diff --git a/mm/vmstat.c b/mm/vmstat.c index 98af82e65ad9..444740605f2f 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1325,6 +1325,9 @@ const char * const vmstat_text[] = { "compact_daemon_migrate_scanned", "compact_daemon_free_scanned", "compact_cross_zone_migrated", + "compact_cross_zone_start", + "compact_cross_zone_migrate_scanned", + "compact_cross_zone_free_scanned", #endif #ifdef CONFIG_HUGETLB_PAGE @@ -1692,7 +1695,9 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, "\n spanned %lu" "\n present %lu" "\n managed %lu" - "\n cma %lu", + "\n cma %lu" + "\n start %lu" + "\n end %lu", zone_page_state(zone, NR_FREE_PAGES), zone->watermark_boost, min_wmark_pages(zone), @@ -1701,7 +1706,9 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, zone->spanned_pages, zone->present_pages, zone_managed_pages(zone), - zone_cma_pages(zone)); + zone_cma_pages(zone), + zone->zone_start_pfn, + zone_end_pfn(zone)); seq_printf(m, "\n protection: (%ld",