From patchwork Tue Apr 18 19:13:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC60EC77B78 for ; Tue, 18 Apr 2023 19:14:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1D9C28000A; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCB85280001; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A449928000A; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 93278280001 for ; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 72FB28021F for ; Tue, 18 Apr 2023 19:13:51 +0000 (UTC) X-FDA: 80695461462.04.DACFD22 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf05.hostedemail.com (Postfix) with ESMTP id A9F8E10000B for ; Tue, 18 Apr 2023 19:13:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KFvVKf3Q; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3kWGHSuKTgkf/KcYFfj9Qt8QOcgzsovuy8ycEW8Qbl8=; b=64q1eAGIMoGl5+j+i9CN/f+1A67QqCU1EzH3Lq2TpDsCZRXDtCsgY0wyG1oC6uvh3kd2DS 1QBvlen62375+QeKYvI5PUBOYtBIfI4idVOrvh3Z1NQhukBYeNeCmsxMsCzLQYv1La9Z66 Uy3tk7uQ++YORvtX2BNrjMH9tVst+m4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KFvVKf3Q; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845229; a=rsa-sha256; cv=none; b=06Jn0FcCZEMMmnheafsbOVKURgiiU1pvLpmNZ8qGeE4xAMUw9O9BnoIP2urG6KWRaPSj9L bSDd0LCnm6p+uTsAZ/oukNK04fdqhNYL4Qw6/4ayeYf+CdzQiba3q9pMRl5SNqHe4A+bI2 FReAwjZ3Sm+uIuKcQliJyqT3QgK+66E= Received: by mail-qt1-f173.google.com with SMTP id ey8so6190226qtb.3 for ; Tue, 18 Apr 2023 12:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845228; x=1684437228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3kWGHSuKTgkf/KcYFfj9Qt8QOcgzsovuy8ycEW8Qbl8=; b=KFvVKf3QkfWQMBDUHAZr4VeBeoisE8O1GGsOodjwgMNuADxrQdnvk+SbprdpW2IP5r ag93cZ6VTR9I76TSMGwf5woII5HGfN9l2WN88UYR1tzcnTBufAlpM3lp/xMmNVG97M43 eNnVONsHRxzcPVv/iZgrUj1Uz8RGQ4JjzKnpXZZNAJj/FsvWfEp9h8m3xbiQ4JGMIzXh Wd0SEOETtzw6tJD+77T1b7ruvzZOu+Nz8JErUfQPxnwB2Zm75Cbp78ju67EmWgaNaQRd j09JTsw/mPs41GthMRykMfarC7ahjfxAE2/z8L+r1gArqpDDy8cskY5OjwSOMNK5kLZu 3VpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845228; x=1684437228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3kWGHSuKTgkf/KcYFfj9Qt8QOcgzsovuy8ycEW8Qbl8=; b=FYy+QOptDCUIYkdrlMtU5iMP8fs9qvMfMJ5GpKQi0n1tb9rtrj/ATN4vfnOqQbyfrG mGmv6L8bCBG+9j/4B8QKota7Cy/saYkDXF8GeI/moL0JxxnGLihSLlYkB8PEiGGzZAmn ZH91pdE7x1CAiq7cAg8hf4fliMSdjUGsKcB3EYikRrbsn6gE0WVz7JHPRfh3nU6MOvMp NvaCFQW0OrKfyF5/TP1g7/KMgVCnZ3ATAIREnSUfH6djEpqbrgZYlPdoWqovEyYpFa4m eb72+VBYQEDjeu6EyvPvgM4Fed9wA9alL4eOXa7qn2o01/+rMRZgtxDDxim9C4Z7zzQV Hkyw== X-Gm-Message-State: AAQBX9eBRWIo7Uz9sQyIcX2RJ4COQeGNV7++cxlF3bTQSNWr4gaEp5SY 82hzKJg+/7uKVFso8vZm6qfwPIfIxV1fCe9yfWI= X-Google-Smtp-Source: AKy350ZnZudpfigs0VHBHaysAID+48G8C5CQkmg6KTT32vxOaKM82OTa8s5aOXsXXAzcGRMb+jKq5A== X-Received: by 2002:ac8:5f0b:0:b0:3e6:4069:9136 with SMTP id x11-20020ac85f0b000000b003e640699136mr1124473qta.45.1681845228420; Tue, 18 Apr 2023 12:13:48 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id x6-20020ac84d46000000b003ef231cceeasm1122594qtv.23.2023.04.18.12.13.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:48 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 25/26] mm: page_alloc: disallow fallbacks when 2M defrag is enabled Date: Tue, 18 Apr 2023 15:13:12 -0400 Message-Id: <20230418191313.268131-26-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Stat-Signature: 6se4p1yeqgd5fc6xo8asqyf76cqzrsrd X-Rspam-User: X-Rspamd-Queue-Id: A9F8E10000B X-Rspamd-Server: rspam06 X-HE-Tag: 1681845229-387765 X-HE-Meta: U2FsdGVkX19jRxbaVibv87PlpG4pa/O4M98Y/OBoW3W4Ly9AmSiwuVodoIsDil9fpzqH/wqPYaUxiTvDgEVtA6Jk+P/tBfwagbBHj9jHwVjDlxRTlSo9ZI+J7V2R1lGV4XS73DoMQd0rFmouRhR7zqUom2AGTyPgufIn2kDqo71s6n5MJoXamamPGL+dNGFCjaze+vf/+ozzRQXYk2bZHSjcgjXoj+mXM0Bm8jcHLWC/KfBHaJAGLUNbEVVHjCt/o5E11lotN0gckACeqyZNcz6oK944hlDDtmWMbB8yuVs3fk6jiN7P/vy7bePD+6oqxpAdeyMB+gsE68/dvbUfh4swuBQMCUM/xbc4pO529Pbvf6wyMTv+UHOeQuIUGtS39mohFYeCoU6dO6RS0qlBbqAnbxRv6wet4qFax7CPrduhZuz5xAZcdwwl1WdVb5tYL36KE0iaYpqciO9u7KwS6xVsbsguJ81OJoQcxGbErb6+I9Sap4yUpvuaIQ4ZG0G5RN9pNF6JFDr0mkpRi6GDbVFOpd8BoazrT+m9dsleYNVkBigE421OVxtahOzZwRMfsbh6ohGu0MKMuXXxqGaFRGpH/6uxcSfNyIC3aydyG2DdskaW0I+/ZVMiLthA8gCYQv45fUIPUstMa+7RoeZqFDR8PXiJZWDXh7m8J2gCb/A16Q5wvIRyaGaEXUIm3Zn0C+E8NwsV+lkNoq/6W/EDpZBdFijSp3PbOi5IfAfJmhYvlOolFXTkRID8blCMkCI5U1lEEhm9Lti0MIDTYuKfpuqtjuJSbZAlp+ukenxgRBK47dSABqtSyIseKTceeyvB6j7G1UVK2b33WndcCDDvxXNarcT5Vz8u+yojlSWaBgPwklhHdN4cbuRMeOBf79LcTgslBBaq1p5YyBjX29aoLG7vSNJSCZFVKN+Uw92r/BuSoyIfUs+JmpHHPH0Fl6uyWWM3+UHfQK7hz2jIYDC YoNc1DoM Bosl1EHgmsIcbSThuZJjFeI+4iXXNCIgYXnI+CrSqOmN16/HQ6xOFhnau4jr9YMqHfLBOkjZAYRRxC1dNKP4TOznD9VSut8tzplvZKBXlzPFJW0imbMJjDglpS4Pueb4jm1TmrK5z46DhU0FoTTMoooZxucYeTVZcj0Th0EdOGvkcDrQkofQE7i6Nr2ANfojgYB/7hX5TdTHWT88gzsYnyXQkNdrOoD+jhwKvllcwR5+2GFEDSgZFBj2CXPQ1w8JmkAqE2IBnLGgPE4uT8KkfjA4LjMCSseOHTvuEduTrOHpKVgfd1Sf3eYy1zy/BP9R7b002yJOvLk9BvrVvHT8zB+gkjG4/z/k117a8SYwl3KPaxd7XR3I/j+CfbsDcQ58b0TNk+FPpp01NAr8CpiqRS/YzgnS/+Zvgd8p6xVbo4/ijOwQbZDAmtFTxaPEPcT+VJFSyNZ5SvrQlSixhheXLOMMVWuDS6z0F7U4cQRWgOko52eTfwcnWUaaL/rV2SynUV34aEG6A5myhrVKsV/eE6+joD1NT3EKCSs7I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fallbacks are already unlikely due to watermarks being enforced against MIGRATE_FREE blocks. Eliminate them altogether. This allows compaction to look exclusively at movable blocks, reducing the number of pageblocks it needs to scan on an ongoing basis. Signed-off-by: Johannes Weiner --- mm/compaction.c | 52 +++++-------------------------------------------- mm/internal.h | 2 +- mm/page_alloc.c | 8 ++++++++ 3 files changed, 14 insertions(+), 48 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index e33c99eb34a8..37dfd1878bef 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1258,46 +1258,6 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn, #endif /* CONFIG_COMPACTION || CONFIG_CMA */ #ifdef CONFIG_COMPACTION -static bool suitable_migration_source(struct compact_control *cc, - struct page *page) -{ - int block_mt; - - if (pageblock_skip_persistent(page)) - return false; - - if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction) - return true; - - block_mt = get_pageblock_migratetype(page); - - if (cc->migratetype == MIGRATE_MOVABLE) - return is_migrate_movable(block_mt); - else - return block_mt == cc->migratetype; -} - -/* Returns true if the page is within a block suitable for migration to */ -static bool suitable_migration_target(struct compact_control *cc, - struct page *page) -{ - int mt = get_pageblock_migratetype(page); - - /* If the page is a large free page, then disallow migration */ - if (mt == MIGRATE_FREE) - return false; - - if (cc->ignore_block_suitable) - return true; - - /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */ - if (is_migrate_movable(mt)) - return true; - - /* Otherwise skip the block */ - return false; -} - static inline unsigned int freelist_scan_limit(struct compact_control *cc) { @@ -1620,7 +1580,7 @@ static void isolate_freepages(struct compact_control *cc) continue; /* Check the block is suitable for migration */ - if (!suitable_migration_target(cc, page)) + if (!is_migrate_movable(get_pageblock_migratetype(page))) continue; /* If isolation recently failed, do not retry */ @@ -1927,14 +1887,12 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) continue; /* - * For async direct compaction, only scan the pageblocks of the - * same migratetype without huge pages. Async direct compaction - * is optimistic to see if the minimum amount of work satisfies - * the allocation. The cached PFN is updated as it's possible - * that all remaining blocks between source and target are + * The cached PFN is updated as it's possible that all + * remaining blocks between source and target are * unsuitable and the compaction scanners fail to meet. */ - if (!suitable_migration_source(cc, page)) { + if (pageblock_skip_persistent(page) || + !is_migrate_movable(get_pageblock_migratetype(page))) { update_cached_migrate(cc, block_end_pfn); continue; } diff --git a/mm/internal.h b/mm/internal.h index 24f43f5db88b..1c0886c3ce0e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -741,7 +741,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_HIGH 0x20 /* __GFP_HIGH set */ #define ALLOC_CPUSET 0x40 /* check for correct cpuset */ #define ALLOC_CMA 0x80 /* allow allocations from CMA areas */ -#ifdef CONFIG_ZONE_DMA32 +#if defined(CONFIG_ZONE_DMA32) && !defined(CONFIG_COMPACTION) #define ALLOC_NOFRAGMENT 0x100 /* avoid mixing pageblock types */ #else #define ALLOC_NOFRAGMENT 0x0 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f835a5548164..9db588a1de3b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2622,11 +2622,19 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, * * The other migratetypes do not have fallbacks. */ +#ifdef CONFIG_COMPACTION +static int fallbacks[MIGRATE_TYPES][2] = { + [MIGRATE_UNMOVABLE] = { MIGRATE_FREE, MIGRATE_TYPES }, + [MIGRATE_MOVABLE] = { MIGRATE_FREE, MIGRATE_TYPES }, + [MIGRATE_RECLAIMABLE] = { MIGRATE_FREE, MIGRATE_TYPES }, +}; +#else static int fallbacks[MIGRATE_TYPES][4] = { [MIGRATE_UNMOVABLE] = { MIGRATE_FREE, MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, [MIGRATE_MOVABLE] = { MIGRATE_FREE, MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES }, [MIGRATE_RECLAIMABLE] = { MIGRATE_FREE, MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, }; +#endif #ifdef CONFIG_CMA static __always_inline struct page *__rmqueue_cma_fallback(struct zone *zone,