From patchwork Tue Apr 18 19:13:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59513C6FD18 for ; Tue, 18 Apr 2023 19:13:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2C0190000A; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED961900009; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA0F790000A; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C5DC1900009 for ; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8FB861C653C for ; Tue, 18 Apr 2023 19:13:41 +0000 (UTC) X-FDA: 80695461042.12.8917F48 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf10.hostedemail.com (Postfix) with ESMTP id ABF3DC0021 for ; Tue, 18 Apr 2023 19:13:39 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=PrIghyUC; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EFgZ3D8VKBLVvaJCzeMlC8+6IsY+n+UHK2kcdHMAd6Y=; b=YP/shsz0ozbiKVD0iH8zpNxUUANmDsMzMT7dJZK0oa6diQCEVmFKWE+sudBWj2Uz+rTu4o /H94kJ17TnjyFAHxTjYfJ9/lkaTKToOIElgb1Ub2JdVrZFju0LWVotMEauq7E9MqFyjPlg Ta7EOQwEg5ErxcW0VQcwpzp9DJ7C4Gw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=PrIghyUC; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845219; a=rsa-sha256; cv=none; b=i79zzm30A4FDZI4ccy6b60+BgkaoUv4mOneVTty7IVAdHzDfk3RMmBt7LCNkiYs4Zd7CW1 rTRNs9yvgmU1cPD+UPgbwGAKGB7GDlKNiNgW9nrfnAX95SBtcon9IvKGR3OKALcS+ZE2Wa tW7hl6ah1uYzZIJiyO53p4wfkQPcxu0= Received: by mail-qv1-f46.google.com with SMTP id h14so18464600qvr.7 for ; Tue, 18 Apr 2023 12:13:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845219; x=1684437219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EFgZ3D8VKBLVvaJCzeMlC8+6IsY+n+UHK2kcdHMAd6Y=; b=PrIghyUCPpn833RcxLlSVfU5Tk3SpnQwff6jbo/QPuj/XkyPt6wBzL/Dp9O4CsPEIx vlbtgOz19a8Bj5Q9Tvz3XxufWwoP1pyylG1i+Eubg/l0s5lNHABn8jx+RIo3qNp8yvJd xxmWan9BQx+LKGsE5QvB32mS3Fey4Ysn5JLrhFNxDsTGokonj/n7lqrpIjGCmJsUuM9+ Ab+1ebZi1/6VKaIDC4ffD88G61bRq54CAS9nhM+89W7ACAiv40xaJvzX03Xeni/fkhUD S0GccLFy6IRYr+8AC7mboIhQozDrEx6WmyjvaCp6CXY+ejn1Dw19uqM1MKn105sfCPkW reiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845219; x=1684437219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EFgZ3D8VKBLVvaJCzeMlC8+6IsY+n+UHK2kcdHMAd6Y=; b=UqlOyX7HCvDsRRSc6xoJcCijS5CAMuPo6TgqZZujDgGhZdOR8kfUS6PW/O2xSq8Bv9 3jE6VEqmMOt9gv4I7dbNMVRGNIMRB/bUuUolcfHpLli+ZXSpQvMiSPhEUOHLdBBz6BOP bQVYy7YXmWrLpYiqcRf0sbTFX9JGgb9YrM4djzaU+87uMTKfOF4mJCfGFpzSZozr0kne lsHKge6ugQH8R1qSoIM8AGfRIrH2WP6v/7e5MIg0iaYOP80Ng1fZ4JZ/ZlWFzztlyCRK l9RagVvWY3WmtlkhQN6YyVxeexgO6qI9MleZyD95nJR/IjlToZQPB5x0Ra5MKcm/hfEo Z4Yg== X-Gm-Message-State: AAQBX9fybOCKxS3N0OFz3bRBRhm/vrScik+m/3jLP2uNfmbnDX8+pSaz l1OyfgE6JLncQ7ONjTtkkjhee1LkMytiB3WmWF4= X-Google-Smtp-Source: AKy350YOTexSjADk98sBY4JcGk1SQDwC9c/+fGeG6k+kXfYRIwBupTZdOSOM2/pP7fHju0r9Wjpp2g== X-Received: by 2002:a05:6214:20e2:b0:56a:b623:9b09 with SMTP id 2-20020a05621420e200b0056ab6239b09mr22229605qvk.14.1681845218901; Tue, 18 Apr 2023 12:13:38 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id z9-20020a0ce609000000b005dd8b9345c7sm3914040qvm.95.2023.04.18.12.13.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:38 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 17/26] mm: compaction: refactor __compaction_suitable() Date: Tue, 18 Apr 2023 15:13:04 -0400 Message-Id: <20230418191313.268131-18-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ABF3DC0021 X-Stat-Signature: 5u3wcgupoe8fhbzmdpcf6f9cdshgyt7x X-Rspam-User: X-HE-Tag: 1681845219-18852 X-HE-Meta: U2FsdGVkX18o9d0RBi0prcn8IXk4VrHPU+imS1nxO664yvbGW52KWz9yXLtuOV7fVonDTD6/0IRzHqVnrD2VkF8VNCiuDf0l43gz7R3yJPeMSd2G678X61QcRxOE+cB1NiMHMLdYbZbI25TyY5qvmdoHO04kcWL/dGimg1cgFguSYIrkuqt2WZOR+Re5BR8mRylePWj9o3fvCurPlJ/T80GUcSRIwvHvlX4YR0Y6DMQ7xSHJZXrqiPNCst7WuEEEKvR35zTmK3UqI9DwhcNGN0x4Qanq6AU6ofDZYCgpJeNOiio6qZ+3Z+OqLg0xxG8hXNew18CIoH2AcM/Qwp/vEM1mcGJXpdm5mrytlqa5crjofRm5+Bk/99qbysg2jLMaKOConUUqP3/wMpJc4KHDfwmgOE+QpTHJ88yBeVE9qqMB5yvXPCTBMSMQbae7L76SvLA4VmceEZsMCIEhjlEkMr6Y5hUM1h3Gt/gDeXKhaXggJLAG9pgyvvjkHHVDUK+HlcYbPJcACV02kXTrfvH/lrlmcjSA0ugfwNi50CXcCqFtIMwIu7/Ybkd8LQw08HKTUtYLrmATyO0FFNHwAGl7MUobeb6TYRLLq+3IuZgEJ5IE5Cb+uhZ95MntEeFYDIe1k9Cs0H+MojzIf5IIwFv9gutuoNq9PQeeEC+buULXttgLHdmYqPhvKS4lD4GoKX4aBvl6kwgn4zZS2sOKEFsUL/O7V3dyAh6B9uOfVEu0S/52/6qPzzM8gGk5rC9fSsHBYoFUuR8VPRg2yqeeVbwdcyxt45YDnAc6fsxt+vwhCB9k7Uo9NhSrF9c0IALlxvjlXfMrBTfYp5Mo/rQSKyboE7cutBFhldwcVGXSeLWFIj5Cw9J/n6/oOmvqxlMXg977avOlF4xw/1LX7cnPsH/cTiguw7MSlr+urB4Bof02LtCcLfcFy9pzbXHKwYwjCkT5DKYqR56RK30cUlA6xaB z1ePmkm4 7Ve5nJslQqNGRY9vO+fnDvml+uDH4/qHGWj5y1ncfgj4p36UdnPxa1ZUgqUHsTBPHpuHcpc80uwnBOplam1Xx3KKT3RZmbq/eOMP5kUP1wvH4OC2jZbTUzLH4JQFIt6ot/O+TElFV235MGabtwkpzFu4jnKDL7ojnc80KyHXLtrWwnBneymJ20QhmmvEM8gHHmGlmzSaLxNJrKw2/36jKE/cY6pNrjo57G6JejZO2jKZmWp0Rlth6v0UrYduSP/66H1lHV63uFwjNWLqFkkLYGnFV/owKAlbJmu7pWIYTNrzrifDcmlwQAZSZ99mpxwCRJNTQKrMjLP/CxgleIdQHizp553tcf7zhTwkW6n793nmLJsJGX5NoU9JhBbDr+uK1jAcl/n1IEvjxJscjzb8CJH7TH9vibVtq63jEdebSszNQzaRkwgIr2M6vZnK+678KdEcv4b2YTHJP1rSVPbpj1KkDji7NVRt2arPVbAwijwsbRr1dN2Z57PqnoiAVMxd11uD+GRyZMj9E9U24naE+6Lb3qw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __compaction_suitable() is supposed to check for available migration targets. However, it also checks whether the operation was requested via /proc/sys/vm/compact_memory, and whether the original allocation request can already succeed. These don't apply to all callsites. Move the checks out to the callers, so that later patches can deal with them one by one. No functional change intended. Signed-off-by: Johannes Weiner --- include/linux/compaction.h | 4 +- mm/compaction.c | 80 ++++++++++++++++++++++++-------------- mm/vmscan.c | 35 ++++++++++------- 3 files changed, 74 insertions(+), 45 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 7635e220215a..9e1b2c56df62 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -98,7 +98,7 @@ extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, struct capture_control *capc); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx); + int highest_zoneidx); extern void compaction_defer_reset(struct zone *zone, int order, bool alloc_success); @@ -116,7 +116,7 @@ static inline void reset_isolation_suitable(pg_data_t *pgdat) } static inline enum compact_result compaction_suitable(struct zone *zone, int order, - int alloc_flags, int highest_zoneidx) + int highest_zoneidx) { return COMPACT_SKIPPED; } diff --git a/mm/compaction.c b/mm/compaction.c index f637b4ed7f3c..d4b7d5b36600 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2173,24 +2173,10 @@ static enum compact_result compact_finished(struct compact_control *cc) } static enum compact_result __compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx, unsigned long wmark_target) { unsigned long watermark; - - if (is_via_compact_memory(order)) - return COMPACT_CONTINUE; - - watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); - /* - * If watermarks for high-order allocation are already met, there - * should be no need for compaction at all. - */ - if (zone_watermark_ok(zone, order, watermark, highest_zoneidx, - alloc_flags)) - return COMPACT_SUCCESS; - /* * Watermarks for order-0 must be met for compaction to be able to * isolate free pages for migration targets. This means that the @@ -2223,7 +2209,6 @@ static enum compact_result __compaction_suitable(struct zone *zone, int order, * COMPACT_CONTINUE - If compaction should run now */ enum compact_result compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx) { unsigned long free_pages; @@ -2234,8 +2219,7 @@ enum compact_result compaction_suitable(struct zone *zone, int order, free_pages = zone_page_state(zone, NR_FREE_MOVABLE); free_pages += zone_page_state(zone, NR_FREE_CMA_PAGES); - ret = __compaction_suitable(zone, order, alloc_flags, - highest_zoneidx, free_pages); + ret = __compaction_suitable(zone, order, highest_zoneidx, free_pages); /* * fragmentation index determines if allocation failures are due to @@ -2279,6 +2263,16 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->highest_zoneidx, ac->nodemask) { unsigned long available; + unsigned long watermark; + + if (is_via_compact_memory(order)) + return true; + + /* Allocation can already succeed, nothing to do */ + watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); + if (zone_watermark_ok(zone, order, watermark, + ac->highest_zoneidx, alloc_flags)) + continue; available = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); available += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); @@ -2290,8 +2284,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, */ available += zone_reclaimable_pages(zone) / order; - if (__compaction_suitable(zone, order, alloc_flags, - ac->highest_zoneidx, + if (__compaction_suitable(zone, order, ac->highest_zoneidx, available) == COMPACT_CONTINUE) return true; } @@ -2322,14 +2315,26 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) INIT_LIST_HEAD(&cc->migratepages); cc->migratetype = gfp_migratetype(cc->gfp_mask); - ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, - cc->highest_zoneidx); - /* Compaction is likely to fail */ - if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED) - return ret; - /* huh, compaction_suitable is returning something unexpected */ - VM_BUG_ON(ret != COMPACT_CONTINUE); + if (!is_via_compact_memory(cc->order)) { + unsigned long watermark; + + /* Allocation can already succeed, nothing to do */ + watermark = wmark_pages(cc->zone, + cc->alloc_flags & ALLOC_WMARK_MASK); + if (zone_watermark_ok(cc->zone, cc->order, watermark, + cc->highest_zoneidx, cc->alloc_flags)) + return COMPACT_SUCCESS; + + ret = compaction_suitable(cc->zone, cc->order, + cc->highest_zoneidx); + /* Compaction is likely to fail */ + if (ret == COMPACT_SKIPPED) + return ret; + + /* huh, compaction_suitable is returning something unexpected */ + VM_BUG_ON(ret != COMPACT_CONTINUE); + } /* * Clear pageblock skip if there were failures recently and compaction @@ -2803,7 +2808,16 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) if (!populated_zone(zone)) continue; - if (compaction_suitable(zone, pgdat->kcompactd_max_order, 0, + if (is_via_compact_memory(pgdat->kcompactd_max_order)) + return true; + + /* Allocation can already succeed, check other zones */ + if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, + min_wmark_pages(zone), + highest_zoneidx, 0)) + continue; + + if (compaction_suitable(zone, pgdat->kcompactd_max_order, highest_zoneidx) == COMPACT_CONTINUE) return true; } @@ -2841,10 +2855,18 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (compaction_deferred(zone, cc.order)) continue; - if (compaction_suitable(zone, cc.order, 0, zoneid) != - COMPACT_CONTINUE) + if (is_via_compact_memory(cc.order)) + goto compact; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, cc.order, + min_wmark_pages(zone), zoneid, 0)) continue; + if (compaction_suitable(zone, cc.order, + zoneid) != COMPACT_CONTINUE) + continue; +compact: if (kthread_should_stop()) return; diff --git a/mm/vmscan.c b/mm/vmscan.c index 9ecf29f4dab8..a0ebdbf3efcf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6076,14 +6076,17 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, if (!managed_zone(zone)) continue; - switch (compaction_suitable(zone, sc->order, 0, sc->reclaim_idx)) { - case COMPACT_SUCCESS: - case COMPACT_CONTINUE: + if (sc->order == -1) /* is_via_compact_memory() */ + return false; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), + sc->reclaim_idx, 0)) + return false; + + if (compaction_suitable(zone, sc->order, + sc->reclaim_idx) == COMPACT_CONTINUE) return false; - default: - /* check next zone */ - ; - } } /* @@ -6271,16 +6274,20 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) { unsigned long watermark; unsigned long free_pages; - enum compact_result suitable; - suitable = compaction_suitable(zone, sc->order, 0, sc->reclaim_idx); - if (suitable == COMPACT_SUCCESS) - /* Allocation should succeed already. Don't reclaim. */ + if (sc->order == -1) /* is_via_compact_memory() */ + goto suitable; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), + sc->reclaim_idx, 0)) return true; - if (suitable == COMPACT_SKIPPED) - /* Compaction cannot yet proceed. Do reclaim. */ - return false; + /* Compaction cannot yet proceed. Do reclaim. */ + if (compaction_suitable(zone, sc->order, + sc->reclaim_idx) == COMPACT_SKIPPED) + return false; +suitable: /* * Compaction is already possible, but it takes time to run and there * are potentially other callers using the pages just freed. So proceed