From patchwork Fri May 19 12:39:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13248285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C918CC7EE23 for ; Fri, 19 May 2023 12:40:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0659D900008; Fri, 19 May 2023 08:40:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0183A900003; Fri, 19 May 2023 08:40:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAA1A900008; Fri, 19 May 2023 08:40:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C9BDA900003 for ; Fri, 19 May 2023 08:40:13 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9F1C8160110 for ; Fri, 19 May 2023 12:40:13 +0000 (UTC) X-FDA: 80806962306.14.6F926DE Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf28.hostedemail.com (Postfix) with ESMTP id 9EFE2C000F for ; Fri, 19 May 2023 12:40:11 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=j8nprdhT; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.218.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684500011; a=rsa-sha256; cv=none; b=A0FENVKup280g3FJUgBHHncxrIMhZJroLUwoKDUZzEuBvG1CaAx1XVBsRAq1FXQvSuLtNS KUH3ysxM/NJ225cQKGf9AFIlyfMl/8xu7ZUdd5tLm8giIyegUATU7N4ZqU1RSVDY6F5ybX Td/estnNhcmS+eJExnvxtL9aJa/Ge/4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=j8nprdhT; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.218.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684500011; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vXkGmoOKMV9wp5TM6FJc6/QwLR/CY7AZ1q5R5QORiXk=; b=wVHXSVIEeC7jepoUkseepA6x1IAaV23SuRTLE1ZiQbC0WGK6GfAPTNXbUGerdPd42Wjq7o 4Vfnhob4I555/TK/1XPIFgjVSWOPMfH7aag+c1Z1ylGl7+Y5swzE9yL0/5Q+XwCHyIvexK ejhB3OBVQJQWc3OCZwD0RM7XrmQaE0E= Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-94a342f7c4cso610442066b.0 for ; Fri, 19 May 2023 05:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1684500010; x=1687092010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vXkGmoOKMV9wp5TM6FJc6/QwLR/CY7AZ1q5R5QORiXk=; b=j8nprdhTRVRZX+b+MiJ39vvsPB8y0GGrbPhI6lsexF0dvkO4amR6WItiuegG9BfiRU MN8A49KVx8o86hTYpRXpcGjnSKD25lzyyj/NH4c3kzpIu8FH29Fwu9YXWODBkI/cR2tZ tOWDqP3eVUjsGYrXv9d6xlJKrPpVGZlF4AoFmjHQ8nIIaKIWelJHj4SHtHzvkTfTetw7 uwZhgOO/hTAb8A1ZkSqmIj2ZGQM1X+5/b7c6kdyR6bMbj9hfiKlwreo9cF2y7+qx1WJD 81n9YGKCYvomlnft2jGfo59NweehDvgmGgPIPXd9/QQdknjD+xJPIaq182eUfreAbQ8C sHmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684500010; x=1687092010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vXkGmoOKMV9wp5TM6FJc6/QwLR/CY7AZ1q5R5QORiXk=; b=UOMhBwyjc9jRTDGG8KjUyO81bg2bezQCTdgkfLjEYOIihPeqTuYmTTiX8ANtugzE5E XcdzFs92hE6vGJknxl38hjhpwY3bOXEfmOR0xrhjzchVli8WAPP/y0l1L5XKFrMsHJsh u6FIkK0vC6hx/zn5c0NHrtHoDrTXEa94qgQ/sL+MxR84H/KFXP6+f/RuZIuEcFudYlKU SiQDL/e9AING5n/LWc4ae+O9Vem7dLUBHGWAYPfqxpftwn/MCNq+wv+x4fCNWqkcj7Fg e9Zwv2SOxlzlseo1XLi1vC/Q+X/lxvTGv1duVBFB0OAZ4zSZS2zO/vZtJrc/qOhpSszO tDyw== X-Gm-Message-State: AC+VfDw1uGdMhuTIK1Y3kSqxha8gvsvOFJ3/nHCK8yqK7fFpdSkicYRs Idq5kp723Zj8+vurnvY/G/poRQ== X-Google-Smtp-Source: ACHHUZ5OI99u6ZjRHk2L81FZPXX2dcOxJ0p3S0XoqUDF0BmNWfVXDxnHp4ONjafiSuyLZ1d4i1Us2w== X-Received: by 2002:a17:906:ee88:b0:94a:5819:5a2b with SMTP id wt8-20020a170906ee8800b0094a58195a2bmr1567192ejb.33.1684500010022; Fri, 19 May 2023 05:40:10 -0700 (PDT) Received: from localhost ([2a02:8070:6389:7d40:e266:3092:9afb:a7b1]) by smtp.gmail.com with ESMTPSA id u20-20020a170906125400b009571293d6acsm2236343eja.59.2023.05.19.05.40.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 May 2023 05:40:09 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Mel Gorman , Vlastimil Babka , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 3/5] mm: compaction: refactor __compaction_suitable() Date: Fri, 19 May 2023 14:39:57 +0200 Message-Id: <20230519123959.77335-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230519123959.77335-1-hannes@cmpxchg.org> References: <20230519123959.77335-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9EFE2C000F X-Stat-Signature: c84wffxgoqb37ioykch3kh8hnsbz1cp8 X-Rspam-User: X-HE-Tag: 1684500011-173792 X-HE-Meta: U2FsdGVkX1/10FeCXYpLZiWdarHOacdbXVbFt+0UGlWg4KOcnRpJMqtiFgHtJQwFFSDTd0+52wMEIpVAr/WTAcDzIarq5Wi7J9eBIddvgSyU3uGwg4n4hzfwVfk9mtzz+wZGjCVpcl94I5SdrPflMuq1mLG6EBK6kJyrjPuqZ21I/r5kacW+8SGltVVmXOmXdbUYiLad2iDV43QsJwx/qXDjXRm5zPAVsUlKU/xl6hbAG/vWZ4W6dSKKcqYtTMUsHnpyglI6yxCBOTaKKStj99r94OvDi4mVTBmdq7z6sdWGb5AEyc/lphxH/l0hO5f9ogZqgfoF5oAqHrDz/HZIvCq76KXd7XBjmDyMFE5B/fWWwSxphbonAYlcMYfUlSizDc6SxlX7D+vr/qOAbuoXr/okbeTdG7oFflbmyC4CL7XvhAEjDsqsifrjmYRrVTbL5vh5fAJEQQwqNrwpn+snEkWsO68RTyY/d4+XL/B7DHQlrPalPifXvIalSPTQAnMEAbSSstGKSHLPWgUsNOwECEukGsGWouV7Wqo2YtagoUfoyQeCKW2B34u3jKfTqKlVy+T669RzLEc5GUhepummqlExAV6CZ2Bp3ZT+3BxfVfvZ4lCWAU+T4Sy6YIQMnnwtXtDxavvQMyo0bKoSyjPf3tXfz72xPSJJxfTrbKqztT6RazscHmYQNgfwq/QT3L+5mjyrlwMjdWCWsk3u+WEdGvIh8bfTBDlPgM7rLEDxRBiA9UeXGIEUXz3Q9JwGCaHhTMvm/LGQQSs7czgN4+Fpr7vhc0TYb5rIMYkCfmrahgdVznIgpu8oZujtBL7fMwBsUA9NIe9KWjnT65zimvV3pB4hzBAnD93pdCgtAck0y9vRYCDqZbAfbAN9ZkfliBnF5b4fVCydR9UjkRIosqymBmFfvAkTvOjnTjzA+HFa+LvrqUs6xK7ulBy9csBz2/cY/ys5twOH44RcscXPhB5 x19SuGGK nowDIc8a1S9tsmMi/XlIihwhixyKOsblXhbqXRNf2apD8nYRE4coHSA1bhJfk8haM3lz0HifY9vmd4sSuynuk3Rn/5QDx13Mdg9DN6bAANpHGLnJnP3t4uRRCzmCjmtwpp5B7gKkrnuOE5g6skC/a0h26w4m0dhy0j5w3lnJ/JRVh2x+YpYD7JL/u38Uc612LFXKwrtv1jLgCmdDtwOdz6HoG8aPwQMJWRDk8f39hvUBjVSIgy04ZJBsjZqwlPa0UL+MBSAB6tfRqbYx53Om+IzbUtWxmnNqT1ajnZOvlWWIAhm6JMOQ6KjrOLKP7hVmOlK3VaB2/myYHK7iZrI7DX2zO/qkIRumvf1gUKB1QhDKgb3UTXca5lr4GArN0im1XsZMfbIaVfYtO1JhtZpJN98STAJDpbvUDIJYe0PNj84aqvFh3/0WgSEQJsBxxp0xXWsd45W7eyai+HR6oVroa3dnAQMXg03dmK07uJcva63H/L/QyAmMAlsDpEtE8mzuLPNi9cG37nqag/ng= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __compaction_suitable() is supposed to check for available migration targets. However, it also checks whether the operation was requested via /proc/sys/vm/compact_memory, and whether the original allocation request can already succeed. These don't apply to all callsites. Move the checks out to the callers, so that later patches can deal with them one by one. No functional change intended. Signed-off-by: Johannes Weiner Acked-by: Vlastimil Babka --- include/linux/compaction.h | 4 +- mm/compaction.c | 78 ++++++++++++++++++++++++-------------- mm/vmscan.c | 35 ++++++++++------- 3 files changed, 73 insertions(+), 44 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 1f0328a2ba48..9f7cf3e1bf89 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -90,7 +90,7 @@ extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, struct page **page); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx); + int highest_zoneidx); extern void compaction_defer_reset(struct zone *zone, int order, bool alloc_success); @@ -108,7 +108,7 @@ static inline void reset_isolation_suitable(pg_data_t *pgdat) } static inline enum compact_result compaction_suitable(struct zone *zone, int order, - int alloc_flags, int highest_zoneidx) + int highest_zoneidx) { return COMPACT_SKIPPED; } diff --git a/mm/compaction.c b/mm/compaction.c index c9a4b6dffcf2..8f61cfa87c49 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2206,24 +2206,10 @@ static enum compact_result compact_finished(struct compact_control *cc) } static enum compact_result __compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx, unsigned long wmark_target) { unsigned long watermark; - - if (is_via_compact_memory(order)) - return COMPACT_CONTINUE; - - watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); - /* - * If watermarks for high-order allocation are already met, there - * should be no need for compaction at all. - */ - if (zone_watermark_ok(zone, order, watermark, highest_zoneidx, - alloc_flags)) - return COMPACT_SUCCESS; - /* * Watermarks for order-0 must be met for compaction to be able to * isolate free pages for migration targets. This means that the @@ -2256,13 +2242,12 @@ static enum compact_result __compaction_suitable(struct zone *zone, int order, * COMPACT_CONTINUE - If compaction should run now */ enum compact_result compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx) { enum compact_result ret; int fragindex; - ret = __compaction_suitable(zone, order, alloc_flags, highest_zoneidx, + ret = __compaction_suitable(zone, order, highest_zoneidx, zone_page_state(zone, NR_FREE_PAGES)); /* * fragmentation index determines if allocation failures are due to @@ -2306,7 +2291,16 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->highest_zoneidx, ac->nodemask) { unsigned long available; - enum compact_result compact_result; + unsigned long watermark; + + if (is_via_compact_memory(order)) + return true; + + /* Allocation can already succeed, nothing to do */ + watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); + if (zone_watermark_ok(zone, order, watermark, + ac->highest_zoneidx, alloc_flags)) + continue; /* * Do not consider all the reclaimable memory because we do not @@ -2316,9 +2310,8 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, */ available = zone_reclaimable_pages(zone) / order; available += zone_page_state_snapshot(zone, NR_FREE_PAGES); - compact_result = __compaction_suitable(zone, order, alloc_flags, - ac->highest_zoneidx, available); - if (compact_result == COMPACT_CONTINUE) + if (__compaction_suitable(zone, order, ac->highest_zoneidx, + available) == COMPACT_CONTINUE) return true; } @@ -2348,11 +2341,23 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) INIT_LIST_HEAD(&cc->migratepages); cc->migratetype = gfp_migratetype(cc->gfp_mask); - ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, - cc->highest_zoneidx); - /* Compaction is likely to fail */ - if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED) - return ret; + + if (!is_via_compact_memory(cc->order)) { + unsigned long watermark; + + /* Allocation can already succeed, nothing to do */ + watermark = wmark_pages(cc->zone, + cc->alloc_flags & ALLOC_WMARK_MASK); + if (zone_watermark_ok(cc->zone, cc->order, watermark, + cc->highest_zoneidx, cc->alloc_flags)) + return COMPACT_SUCCESS; + + ret = compaction_suitable(cc->zone, cc->order, + cc->highest_zoneidx); + /* Compaction is likely to fail */ + if (ret == COMPACT_SKIPPED) + return ret; + } /* * Clear pageblock skip if there were failures recently and compaction @@ -2845,7 +2850,16 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) if (!populated_zone(zone)) continue; - if (compaction_suitable(zone, pgdat->kcompactd_max_order, 0, + if (is_via_compact_memory(pgdat->kcompactd_max_order)) + return true; + + /* Allocation can already succeed, check other zones */ + if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, + min_wmark_pages(zone), + highest_zoneidx, 0)) + continue; + + if (compaction_suitable(zone, pgdat->kcompactd_max_order, highest_zoneidx) == COMPACT_CONTINUE) return true; } @@ -2883,10 +2897,18 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (compaction_deferred(zone, cc.order)) continue; - if (compaction_suitable(zone, cc.order, 0, zoneid) != - COMPACT_CONTINUE) + if (is_via_compact_memory(cc.order)) + goto compact; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, cc.order, + min_wmark_pages(zone), zoneid, 0)) continue; + if (compaction_suitable(zone, cc.order, + zoneid) != COMPACT_CONTINUE) + continue; +compact: if (kthread_should_stop()) return; diff --git a/mm/vmscan.c b/mm/vmscan.c index d257916f39e5..c9c0f3e081f5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6397,14 +6397,17 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, if (!managed_zone(zone)) continue; - switch (compaction_suitable(zone, sc->order, 0, sc->reclaim_idx)) { - case COMPACT_SUCCESS: - case COMPACT_CONTINUE: + if (sc->order == -1) /* is_via_compact_memory() */ + return false; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), + sc->reclaim_idx, 0)) + return false; + + if (compaction_suitable(zone, sc->order, + sc->reclaim_idx) == COMPACT_CONTINUE) return false; - default: - /* check next zone */ - ; - } } /* @@ -6592,16 +6595,20 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) { unsigned long watermark; - enum compact_result suitable; - suitable = compaction_suitable(zone, sc->order, 0, sc->reclaim_idx); - if (suitable == COMPACT_SUCCESS) - /* Allocation should succeed already. Don't reclaim. */ + if (sc->order == -1) /* is_via_compact_memory() */ + goto suitable; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), + sc->reclaim_idx, 0)) return true; - if (suitable == COMPACT_SKIPPED) - /* Compaction cannot yet proceed. Do reclaim. */ - return false; + /* Compaction cannot yet proceed. Do reclaim. */ + if (compaction_suitable(zone, sc->order, + sc->reclaim_idx) == COMPACT_SKIPPED) + return false; +suitable: /* * Compaction is already possible, but it takes time to run and there * are potentially other callers using the pages just freed. So proceed