From patchwork Tue Apr 18 19:13:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6D21C77B75 for ; Tue, 18 Apr 2023 19:14:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8987D280008; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 84891280001; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7104C280008; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 60E42280001 for ; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 41026AAE45 for ; Tue, 18 Apr 2023 19:13:46 +0000 (UTC) X-FDA: 80695461252.24.84B265F Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf10.hostedemail.com (Postfix) with ESMTP id 6C672C001F for ; Tue, 18 Apr 2023 19:13:44 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=SD0CfPNo; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845224; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hsTBnxogc0dXbXldiQnTcfaSMnaw37pvCiHuIsUwhKs=; b=1AR7P9x05XVNEZazx/71+1W44dg7SzJKFkskkKjpv8Vj072IkvxJA+aSyW+LvriI2oAzqy 3BHikv6AmW9BH+QEcKErOf1pP8OlM7/CCcdbfSlQN+R4hZqlOPiTBPT0jxrZWJaqfzSbdU 64pUTJMQetyPA02cLLeiqpZKoEIXVcs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=SD0CfPNo; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845224; a=rsa-sha256; cv=none; b=bbI75V7v+s8MQEkk6Aoo1f4Dgr4TumSiVoCgBGtIWNqT6AjTzqhWwNkNmcGzXmZzDlXFAr AcCp/iATIGSY8z8lDu5wRgQJg8iNaMP+Tk1Ef7b2Yh7/EEGdWe//NO40/hnveoQKsrCeq8 aa45uF6ZWg1LXZ2m3A96dKH6KRgG0Wg= Received: by mail-qv1-f46.google.com with SMTP id h14so18464855qvr.7 for ; Tue, 18 Apr 2023 12:13:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845223; x=1684437223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hsTBnxogc0dXbXldiQnTcfaSMnaw37pvCiHuIsUwhKs=; b=SD0CfPNo/FtA52mn8PTuMlWwsWPdBrYNWFQm6HroBkjTNWL2qXQf7wrFmhK/WWm7IP xcIXlFfLFGEf/BrGyHz+UE3kHnP2QzzQB7WXPEZFLWPxISNTxWhAaa0h1tT0olaHgEUs 0hpjB9LzjiA5OHWB18U6c4fyqiiCG4R128OFbaQXw6cA+VNmAHuxaR45uultoYesSJx0 CVUbWebDBfJA3QQt/c1BCijGivD+dCjM9agqh7+5bqifw5Letf8PMz5QSYbJeZ5UTCUP tSHckFJNQ1uiBdu7JKbqXKX13Y5T5FnoLNhX7VZobyOD8tLkxHSMFOnr57kBpsZRcWht QozQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845223; x=1684437223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hsTBnxogc0dXbXldiQnTcfaSMnaw37pvCiHuIsUwhKs=; b=fAiwfPo9mZlpSOkeg/DxHBycx9njbDe/uQjrszkaxeVo/3EuGLd6BQlmlxDNCjtTFD /603WTsGVrQ4xPHqLxNaWpP/03JzIJdyHZehyhGSbMXxaVAPIsdQhSyJ5F2t11mEp08p Rqokm1kM+dl0r77OoKqMN82D0BthnHOVrnsnEfZzYJ7qFoFGNi+A47zbA9Qrmmvy7SIm W/AK4l0P3kLZhniIMMPLHR6KZQiJK1Xau2mdxej4gUlcnTrIgtN8Y3vpjoh5GpL4NOIg am/af356EIUCB3TsPHjvh4G/xY3C98TIK1SNAgqTj9DGDgGMGe8SGT5AZaD/KkCSB8DQ Cn6g== X-Gm-Message-State: AAQBX9eKs/QKWoOohwkpdviu7TkSNsyiL7VLfzcx8gadc95AKBc0CXUd eVJsNs03OiEJJTAUYsj5SqUrCFITI0uGWukr1as= X-Google-Smtp-Source: AKy350afvFhbEDO4u0w1iKrPXbytvWWwl+mYLWsq6qZEuaKTZTEbfiazBxZv7n8V0c8QP05wvdn3Kw== X-Received: by 2002:a05:6214:2607:b0:5c2:a8b0:d71a with SMTP id gu7-20020a056214260700b005c2a8b0d71amr27349805qvb.43.1681845223606; Tue, 18 Apr 2023 12:13:43 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id x18-20020a0ce252000000b005dd8b934598sm3884188qvl.48.2023.04.18.12.13.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:43 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 21/26] mm: compaction: align compaction goals with reclaim goals Date: Tue, 18 Apr 2023 15:13:08 -0400 Message-Id: <20230418191313.268131-22-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6C672C001F X-Stat-Signature: mit5erwgrnpeupdbgcu5iqg7xbtmzsn3 X-Rspam-User: X-HE-Tag: 1681845224-321041 X-HE-Meta: U2FsdGVkX18Jp4kw5uU29UcuyU9LTSi6jfgbuO0+8qtQLTUXl2SaDNo+YEqcGWfEHEiV1cncF1WebVK3TJ4s2XHEtCRVOLuXFCtdADG4C/WJAWrHutE6tp+rRLtt6ODfTLe5ZNoksSiCmyLtJBCmOnRyGZZBbvjS6JWxZfsT0LojSED438S8oMsxH10L7jIvaL/libiRFFdn0H0At1y1sn5x8dOl+7qE4EgCtPTAE2EFoQ6JAcxLPE6oLqW3TZVfsZcLQNqPPuegArUF3lzp4knG3GCFuBqnr04W72BjAFbQ6ulOs9PDTh8qseVZtEV2bdl5vFahK37I2zj8o1Chjky3l6O8fpTZqE553ZX80gB8d+XWvE0YyRvolQxTte1mBov9AWnAZZ+KfHMElZ2lBuCraEVdesHUnbPEEPfGvOMNUsBjtQlMm4fgaZCttwdlHpBp7QvmLQ191Rldz7Vf+xEqhIYLgiSs3BSJ8sQwBLxkX5MhvlRDodp6uClSPEYDpEL3mstWKkvXrTwWmKGSzbFA7/1YEZyi+Mjj/Ls/1Z39nv2YZn5TX82F+Fboajmzm0iSWagfSHX0KSmOizfy5GxKjRpoYQ2twZhP/mTQ7mOzmcBU8bHcY2SGzD5t1B8xcP2Ez0/WHLjepOhA9/z86hSI3AwcOJfwKgczXkvQgWJibKZNHmBZmVDsPl4Gpt9z4CBvpfcuPOClVdJnrjauRrapEQWuQFS1HKJQwIpHGRvK8LyfissJDNwEnDUqC1ll/HX4DMK3oNG/J1VNs5i9Ba7QVCS6fFbfP8nHN5n5+B6b6349Am0+DSeSHWet0k09Rm5LGFbaaPvVW75v6F6FPDlv6v4bEwH9bJdrtgBs0jd4c3n99YrMxv/0mQ+0ofp9q+X3AnYnxa8qkZdsCVNKYXpI+eVFfsOdgUK9DkXXKqbR8JEUloznkSC18zO/kKNWTCMrkW84ZWGqjMtAQjG nEcxCs60 rcTI6CRs2ewKoLngGklrk+RXlifkb1EPQeh2/LSYYhR23gQVCFJMIeM4LNfixRlzOYrnkiY770+NlgIazAjDTyHZ5tj7MVJ/uCHfRlp5To+II9dx37CjL70hMaPy25W+fKh5CbZufoYPSEjT2EmVhOQK2YtovuidtcpQiPWrNAxTtfACiuvWdXg4eHdoc6om/jaLxA0Cgw1WkXLWsOT8zWMfTSY4gAmz2gSC0Wr0HpjtFFbG3DEu3AUvEUR48proV/81shcvRhanSqdgvIFCD2PJgLc5kITU1f7cva2QBHyfNWMJ06UY7or2iRwD+IMhEyeirAkvARQiLzAOi3ibrmy5bffNHDZGfDHr5ZjrCEpAXcgmMuyxz4klOw/dlEMKejjqErZhTeCl0ti6yADidtqbJcC21lJJ6MHI1MOgMRqGSfQf9MQVPIZGN8qnsV+6OT84WzBXCMXQ+Dn78SmJC8JezVE94EUnNNkCujF87Hv/7C98= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kswapd's goal is to balance at least one zone in the node for the requested zoneidx, but no more than that. Kcompactd on the other hand compacts all the zones in the node, even if one of them is already compacted for the given request. This can hog kcompactd unnecessarily long on a requested zoneidx. It also has kcompactd working on zones without the cooperation of kswapd. There is a compaction_suitable() check of course, but whether that is true or not depends on luck, risking erratic behavior. Make kcompactd follow the same criteria as kswapd when deciding to work on a node, to keep them working in unison as much as possible. Likewise, direct reclaim can bail as soon as one zone in the zonelist is compaction_ready(), so check up front before hammering lower zones while higher zones might already be suitable. This matches compaction_zonelist_suitable() on the compaction side. Signed-off-by: Johannes Weiner --- mm/compaction.c | 5 +++-- mm/vmscan.c | 35 +++++++++++++++++------------------ 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 52103545d58c..8080c04e644a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2798,12 +2798,13 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) if (!populated_zone(zone)) continue; - /* Allocation can already succeed, check other zones */ + /* Allocation can succeed in any zone, done */ if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, min_wmark_pages(zone), highest_zoneidx, 0)) - continue; + return true; + /* Allocation can't succed, but enough order-0 to compact */ if (compaction_suitable(zone, pgdat->kcompactd_max_order, highest_zoneidx) == COMPACT_CONTINUE) return true; diff --git a/mm/vmscan.c b/mm/vmscan.c index 723705b9e4d9..14d6116384cc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6277,7 +6277,7 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) sc->reclaim_idx, 0)) return true; - /* Compaction cannot yet proceed. Do reclaim. */ + /* Compaction cannot yet proceed, might need reclaim */ if (compaction_suitable(zone, sc->order, sc->reclaim_idx) == COMPACT_SKIPPED) return false; @@ -6357,6 +6357,21 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) sc->reclaim_idx = gfp_zone(sc->gfp_mask); } + /* Bail if any of the zones are already compactable */ + if (IS_ENABLED(CONFIG_COMPACTION) && + sc->order > PAGE_ALLOC_COSTLY_ORDER) { + for_each_zone_zonelist_nodemask(zone, z, zonelist, + sc->reclaim_idx, sc->nodemask) { + if (!cpuset_zone_allowed(zone, + GFP_KERNEL | __GFP_HARDWALL)) + continue; + if (compaction_ready(zone, sc)) { + sc->compaction_ready = true; + goto out; + } + } + } + for_each_zone_zonelist_nodemask(zone, z, zonelist, sc->reclaim_idx, sc->nodemask) { /* @@ -6368,22 +6383,6 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) GFP_KERNEL | __GFP_HARDWALL)) continue; - /* - * If we already have plenty of memory free for - * compaction in this zone, don't free any more. - * Even though compaction is invoked for any - * non-zero order, only frequent costly order - * reclamation is disruptive enough to become a - * noticeable problem, like transparent huge - * page allocations. - */ - if (IS_ENABLED(CONFIG_COMPACTION) && - sc->order > PAGE_ALLOC_COSTLY_ORDER && - compaction_ready(zone, sc)) { - sc->compaction_ready = true; - continue; - } - /* * Shrink each node in the zonelist once. If the * zonelist is ordered by zone (not the default) then a @@ -6420,7 +6419,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) if (first_pgdat) consider_reclaim_throttle(first_pgdat, sc); - +out: /* * Restore to original mask to avoid the impact on the caller if we * promoted it to __GFP_HIGHMEM.