From patchwork Thu Dec 22 04:19:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13079363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67610C4332F for ; Thu, 22 Dec 2022 04:19:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5508A940009; Wed, 21 Dec 2022 23:19:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48905940007; Wed, 21 Dec 2022 23:19:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D9A0940009; Wed, 21 Dec 2022 23:19:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 184B0940007 for ; Wed, 21 Dec 2022 23:19:47 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DADD5404A9 for ; Thu, 22 Dec 2022 04:19:46 +0000 (UTC) X-FDA: 80268638772.17.329C8EB Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 5C1EB140007 for ; Thu, 22 Dec 2022 04:19:45 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YYDj3c5f; spf=pass (imf09.hostedemail.com: domain of 34NqjYwYKCEk95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34NqjYwYKCEk95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671682785; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BDLLEQAWPiuWvu6uAw7SlKRBGnG2Q5GXtGAMB+TrPE8=; b=FW085sihti/MvpY7IpeIT27sSmPDd9xrqn3t3BNrywA9pcyYP55IxUW+EPotouUGD84yzE 8nUJiqJcxZkqvUFwU0QHzsIEaK+4gHP18xoF2oQeH2llanS/W2BH5sP734RkPm179p9HpY JvU8g0nbAU8rEC4q39p/nYvPREjy1uk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YYDj3c5f; spf=pass (imf09.hostedemail.com: domain of 34NqjYwYKCEk95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34NqjYwYKCEk95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671682785; a=rsa-sha256; cv=none; b=dIEfqA46bGVjcJZU/S1hd3arvfGsVxNHJDwNCCPrHQDSXdYkhc55Zg0mUN9zxN6DLaFV8F azL8MJORI2NS48QN+7RCigZPJESCmJ0KZzeKYxG403xNaxTuCI8FM5Hn7WEOQk4PKCfCcV 6sTw0o0tEy5OLd0+SCVlJj51n/jn9N4= Received: by mail-yb1-f202.google.com with SMTP id y6-20020a25b9c6000000b006c1c6161716so685647ybj.8 for ; Wed, 21 Dec 2022 20:19:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BDLLEQAWPiuWvu6uAw7SlKRBGnG2Q5GXtGAMB+TrPE8=; b=YYDj3c5fM6hkrbkgsT5AQ5QuP7PmzqKTtUZOCkQp6tMZe2KtlQXDXuSeID5qZPLv5w uekCpC4/7VqRHl3u7ovnnOyeEqj/EsywNvGk0SU8TWCKdL5T+UUJ8v4uRxtZmS6vPnl7 Wzj8lALjPQjik55Xsborgrd9EdV5KGHkidFuPRxek7BFO5jMvsBbkzoNYwVZleOoZih8 UbO7SNB2W/YpVQufPScszqS4sLh/XpytF5xvF1sxeegGQeYmCWYdyiAVgWPTJHcniUSO tcG3i3xlYkukeP0JsAW7kB7ooTJnC5WmKclKn5JT2ujNOCPHbdsDPc8vVma69pRmczRk 1nkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BDLLEQAWPiuWvu6uAw7SlKRBGnG2Q5GXtGAMB+TrPE8=; b=BE72Rxwyi7aciYciVaamvXEXXPO1PRK8bJNoO4MELUyHeiQacGaa0+sML10cJjurFM i0lkc+tYu4WIvmGILwn4zCjx8Ot9tLpHXhxIbrRTG9FxeErE2kn9KH+a6mq1bJnvgKyx Z1NynjVeQrXEuAAM3cu9TDdN6XdTLsSffTZ4hFD7Zvt4otWLf9yZdezDjsEq5ldauS78 bTu7drn3LFv1+fzTHz0Rg/xwMrkAQN/DbFR+X05iZly0ADN1mmCyWgV4n8y+4qP+elq1 b9PGhw1S7rL3SdSBQWN3yuy5z1Q/h7HhiPnxTaVrMs4WmcgJq+6UitPKnVCm4d+zeI8f jbUg== X-Gm-Message-State: AFqh2kq2Jrtctf8/nPjjr2frfwxv2EFxr979QdlTbqMgx+70LbDgYiHC 9wJrKVpO7iMXabtTjowVl5WAZmdvvMs= X-Google-Smtp-Source: AMrXdXs9f5JEJZRSlLA1c9vebSH3Lts9X7RAt6HHv3GXYgSIKO08I/5btJz8yV9BWIG6W3F5rzxEG2SadvM= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a463:5f7b:440e:5c77]) (user=yuzhao job=sendgmr) by 2002:a25:dcc7:0:b0:6fa:5ab4:12b5 with SMTP id y190-20020a25dcc7000000b006fa5ab412b5mr381185ybe.620.1671682784642; Wed, 21 Dec 2022 20:19:44 -0800 (PST) Date: Wed, 21 Dec 2022 21:19:01 -0700 In-Reply-To: <20221222041905.2431096-1-yuzhao@google.com> Message-Id: <20221222041905.2431096-4-yuzhao@google.com> Mime-Version: 1.0 References: <20221222041905.2431096-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v3 3/8] mm: multi-gen LRU: remove eviction fairness safeguard From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5C1EB140007 X-Stat-Signature: haeh4rwm4r4ofwjng36g4nuwspj7w5rf X-HE-Tag: 1671682785-967073 X-HE-Meta: U2FsdGVkX18U91SCoEkC2dd1gnR6grPd6WtKvnPRUva0ep4BPR7Gp1A1uY4GTfNkiGdQwjNxT4FAgjLF59VM11Emxwwns5VwyQNX/LHwhs2Cthz/mxMzpNa+LqiM5vUFdD1OeWIfZKDB/C1lw1XvVQ8fD8gZGRwQHxdfn9kV3F2QbbV74ONArUEpHMN04IMKNdshZzgkW3KSI8ksMSXyB9tAYOXxO/BDeDXj7QgXfMOt3uj/zPnJjvly6Z3il0cMkBtuc5eif75VDepnma+t3ocAD0I9I5V5cyF91hHiMmBqckJtUml74ObqZRQULcC59nq7gmF/RC9WTQWp9/BrgnwLG9KJ7Gk8AOAe3sJhHgdTTwg/UKo3QBeYIp5gvHD468tvJoP3uVZWIr9jOkgL/kQ8RQbByfRO9O8k+6AGWxx9vxhdSIlAoLiLimDNWsis5Mnhza0Je2yxi1Ce8v8wB3M1RCKLldWdu/r9nRLvZViUVv0vM1TLw4V2hG6vpI3Idn5TnBPV5DXrtbIGlUUV+qZEduX6eRKUUTJy0Z7REmB7UiDgDHnb0hFUVIa06kHp8Jjo1CLFbEKCcFpMH/DFZBAmHYTj1E2MvXzn8Bm96u0riPD7wWoaNpZ2s6GvwZpMXAR52mcFxr77zkcNTlzDBl0YKY/g4h+R3MeQgkRuyEDCkpg0FKEMhNfyZ5Hv8ltBEz1q0PcU9iT7zSNxKCNiYmEBG44SmiIf5QTGF+kY7lF8sjkIuqh24JDu4CYihDlr6fBlqiLtkc7csOK+hzHo5soE0qRiJmDNFQzIyGIIAGoUR+4iGQbsSbpzEUdgQjT84umqFDEStA9jXjDh4+4X4Tya5fcEolIwYgaSVPSILAESUgNSw1pzwxBBA3Z2sc0loyx9Z+blYoruDnklThePig+bC2wy1No8N0Mp4i+5dY+Df1k0q3k2bK++qZQWWmtV4fvXqj2i2HdoyjVLhvB ttg1GfCA NB+e8L1kGwMmNC+SwQir9UbULllbxR5C5UuLKLb43hQH/Oj2eKcntOqzSN90yJoO2RMpQ6MrpMGMCNdYFf+4ViviZ5jKiN6epYTdz7/JOe97j8U55D0D5XjR5JQ42FVl1cI9R9j7kmtwZjoK8TnN1wVPMvAsrzh7/qnhNIjdKbZq87cOTzc5YTLNupI20fWMlR5usPwq7eNRvk+HWITrSYtZeA5YzVAtQWj15mw6ofLfEfpvYln/xJV3s3qo7vbjNQ1Z1O8EyVE5jcH3baa4pYceiNrbBVTzlTPnz2LYxUNmBfc35NLf4wSTlJ3YPeD/yfiqVfk6KiTqLNxQqKpwX+SSYg7gYqgx0uC1X5T9cHF+8ygA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recall that the eviction consumes the oldest generation: first it bucket-sorts folios whose gen counters were updated by the aging and reclaims the rest; then it increments lrugen->min_seq. The current eviction fairness safeguard for global reclaim has a dilemma: when there are multiple eligible memcgs, should it continue or stop upon meeting the reclaim goal? If it continues, it overshoots and increases direct reclaim latency; if it stops, it loses fairness between memcgs it has taken memory away from and those it has yet to. With memcg LRU, the eviction, while ensuring eventual fairness, will stop upon meeting its goal. Therefore the current eviction fairness safeguard for global reclaim will not be needed. Note that memcg LRU only applies to global reclaim. For memcg reclaim, the eviction will continue, even if it is overshooting. This becomes unconditional due to code simplification. Signed-off-by: Yu Zhao --- mm/vmscan.c | 81 +++++++++++++++-------------------------------------- 1 file changed, 23 insertions(+), 58 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d94d9fcabf36..49d7c103906a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -449,6 +449,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return sc->target_mem_cgroup; } +static bool global_reclaim(struct scan_control *sc) +{ + return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); +} + /** * writeback_throttling_sane - is the usual dirty throttling mechanism available? * @sc: scan_control in question @@ -499,6 +504,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return false; } +static bool global_reclaim(struct scan_control *sc) +{ + return true; +} + static bool writeback_throttling_sane(struct scan_control *sc) { return true; @@ -5006,8 +5016,7 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; } -static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, - bool *need_swapping) +static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness) { int type; int scanned; @@ -5096,9 +5105,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap goto retry; } - if (need_swapping && type == LRU_GEN_ANON) - *need_swapping = true; - return scanned; } @@ -5138,67 +5144,26 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * return min_seq[!can_swap] + MIN_NR_GENS <= max_seq ? nr_to_scan : 0; } -static bool should_abort_scan(struct lruvec *lruvec, unsigned long seq, - struct scan_control *sc, bool need_swapping) +static unsigned long get_nr_to_reclaim(struct scan_control *sc) { - int i; - DEFINE_MAX_SEQ(lruvec); + /* don't abort memcg reclaim to ensure fairness */ + if (!global_reclaim(sc)) + return -1; - if (!current_is_kswapd()) { - /* age each memcg at most once to ensure fairness */ - if (max_seq - seq > 1) - return true; + /* discount the previous progress for kswapd */ + if (current_is_kswapd()) + return sc->nr_to_reclaim + sc->last_reclaimed; - /* over-swapping can increase allocation latency */ - if (sc->nr_reclaimed >= sc->nr_to_reclaim && need_swapping) - return true; - - /* give this thread a chance to exit and free its memory */ - if (fatal_signal_pending(current)) { - sc->nr_reclaimed += MIN_LRU_BATCH; - return true; - } - - if (cgroup_reclaim(sc)) - return false; - } else if (sc->nr_reclaimed - sc->last_reclaimed < sc->nr_to_reclaim) - return false; - - /* keep scanning at low priorities to ensure fairness */ - if (sc->priority > DEF_PRIORITY - 2) - return false; - - /* - * A minimum amount of work was done under global memory pressure. For - * kswapd, it may be overshooting. For direct reclaim, the allocation - * may succeed if all suitable zones are somewhat safe. In either case, - * it's better to stop now, and restart later if necessary. - */ - for (i = 0; i <= sc->reclaim_idx; i++) { - unsigned long wmark; - struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; - - if (!managed_zone(zone)) - continue; - - wmark = current_is_kswapd() ? high_wmark_pages(zone) : low_wmark_pages(zone); - if (wmark > zone_page_state(zone, NR_FREE_PAGES)) - return false; - } - - sc->nr_reclaimed += MIN_LRU_BATCH; - - return true; + return max(sc->nr_to_reclaim, compact_gap(sc->order)); } static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { struct blk_plug plug; bool need_aging = false; - bool need_swapping = false; unsigned long scanned = 0; unsigned long reclaimed = sc->nr_reclaimed; - DEFINE_MAX_SEQ(lruvec); + unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); lru_add_drain(); @@ -5222,7 +5187,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (!nr_to_scan) goto done; - delta = evict_folios(lruvec, sc, swappiness, &need_swapping); + delta = evict_folios(lruvec, sc, swappiness); if (!delta) goto done; @@ -5230,7 +5195,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (scanned >= nr_to_scan) break; - if (should_abort_scan(lruvec, max_seq, sc, need_swapping)) + if (sc->nr_reclaimed >= nr_to_reclaim) break; cond_resched(); @@ -5677,7 +5642,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co if (sc->nr_reclaimed >= nr_to_reclaim) return 0; - if (!evict_folios(lruvec, sc, swappiness, NULL)) + if (!evict_folios(lruvec, sc, swappiness)) return 0; cond_resched();