From patchwork Fri Dec 8 06:14:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13484884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01AC1C4167B for ; Fri, 8 Dec 2023 06:14:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75C8E6B007E; Fri, 8 Dec 2023 01:14:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E4906B0081; Fri, 8 Dec 2023 01:14:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55E836B0082; Fri, 8 Dec 2023 01:14:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3B4766B007E for ; Fri, 8 Dec 2023 01:14:18 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0EB6880292 for ; Fri, 8 Dec 2023 06:14:18 +0000 (UTC) X-FDA: 81542636196.03.33F338C Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 57F5380008 for ; Fri, 8 Dec 2023 06:14:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2n6bpTcR; spf=pass (imf02.hostedemail.com: domain of 3N7RyZQYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3N7RyZQYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702016056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MPyzSKrXH5Tqmu9fHEEk3D/5egUUX9zjhxgsMiAOvSY=; b=vXAoQ7d4RGiBH+cWA84Ka7way+pdAS/5NLDuMJ72H+BjrblRg/ZNOeTyXKR8LJeWpFCdYd qKt3zPgUPYscPlpQgMszLNUZVrjS+JyT6FvIELYqkp3jFUf1fbF6p3fgXz83koArr8QKdv GJMMpptDJiwUcJgzr1c6Ca3O6GcuutM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702016056; a=rsa-sha256; cv=none; b=qNqUpFfQgCP3YTcq4S2XvaXJphOmBB7U1FRsNZWvrvyclPMrhDXOiHuEGYbEAkIHVcOO0I DWRW8rpHYZFW2niDS+qSjf4OkzZi7oClV/IRrD7NQ8hQsOzB8LxfT0cxOW54JRCvSXpBmg TUsP8uknppv9Hk7mxmmxMU6KRjgbWZA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2n6bpTcR; spf=pass (imf02.hostedemail.com: domain of 3N7RyZQYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3N7RyZQYKCJwUQVD6KCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-db542ec49d9so2325170276.0 for ; Thu, 07 Dec 2023 22:14:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1702016055; x=1702620855; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MPyzSKrXH5Tqmu9fHEEk3D/5egUUX9zjhxgsMiAOvSY=; b=2n6bpTcRhPSp/vA2qKXmPh1YaZJQLGlTTnk0wmkZJ8vmgzOdlNqipoBGkxvmsnY/JT pceWz7Up1eM296/YElKT6Wlf2rBA5zJGY+j5MIfv6DNyWNlV1V5AIPdd1Fffv2hZjW0i 0f2t1Qc8lTDSctqpc8jQ5B7i+k8rvh+WQ2izBnLdKUQMqOl/KjhVvsIFWfqnxvY7tP3e cz9+OU3MDCLg2OjJhPkxdXS8Rk50G4j8gx7R8Q3S2R8qdxVBbK83i85G9dplO+bB4cNG Sb3mCuNOGin/fj611cOxqhKAh2UVkN6m/RVXJ29+VvPc+oj5iKvvRYy0SYJ0S9DEVr2L Gccw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702016055; x=1702620855; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MPyzSKrXH5Tqmu9fHEEk3D/5egUUX9zjhxgsMiAOvSY=; b=Pnlq1u3qdH7YPfBUMdCkKAgrs2dSjG1V66gTEZAPuE7aNEv+6KuEqiKD+ovaHkoZw+ VUrFZyH93AWF2IMuyCs1JHb1tSIVfifnF8HCv5TZbYd3paO3TfuxvX19RLw+XATT5zfd 9/o/Qg1u82s30wCBWkLy6QxGWSXMen2XKYKJkIvkSqWxHthOxP1KQpNxkSRfqKnP5y0e M2XNFas7r1zHvDJzRzF5eykjiN5LK7KgClNZntqzWGdeUcbhISiT70d5tqWi+mPilg53 zkryT7PvAZXrGJSAdTlzdtm2yuqXUr2gpEzym/ww4bVWd3hg4NEVcw6VnK3oHnSp3oC+ 863Q== X-Gm-Message-State: AOJu0Yx+hJoxxIOOJMlPG05E6NmMoeJQoB9EL3t+sbq6G4/unEI8UI4k pp3Eo5Hvb5TdJnvmTGwA9QOWsY+VeHA= X-Google-Smtp-Source: AGHT+IGdRJ6oJBGbszEwYJdNZWMiTiNDeQOA8ujBPwzvHJj4q8s9r2Akp7Hq9nIVYDsXPZgw2A9/YR7iKD4= X-Received: from yuzhao2.bld.corp.google.com ([100.64.188.49]) (user=yuzhao job=sendgmr) by 2002:a25:76d5:0:b0:da0:3da9:ce08 with SMTP id r204-20020a2576d5000000b00da03da9ce08mr52921ybc.10.1702016055395; Thu, 07 Dec 2023 22:14:15 -0800 (PST) Date: Thu, 7 Dec 2023 23:14:05 -0700 In-Reply-To: <20231208061407.2125867-1-yuzhao@google.com> Mime-Version: 1.0 References: <20231208061407.2125867-1-yuzhao@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208061407.2125867-2-yuzhao@google.com> Subject: [PATCH mm-unstable v1 2/4] mm/mglru: try to stop at high watermarks From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Charan Teja Kalla , Jaroslav Pulchart , Kalesh Singh , stable@vger.kernel.org X-Rspamd-Queue-Id: 57F5380008 X-Rspam-User: X-Stat-Signature: fjt1jrjmngnksuhhqdx4x3tw4d1uo7x5 X-Rspamd-Server: rspam03 X-HE-Tag: 1702016056-101804 X-HE-Meta: U2FsdGVkX1/CHSVylP+WipuUv5AQpH4fqtMw00W24ih8gfdXzkrwCxIsQSS2Wmhss+ITnqjZZmpDjzDekNUexh0QhcbLPvCK5z8X+At/X7K5HGB7Awan5bECDjHmfQ+UzRIb2VpM8UrJOLYp26I2WioossHPXC6JzMs9ggjotLl4Jc9hQugSRZKPYEuM4Qrr8yhq8XPobR6qWmwiUwbLzXsSiR+1yS8DPSE3QzoMabzMX+kDxKNFypf1U4DQ/fF1pllKvgoZi1Pv9/FIyzfCBm4fYhM3jTcw2Q2a9LazjfFUxVghYYktlEYTn2Ot5KWmYdmoFokdbPpN32CuIGrbHBjrwHK4Gk6S7ECo2XF2YcV1gofAe/3taiw76Au8ggo2PPdWAa16F83U8/8gr/qkzkvMeglvdz/DRAdb/yVMP3c78VU8I9CoCzpLeOEnDjUABgoZWII1PNeArkDAry9RykP6Ln3JgydWyBNPA8lQRQbDICEcag6wXAwgsGoBhuRyZMwTdYV4uxu7oghciWdsEAPlKdvUzE4+Zu4S2+zHx/1/sc2Vd9V22doswVgbEuX9ko5iZvSazGzvawlYzFJftJtdygSZMhr93Yf7NLv3QmJ96ihrXk85F+VbohhpyUOg4DND/oCPZUFawuhnaxP21g6CE8q4X/Dv7ZPmmbQaAu9drnGwTNVfKWpCEQhOVSR3HIi1Zu3Fct9tGOUPsJdXB9qFOvyNTRx2aryYzhjClspGxdx+TQ+8s55dVIL72txPWmtJAUJvXIhCUkn7UmHi3ssMFypp1BuQ1mWRozAlRGUVak/bSrCxRmasXhQvKs6q/VXvZsC50FPDeQR6HErsNhmg7UokarS69QzVnFZUUYWN7vhU+jElJCf2K5XhSMVRIumOn6LffEeS1AUb0MtxTnVrDl3yWmFPRH/nDZIFt27xvWovfsl/gj1+yj6RqY23rFHAj+ca5c4LrQnCtdL Y2N6mVkr UHlJib+lBuFFFAUCqNBIRfpHh6y01syqjL6IvWldZ6+GN5CYIqA4PCh895hWPrgOLgwNCcJcF0lbzE25aOdKGW0pWlakVmB2thEGLZ0rj2H7lzO3rfAMNxWYUOW8U0Ug7iBiBUNNh0EFAweE+JMMu0udEhSlmxJ/Bo914kWyaFRxDdaI2Rv2CPr+wLf5Gm6TKW4rH5HhpEC1SkZQ3o4pw5y+ut6ibirx9IfVAkhiPrWVWkjOygUbS2skugoup8WjbYXz5YALQ9xWaIunRY2/Wpxu8Jgh+zH+0125w3YylIIdIMD9Hfl5PovPLSzxsfKEFWBhRUTD4zFQJkpf1eqvZnIQnUTxvqLGD9vBGLGgMwoCJ6dfxcLJQMJnE34zIvXHiKo6CFpuNFu6QlTqAIi2YMBLZSEe/exhASfh7iseI912UPdPpVxz6Dv/nxkLnZNbMQiiIKOeIvOXPd2SThdySFin2uQslOdbKzAW8fhClgL7WBSfV+aI1l4PsNcoKMB65Hj7THTMyZZsguabAiaZC5oLFBCRSICrPuyV7RGWu2aQOCHPHEG6oXlCRMc8Ip7Ld6YYH2XFICd/0IRe6nTwIHdfCeAKKgxgbG2FiEw3GronxWcTr77InX/OEDRBPdDeg6gJ6c+pTZi1jJVlKNGjo+wC6M54HW4V5GB0qJXxrjjB35Hhxkr7Y5a6znTx6+vQYzqYfk1+uCqoi+r8dwHy8BPCntI+ZH4GT5+MG4MM8QRWpzRs9kVd12I3qdtZvbEM935SIlHeAKIurYVjTI2oznCb/ZC7TTr8KUSMzQlhLUEI2H2UISizHRgNBlA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The initial MGLRU patchset didn't include the memcg LRU support, and it relied on should_abort_scan(), added by commit f76c83378851 ("mm: multi-gen LRU: optimize multiple memcgs"), to "backoff to avoid overshooting their aggregate reclaim target by too much". Later on when the memcg LRU was added, should_abort_scan() was deemed unnecessary, and the test results [1] showed no side effects after it was removed by commit a579086c99ed ("mm: multi-gen LRU: remove eviction fairness safeguard"). However, that test used memory.reclaim, which sets nr_to_reclaim to SWAP_CLUSTER_MAX. So it can overshoot only by SWAP_CLUSTER_MAX-1 pages, i.e., from nr_reclaimed=nr_to_reclaim-1 to nr_reclaimed=nr_to_reclaim+SWAP_CLUSTER_MAX-1. Compared with the batch size kswapd sets to nr_to_reclaim, SWAP_CLUSTER_MAX is tiny. Therefore that test isn't able to reproduce the worst case scenario, i.e., kswapd overshooting GBs on large systems and "consuming 100% CPU" (see the Closes tag). Bring back a simplified version of should_abort_scan() on top of the memcg LRU, so that kswapd stops when all eligible zones are above their respective high watermarks plus a small delta to lower the chance of KSWAPD_HIGH_WMARK_HIT_QUICKLY. Note that this only applies to order-0 reclaim, meaning compaction-induced reclaim can still run wild (which is a different problem). On Android, launching 55 apps sequentially: Before After Change pgpgin 838377172 802955040 -4% pgpgout 38037080 34336300 -10% [1] https://lore.kernel.org/20221222041905.2431096-1-yuzhao@google.com/ Fixes: a579086c99ed ("mm: multi-gen LRU: remove eviction fairness safeguard") Signed-off-by: Yu Zhao Reported-by: Charan Teja Kalla Reported-by: Jaroslav Pulchart Closes: https://lore.kernel.org/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/ Tested-by: Jaroslav Pulchart Tested-by: Kalesh Singh Cc: stable@vger.kernel.org --- mm/vmscan.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e67631c60ac0..10e964cd0efe 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4676,20 +4676,41 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool return try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false) ? -1 : 0; } -static unsigned long get_nr_to_reclaim(struct scan_control *sc) +static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) { + int i; + enum zone_watermarks mark; + /* don't abort memcg reclaim to ensure fairness */ if (!root_reclaim(sc)) - return -1; + return false; - return max(sc->nr_to_reclaim, compact_gap(sc->order)); + if (sc->nr_reclaimed >= max(sc->nr_to_reclaim, compact_gap(sc->order))) + return true; + + /* check the order to exclude compaction-induced reclaim */ + if (!current_is_kswapd() || sc->order) + return false; + + mark = sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ? + WMARK_PROMO : WMARK_HIGH; + + for (i = 0; i <= sc->reclaim_idx; i++) { + struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; + unsigned long size = wmark_pages(zone, mark) + MIN_LRU_BATCH; + + if (managed_zone(zone) && !zone_watermark_ok(zone, 0, size, sc->reclaim_idx, 0)) + return false; + } + + /* kswapd should abort if all eligible zones are safe */ + return true; } static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { long nr_to_scan; unsigned long scanned = 0; - unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); int swappiness = get_swappiness(lruvec, sc); /* clean file folios are more likely to exist */ @@ -4711,7 +4732,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) if (scanned >= nr_to_scan) break; - if (sc->nr_reclaimed >= nr_to_reclaim) + if (should_abort_scan(lruvec, sc)) break; cond_resched(); @@ -4772,7 +4793,6 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) struct lru_gen_folio *lrugen; struct mem_cgroup *memcg; const struct hlist_nulls_node *pos; - unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); bin = first_bin = get_random_u32_below(MEMCG_NR_BINS); restart: @@ -4805,7 +4825,7 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) rcu_read_lock(); - if (sc->nr_reclaimed >= nr_to_reclaim) + if (should_abort_scan(lruvec, sc)) break; } @@ -4816,7 +4836,7 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) mem_cgroup_put(memcg); - if (sc->nr_reclaimed >= nr_to_reclaim) + if (!is_a_nulls(pos)) return; /* restart if raced with lru_gen_rotate_memcg() */