From patchwork Tue Aug 2 16:28:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12934900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 733DFC00140 for ; Tue, 2 Aug 2022 16:28:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D2976B0071; Tue, 2 Aug 2022 12:28:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 882726B0072; Tue, 2 Aug 2022 12:28:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 748C18E0001; Tue, 2 Aug 2022 12:28:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 65F1F6B0071 for ; Tue, 2 Aug 2022 12:28:14 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 342AD1208FD for ; Tue, 2 Aug 2022 16:28:14 +0000 (UTC) X-FDA: 79755184908.17.7B3CE6B Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf09.hostedemail.com (Postfix) with ESMTP id 89052140121 for ; Tue, 2 Aug 2022 16:28:13 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id e12so8418521qkl.2 for ; Tue, 02 Aug 2022 09:28:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc; bh=2DU8LSMpwhI42VEzkmQDivhyfVLUKtdex6r6frtpBtk=; b=Hz4tGLPdmHI5Ynw233elpIWwl+IpLPuYjEvvnz6WSDcQf/94BXv1qgzKy6MtcKo+HZ eNS8xFEqLWEuuwbCRwG2IeGzNAFntMpC9A0GN+YBE5W7rNHzxYS6ZDJBfGrcS8NXYJUM SFDA7qBUDy6tHx/JcMqsgK8Ow06+7TBgxtqXINm/IZlQ67HmYMjUtNXfySPL+uYqXTpd EWn4IsyAIpub0d1CdHY2kLyluZ0t0ycl1OBkIRjRLY69ysaAufC1IeKQR4a4S/87JLEL ygLw7I5gpuioDUuEWc+vZZk/20Ybjbv2kXGva9wdJHBp867smljqHOYLC72XM3Ejgp/s c2fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc; bh=2DU8LSMpwhI42VEzkmQDivhyfVLUKtdex6r6frtpBtk=; b=EJpW/aMxPoR7215KyO7RSzlwK0t6V2TmFESGWmTbt8nvlLCRhZpQty8bgYwrZB7/Gj /I0StghXLegIDsfb9pMQUYLnsLft4mRdvAWco6XybHncc853ra38thvFQ8WhftKIzzp1 EJRQEBtC3Oy09s0jBiqztpoAOnJXTsnIf9it+ULWxoqFqfwq1/BnUsV/4ea0+AFmsx1E UqCfo7G0xU62S4HAqptpiw6MCu1vqyV/Z8F2ev8qXDGm79bm2zml06jAfaWGSDWpY1Qm f1dCdWA24k6HCBPOn1NlqVMBQdF8xO6UW5IeULSwY8ZgSvnAyf5XOSRFoVqUtMmhzFt7 Gw/Q== X-Gm-Message-State: AJIora++ok96IS4iAZtxj+NZacLRHf5rB4AEW9afgkRnQ1Ho5z1+heM+ hzPpP0VNdglFPrJVa5rQ/Y8M1g== X-Google-Smtp-Source: AGRyM1vX3o1EcrOzawr4QvYYmJcA8+LAKFnQxVBxckSLXHI9X12qP84pcIaArycJDHZVaP7nIPdUIQ== X-Received: by 2002:a05:620a:143c:b0:6b8:8d43:5581 with SMTP id k28-20020a05620a143c00b006b88d435581mr10928966qkj.179.1659457692698; Tue, 02 Aug 2022 09:28:12 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::ad0f]) by smtp.gmail.com with ESMTPSA id b23-20020ac86797000000b0031ee34ca176sm9394404qtp.59.2022.08.02.09.28.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Aug 2022 09:28:12 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Mel Gorman , Hugh Dickins , Joonsoo Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH] mm: vmscan: fix extreme overreclaim and swap floods Date: Tue, 2 Aug 2022 12:28:11 -0400 Message-Id: <20220802162811.39216-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659457693; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=2DU8LSMpwhI42VEzkmQDivhyfVLUKtdex6r6frtpBtk=; b=RxYArJzucZ0uRDwM7M8HPWO9pry41VaJrS87uaTq4XBordKlFDCJXpikx1cL+itHz5yiho mB22UDBsAkPzc0hotZVt8k+11KeV0QZLt5lNhYPi6tO+WSPha2n+0SodKb/oD10Gr9HOML oKuxvrMkt+aYlF34Kk9X43HiVgu/h7Y= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=Hz4tGLPd; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf09.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659457693; a=rsa-sha256; cv=none; b=sQkcN8H40BkkNfeXyV8KE+dj/X6gfI1qCsOukl8OALRIVDFE6QjwlGcNKsFu8SaAI7L+Ef QnPeLXHe4niY2TeLDtNNhFQIMPLCqW0fxbQnKPhwKv7j3h0CM9kBmuQ5lnY1W+CMpfmkEx zmHmLYpltFBxD19E0i3F9OdXNxaG2ts= X-Stat-Signature: m87dsqnhdmqg9rwq9muru46gpouj67zi X-Rspamd-Queue-Id: 89052140121 X-Rspamd-Server: rspam01 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=Hz4tGLPd; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf09.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-Rspam-User: X-HE-Tag: 1659457693-6704 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During proactive reclaim, we sometimes observe severe overreclaim, with several thousand times more pages reclaimed than requested. This trace was obtained from shrink_lruvec() during such an instance: prio:0 anon_cost:1141521 file_cost:7767 nr_reclaimed:4387406 nr_to_reclaim:1047 (or_factor:4190) nr=[7161123 345 578 1111] While he reclaimer requested 4M, vmscan reclaimed close to 16G, most of it by swapping. These requests take over a minute, during which the write() to memory.reclaim is unkillably stuck inside the kernel. Digging into the source, this is caused by the proportional reclaim bailout logic. This code tries to resolve a fundamental conflict: to reclaim roughly what was requested, while also aging all LRUs fairly and in accordance to their size, swappiness, refault rates etc. The way it attempts fairness is that once the reclaim goal has been reached, it stops scanning the LRUs with the smaller remaining scan targets, and adjusts the remainder of the bigger LRUs according to how much of the smaller LRUs was scanned. It then finishes scanning that remainder regardless of the reclaim goal. This works fine if priority levels are low and the LRU lists are comparable in size. However, in this instance, the cgroup that is targeted by proactive reclaim has almost no files left - they've already been squeezed out by proactive reclaim earlier - and the remaining anon pages are hot. Anon rotations cause the priority level to drop to 0, which results in reclaim targeting all of anon (a lot) and all of file (almost nothing). By the time reclaim decides to bail, it has scanned most or all of the file target, and therefor must also scan most or all of the enormous anon target. This target is thousands of times larger than the reclaim goal, thus causing the overreclaim. The bailout code hasn't changed in years, why is this failing now? The most likely explanations are two other recent changes in anon reclaim: 1. Before the series starting with commit 5df741963d52 ("mm: fix LRU balancing effect of new transparent huge pages"), the VM was overall relatively reluctant to swap at all, even if swap was configured. This means the LRU balancing code didn't come into play as often as it does now, and mostly in high pressure situations where pronounced swap activity wouldn't be as surprising. 2. For historic reasons, shrink_lruvec() loops on the scan targets of all LRU lists except the active anon one, meaning it would bail if the only remaining pages to scan were active anon - even if there were a lot of them. Before the series starting with commit ccc5dc67340c ("mm/vmscan: make active/inactive ratio as 1:1 for anon lru"), most anon pages would live on the active LRU; the inactive one would contain only a handful of preselected reclaim candidates. After the series, anon gets aged similarly to file, and the inactive list is the default for new anon pages as well, making it often the much bigger list. As a result, the VM is now more likely to actually finish large anon targets than before. Change the code such that only one SWAP_CLUSTER_MAX-sized nudge toward the larger LRU lists is made before bailing out on a met reclaim goal. This fixes the extreme overreclaim problem. Fairness is more subtle and harder to evaluate. No obvious misbehavior was observed on the test workload, in any case. Conceptually, fairness should primarily be a cumulative effect from regular, lower priority scans. Once the VM is in trouble and needs to escalate scan targets to make forward progress, fairness needs to take a backseat. This is also acknowledged by the myriad exceptions in get_scan_count(). This patch makes fairness decrease gradually, as it keeps fairness work static over increasing priority levels with growing scan targets. This should make more sense - although we may have to re-visit the exact values. Signed-off-by: Johannes Weiner Reviewed-by: Rik van Riel Acked-by: Mel Gorman --- mm/vmscan.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f7d9a683e3a7..1cc0c6666787 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2897,8 +2897,8 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) enum lru_list lru; unsigned long nr_reclaimed = 0; unsigned long nr_to_reclaim = sc->nr_to_reclaim; + bool proportional_reclaim; struct blk_plug plug; - bool scan_adjusted; get_scan_count(lruvec, sc, nr); @@ -2916,8 +2916,8 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) * abort proportional reclaim if either the file or anon lru has already * dropped to zero at the first pass. */ - scan_adjusted = (!cgroup_reclaim(sc) && !current_is_kswapd() && - sc->priority == DEF_PRIORITY); + proportional_reclaim = (!cgroup_reclaim(sc) && !current_is_kswapd() && + sc->priority == DEF_PRIORITY); blk_start_plug(&plug); while (nr[LRU_INACTIVE_ANON] || nr[LRU_ACTIVE_FILE] || @@ -2937,7 +2937,7 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) cond_resched(); - if (nr_reclaimed < nr_to_reclaim || scan_adjusted) + if (nr_reclaimed < nr_to_reclaim || proportional_reclaim) continue; /* @@ -2988,8 +2988,6 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) nr_scanned = targets[lru] - nr[lru]; nr[lru] = targets[lru] * (100 - percentage) / 100; nr[lru] -= min(nr[lru], nr_scanned); - - scan_adjusted = true; } blk_finish_plug(&plug); sc->nr_reclaimed += nr_reclaimed;