From patchwork Thu Dec 1 22:39:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13061874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6DF1C3A5A7 for ; Thu, 1 Dec 2022 22:39:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 142088E0001; Thu, 1 Dec 2022 17:39:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 11BD46B0078; Thu, 1 Dec 2022 17:39:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAE388E0001; Thu, 1 Dec 2022 17:39:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DD2746B0075 for ; Thu, 1 Dec 2022 17:39:45 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B74CEA02C3 for ; Thu, 1 Dec 2022 22:39:45 +0000 (UTC) X-FDA: 80195205930.07.AC5DCAB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 59891180009 for ; Thu, 1 Dec 2022 22:39:45 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZZB8O77v; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3MC2JYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3MC2JYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669934385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n1CAWLF/Pr37Q+tUzT+ZYK1OUBV6p0czcCCYFvheAt8=; b=WK9zLOSIUCJAuVPG38Qnm1bXdnCgBslOt/ECodVZVfFhUazF66jCq5QgQ8lV+QTuyRdpSS DZoWUoA+E0etAWOZszZSht8m5WW8taJQk6XJnCB2Lx+Tivmu3ZbVtifya0Nax/RmL7oCA9 61oqnD6mF0jb99FtZlYa3xRSaJj7Pnk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZZB8O77v; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3MC2JYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3MC2JYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669934385; a=rsa-sha256; cv=none; b=lI1ulBgrQEzh32+zZrqHHnb2iveCmQLvZ9HtH/w6QKWTbP/ylq0nlaKDDB17E2zcRlPmHw TO0b4I6II0ANe8dRLLZ3G3ZQU7Xr40843WzkiFPIV1TJYBwkKOnPw7ZO4nrKdi/PQl36HF +p+hsBe5yWegUmtVYS8eImSA0KBGpDY= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-37360a6236fso30704557b3.12 for ; Thu, 01 Dec 2022 14:39:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n1CAWLF/Pr37Q+tUzT+ZYK1OUBV6p0czcCCYFvheAt8=; b=ZZB8O77vizkwbL+BtNrQny9vAyzGkEzi6VNYlAJ8o3lRHt9ct0ErfZZsrVGL3AcXQ1 g6b3y+/raQL/iqTo2KKKu4ie98C0Ave3f8NrVN4wj/zrtMmGBYSvl4T6CxGDB7tYzaQr Mq6iEjuAJj1LYaWFXS1ohNm+DICTB3/SMAmq6gEM6dCOOmmlvoh7c2ITyousM2yV6+/7 P+ER0w42T/CXOrOJHoQD91ZeM7pLhdUBj5LaNFQ+gfq1SEj9ZiGnCzeY+9hfLaeDKMPw aN+UMB3gHzXUEbX2Sk1x5SS7+I0WdPDFmnbV5uPjCh5o+bqRBhMmtBtIQkXnnW5Ve/GZ 4e4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n1CAWLF/Pr37Q+tUzT+ZYK1OUBV6p0czcCCYFvheAt8=; b=A4w5ZfyMOrHa09ciWoXc/EXM6jFsBlPsakkFEo38dPMhm/V1DS20iotNowo4AyASty XrP+MXhprs909cn7yjvooDoT8vkUUD3eQsFFb+JmzXigxylRYG6zhpz24aku6CXXkDKX Rh5o57TEnCXdAJqHyxg6UNwfRirkK2wt7AQT5eryqUSHcI4h4lHNUTUnOwNtylAYdYlQ vod6pzKSOZanwdAkuvxlSFV5q3gnGEHww2WO/VGolx8Ie0wPZIhMxcQXsKgD5wLXiIvn iCTNe20qZ0f/x9O2vQEOLf/Lk8IsLl7Oz4tm5Hs2B3zVGU6o4/qQuTwSYatkPeGk006V dTqw== X-Gm-Message-State: ANoB5pkYXkU+aoHDhcNUaK8Ysb37gScHKl77+8bKp1b15d8selLAm3+6 50NUFKHes9vFwmitVuRj0kmaHpefs/o= X-Google-Smtp-Source: AA0mqf59sgS0ZhfbHLTxSsJYv8pDkOxnnBfFny9mVriiFvV1ECl7PNXEtC23rhvo7tBw7R7x1gSSmc1Y5OE= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:1d8c:fe8c:ee3e:abb]) (user=yuzhao job=sendgmr) by 2002:a81:5f83:0:b0:367:a786:8318 with SMTP id t125-20020a815f83000000b00367a7868318mr47352781ywb.367.1669934384612; Thu, 01 Dec 2022 14:39:44 -0800 (PST) Date: Thu, 1 Dec 2022 15:39:19 -0700 In-Reply-To: <20221201223923.873696-1-yuzhao@google.com> Message-Id: <20221201223923.873696-4-yuzhao@google.com> Mime-Version: 1.0 References: <20221201223923.873696-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.rc0.267.gcb52ba06e7-goog Subject: [PATCH mm-unstable v1 3/8] mm: multi-gen LRU: remove eviction fairness safeguard From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Spamd-Result: default: False [1.90 / 9.00]; SORBS_IRL_BL(3.00)[209.85.128.202:from]; BAYES_HAM(-3.00)[99.97%]; MID_CONTAINS_FROM(1.00)[]; MV_CASE(0.50)[]; FORGED_SENDER(0.30)[yuzhao@google.com,3MC2JYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_DKIM_ALLOW(0.00)[google.com:s=20210112]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; RCPT_COUNT_TWELVE(0.00)[12]; DKIM_TRACE(0.00)[google.com:+]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DMARC_POLICY_ALLOW(0.00)[google.com,reject]; FROM_NEQ_ENVFROM(0.00)[yuzhao@google.com,3MC2JYwYKCNELHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[] X-Stat-Signature: 9typ77i9nww98xwhy53daewxkbiwxbbo X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 59891180009 X-Rspam-User: X-HE-Tag: 1669934385-474885 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recall that the eviction consumes the oldest generation: first it bucket-sorts folios whose gen counters were updated by the aging and reclaims the rest; then it increments lrugen->min_seq. The current eviction fairness safeguard for global reclaim has a dilemma: when there are multiple eligible memcgs, should it continue or stop upon meeting the reclaim goal? If it continues, it overshoots and increases direct reclaim latency; if it stops, it loses fairness between memcgs it has taken memory away from and those it has yet to. With memcg LRU, the eviction, while ensuring eventual fairness, will stop upon meeting its goal. Therefore the current eviction fairness safeguard for global reclaim will not be needed. Note that memcg LRU only applies to global reclaim. For memcg reclaim, the eviction will continue, even if it is overshooting. This becomes unconditional due to code simplification. Signed-off-by: Yu Zhao --- mm/vmscan.c | 81 +++++++++++++++-------------------------------------- 1 file changed, 23 insertions(+), 58 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ebab1ec3d400..d714a777c88b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -449,6 +449,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return sc->target_mem_cgroup; } +static bool global_reclaim(struct scan_control *sc) +{ + return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); +} + /** * writeback_throttling_sane - is the usual dirty throttling mechanism available? * @sc: scan_control in question @@ -499,6 +504,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return false; } +static bool global_reclaim(struct scan_control *sc) +{ + return true; +} + static bool writeback_throttling_sane(struct scan_control *sc) { return true; @@ -4991,8 +5001,7 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; } -static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, - bool *need_swapping) +static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness) { int type; int scanned; @@ -5081,9 +5090,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap goto retry; } - if (need_swapping && type == LRU_GEN_ANON) - *need_swapping = true; - return scanned; } @@ -5122,67 +5128,26 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * return min_seq[!can_swap] + MIN_NR_GENS <= max_seq ? nr_to_scan : 0; } -static bool should_abort_scan(struct lruvec *lruvec, unsigned long seq, - struct scan_control *sc, bool need_swapping) +static unsigned long get_nr_to_reclaim(struct scan_control *sc) { - int i; - DEFINE_MAX_SEQ(lruvec); + /* don't abort memcg reclaim to ensure fairness */ + if (!global_reclaim(sc)) + return -1; - if (!current_is_kswapd()) { - /* age each memcg at most once to ensure fairness */ - if (max_seq - seq > 1) - return true; + /* discount the previous progress for kswapd */ + if (current_is_kswapd()) + return sc->nr_to_reclaim + sc->last_reclaimed; - /* over-swapping can increase allocation latency */ - if (sc->nr_reclaimed >= sc->nr_to_reclaim && need_swapping) - return true; - - /* give this thread a chance to exit and free its memory */ - if (fatal_signal_pending(current)) { - sc->nr_reclaimed += MIN_LRU_BATCH; - return true; - } - - if (cgroup_reclaim(sc)) - return false; - } else if (sc->nr_reclaimed - sc->last_reclaimed < sc->nr_to_reclaim) - return false; - - /* keep scanning at low priorities to ensure fairness */ - if (sc->priority > DEF_PRIORITY - 2) - return false; - - /* - * A minimum amount of work was done under global memory pressure. For - * kswapd, it may be overshooting. For direct reclaim, the allocation - * may succeed if all suitable zones are somewhat safe. In either case, - * it's better to stop now, and restart later if necessary. - */ - for (i = 0; i <= sc->reclaim_idx; i++) { - unsigned long wmark; - struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; - - if (!managed_zone(zone)) - continue; - - wmark = current_is_kswapd() ? high_wmark_pages(zone) : low_wmark_pages(zone); - if (wmark > zone_page_state(zone, NR_FREE_PAGES)) - return false; - } - - sc->nr_reclaimed += MIN_LRU_BATCH; - - return true; + return max(sc->nr_to_reclaim, compact_gap(sc->order)); } static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { struct blk_plug plug; bool need_aging = false; - bool need_swapping = false; unsigned long scanned = 0; unsigned long reclaimed = sc->nr_reclaimed; - DEFINE_MAX_SEQ(lruvec); + unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); lru_add_drain(); @@ -5206,7 +5171,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (!nr_to_scan) goto done; - delta = evict_folios(lruvec, sc, swappiness, &need_swapping); + delta = evict_folios(lruvec, sc, swappiness); if (!delta) goto done; @@ -5214,7 +5179,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (scanned >= nr_to_scan) break; - if (should_abort_scan(lruvec, max_seq, sc, need_swapping)) + if (sc->nr_reclaimed >= nr_to_reclaim) break; cond_resched(); @@ -5661,7 +5626,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co if (sc->nr_reclaimed >= nr_to_reclaim) return 0; - if (!evict_folios(lruvec, sc, swappiness, NULL)) + if (!evict_folios(lruvec, sc, swappiness)) return 0; cond_resched();