From patchwork Sun Sep 18 20:47:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12979593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EBC6ECAAD8 for ; Sun, 18 Sep 2022 20:48:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B003940011; Sun, 18 Sep 2022 16:48:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 137DF940008; Sun, 18 Sep 2022 16:48:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7F42940011; Sun, 18 Sep 2022 16:48:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D89E3940008 for ; Sun, 18 Sep 2022 16:48:14 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B64B0A0353 for ; Sun, 18 Sep 2022 20:48:14 +0000 (UTC) X-FDA: 79926393708.19.A1DC0A5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 75EA0C0004 for ; Sun, 18 Sep 2022 20:48:14 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-349f88710b2so80747247b3.20 for ; Sun, 18 Sep 2022 13:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date; bh=4dMrFmMaEOJAkK/RVbE/otUUczJ0DfDH7kAUZcwg9Yo=; b=mic9okYpjd7t6Jy5SOgxblMABXWu8QZ9mNcw9JTed6mYg+nmY7RzlqvVB2ErsX6N4a yAB4ietgY9OuD9fXPDEDPy0eToAZXtqKs/HCTWY2T8YtgmimfMEjFHyKlZ3J1c3Bh+/F yDHNs3ETSKaBkxzO3Vnq0/rKmhydea7Fa4MjlC61kjBSNXAIjgpRMvQmTc5OeBgB6v1O 8C5jJrAFdrqUM8VKYasNHPNHq2UVOMeSsx0mLqNPrcbNniW+xgU7kyUtHqO++Bnc/Tmd nzC4EG5g7KoIuQDITJteNalaKHst/GzH45goMiHGNz53zLsTlp/9NkinSzaTjk7xT25Z qpeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=4dMrFmMaEOJAkK/RVbE/otUUczJ0DfDH7kAUZcwg9Yo=; b=6uNu6OzDLOnfeWXQJ0Dm2H9YVXUF+JtjzHoZw7pjPRss90bQr0ePrBsyfUlE995mvO Fph+T3XhVn6Fqglyhw+Q3kgFdu563299T3VRhgVO3ypvKCmSrB5gXqm8fqjQxM7ZViCw lPbUZ6C3r3slIFfzxPnW0RVYkW/9HwvbCjvYDdWk3M2FizZJ9ugh844vmhmhveV+H4lw v2mJGp8MbGJW6r0WPLsI/Vsgxq9I84f4s1edJiv5RALvGOI+QvaXLNEDM7/9Zjb6WcCN bjkAuAjSIvpKXuDMtv/wDL7pRdX6bWUvjQIxP6wlucpge2v6q6eDo/kWigr27yX3w+vu JpDw== X-Gm-Message-State: ACrzQf3GUhpCsX0DO/+0tvaf5De2uGb/HDr30aWykkLFa8ZWmuD/sSKI nvYUGHaVVzxjhWeYUp5KoCOpl4t8CoM= X-Google-Smtp-Source: AMsMyM69sdTfA4Q/ZiGuBDxMN+9Z3VE4aZlnyp0qY7hf8grZciFnwRGs3pmk9Nqfr2T6UdRBc3cvPBYSRRA= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:ccff:7b62:4ed6:2946]) (user=yuzhao job=sendgmr) by 2002:a25:3491:0:b0:6ae:e4dc:93ec with SMTP id b139-20020a253491000000b006aee4dc93ecmr11873221yba.26.1663534093878; Sun, 18 Sep 2022 13:48:13 -0700 (PDT) Date: Sun, 18 Sep 2022 14:47:55 -0600 In-Reply-To: <20220918204755.3135720-1-yuzhao@google.com> Message-Id: <20220918204755.3135720-11-yuzhao@google.com> Mime-Version: 1.0 References: <20220918204755.3135720-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog Subject: [PATCH v14-fix 11/11] mm: multi-gen LRU: refactor get_nr_evictable() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663534094; a=rsa-sha256; cv=none; b=muAeCD+q7FzSjhX+zcCHA6+fGhVLQZM+VGGAdVba8lHzL6iCjih5w+wMShIST6cw/jTTbw lRhqMpt2Y2yneGdIbfg849KpunaHGJ2DuJN2pKp/k0C7SbsAqZZ6Zz5x6YAzQBjyF7ICFu h/LbERvDucXHiHEdsOLHe8641GfGrL4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mic9okYp; spf=pass (imf28.hostedemail.com: domain of 3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663534094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4dMrFmMaEOJAkK/RVbE/otUUczJ0DfDH7kAUZcwg9Yo=; b=PmFz7yCHJtu5kIjwILxdVFpsmDdhUquUUEr69fHK2A91YbsJFgedkFzu8vhVG7nRde6bXU fbt1SdZbQPe4BJ+bykixq0iXsWTXMWxduxc0JxpAA4xkgJ9x7SJN2j/jXY8U8a+sksG8I1 IbPDUy9mapXyzhe3Mqb294TBCY7tYLM= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mic9okYp; spf=pass (imf28.hostedemail.com: domain of 3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DYQnYwYKCNIMIN5yC4CC492.0CA96BIL-AA8Jy08.CF4@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: kdz4jantwsuriickbj74z8o8rgajmipt X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 75EA0C0004 X-HE-Tag: 1663534094-673552 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous commit made this refactor possible: 28 insertions and 36 deletions. Signed-off-by: Yu Zhao --- mm/vmscan.c | 64 +++++++++++++++++++++++------------------------------ 1 file changed, 28 insertions(+), 36 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 50764b2d462f..a8fd6300fa7e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4418,14 +4418,15 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static unsigned long get_nr_evictable(struct lruvec *lruvec, unsigned long max_seq, - unsigned long *min_seq, bool can_swap, bool *need_aging) +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsigned long *min_seq, + struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) { int gen, type, zone; unsigned long old = 0; unsigned long young = 0; unsigned long total = 0; struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); for (type = !can_swap; type < ANON_AND_FILE; type++) { unsigned long seq; @@ -4441,35 +4442,37 @@ static unsigned long get_nr_evictable(struct lruvec *lruvec, unsigned long max_s total += size; if (seq == max_seq) young += size; - if (seq + MIN_NR_GENS == max_seq) + else if (seq + MIN_NR_GENS == max_seq) old += size; } } + /* try to scrape all its memory if this memcg was deleted */ + *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; + /* - * The aging tries to be lazy to reduce the overhead. On the other hand, - * the eviction stalls when the number of generations reaches - * MIN_NR_GENS. So ideally, there should be MIN_NR_GENS+1 generations, - * hence the first two if's. - * - * Also it's ideal to spread pages out evenly, meaning 1/(MIN_NR_GENS+1) + * The aging tries to be lazy to reduce the overhead, while the eviction + * stalls when the number of generations reaches MIN_NR_GENS. Hence, the + * ideal number of generations is MIN_NR_GENS+1. + */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) + return true; + if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + return false; + + /* + * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) * of the total number of pages for each generation. A reasonable range * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * eviction cares about the lower bound of cold pages, whereas the aging - * cares about the upper bound of hot pages. + * aging cares about the upper bound of hot pages, while the eviction + * cares about the lower bound of cold pages. */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) - *need_aging = true; - else if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - *need_aging = false; - else if (young * MIN_NR_GENS > total) - *need_aging = true; - else if (old * (MIN_NR_GENS + 2) < total) - *need_aging = true; - else - *need_aging = false; + if (young * MIN_NR_GENS > total) + return true; + if (old * (MIN_NR_GENS + 2) < total) + return true; - return total; + return false; } static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned long min_ttl) @@ -4488,11 +4491,7 @@ static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned if (mem_cgroup_below_min(memcg)) return false; - nr_to_scan = get_nr_evictable(lruvec, max_seq, min_seq, swappiness, &need_aging); - if (!nr_to_scan) - return false; - - nr_to_scan >>= mem_cgroup_online(memcg) ? sc->priority : 0; + need_aging = should_run_aging(lruvec, max_seq, min_seq, sc, swappiness, &nr_to_scan); if (min_ttl) { int gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); @@ -4506,7 +4505,7 @@ static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned return false; } - if (nr_to_scan && need_aging) + if (need_aging) try_to_inc_max_seq(lruvec, max_seq, sc, swappiness, false); return true; @@ -5054,14 +5053,7 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * (mem_cgroup_below_low(memcg) && !sc->memcg_low_reclaim)) return 0; - nr_to_scan = get_nr_evictable(lruvec, max_seq, min_seq, can_swap, need_aging); - if (!nr_to_scan) - return 0; - - nr_to_scan >>= mem_cgroup_online(memcg) ? sc->priority : 0; - if (!nr_to_scan) - return 0; - + *need_aging = should_run_aging(lruvec, max_seq, min_seq, sc, can_swap, &nr_to_scan); if (!*need_aging) return nr_to_scan;