From patchwork Wed Dec 21 00:12:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A72A5C4167B for ; Wed, 21 Dec 2022 00:12:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61FD88E000A; Tue, 20 Dec 2022 19:12:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AB3E8E0007; Tue, 20 Dec 2022 19:12:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F9BB8E000A; Tue, 20 Dec 2022 19:12:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 27DCD8E0007 for ; Tue, 20 Dec 2022 19:12:45 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DE8AA4049A for ; Wed, 21 Dec 2022 00:12:44 +0000 (UTC) X-FDA: 80264387448.09.9EE804D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf03.hostedemail.com (Postfix) with ESMTP id 48C4920005 for ; Wed, 21 Dec 2022 00:12:43 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pYjtLo94; spf=pass (imf03.hostedemail.com: domain of 3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581563; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5gz0YgVLNDGFuqc3Lh4NnPiIlhtb3z+cq+FKhQZOoZk=; b=xtUiBQ45wV/jD8r4n3J8UozOTg4I1Vc0eL3DrYSFaroH7AGCmdZj0CdgSmgCc5OAmGrtJn X+/thDnjtwlWfIyi0DAvBuwwmpB4uJ02n2mcIRzD1wttszJTLnWbgmiMhiCaiOKDUoDfNs LbLFNiZsHGcOXvXqFNhrgpBEKEUkjPI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pYjtLo94; spf=pass (imf03.hostedemail.com: domain of 3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581563; a=rsa-sha256; cv=none; b=xQ14UdPyn/DWJ0fG9lzI2Nb4gGog7quBl0wA5xOekU+h54W58iU912fsk/AAJbvcHFD5Mq sRuc8SUaJtw/V1Una+OidZjZWWSqunviYsulDNRJwbjXuljKit2m1mFgWFQEtrm8KXPqik ByfIrpeNe1yjKCSivGLZUJa64eU/na8= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a25e311000000b00719e04e59e1so15991219ybd.10 for ; Tue, 20 Dec 2022 16:12:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5gz0YgVLNDGFuqc3Lh4NnPiIlhtb3z+cq+FKhQZOoZk=; b=pYjtLo94emkX9HO5dNLwhnNl1q+/QkmPT4BV1Q2CYxKbEqjmuBFe+6E8hDAD23TmkJ rM8E5hVkIT1iJndDE4GFfnTHJmPy8rBNDiU6xKV12vavnbPR7PD/inFgUi8kelq6KFXs 0OyI9IPv8w8vh7ij2xerLvGSfzs7TWadUGqYgShpNGJ5FVlZQd8389NLfnqGGFtFYhZ5 yMAQh71Vn5/4k+ix3kKLG2LnpGcKhFJb6ylWem23nVTzY3PJF0dMSMUj/ohq835xZL4p HniHGo7EmzRDdAifMnyP6l1XmBjesx3FTwryw+4a+i46NpHmyyWubfVZzsndi55Y/SZd SgAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5gz0YgVLNDGFuqc3Lh4NnPiIlhtb3z+cq+FKhQZOoZk=; b=0muzJXK0JGfizLWKe+EN3pfSAZtqO3wU736PUqGLXPQ4in39rDa+kDVtFxK7AXCSJl C3a0VdZ7yW1FTgE8ieqxrwBap+B9FEQbhAs1TWjcMnmuvOVOuMcXvv7sJho4n2JbnqrK 9UQ2SKbsw8xoFwlRRBm8ESBvIM/BHsKZTyd6YCNyl6dEjccoyqXAqMcx43SWJ5ulX21O Q8Oi5hE9MAtLQKGs6pL32y1ZNOz3ypB6ljl/3x4Q3kG9NbWTA2t1l0AZsTcjom2iL0O4 PDImu5rfEBGe+60ADbISSHRuO2cuK6QJe/vPCvucBZ7U+eHAJ5yCwnKkpBh1is8eTkNZ v/AA== X-Gm-Message-State: ANoB5pmO4ZiCm0+Fe/ruxP0sZJguqnflIkLlzCE6kycwTNT+CMKQl8MX EgHT8Y/bEOzCrO4t5muElW5PAvokcIk= X-Google-Smtp-Source: AA0mqf5J92WPjbmjq0S0/8L8nqyD/H5zHMKEWJbsreI8pJ3tp81Dmd741SGe/PrsNv8NhFNSwOv39dKlDJo= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a81:4d07:0:b0:427:c319:1abe with SMTP id a7-20020a814d07000000b00427c3191abemr3853654ywb.328.1671581562522; Tue, 20 Dec 2022 16:12:42 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:05 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-6-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 5/8] mm: multi-gen LRU: shuffle should_run_aging() From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 48C4920005 X-Stat-Signature: tzw43uh7b8kaydg3kmkap7oye5c4mjx4 X-Rspam-User: X-HE-Tag: 1671581563-663207 X-HE-Meta: U2FsdGVkX19ShBiq8CtWXBi4+/u6JeQHmojKcd+6KH4a8+LvIZhpS9ZFKuG/Qe7Wo3oCkB5s7CdJJq+V0P5x/RnB2rhEyq/jyusOcEvLBjOCVU3jFseim5LriiM74cIRWJeJ1UOvnPXCdl5VNDYX1HzJ90ZsYTZ5z1GM5toFcoDYx8zVBTBdtjth3beoGEA6ZQQb8DILItGfR4xmCiy0X3AlqTgAqVogr6gK49j3mbvIi944RNqETYjLhOg8bl+04tL0WjjgAZQGrW0tGC4hOlxMLsk5ZKZP0D7gN1+5gbFwU9NJjHomOGboIXHbMO5cObn587+s2TD9hjHJBFCYDs7s617wktU6fymD/C4M/5hR3cOBCl9S85QZSHbMtk8q7XCvxUdnnVoR+Rzt9mqKK2ER9DbOxwwGdYmXAro/K6eFF4AkyHiu3wRn4/LdEbdPr2uF/TeSTL1Ujp+TOT9ty3S7TZkNDxcR/SYMb2HfDO7wU4PK/fDKT1ZvUV8F+h0Oi2X01JafR4YBkSXDfDXPw2axlwGLPkxjzr9OIJkB8CM42ygaQxgNMKP5PxFGvzBdLufMxnjtsFfUO3M+5jRKYM9axJH4saRLcUxZMADePiIcS77pww6NQzxnk0mDO4biP3xCZwZlL++j6O/BSrQNFyWl3xk27AlOQAS04TCtVTnsEmR/GMMn/6mi+9+0rvs752afS1bROqwRcNSX4t/bxWdqv/55nO9qcq7pt9UF4L4/THHuFkzWGj6A6TZ22ZRIET+M2AQiIvw3PyW1ZRSbfFhvx1bMAWAk+VYG8kHMiKfINUcZlvLaNRMNXgsFxlGjk+OYy3Bq7wdMagQQt6pcHWl/zLR97blVBld1704ecspfmfTDSH/eV9pfNYAgRnJn0PNjT0eo7Hbt3jvQUzenF7Blbuwp3GgpR0ZyP5xZIxIvJTINAFEZ6+wbvOOdzJGsTkab+v3xa7HLhSUyiyV Ipnt2iWH YC3QB1ejnEWFnw26cXuLxMj93yFqvlJVHTnYo3to8agV8ykjwdoTYiOxtbazwI3f9XB/qoL/PwqtoYecI4K+7Ou2ipBNLp6bpLpfV0E3ontQE+Gb0L2LfXyD9EWEUcbpxyRhro0wfmKXpP0mmYb5ff7WhH3/VRqIMtcBEbPaeJEpqtgrbG4CStFiTmwoD1vUAO+XfED2q1H1CDBKoO82qb7bk9ifilzCj+jhw+XFwl4Z6dP+bGndK2padfbV8NLIjAjuC5dRqkAIgg+tRzPmAOrED+rlMxnKtB9UzWDtp28lZRQDebAt9gAVUUExedbNT14iVuJo+/iFsWW5GT61YlUz6RcoOkijz2kXtAO6na0NS9sVpsS19VC8eWpRxBpKFcdtMJWYULtv9G2f4qoLfcDG4zWOe2fCz3Mq4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move should_run_aging() next to its only caller left. Signed-off-by: Yu Zhao Change-Id: I07e5372b98ca28c003861fdeddadde4304abcfe4 --- mm/vmscan.c | 124 ++++++++++++++++++++++++++-------------------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a2f71400b8be..c424cc06f8c6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4470,68 +4470,6 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, - struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) -{ - int gen, type, zone; - unsigned long old = 0; - unsigned long young = 0; - unsigned long total = 0; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - struct mem_cgroup *memcg = lruvec_memcg(lruvec); - DEFINE_MIN_SEQ(lruvec); - - /* whether this lruvec is completely out of cold folios */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { - *nr_to_scan = 0; - return true; - } - - for (type = !can_swap; type < ANON_AND_FILE; type++) { - unsigned long seq; - - for (seq = min_seq[type]; seq <= max_seq; seq++) { - unsigned long size = 0; - - gen = lru_gen_from_seq(seq); - - for (zone = 0; zone < MAX_NR_ZONES; zone++) - size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - - total += size; - if (seq == max_seq) - young += size; - else if (seq + MIN_NR_GENS == max_seq) - old += size; - } - } - - /* try to scrape all its memory if this memcg was deleted */ - *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; - - /* - * The aging tries to be lazy to reduce the overhead, while the eviction - * stalls when the number of generations reaches MIN_NR_GENS. Hence, the - * ideal number of generations is MIN_NR_GENS+1. - */ - if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - return false; - - /* - * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) - * of the total number of pages for each generation. A reasonable range - * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * aging cares about the upper bound of hot pages, while the eviction - * cares about the lower bound of cold pages. - */ - if (young * MIN_NR_GENS > total) - return true; - if (old * (MIN_NR_GENS + 2) < total) - return true; - - return false; -} - static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) { int gen, type, zone; @@ -5115,6 +5053,68 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap return scanned; } +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, + struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) +{ + int gen, type, zone; + unsigned long old = 0; + unsigned long young = 0; + unsigned long total = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MIN_SEQ(lruvec); + + /* whether this lruvec is completely out of cold folios */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { + *nr_to_scan = 0; + return true; + } + + for (type = !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq = min_seq[type]; seq <= max_seq; seq++) { + unsigned long size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + + total += size; + if (seq == max_seq) + young += size; + else if (seq + MIN_NR_GENS == max_seq) + old += size; + } + } + + /* try to scrape all its memory if this memcg was deleted */ + *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; + + /* + * The aging tries to be lazy to reduce the overhead, while the eviction + * stalls when the number of generations reaches MIN_NR_GENS. Hence, the + * ideal number of generations is MIN_NR_GENS+1. + */ + if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + return false; + + /* + * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) + * of the total number of pages for each generation. A reasonable range + * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The + * aging cares about the upper bound of hot pages, while the eviction + * cares about the lower bound of cold pages. + */ + if (young * MIN_NR_GENS > total) + return true; + if (old * (MIN_NR_GENS + 2) < total) + return true; + + return false; +} + /* * For future optimizations: * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg