From patchwork Thu Dec 22 04:19:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13079365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A029DC4332F for ; Thu, 22 Dec 2022 04:19:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B412F94000B; Wed, 21 Dec 2022 23:19:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ACA2E940007; Wed, 21 Dec 2022 23:19:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87EF394000B; Wed, 21 Dec 2022 23:19:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6D862940007 for ; Wed, 21 Dec 2022 23:19:50 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 45EDDA0485 for ; Thu, 22 Dec 2022 04:19:50 +0000 (UTC) X-FDA: 80268638940.04.8D450C7 Received: from mail-io1-f73.google.com (mail-io1-f73.google.com [209.85.166.73]) by imf19.hostedemail.com (Postfix) with ESMTP id A874E1A0004 for ; Thu, 22 Dec 2022 04:19:48 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=awokrXW2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 349qjYwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.166.73 as permitted sender) smtp.mailfrom=349qjYwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671682788; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S9ZuAyfwqYLskluTxeMZ1Oyj25JaC3jpnfC7bM5WyD4=; b=w0NF4mBi0CWTNy7hEMaVGxG4ei4huDnOYYg+dQJM2JP46G3wzT61gQvsifK8vVK3420RVu 8N9C/LvdZa0p1ucAw0jGOhtOq4NeJ0gXQgEltNsVHYCVSTdvFnv96/tNTsNp8P1sQOtJmq DKRHd/4ehYV9ebRdtIDbEeiaQ5TrPcg= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=awokrXW2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 349qjYwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.166.73 as permitted sender) smtp.mailfrom=349qjYwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671682788; a=rsa-sha256; cv=none; b=r4xljuh1ThHh3qqmO8Qk0lxTUvOPiXfoARPM+QcudJiFnggapRfGYdlCacp7T9sZrGfU4s JlWdS+ssX1LDZ7dazxK+1+P0JSNEhyiL59e5oniUy59vdHg+oCc/+CkUY2+sLHssPZYEGN 5+2/9QbMXMnXO457VQrqG/noVnzClWA= Received: by mail-io1-f73.google.com with SMTP id z9-20020a6be009000000b006e0577c3686so340127iog.0 for ; Wed, 21 Dec 2022 20:19:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S9ZuAyfwqYLskluTxeMZ1Oyj25JaC3jpnfC7bM5WyD4=; b=awokrXW2IaKsda0feV8JW/aYhMrUqKzEHQX+f5UsTBU2+8KVpH/XEmCKdVW/3j64+P LX4/xKKwf3yGaOd1RMWjLx+SBKbq5b79RaWXbvO3uvqC1rK19G2lTd+XOyNc4OAudt0G BNVuAfZmPTiS12q4Pp+vrGp7KjeUeKAD9fMxEibkXqvB9XoQo6+4i1rCFGj5s/wHCCxE YvcF1Rui1WagXwkNShtpDS8/0ZLm9UBhXa7tsws0qPh8YIdTW6gtnBpxOGzSDL8IeNie vd507zZFTZU4yPE5FV0HqFBmaZ/37IUyHVnBJLcsj1LUs4dMyMtEtQhrG/k+tqR6EKRe 4JXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S9ZuAyfwqYLskluTxeMZ1Oyj25JaC3jpnfC7bM5WyD4=; b=n/07zFBin2Gd1CtW5WPQBnE9RGWWVkluBJ6KITjybh6TfC3Su9ixIVB4wHhhTg1d+g Ftg3fPj4Fvp2+jkYrQbGrGf5//8lz/vm8AgpErEYcOt2o5Hf24rq/QnGAbc9Y7DdqUQg qWDbh8/Blh5A3IrStdq6tmgiIQjZ7Cf6CGm2dofY2FUqx9SXUebsUZUPTPE4aWDbk0l8 YME7Y1N/M5kXSbx1d4V1cehXyNiG6yycxDcYJlXxDGZMSDNkgkoccluUxTjdevbF34h5 CtRI5bn29jmG4pThRa+ztUSssdAW9WYz2xWaOGKWVhGbFf+zcg61Dtx04GPjZpKKuINh tkHA== X-Gm-Message-State: AFqh2krN5crQl1h+Dpskq7JBjnhNwWJhfdhDfiVdadYuKglRO0fu13rW cvBLmGHZ9+yamOXQo4DkFMSc4E7UR5A= X-Google-Smtp-Source: AMrXdXuEY3YkuSyiXYaG/rF71AtXpvUi99HGEHzJyq69UdzONGfza3jeMcIIHRBjZ3l13oxwVLYTLfNu04M= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a463:5f7b:440e:5c77]) (user=yuzhao job=sendgmr) by 2002:a92:cb42:0:b0:305:eba6:78ab with SMTP id f2-20020a92cb42000000b00305eba678abmr300227ilq.316.1671682787850; Wed, 21 Dec 2022 20:19:47 -0800 (PST) Date: Wed, 21 Dec 2022 21:19:03 -0700 In-Reply-To: <20221222041905.2431096-1-yuzhao@google.com> Message-Id: <20221222041905.2431096-6-yuzhao@google.com> Mime-Version: 1.0 References: <20221222041905.2431096-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v3 5/8] mm: multi-gen LRU: shuffle should_run_aging() From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: A874E1A0004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: rbk9ta6e96y8kiosi3j61pbunh9agnae X-HE-Tag: 1671682788-215688 X-HE-Meta: U2FsdGVkX1+M3HnGePsXkyIZJgJ2gAq3hDe/HtHxqbmcky4xQ5Wg+4k4gnQjRcHdV32dpLA5pnR4Qkz06IeGwrXBhtc2wlcU2bu0t6+3TpGv815C0PztRdzSa246+hRw1xiFqss/FJv0LRitiWK3i5itvLf9zoN7cD9fcWaVb2dDuMb2kZohQaZ9mow9LFGq5tOmpz0kD4UJCvEqH71PAmzr1pHQgl3k0nCT0u8SnxTT9ll8W5kCkMw3XjDAlyDm8UHcnvJCYY37pMUrCqd8bLDqla1+0qCYJgn2BRfojLlDI5ZfKI/sI9nwmQG8L9kdKniT31Hk7aMhkACJGalO5bZPe/PvyXS0cGVH+7HWZp8OIV2JSRG9zMFXk4kby138XWGhMO4e9PDjbeMR51su2HeoOGj568I5NlAIKV3YBVGzGCJmNp1Zg8VQLd8FNNtfIXHymBQ235tPB3ojmh9g1za8UEhySr/yY3knixDPadMkFlTUAIt96OEsy9kfAwUxOutUv4Utq2m3cN31KCzQnBImk2iWsk3W2vA6FdrPHr2aLneQX34FnDiFOsdlMgKdqg5U3ST/f58uVWJXHvlMuTdHyHjfRqdyqXlR5+j0+d61lDUDrZLp0HZcFajc1vv9qOhuEul3rDUokzkFdFqnIPY9MkWyv7I5HVRlM9FCLzxstL3jgH3Fz8h56ChY1lPDy73bFDWF5QTiEcO09KUnkE4rU4D+Eb9+vyxA1Akirupp6W35qaZEZIMY+uC5rKKN3rLEjJDk7ZgNiOUF0YL2MojmCl8Bc8FTU9ZViifr61OrZGtW95p6qcbkvRE27EEiFUX5s1I5+UkdYIm4RuG6K/8pwMdF8HhAInMkF7af90fzgk8Xs4XTEBNI8WmuseypyuOBnwspw/WJc8hAB0l3OfTwNcp53B4WzVwsO/UaqMNkwG6gbc4DmpExzI7UVrN3upwgCzbSv2ZbiHSLexj JGXlikm4 zYDm803ro6kWeuJew/Ebt88sjQW7I1bDlc7zWtP64vgNoUt+VhaOfHhyzVpbcs+KXaM6ZW2RsV196I4Jo8WMqwX7z1G/QLQvUigydHcYreJajxP9/0kRJVls02vXWkMjgYwzTTAWIxfHLa1fh+E88tantjWn0ZONK+lRvggTPYDeqXacExjZLPsW5cWcGXH3/izvPlVBnx6MHJHXYByZt4ILIz9k2VnMkcC+/3wbcZimPCWPctCdyuJcO7k2uN4EG7/k/MUAE/P0ciFgZ0AJjykqM2/CnClg9ovoSybniYS/EnUqRGXpQCq/kMkST7VkLAR0agYYvQA+Iu8tG7aQZKNzI/YcWt5O3s5+gCxpIiLApSJX4egkeHwOT2OmztUsjvTsQpasg73EoTmpqWkGBciEiMhrq+NSz1dnH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move should_run_aging() next to its only caller left. Signed-off-by: Yu Zhao --- mm/vmscan.c | 124 ++++++++++++++++++++++++++-------------------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 65cc82208b6e..dd9f7b7abe1c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4467,68 +4467,6 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, - struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) -{ - int gen, type, zone; - unsigned long old = 0; - unsigned long young = 0; - unsigned long total = 0; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - struct mem_cgroup *memcg = lruvec_memcg(lruvec); - DEFINE_MIN_SEQ(lruvec); - - /* whether this lruvec is completely out of cold folios */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { - *nr_to_scan = 0; - return true; - } - - for (type = !can_swap; type < ANON_AND_FILE; type++) { - unsigned long seq; - - for (seq = min_seq[type]; seq <= max_seq; seq++) { - unsigned long size = 0; - - gen = lru_gen_from_seq(seq); - - for (zone = 0; zone < MAX_NR_ZONES; zone++) - size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - - total += size; - if (seq == max_seq) - young += size; - else if (seq + MIN_NR_GENS == max_seq) - old += size; - } - } - - /* try to scrape all its memory if this memcg was deleted */ - *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; - - /* - * The aging tries to be lazy to reduce the overhead, while the eviction - * stalls when the number of generations reaches MIN_NR_GENS. Hence, the - * ideal number of generations is MIN_NR_GENS+1. - */ - if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - return false; - - /* - * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) - * of the total number of pages for each generation. A reasonable range - * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * aging cares about the upper bound of hot pages, while the eviction - * cares about the lower bound of cold pages. - */ - if (young * MIN_NR_GENS > total) - return true; - if (old * (MIN_NR_GENS + 2) < total) - return true; - - return false; -} - static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) { int gen, type, zone; @@ -5112,6 +5050,68 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap return scanned; } +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, + struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) +{ + int gen, type, zone; + unsigned long old = 0; + unsigned long young = 0; + unsigned long total = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MIN_SEQ(lruvec); + + /* whether this lruvec is completely out of cold folios */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { + *nr_to_scan = 0; + return true; + } + + for (type = !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq = min_seq[type]; seq <= max_seq; seq++) { + unsigned long size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + + total += size; + if (seq == max_seq) + young += size; + else if (seq + MIN_NR_GENS == max_seq) + old += size; + } + } + + /* try to scrape all its memory if this memcg was deleted */ + *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; + + /* + * The aging tries to be lazy to reduce the overhead, while the eviction + * stalls when the number of generations reaches MIN_NR_GENS. Hence, the + * ideal number of generations is MIN_NR_GENS+1. + */ + if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + return false; + + /* + * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) + * of the total number of pages for each generation. A reasonable range + * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The + * aging cares about the upper bound of hot pages, while the eviction + * cares about the lower bound of cold pages. + */ + if (young * MIN_NR_GENS > total) + return true; + if (old * (MIN_NR_GENS + 2) < total) + return true; + + return false; +} + /* * For future optimizations: * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg