From patchwork Thu Dec 1 22:39:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13061876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD78BC4321E for ; Thu, 1 Dec 2022 22:39:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49DF96B0075; Thu, 1 Dec 2022 17:39:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B2DA6B0078; Thu, 1 Dec 2022 17:39:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 168C26B007B; Thu, 1 Dec 2022 17:39:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 04AF06B0075 for ; Thu, 1 Dec 2022 17:39:49 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C1FEFC1205 for ; Thu, 1 Dec 2022 22:39:48 +0000 (UTC) X-FDA: 80195206056.16.1532FE8 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 6A01940004 for ; Thu, 1 Dec 2022 22:39:48 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RHNDbvsz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3My2JYwYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3My2JYwYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669934388; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JHZglG5ilwT7hsqyBjm+nL4wtqZp55RCSn8BZ+7jNpg=; b=DNicPxhSfAcyGIMySgu103R4S7b0oKtXzHN7SSp7OI0iJIq2bSsxYKzw/VPSqgK6uvVtEe 8JB3lipY3Epzy62beUICloToj1DVcvNm8+YlH7cYTXd8ZPCDe6oucRA284zzmnh9k7taoQ CJbYFt5r2Yra/0NlR13VyPhREd2x8KY= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RHNDbvsz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3My2JYwYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3My2JYwYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669934388; a=rsa-sha256; cv=none; b=X/2bGjn8/huCLoFbC87Q9ihYlScln8RZtTo+StFhroBHEWNJ14Mh3q22iCj5WwFYw4EkG/ v7u7aa2TmdmiUMPPfBxpCJIpyKQB3qKqiKXcO10JqdFUiXfuqRcV2+A7o2YF5/Vali3p+c ooTOBZe/TXpvMmOIG1J2p1LXLc/0iok= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-3dddef6adb6so15786177b3.11 for ; Thu, 01 Dec 2022 14:39:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JHZglG5ilwT7hsqyBjm+nL4wtqZp55RCSn8BZ+7jNpg=; b=RHNDbvszyBbPFEB77yE6AJGQfBt0EWpkE3qN5H9J3Pp5Q77hSPzW8ArtLS4SmB33mD 8SXDsp5hzK2/hxWT7MKoqC3fSW77JLZagIWlq/EqS/blNXzBtdfoH26kkDNbrKgwi2Tm JjamnWumeGYaJKFjwgbEqgqlyxC6Jx2bu0SUugLD26lLkdaQpsAYAiFUKxkw19M6otbb 1Z3qS4bBjRYFp6/H6vgSb6BjlQ2H8QkpZ1WmZkealqf3ubG/R58XI2SQtnO7fhHTTiOr 1saN6mqKZeo9AQuP9prToIk9MQu8V2Z8qFrUVWJgcECwRdBaWIGCcT4+RTIcmpvH7gmP 0EVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JHZglG5ilwT7hsqyBjm+nL4wtqZp55RCSn8BZ+7jNpg=; b=FhwPNS4N44zE3k7pNj68sJjk2+HUp8WCt8WAcQ6bVoS3QrPu1LGmTaoveiZ3OW0myd qWBFyih+PXIackeF729THSv2TP1rffoUXnj1UpVl5Hifl+yf+kcieSMXUDVY6l3W2uhS 9UVsev75f5wATaQpTkwEAqSRGiRSNRTMLqPxiS2sdwLecQOc3sAu2AsBIMq9BOK7MvHu HiSDtCnokhQNlax55QrRFiZux15H7nnaDdaRG2d2gIih4dElJDR3ByGnz5P1wZFzqG1v hF0zidxgovNtRq5YHzxhVVWR1JutSuOmBHcAaKCgMnFKTJImyg9ov8QqpsioFN03x0J0 Yecg== X-Gm-Message-State: ANoB5pllMempMxgUylM/gaZVSc5zbA4B3b1VqxVB5jYSpW7CVkGZV2Z7 dZKasHzUSQnCVYMdQIPaUhuZFFoMON8= X-Google-Smtp-Source: AA0mqf7944UNERKD5m4EdoDunAUNDT5CtA5j6zI05nuoJYATtJBJKXDdLhv1W54WaUpHG3XVTXdKF39O88I= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:1d8c:fe8c:ee3e:abb]) (user=yuzhao job=sendgmr) by 2002:a81:4789:0:b0:39a:adfe:bf5c with SMTP id u131-20020a814789000000b0039aadfebf5cmr46711595ywa.403.1669934387675; Thu, 01 Dec 2022 14:39:47 -0800 (PST) Date: Thu, 1 Dec 2022 15:39:21 -0700 In-Reply-To: <20221201223923.873696-1-yuzhao@google.com> Message-Id: <20221201223923.873696-6-yuzhao@google.com> Mime-Version: 1.0 References: <20221201223923.873696-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.rc0.267.gcb52ba06e7-goog Subject: [PATCH mm-unstable v1 5/8] mm: multi-gen LRU: shuffle should_run_aging() From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Spamd-Result: default: False [3.40 / 9.00]; SORBS_IRL_BL(3.00)[209.85.128.202:from]; BAYES_HAM(-2.50)[97.73%]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; MV_CASE(0.50)[]; FORGED_SENDER(0.30)[yuzhao@google.com,3My2JYwYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_DKIM_ALLOW(0.00)[google.com:s=20210112]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17:c]; RCPT_COUNT_TWELVE(0.00)[12]; DKIM_TRACE(0.00)[google.com:+]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DMARC_POLICY_ALLOW(0.00)[google.com,reject]; FROM_NEQ_ENVFROM(0.00)[yuzhao@google.com,3My2JYwYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[] X-Stat-Signature: xyrdgixsek64df6y4azzixg3bm6b1bee X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 6A01940004 X-Rspam-User: X-HE-Tag: 1669934388-864756 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move should_run_aging() next to its only caller left. Signed-off-by: Yu Zhao --- mm/vmscan.c | 124 ++++++++++++++++++++++++++-------------------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 67967a4b18a9..0557adce75c5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4452,68 +4452,6 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, - struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) -{ - int gen, type, zone; - unsigned long old = 0; - unsigned long young = 0; - unsigned long total = 0; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - struct mem_cgroup *memcg = lruvec_memcg(lruvec); - DEFINE_MIN_SEQ(lruvec); - - /* whether this lruvec is completely out of cold folios */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { - *nr_to_scan = 0; - return true; - } - - for (type = !can_swap; type < ANON_AND_FILE; type++) { - unsigned long seq; - - for (seq = min_seq[type]; seq <= max_seq; seq++) { - unsigned long size = 0; - - gen = lru_gen_from_seq(seq); - - for (zone = 0; zone < MAX_NR_ZONES; zone++) - size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - - total += size; - if (seq == max_seq) - young += size; - else if (seq + MIN_NR_GENS == max_seq) - old += size; - } - } - - /* try to scrape all its memory if this memcg was deleted */ - *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; - - /* - * The aging tries to be lazy to reduce the overhead, while the eviction - * stalls when the number of generations reaches MIN_NR_GENS. Hence, the - * ideal number of generations is MIN_NR_GENS+1. - */ - if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - return false; - - /* - * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) - * of the total number of pages for each generation. A reasonable range - * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * aging cares about the upper bound of hot pages, while the eviction - * cares about the lower bound of cold pages. - */ - if (young * MIN_NR_GENS > total) - return true; - if (old * (MIN_NR_GENS + 2) < total) - return true; - - return false; -} - static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) { int gen, type, zone; @@ -5097,6 +5035,68 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap return scanned; } +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, + struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) +{ + int gen, type, zone; + unsigned long old = 0; + unsigned long young = 0; + unsigned long total = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MIN_SEQ(lruvec); + + /* whether this lruvec is completely out of cold folios */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { + *nr_to_scan = 0; + return true; + } + + for (type = !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq = min_seq[type]; seq <= max_seq; seq++) { + unsigned long size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + + total += size; + if (seq == max_seq) + young += size; + else if (seq + MIN_NR_GENS == max_seq) + old += size; + } + } + + /* try to scrape all its memory if this memcg was deleted */ + *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; + + /* + * The aging tries to be lazy to reduce the overhead, while the eviction + * stalls when the number of generations reaches MIN_NR_GENS. Hence, the + * ideal number of generations is MIN_NR_GENS+1. + */ + if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + return false; + + /* + * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) + * of the total number of pages for each generation. A reasonable range + * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The + * aging cares about the upper bound of hot pages, while the eviction + * cares about the lower bound of cold pages. + */ + if (young * MIN_NR_GENS > total) + return true; + if (old * (MIN_NR_GENS + 2) < total) + return true; + + return false; +} + /* * For future optimizations: * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg