From patchwork Wed Dec 21 00:12:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 488EBC4332F for ; Wed, 21 Dec 2022 00:12:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0F60900003; Tue, 20 Dec 2022 19:12:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBE6E900002; Tue, 20 Dec 2022 19:12:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9BAD900003; Tue, 20 Dec 2022 19:12:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A2B3B900002 for ; Tue, 20 Dec 2022 19:12:47 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 67667140152 for ; Wed, 21 Dec 2022 00:12:47 +0000 (UTC) X-FDA: 80264387574.23.7D0A526 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf15.hostedemail.com (Postfix) with ESMTP id D2C63A0007 for ; Wed, 21 Dec 2022 00:12:45 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BCLok8Tn; spf=pass (imf15.hostedemail.com: domain of 3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581565; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gzZZ7Pl3axic12Be6+Sv18aJSfohVNrr9lclEYQlfdA=; b=QhGZrcPfIC/uTBXN0jhX/gkFNl4sxo+xXwsvs4In1M8FS8lXqkXd6OQgDpFF5PQAcO+ltm bT37dj32gG7C8cKdPBISU9667EN00T45n34DldNF7YRgUp9AOCLTl+7eh9DJQrSW1Vmflt a965nPY8BfkAttpHGzUdat4kd5RspM0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BCLok8Tn; spf=pass (imf15.hostedemail.com: domain of 3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581565; a=rsa-sha256; cv=none; b=VzWWcixRm91ePmmEDGm30s8EFxuclMTUjlxX4tFa6WN6ejvu2PVtx0Z5rwO+bzwXnyNimf 85HZ1OtR4Xc9jwLCa3VnE3LSMOoCIDNvllDEjCM+Ek9XXCzQ4IlG9HzYWFIaQAL5D7PDFs 7VaZKpDDtgMoM7S00uMBMRt64MYTZe0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-3ceb4c331faso158312607b3.2 for ; Tue, 20 Dec 2022 16:12:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gzZZ7Pl3axic12Be6+Sv18aJSfohVNrr9lclEYQlfdA=; b=BCLok8Tn1XTfQ1A8R6b7C5Ad/joLM/zJ52a9in4TvZQxFue8D9E+7yIFrIer98Eqj4 IT7c2Y7QptfbqvSrriBtpoLImc2rymZxl/aSRPL6iprTxYnoZ8GkLd6dpko1fT3qYpl3 j5aIasFOsO6exHTJNjrCfB/WGow8hUzKcSXoqt+CqvmuLFzcGhf9io/+i2C6Ss/e2mQV laXDGRPFMp9s74HpTyKqXCKzcyFji0RrSIXbOT0lve1VJZR/j3f6ef/UFnLZwT02BvE1 YIHTiq2hbzPBFT/mmXk6dbtrE2QduGkU5U0EBz06MoDqQSe1Flej1GG8CgMufMWwSm9d /umw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gzZZ7Pl3axic12Be6+Sv18aJSfohVNrr9lclEYQlfdA=; b=q5Ek5qEt6EZlC2vABFLUJTcrbqzz54qdQZ7poaJZI1Jnfx1wzgtoZd1xMWMwcjzwNg 24Lwi9N/NEqz/8VhB37rwU4DYAx0mGBTWBlQ3Wr7J0p685DDdSDzCdHIQo0JfX0Z6FAb mCy3P5EC2a7EtfhaliNMvJ7yZ08nfTK7m8KDchnHhK7krUcO81tTz2XUTjjEID+uNwmU nABUtl0SFDRAg4iGux4Ph8twsH859pLfffmG2BI4NExO8Hf0w3kpqk/q6vGTFLPQIQ3O 6xIVta9AFTB9xbnLuOVF2KVkKNDaJ8eLeJL1slAsuTib0QiZUG5tQ7YvE4uDgFpa59eM dANA== X-Gm-Message-State: ANoB5pmtZlYDH4Zv3kfwTFC+lgGu92Ec87UIbFIKJVX7z3TNl6hroQJH OkSrbMiLZ3OqWMC/mu1ZNvtvfMnn9Qs= X-Google-Smtp-Source: AA0mqf6rnjM1gkiEH2cSoCaqtQ9HV2F4qXN/ljYPU8+/Y/VmAHrOUkrolCEScMqy2/MGCZCUbXIT85hA/c0= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a81:f90d:0:b0:404:7030:e1c0 with SMTP id x13-20020a81f90d000000b004047030e1c0mr6939777ywm.53.1671581565140; Tue, 20 Dec 2022 16:12:45 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:07 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-8-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 7/8] mm: multi-gen LRU: clarify scan_control flags From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D2C63A0007 X-Rspam-User: X-Stat-Signature: c7hm75g4y5qoxw76grdegfnuj6mhny55 X-HE-Tag: 1671581565-332829 X-HE-Meta: U2FsdGVkX19X9PDv+UCdHls3nfMbHv0WTkNXUh7n+XQIsFpfJoqinNH6ZQmMKVOq2eTTBZtV4poIb28REQMZJwbmnYvtBytielGlF3hcnDEg4rblLpV25XDoMIEsr+Q7TxsZnsF50Y4zYimxcQ7UdEqR9Ojs9HRngUTQNr23oXkwb6csHa0/FyLkwVFOv1ICjy09M3US0GStnnylWk67KbH4rJ65h5u4c6SLRxBxKtRi1qJN7vShhkH7hOluwZHZKlOG4omhzx43iNtoT9kQaYEdN2u6UvJT9nDnCq5AhWOEb2PRGzdhK0KYqKrzM+8/jwUB6Tu+R0ILKZ4BP2t1rdq9RWpXryjbAAZ/EgV0Dc4T4WPXpgdj5HRt4wLBRrygbS5FeOn8TIgP6Sybz3L6DtmvWhdFlwIDn2pV01BMz+Epr02o5We3uYd/xmfOh99oNWXRuYuk99Ha6aziKDDPBGclCxzCA/R2ARaBGCrINO5z0+pj+W83bNTSv3NO00KwCd34FNgXwbQdkP8GmYPyF0rJz07yPcEq5SMQd0S3G0gmfd4e88XdnFYKVY7FCvkqyKNsORAa32cDyxLAAWUnfyohfalkmJOr5QyksBPJagxp16ZIrln801TNvfA5WOKyTiojTpPYl4pUNBe49lBgfVYAa6YnSjibQsiFa9KkkTOI2+7z7zOAzr9EHmczFDGWS0Fpg8NyqFBZEEOqZOY+RTTWU1AWFq/KVwP0+PdDmlvYSDrnvdGKdlP2AcVn4AR+1rfWR2SPptK+tKg7nEi7SGtkDYtzF6WcIpfAhqpGSrHUGBV2y7NdnCjMVKGPxgozaXMEvsjpsorpk37PM4qhk5fux7JlNzskWM+JgGXl8lCxwZIlCm28OSTwiONv5VqD8vB1fWhlMjb5e9X9OVbifM9QO4hwfgNNeVgnRpVXumLCPtLqSJOaWczvTx7pfUwgn4mixoyeV3mIoitbuuU h6SmqW4i wu5jYZb6YPqxJQZt1MFuc7v4esvrWm7mhk25FgkHSRldngcC9kSRux5Z7sbNWtBQSPVxJqxpbkVfbaWvlbWTnkiNZ/tV82tKPouH/0DQAwcKci7keRVuQELouqHFEL3UVxPj2BSHRN/EunX/xSfvUmGOWgeCnhbE7LJIUJqPYxkBx9FScaEfLQOAyIoEJ0AS95Kul/dPw07xOYa2NoQFvRpOvtriCkTd3/uZuFVq9EcdCMCuGX4yn/JcaAGjuDLePhIdzSIJDnguxNgRch2hgBI23QW6RAQoxrzhwddaPLVugAlffwT/eoSXt4Cz6iJnP1W1ruMsOkJWPL04QkagtYep559/Duv+OdHmuEcr13N0nkSbGsLXxw/vButfrKc3GyI1u+BrYK5a8mazepLU4qLaN9pxmieZ2wvGt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Among the flags in scan_control: 1. sc->may_swap, which indicates swap constraint due to memsw.max, is supported as usual. 2. sc->proactive, which indicates reclaim by memory.reclaim, may not opportunistically skip the aging path, since it is considered less latency sensitive. 3. !(sc->gfp_mask & __GFP_IO), which indicates IO constraint, lowers swappiness to prioritize file LRU, since clean file folios are more likely to exist. 4. sc->may_writepage and sc->may_unmap, which indicates opportunistic reclaim, are rejected, since unmapped clean folios are already prioritized. Scanning for more of them is likely futile and can cause high reclaim latency when there is a large number of memcgs. The rest are handled by the existing code. Signed-off-by: Yu Zhao Change-Id: Ic3b1a13ad1f88853427962b37669aa99942c9fb5 --- mm/vmscan.c | 56 ++++++++++++++++++++++++++--------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index fd837a13617c..308c3c16d81d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3209,6 +3209,9 @@ static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); + if (!sc->may_swap) + return 0; + if (!can_demote(pgdat->node_id, sc) && mem_cgroup_get_nr_swap_pages(memcg) < MIN_LRU_BATCH) return 0; @@ -4238,7 +4241,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ } while (err == -EAGAIN); } -static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat) +static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) { struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; @@ -4246,7 +4249,7 @@ static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat) VM_WARN_ON_ONCE(walk); walk = &pgdat->mm_walk; - } else if (!pgdat && !walk) { + } else if (!walk && force_alloc) { VM_WARN_ON_ONCE(current_is_kswapd()); walk = kzalloc(sizeof(*walk), __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); @@ -4432,7 +4435,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, goto done; } - walk = set_mm_walk(NULL); + walk = set_mm_walk(NULL, true); if (!walk) { success = iterate_mm_list_nowalk(lruvec, max_seq); goto done; @@ -4501,8 +4504,6 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MIN_SEQ(lruvec); - VM_WARN_ON_ONCE(sc->memcg_low_reclaim); - /* see the comment on lru_gen_folio */ gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); @@ -4758,12 +4759,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca { bool success; - /* unmapping inhibited */ - if (!sc->may_unmap && folio_mapped(folio)) - return false; - /* swapping inhibited */ - if (!(sc->may_writepage && (sc->gfp_mask & __GFP_IO)) && + if (!(sc->gfp_mask & __GFP_IO) && (folio_test_dirty(folio) || (folio_test_anon(folio) && !folio_test_swapcache(folio)))) return false; @@ -4860,9 +4857,8 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, __count_vm_events(PGSCAN_ANON + type, isolated); /* - * There might not be eligible pages due to reclaim_idx, may_unmap and - * may_writepage. Check the remaining to prevent livelock if it's not - * making progress. + * There might not be eligible folios due to reclaim_idx. Check the + * remaining to prevent livelock if it's not making progress. */ return isolated || !remaining ? scanned : 0; } @@ -5122,9 +5118,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); - if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg) || - (mem_cgroup_below_low(sc->target_mem_cgroup, memcg) && - !sc->memcg_low_reclaim)) + if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) return 0; if (!should_run_aging(lruvec, max_seq, sc, can_swap, &nr_to_scan)) @@ -5152,17 +5146,14 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) long nr_to_scan; unsigned long scanned = 0; unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); + int swappiness = get_swappiness(lruvec, sc); + + /* clean file folios are more likely to exist */ + if (swappiness && !(sc->gfp_mask & __GFP_IO)) + swappiness = 1; while (true) { int delta; - int swappiness; - - if (sc->may_swap) - swappiness = get_swappiness(lruvec, sc); - else if (!cgroup_reclaim(sc) && get_swappiness(lruvec, sc)) - swappiness = 1; - else - swappiness = 0; nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness); if (nr_to_scan <= 0) @@ -5293,12 +5284,13 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc struct blk_plug plug; VM_WARN_ON_ONCE(global_reclaim(sc)); + VM_WARN_ON_ONCE(!sc->may_writepage || !sc->may_unmap); lru_add_drain(); blk_start_plug(&plug); - set_mm_walk(lruvec_pgdat(lruvec)); + set_mm_walk(NULL, sc->proactive); if (try_to_shrink_lruvec(lruvec, sc)) lru_gen_rotate_memcg(lruvec, MEMCG_LRU_YOUNG); @@ -5354,11 +5346,19 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * VM_WARN_ON_ONCE(!global_reclaim(sc)); + /* + * Unmapped clean folios are already prioritized. Scanning for more of + * them is likely futile and can cause high reclaim latency when there + * is a large number of memcgs. + */ + if (!sc->may_writepage || !sc->may_unmap) + goto done; + lru_add_drain(); blk_start_plug(&plug); - set_mm_walk(pgdat); + set_mm_walk(pgdat, sc->proactive); set_initial_priority(pgdat, sc); @@ -5376,7 +5376,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * clear_mm_walk(); blk_finish_plug(&plug); - +done: /* kswapd should never fail */ pgdat->kswapd_failures = 0; } @@ -5945,7 +5945,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, set_task_reclaim_state(current, &sc.reclaim_state); flags = memalloc_noreclaim_save(); blk_start_plug(&plug); - if (!set_mm_walk(NULL)) { + if (!set_mm_walk(NULL, true)) { err = -ENOMEM; goto done; }