From patchwork Mon Dec 9 08:36:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chen Ridong X-Patchwork-Id: 13899079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF59EE77173 for ; Mon, 9 Dec 2024 08:46:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FA3E6B03DC; Mon, 9 Dec 2024 03:46:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53D246B03DF; Mon, 9 Dec 2024 03:46:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 364306B03DC; Mon, 9 Dec 2024 03:46:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 08E576B03DD for ; Mon, 9 Dec 2024 03:46:22 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ABA88140227 for ; Mon, 9 Dec 2024 08:46:21 +0000 (UTC) X-FDA: 82874788500.05.99C4400 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf10.hostedemail.com (Postfix) with ESMTP id 05C63C0010 for ; Mon, 9 Dec 2024 08:46:07 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733733960; a=rsa-sha256; cv=none; b=ZI6W7Nw3a57nnbyL/zote4JnBozXLsr37GUceaXaUMD7MUTzoMgBmoIdgWlEeUbFnvMALM 9jBvuK1E0fxoqhCJXDgLaKThaSkSjkqhuMzcdoLqajn3YNbfImfDVJQZQEvI1UiLxG5k7j cLkj4KP9d/daCPFiqlMrS/ILHlPqK4Y= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733733960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A2OqytWmdVEA9FIFdmoqO1MSpbfDCr9bEJRNJX4iV6I=; b=G9oLj0eU4ex8Xxg3zA1uPjlbOVAAFZyAEyGo4ChAxr7n9CcLiryI+c+MqCUCHByusupQRy HllS0hsps3+NxEHgwu8K2H55n52UFwez2PBU2M0ck0pVEKjhgT+Crr7EhsJaYAWJZ9Jiuu tIbKuKlKqEanrKpFNgd9xxJmsjyGllg= Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4Y6FnV5M5Rz4f3lfX for ; Mon, 9 Dec 2024 16:45:50 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id E95211A06D7 for ; Mon, 9 Dec 2024 16:46:10 +0800 (CST) Received: from hulk-vt.huawei.com (unknown [10.67.174.121]) by APP4 (Coremail) with SMTP id gCh0CgAX8oVArlZn1gthEA--.6675S3; Mon, 09 Dec 2024 16:46:10 +0800 (CST) From: Chen Ridong To: akpm@linux-foundation.org, mhocko@suse.com, hannes@cmpxchg.org, yosryahmed@google.com, yuzhao@google.com, david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, baohua@kernel.org, 21cnbao@gmail.com, wangkefeng.wang@huawei.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com, xieym_ict@hotmail.com Subject: [PATCH v4 1/1] mm: vmascan: retry folios written back while isolated for traditional LRU Date: Mon, 9 Dec 2024 08:36:18 +0000 Message-Id: <20241209083618.2889145-2-chenridong@huaweicloud.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241209083618.2889145-1-chenridong@huaweicloud.com> References: <20241209083618.2889145-1-chenridong@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: gCh0CgAX8oVArlZn1gthEA--.6675S3 X-Coremail-Antispam: 1UD129KBjvJXoW3Jr4DWFWfur4UAFyxXr4rAFb_yoWxKr1fpF Z8Krnrta18AryfKrsxXF1Uur1akas8Wr4UJFy2kr42yF1Yqw4YgFyIk345tF43WrWkZFyS qa12gFyUXa1UJrJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUPFb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMc Ij6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_ Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxVAaw2AFwI 0_Jw0_GFyl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG 67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MI IYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E 14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJV W8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxU2HGQ DUUUU X-CM-SenderInfo: hfkh02xlgr0w46kxt4xhlfz01xgou0bp/ X-Rspamd-Queue-Id: 05C63C0010 X-Stat-Signature: f5uik34u75jfgtnrc116tniq9ckxctth X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733733967-384132 X-HE-Meta: U2FsdGVkX1+x4SVXdtpciCkLQOOa9mn8axSydaLofzfOGnXCkkyH9TT3EERZPbEYla4l1HhVFNKiPQzDGYL2ctGzNlTQ6wogw27836ED0NZRIpEVKgsJaawH3URbjUZdp1zOvfSbKzsCq1kX0ccqVRTKnFU5psYYHxyNzjHh1HKCea7/e1ljNGuYDaGHcnIz1ljkJxGvMnLHdeVPlYCUiewsmei7AIeqtYSCkA2smeSqASai+BZvvuZ8Atsk2Y2yeJTMOhc6Y70qi6zC86dFXf+7gDWQ8shGZW/G4Hd7nPigXEsCeKroZq4+K0Wgwk7TgBTR9b1AniUOhLkfhOf9utzCHnSXaWmhI4sLPHw7LMVY/8hKijy4Y0sWUPpxTEzQXwmx3WZlHQ8jojvECaisKyGrjojcxM8EWE+7LsFKMDpymOgOzy3DwH1bjH+gauS3QMa3DtQ8zYKXCcM0SOxCsHwlqNn+iqMXg+3S7CqMwql5Z8WeT9v8rIWaYBuhrt7GTHkBZaP87IbZprYErZFHSa4quuQ8CAkT3Bbk5iHVK3e3er3/Zs68slgWub5h0Wd/OjyyR8dCHiiJVzR5pXHmqLha1RrbOgDOcJLXzIqZPZAtYBQQc5Nsrf7hX6CrbJt89gUqgaSqJIaZjhrmXBVZoX1yDMODvTP95WnB/atmsXICQiSr734uc7yX+ApP5trsNWdPtLrIO+8DjOdxCDf8902jYps9sRj9u18cKzLPFkFvLQ1XAgWsrLsI0O67DGWOHZsfnMeptqoM8ynMFNdeUkslqazPfKHKHOcwu+V44ykPJloRka4T4GNBaOERfJx6QJEd2s0kQUBNinmwOEHz0QyMmg6kfSFKsskY44Ne1Zntpo2oS53Wj6i2kGeuEjWZxWOb8HZdDRBaxwdaZYtanllmj0jHSmfOC3lqCOnSO9Q8a4LJ4kMujY0eVcFKtIjw1M2R5La+QAgMn3kDrnq rOdWjkRm CJ/dpENPMX24abuSX6Lzqe7hc47Cwthtkruy5nZMSlSctpUW0tHyociCqXDUayrmzcykXLHjqk56vCZY9y+B/HbGqsqy4ICJ9oSFTJJV2rxhYCrk1cRrcrz8ZI63NWm0hiDgQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chen Ridong The commit 359a5e1416ca ("mm: multi-gen LRU: retry folios written back while isolated") only fixed the issue for mglru. However, this issue also exists in the traditional active/inactive LRU. This issue will be worse if THP is split, which makes the list longer and needs longer time to finish a batch of folios reclaim. This issue should be fixed in the same way for the traditional LRU. Therefore, the common logic was extracted to the 'find_folios_written_back' function firstly, which is then reused in the 'shrink_inactive_list' function. Finally, retry reclaiming those folios that may have missed the rotation for traditional LRU. Signed-off-by: Chen Ridong --- include/linux/mmzone.h | 3 +- mm/vmscan.c | 108 +++++++++++++++++++++++++++++------------ 2 files changed, 77 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..47c6e8c43dcd 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -391,6 +391,7 @@ struct page_vma_mapped_walk; #define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) #define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) #ifdef CONFIG_LRU_GEN @@ -406,8 +407,6 @@ enum { NR_LRU_GEN_CAPS }; -#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) - #define MIN_LRU_BATCH BITS_PER_LONG #define MAX_LRU_BATCH (MIN_LRU_BATCH * 64) diff --git a/mm/vmscan.c b/mm/vmscan.c index 76378bc257e3..1f0d194f8b2f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -283,6 +283,48 @@ static void set_task_reclaim_state(struct task_struct *task, task->reclaim_state = rs; } +/** + * find_folios_written_back - Find and move the written back folios to a new list. + * @list: filios list + * @clean: the written back folios list + * @skip: whether skip to move the written back folios to clean list. + */ +static inline void find_folios_written_back(struct list_head *list, + struct list_head *clean, bool skip) +{ + struct folio *folio; + struct folio *next; + + list_for_each_entry_safe_reverse(folio, next, list, lru) { + if (!folio_evictable(folio)) { + list_del(&folio->lru); + folio_putback_lru(folio); + continue; + } + + if (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio))) { + /* restore LRU_REFS_FLAGS cleared by isolate_folio() */ + if (lru_gen_enabled() && folio_test_workingset(folio)) + folio_set_referenced(folio); + continue; + } + + if (skip || folio_test_active(folio) || folio_test_referenced(folio) || + folio_mapped(folio) || folio_test_locked(folio) || + folio_test_dirty(folio) || folio_test_writeback(folio)) { + /* don't add rejected folios to the oldest generation */ + if (lru_gen_enabled()) + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, + BIT(PG_active)); + continue; + } + + /* retry folios that may have missed folio_rotate_reclaimable() */ + list_move(&folio->lru, clean); + } +} + /* * flush_reclaim_state(): add pages reclaimed outside of LRU-based reclaim to * scan_control->nr_reclaimed. @@ -1907,6 +1949,25 @@ static int current_may_throttle(void) return !(current->flags & PF_LOCAL_THROTTLE); } +static inline void acc_reclaimed_stat(struct reclaim_stat *stat, + struct reclaim_stat *curr) +{ + int i; + + stat->nr_dirty += curr->nr_dirty; + stat->nr_unqueued_dirty += curr->nr_unqueued_dirty; + stat->nr_congested += curr->nr_congested; + stat->nr_writeback += curr->nr_writeback; + stat->nr_immediate += curr->nr_immediate; + stat->nr_pageout += curr->nr_pageout; + stat->nr_ref_keep += curr->nr_ref_keep; + stat->nr_unmap_fail += curr->nr_unmap_fail; + stat->nr_lazyfree_fail += curr->nr_lazyfree_fail; + stat->nr_demoted += curr->nr_demoted; + for (i = 0; i < ANON_AND_FILE; i++) + stat->nr_activate[i] = curr->nr_activate[i]; +} + /* * shrink_inactive_list() is a helper for shrink_node(). It returns the number * of reclaimed pages @@ -1916,14 +1977,16 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, enum lru_list lru) { LIST_HEAD(folio_list); + LIST_HEAD(clean_list); unsigned long nr_scanned; unsigned int nr_reclaimed = 0; unsigned long nr_taken; - struct reclaim_stat stat; + struct reclaim_stat stat, curr; bool file = is_file_lru(lru); enum vm_event_item item; struct pglist_data *pgdat = lruvec_pgdat(lruvec); bool stalled = false; + bool skip_retry = false; while (unlikely(too_many_isolated(pgdat, file, sc))) { if (stalled) @@ -1957,10 +2020,20 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, if (nr_taken == 0) return 0; - nr_reclaimed = shrink_folio_list(&folio_list, pgdat, sc, &stat, false); + memset(&stat, 0, sizeof(stat)); +retry: + nr_reclaimed += shrink_folio_list(&folio_list, pgdat, sc, &curr, false); + find_folios_written_back(&folio_list, &clean_list, skip_retry); + acc_reclaimed_stat(&stat, &curr); spin_lock_irq(&lruvec->lru_lock); move_folios_to_lru(lruvec, &folio_list); + if (!list_empty(&clean_list)) { + list_splice_init(&clean_list, &folio_list); + skip_retry = true; + spin_unlock_irq(&lruvec->lru_lock); + goto retry; + } __mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(), stat.nr_demoted); @@ -4567,8 +4640,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap int reclaimed; LIST_HEAD(list); LIST_HEAD(clean); - struct folio *folio; - struct folio *next; enum vm_event_item item; struct reclaim_stat stat; struct lru_gen_mm_walk *walk; @@ -4597,34 +4668,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap scanned, reclaimed, &stat, sc->priority, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); - list_for_each_entry_safe_reverse(folio, next, &list, lru) { - if (!folio_evictable(folio)) { - list_del(&folio->lru); - folio_putback_lru(folio); - continue; - } - - if (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio))) { - /* restore LRU_REFS_FLAGS cleared by isolate_folio() */ - if (folio_test_workingset(folio)) - folio_set_referenced(folio); - continue; - } - - if (skip_retry || folio_test_active(folio) || folio_test_referenced(folio) || - folio_mapped(folio) || folio_test_locked(folio) || - folio_test_dirty(folio) || folio_test_writeback(folio)) { - /* don't add rejected folios to the oldest generation */ - set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, - BIT(PG_active)); - continue; - } - - /* retry folios that may have missed folio_rotate_reclaimable() */ - list_move(&folio->lru, &clean); - } - + find_folios_written_back(&list, &clean, skip_retry); spin_lock_irq(&lruvec->lru_lock); move_folios_to_lru(lruvec, &list);