From patchwork Tue Jun 13 12:00:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13278671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D534C7EE2E for ; Tue, 13 Jun 2023 12:01:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABC396B007B; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6CFA8E0005; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E60C8E0003; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7C5226B007B for ; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 38507120499 for ; Tue, 13 Jun 2023 12:01:26 +0000 (UTC) X-FDA: 80897584572.04.21856B9 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf23.hostedemail.com (Postfix) with ESMTP id 113A914003F for ; Tue, 13 Jun 2023 12:01:03 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="ffXFpK9/"; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686657664; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n66WQDyiXSrXuj4/rlzufpSs1yap/rsndLr+ifqgiZc=; b=Dd+O+wlYUb6Jfh4P98dlDzaJ3QgiHfSZE+x2+dDaOyCJvBZek1mHNUU4dQZohYyzRIJjL2 9NJLxezK6ZdCUC++9Ji8JedQWWTVI09QzpMTwmhXY2ToBfN/rHpuJni1AceirHnOOXW6le bdMDDhjzWoFhq0qB+UxZ9KU217SIjGg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686657664; a=rsa-sha256; cv=none; b=DSNOGBfNzxhuqu9epYuPYJ6kYBimPFoPDTZL5QNkLVEvsQhdu8dANUC2OcKULMeD7FlI7/ b1d9De9fi1G6dNq9KUPuqCeIQa+6oZYBONQzABDNwI+eh5AS6B/8LyAhnR+c/fbuyvU+L9 qi2Br4VuryI9/8IrjVSLfiVc1fwMpKk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="ffXFpK9/"; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35DBqP7F021068; Tue, 13 Jun 2023 12:01:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=n66WQDyiXSrXuj4/rlzufpSs1yap/rsndLr+ifqgiZc=; b=ffXFpK9/6yjKKdv7blcX3BKiDAAeonm/i3HV9PxBwzEpMBDcDI4+YQUlKpKDffXPAXKD xdXQ6/OIwto5BZlA+ddcGXRJfbMJ1f3IV9hzUgArMHvNHxW5DXu9tbjb+2vxojFdbdUu uzPdwAkD2tIKMFSTNzKbWyCg+84uwndEgBZsqW19TPvb5GaVUK5g4BUEIwNouGjqVeja Q4yASFdIXaohDvg1WeK4s7vkzzL14TcSddO7TMSwHA+vs8TlV5/ot1buM/spLIUwEsj/ flVocH97cTZzi1cApcAfhiPiiGeA1I0MHl0EJSw3KyFB0gHuan9oWf/T943jbe3E/IF9 Jw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485yx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:01:02 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35DBwj0p005967; Tue, 13 Jun 2023 12:01:01 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485y7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:01:01 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35DA0qKB012072; Tue, 13 Jun 2023 12:01:00 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([9.208.129.118]) by ppma01dal.us.ibm.com (PPS) with ESMTPS id 3r4gt61dce-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:01:00 +0000 Received: from smtpav03.wdc07v.mail.ibm.com (smtpav03.wdc07v.mail.ibm.com [10.39.53.230]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35DC0xB556492304 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 12:00:59 GMT Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F0C5E5805D; Tue, 13 Jun 2023 12:00:58 +0000 (GMT) Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8D5CE58068; Tue, 13 Jun 2023 12:00:56 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.55.172]) by smtpav03.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 12:00:56 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "T . J . Alumbaugh" , "Aneesh Kumar K.V" Subject: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported Date: Tue, 13 Jun 2023 17:30:47 +0530 Message-Id: <20230613120047.149573-3-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> References: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 9DxxB5lLQ9em5fDUc3HQGKb4Rykxa5EU X-Proofpoint-GUID: XdftFgSUxaLQfUfyZuzU17Ncp8mnMCBU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-13_04,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 priorityscore=1501 adultscore=0 phishscore=0 mlxlogscore=853 malwarescore=0 mlxscore=0 bulkscore=0 spamscore=0 suspectscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130102 X-Rspamd-Queue-Id: 113A914003F X-Rspam-User: X-Stat-Signature: tdppbx8imw6s9f3abk7dttj8djyjfi97 X-Rspamd-Server: rspam03 X-HE-Tag: 1686657663-706344 X-HE-Meta: U2FsdGVkX1/dQQd3cBq3LBIx2x9WCCwIaWiKLgWHD0xm8Nmg0YACH7dwSF0324yqAe5M0i9iDMa0AJkrn2eUv6JGrANrYEpWB+C/YD/TD3k5dfEp5NV+Q7vb6SCJwFNUoPL3tGcS3z8uAigNpeL1oN/b8TqhutvTkUANA7Vg4ZC6SgbkXrWfv1pbhJQqId2qFU6iIUTWzEhhNZXi4LJhGzjlb98PL1bxm8bfM/phqYEbYA8eOVNg4oVt0ekJzwlcz74k+CXEGdDz17vS7kM4daw+cB65k9iJFh0RCGf4kxNeGCkZspGcz2gFBgTkmqIgPHPEhK1ikba4AHROPCuDiFrMyHwWPWpZNRC3UGBkNWpKrO67P1AuwXORWKTHoRZLw3rnl4ovqfq4XClfXekKJBuMEDO5HvxW7emuopJEwY2s7T3YTLVytwVKsnk74ro+HbCETy9/yB/BwYRjqzIIVmPDf528aoj+2BaHoUNNWm6BB6MFAkXurHG5ZhgzobtNopMw6kUW/ORMIHAx5XjDQfLse1YEoSRSEsERUwWMHUWt8wPFJATc9OXhkmbXeo47kYpuFD5Ep+/YKcVELgzLN73ZusIL7zFjTuDM/I1iOUXT0bTh9mXkbaWEeVbE7ZSBAiAPAhvS4cZxr8DYQQOJAiH1sW5xDmGbJs2HXPZZj5ESKHiw7NYMAFg0v/269Ky2CzxTpSOlTaD9MPgkN0hAhZkKiKiF2cu5+eSfQJgHvuk/1Hs32lc8xxDuLoFheKClJXJcKNNsUE6j+ucaZ+NqIy/NJSQoxoM1ZbONTXvgEG7ZLyi/A13LCbjPgvMNmZvK3zFsz11fwmfOAawrcK83gmiEJfqlfOdRGD7AT92HmEsXAio141jYOEf1xui/Qo5JMLiDKtepInz9Wms+D0V+pjSpBdIYpu/dScJHIGNnNkhAV6wR1Ol+YGeZpaBY4SX2QO1Rnr0qesN2jAdzP7S krbmQJz5 OwCwDNFYKh911t8kxmAPcv31/89dzYsgQzYm4bEThMyjl7989Jc7R4FmU9Ts7V4hrJGqwRlU1QezAVsn4sumeSjA02iId0OHSKS2IBh3LHaG8UehYxCRvM0XzzcriBPNzvNHxqvBOHd3WA+WhTbcQJDWqevCxFFaLcpvaNxawp2RCx2cNUPMJlDjoLQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture. No preformance change observed with mongodb ycsb test: Patch details Throughput(Ops/sec) without patch 93278 With patch 93400 Without patch: $ size mm/vmscan.o text data bss dec hex filename 112102 42721 40 154863 25cef mm/vmscan.o With patch $ size mm/vmscan.o text data bss dec hex filename 105430 41333 24 146787 23d63 mm/vmscan.o Signed-off-by: Aneesh Kumar K.V --- arch/Kconfig | 3 + arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/memcontrol.h | 2 +- include/linux/mm_types.h | 8 +-- include/linux/mmzone.h | 8 +++ include/linux/swap.h | 2 +- kernel/fork.c | 2 +- mm/memcontrol.c | 2 +- mm/vmscan.c | 128 +++++++++++++++++++++++++++++++++---- 10 files changed, 137 insertions(+), 20 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 205fd23e0cad..5cdd98731298 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1458,6 +1458,9 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config LRU_TASK_PAGE_AGING + bool + config ARCH_HAS_NONLEAF_PMD_YOUNG bool help diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b1201d25a8a4..e0994fb3504b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -225,6 +225,7 @@ config ARM64 select IRQ_DOMAIN select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN + select LRU_TASK_PAGE_AGING if LRU_GEN select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE select NEED_SG_DMA_LENGTH diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 53bab123a8ee..bde9e6f33b22 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -276,6 +276,7 @@ config X86 select HAVE_GENERIC_VDSO select HOTPLUG_SMT if SMP select IRQ_FORCED_THREADING + select LRU_TASK_PAGE_AGING if LRU_GEN select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK select NEED_SG_DMA_LENGTH diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 43d4ec8445d4..ea5d1d7bfb8b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -321,7 +321,7 @@ struct mem_cgroup { struct deferred_split deferred_split_queue; #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-memcg mm_struct list */ struct lru_gen_mm_list mm_list; #endif diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..f90a4860a792 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -786,7 +786,7 @@ struct mm_struct { */ unsigned long ksm_rmap_items; #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct { /* this mm_struct is on lru_gen_mm_list */ struct list_head list; @@ -801,7 +801,7 @@ struct mm_struct { struct mem_cgroup *memcg; #endif } lru_gen; -#endif /* CONFIG_LRU_GEN */ +#endif /* CONFIG_LRU_TASK_PAGE_AGING */ } __randomize_layout; /* @@ -830,7 +830,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) return (struct cpumask *)&mm->cpu_bitmap; } -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct lru_gen_mm_list { /* mm_struct list for page table walkers */ @@ -864,7 +864,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm) WRITE_ONCE(mm->lru_gen.bitmap, -1); } -#else /* !CONFIG_LRU_GEN */ +#else /* !CONFIG_LRU_TASK_PAGE_AGING */ static inline void lru_gen_add_mm(struct mm_struct *mm) { diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a4889c9d4055..b35698148d3c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -427,6 +427,7 @@ struct lru_gen_folio { #endif }; +#ifdef CONFIG_LRU_TASK_PAGE_AGING enum { MM_LEAF_TOTAL, /* total leaf entries */ MM_LEAF_OLD, /* old leaf entries */ @@ -469,6 +470,7 @@ struct lru_gen_mm_walk { bool can_swap; bool force_scan; }; +#endif void lru_gen_init_lruvec(struct lruvec *lruvec); void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); @@ -613,8 +615,12 @@ struct lruvec { #ifdef CONFIG_LRU_GEN /* evictable pages divided into generations */ struct lru_gen_folio lrugen; +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* to concurrently iterate lru_gen_mm_list */ struct lru_gen_mm_state mm_state; +#else + bool seq_update_progress; +#endif #endif #ifdef CONFIG_MEMCG struct pglist_data *pgdat; @@ -1354,8 +1360,10 @@ typedef struct pglist_data { unsigned long flags; #ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* kswap mm walk data */ struct lru_gen_mm_walk mm_walk; +#endif /* lru_gen_folio list */ struct lru_gen_memcg memcg_lru; #endif diff --git a/include/linux/swap.h b/include/linux/swap.h index 3c69cb653cb9..ce09b1e44275 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -155,7 +155,7 @@ union swap_header { struct reclaim_state { /* pages reclaimed outside of LRU-based reclaim */ unsigned long reclaimed; -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-thread mm walk data */ struct lru_gen_mm_walk *mm_walk; #endif diff --git a/kernel/fork.c b/kernel/fork.c index ed4e01daccaa..2c9e21e39f84 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2939,7 +2939,7 @@ pid_t kernel_clone(struct kernel_clone_args *args) get_task_struct(p); } - if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) { + if (IS_ENABLED(CONFIG_LRU_TASK_PAGE_AGING) && !(clone_flags & CLONE_VM)) { /* lock the task to synchronize with memcg migration */ task_lock(p); lru_gen_add_mm(p->mm); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 959d6a27e23d..d8fe30d880c6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6404,7 +6404,7 @@ static void mem_cgroup_move_task(void) } #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING static void mem_cgroup_attach(struct cgroup_taskset *tset) { struct task_struct *task; diff --git a/mm/vmscan.c b/mm/vmscan.c index f277beba556c..207e62d42888 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3304,6 +3304,7 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) get_nr_gens(lruvec, LRU_GEN_ANON) <= MAX_NR_GENS; } +#ifdef CONFIG_LRU_TASK_PAGE_AGING /****************************************************************************** * Bloom filters ******************************************************************************/ @@ -3650,6 +3651,7 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) return success; } +#endif /****************************************************************************** * PID controller @@ -3819,6 +3821,8 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, return folio; } +#ifdef CONFIG_LRU_TASK_PAGE_AGING + /* promote pages accessed through page tables */ static int folio_update_gen(struct folio *folio, int gen) { @@ -3882,6 +3886,16 @@ static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk } } +static void reset_current_reclaim_batch_size(struct lruvec *lruvec) +{ + struct lru_gen_mm_walk *walk; + + walk = current->reclaim_state->mm_walk; + if (walk && walk->batched) + return reset_batch_size(lruvec, walk); + +} + static int should_skip_vma(unsigned long start, unsigned long end, struct mm_walk *args) { struct address_space *mapping; @@ -4304,7 +4318,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ } while (err == -EAGAIN); } -static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) +static void *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) { struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; @@ -4335,6 +4349,23 @@ static void clear_mm_walk(void) if (!current_is_kswapd()) kfree(walk); } +#else + +static void reset_current_reclaim_batch_size(struct lruvec *lruvec) +{ + +} + +static inline void *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) +{ + return NULL; +} + +static inline void clear_mm_walk(void) +{ + +} +#endif static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { @@ -4468,11 +4499,15 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) /* make sure preceding modifications appear */ smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1); +#ifndef CONFIG_LRU_TASK_PAGE_AGING + lruvec->seq_update_progress = false; +#endif spin_unlock_irq(&lruvec->lru_lock); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, - struct scan_control *sc, bool can_swap, bool force_scan) + bool can_swap, bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -4498,7 +4533,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, goto done; } - walk = set_mm_walk(NULL, true); + walk = (struct lru_gen_mm_walk *)set_mm_walk(NULL, true); if (!walk) { success = iterate_mm_list_nowalk(lruvec, max_seq); goto done; @@ -4520,6 +4555,51 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return success; } +#else + +/* + * inc_max_seq can drop the lru_lock in between. So use a waitqueue seq_update_progress + * to allow concurrent access. + */ +bool __try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + bool success = false; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + + VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq)); + + /* see the comment in iterate_mm_list() */ + if (lruvec->seq_update_progress) + success = false; + else { + spin_lock_irq(&lruvec->lru_lock); + + if (max_seq != lrugen->max_seq) + goto done; + + if (lruvec->seq_update_progress) + goto done; + + success = true; + lruvec->seq_update_progress = true; +done: + spin_unlock_irq(&lruvec->lru_lock); + } + + if (success) + inc_max_seq(lruvec, can_swap, force_scan); + + return success; +} + +static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + return __try_to_inc_max_seq(lruvec, max_seq, can_swap, force_scan); +} +#endif + /****************************************************************************** * working set protection @@ -4630,6 +4710,7 @@ static void __look_around_gen_update(struct folio *folio, int new_gen) folio_activate(folio); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING static inline bool current_reclaim_state_can_swap(void) { if (current->reclaim_state) @@ -4651,6 +4732,18 @@ static void look_around_gen_update(struct folio *folio, int new_gen) } return __look_around_gen_update(folio, new_gen); } +#else + +static inline bool current_reclaim_state_can_swap(void) +{ + return true; +} + +static inline void look_around_gen_update(struct folio *folio, int new_gen) +{ + return __look_around_gen_update(folio, new_gen); +} +#endif /* * This function exploits spatial locality when shrink_folio_list() walks the @@ -4714,7 +4807,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) folio = get_pfn_folio(pfn, memcg, pgdat, current_reclaim_state_can_swap()); - if (!folio) continue; @@ -4734,9 +4826,11 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) arch_leave_lazy_mmu_mode(); mem_cgroup_unlock_pages(); +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* feedback from rmap walkers to page table walkers */ if (suitable_to_scan(i, young)) update_bloom_filter(lruvec, max_seq, pvmw->pmd); +#endif } /****************************************************************************** @@ -5156,7 +5250,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap struct folio *next; enum vm_event_item item; struct reclaim_stat stat; - struct lru_gen_mm_walk *walk; bool skip_retry = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -5211,9 +5304,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap move_folios_to_lru(lruvec, &list); - walk = current->reclaim_state->mm_walk; - if (walk && walk->batched) - reset_batch_size(lruvec, walk); + reset_current_reclaim_batch_size(lruvec); item = PGSTEAL_KSWAPD + reclaimer_offset(); if (!cgroup_reclaim(sc)) @@ -5321,7 +5412,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool return nr_to_scan; /* skip this lruvec as it's low on cold folios */ - return try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false) ? -1 : 0; + return try_to_inc_max_seq(lruvec, max_seq, can_swap, false) ? -1 : 0; } static unsigned long get_nr_to_reclaim(struct scan_control *sc) @@ -5929,6 +6020,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, seq_putc(m, '\n'); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING seq_puts(m, " "); for (i = 0; i < NR_MM_STATS; i++) { const char *s = " "; @@ -5945,6 +6037,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, seq_printf(m, " %10lu%c", n, s[i]); } seq_putc(m, '\n'); +#endif } /* see Documentation/admin-guide/mm/multigen_lru.rst for details */ @@ -6026,7 +6119,7 @@ static int run_aging(struct lruvec *lruvec, unsigned long seq, struct scan_contr if (!force_scan && min_seq[!can_swap] + MAX_NR_GENS - 1 <= max_seq) return -ERANGE; - try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, force_scan); + try_to_inc_max_seq(lruvec, max_seq, can_swap, force_scan); return 0; } @@ -6218,7 +6311,12 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) for_each_gen_type_zone(gen, type, zone) INIT_LIST_HEAD(&lrugen->folios[gen][type][zone]); +#ifdef CONFIG_LRU_TASK_PAGE_AGING lruvec->mm_state.seq = MIN_NR_GENS; +#else + lruvec->seq_update_progress = false; +#endif + } #ifdef CONFIG_MEMCG @@ -6237,16 +6335,20 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat) void lru_gen_init_memcg(struct mem_cgroup *memcg) { +#ifdef CONFIG_LRU_TASK_PAGE_AGING INIT_LIST_HEAD(&memcg->mm_list.fifo); spin_lock_init(&memcg->mm_list.lock); + +#endif } void lru_gen_exit_memcg(struct mem_cgroup *memcg) { - int i; int nid; +#ifdef CONFIG_LRU_TASK_PAGE_AGING VM_WARN_ON_ONCE(!list_empty(&memcg->mm_list.fifo)); +#endif for_each_node(nid) { struct lruvec *lruvec = get_lruvec(memcg, nid); @@ -6256,10 +6358,12 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg) lruvec->lrugen.list.next = LIST_POISON1; - for (i = 0; i < NR_BLOOM_FILTERS; i++) { +#ifdef CONFIG_LRU_TASK_PAGE_AGING + for (int i = 0; i < NR_BLOOM_FILTERS; i++) { bitmap_free(lruvec->mm_state.filters[i]); lruvec->mm_state.filters[i] = NULL; } +#endif } }