From patchwork Tue Jun 13 12:00:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13278670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 067CDC7EE29 for ; Tue, 13 Jun 2023 12:01:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58BCD8E0002; Tue, 13 Jun 2023 08:01:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53C856B007B; Tue, 13 Jun 2023 08:01:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DDEE8E0002; Tue, 13 Jun 2023 08:01:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2D7606B0078 for ; Tue, 13 Jun 2023 08:01:18 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CFD86140470 for ; Tue, 13 Jun 2023 12:01:17 +0000 (UTC) X-FDA: 80897584194.05.A260EC8 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf08.hostedemail.com (Postfix) with ESMTP id 5B2C0160054 for ; Tue, 13 Jun 2023 12:00:57 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=aC8Ml0Tc; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf08.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686657658; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=7oLSxXgTrewlxnbryHws7s+RBbPlO/62vVlC+aUbgtI=; b=4FoVkoopRPx27hDn4UEhj0IXw1t4oTtiYvRJ3sFi9ew0ab5pNVnQzDBbdfazioG3pCIaD9 N8icqgCDyfhWHLFKzMJ/DiW7lQOTQNscYBkRuErW6sY0myrsFVH2Xb2F/OQskx+PJAFP5w SseN4UCLmDxdmrLlwD/XcNdES2t+V3s= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=aC8Ml0Tc; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf08.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686657658; a=rsa-sha256; cv=none; b=IUL6cjrfHHsRs0bJb0t7LQdhBowyowam/Zt14h+P7XlL0m6FuRK186BaNRBJbBwgKsEmvl KsXWY5VyEnhNHcAdNn/DpWNBBa9mbdzAsVYvlpAZ2ICegmBC+kYZcZl5iBnAPsAu9V4vbL rW5NpQnxQFFaMqlXe/TDrrFH8M6Gxnc= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35DBlMkg021969; Tue, 13 Jun 2023 12:00:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=7oLSxXgTrewlxnbryHws7s+RBbPlO/62vVlC+aUbgtI=; b=aC8Ml0TcW4Noq/Uv8Zi0paUik0TdsRMgItsF5w19Fq/zlHPwyLFqNmH3EICjdqAD5Svz h7H8a7q9Ky1tSqDwKH6XXublvOjw6C9P2sFOB4Y2yqMooIxgF2xOYuJJCjBYHXuKSP4+ MyD4jdz/vufcKDTIrlEZ5/8lzii/lFBVQXlscNszXW4mWjRBRlu4DtEF7lvvkgIvtfcr 5KfPfCmh9XRT3v3v7gfcAdM7HlW7tApiXEeUWnqVzvcX6i19saH2hrwfArB4enrl88Vu 8AMCOi7VyJd8JjLN4A5U3ab+9mwlXTbXFVU5ySKvL2oJQaokosdNQMe7kqBYbVUyGCgY KA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qvrr9xx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:55 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35DBma6u024728; Tue, 13 Jun 2023 12:00:55 GMT Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qvrr9x0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:55 +0000 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35D959cP002745; Tue, 13 Jun 2023 12:00:54 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([9.208.130.100]) by ppma01wdc.us.ibm.com (PPS) with ESMTPS id 3r4gt587wk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:54 +0000 Received: from smtpav03.wdc07v.mail.ibm.com (smtpav03.wdc07v.mail.ibm.com [10.39.53.230]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35DC0rAx48562676 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 12:00:53 GMT Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D077A5806D; Tue, 13 Jun 2023 12:00:52 +0000 (GMT) Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 866E15805D; Tue, 13 Jun 2023 12:00:50 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.55.172]) by smtpav03.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 12:00:50 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "T . J . Alumbaugh" , "Aneesh Kumar K.V" Subject: [PATCH 1/3] mm/lru_gen: Move some code around so that next patch is simpler Date: Tue, 13 Jun 2023 17:30:45 +0530 Message-Id: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ib33NBrmlB9Zk9Y0iew6vR6I9R3C4mY2 X-Proofpoint-ORIG-GUID: wyYBzSVhJMwJdmTI25H5OLdcF_YSXOsA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-13_04,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 adultscore=0 priorityscore=1501 mlxlogscore=999 spamscore=0 malwarescore=0 clxscore=1015 phishscore=0 suspectscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130102 X-Rspam-User: X-Stat-Signature: boujamges5fngabar961p4h6p4hbibso X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5B2C0160054 X-HE-Tag: 1686657657-848471 X-HE-Meta: U2FsdGVkX1//zbJvsYL65m37v5RuqQTAPaKOsT/GEhlile2jfhCBzWNClGL8gvpI2/AKG1j+Xld5Ky+w5kmGVJp6YZMEexKH4NT0qlTGUCgE85u0tuMec0PFOCNeBAI7Ra9FW/sGkB6voRdAMOq4G4qSKxOs7sh4SaBtcr8GdrAme1AfN35N1/amoJRccmmDI3ku5BWWRWS9VZSkWAOfhsqZL+L/pQIwo+nsYFNFrNTPeLA+D4XzPQGzCIAfOLtW26r9lUXHDp3R8zD/Y0FrKwhfsDbadIj4qtdKYHQ7gq84nYC8cDSgxbp+T53hhEgi/s+2/sh+a9rYl74tWSC4KouvgJLeVGgoWdtarMmbD/MnLs7/CyC+6Youn3GjYZxwe2YnKOZ+lsHndnm/kAVGPtlDweBdU+7M2IB13zS728t6yXEj0LJULSjwWazBOJYnuwQTppYZd0Vsb84FgvHe2GVzmgJd3BVUq+2XWgnRkS/GBEAbZsAqnZEK0Ik0El9CTA2AS+29CSNWRvtuReWsP6vKQc6otAAXEUgSJ0uJe44bV3aWK003EGwjnJofVcNbt1thmuRIQPlEIdbKG6yBLlxey2CZnKdQnCjWI47tWhyblxC3uJ40204RCE+8XsR/u5wVQW8eoshBZ7a4Cv2R5qz+AT7H74NnHKEEwsDxto0BTL/en3d3ln6wuHIx1uWjegDt8CoKxnHgJcDsgcQjdgsOUK1J8EpyCBf1KkxEZ/Cw12AOT23e3bEyhSsfq3fOGibt4cABEw8XK/34orTPj/E0MLheU1Ezv4NOy9C1jovpCLEjipi2wsZCbOT9qMCm54LNkTl8FaXBcQuEqGX5pTbr3Jrbd11dDmcjQABzRyFphVcLgGNs6TjZp4ZDpQMRbjGXSBRsvmGBWrp8PZLJlgJOjdTEGJJxRwSWBntngBGYJQtz/WaJW7Fat512OeyLAptHkEbgkXER+cfjtMT IjnK3WOO 3IMsHQmxMOMXRR2v0g9Xp62m3d843P0tmX1tbo6EqsaMh3aR17KMGjfXz49ARG7dEafEKzM1X0Kat66ECU4wv/alqwJOkCUGdgu2iFEtY5BrmHn177HHLlQ2+RqeoUDgZSWgpQKQUxh/sWbyuz2jtklvYCBZc5OCy+FCp+XoyBKj3XC3bI7+lsk1RAQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move lrur_gen_add_folio to .c. We will support arch specific mapping of page access count to generation in a later patch and will use that when adding folio to lruvec. This move enables that. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- include/linux/mm_inline.h | 47 +---------- mm/vmscan.c | 172 ++++++++++++++++++++++++-------------- 2 files changed, 110 insertions(+), 109 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 0e1d239a882c..2a86dc4d96ab 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -217,52 +217,7 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen)); } -static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) -{ - unsigned long seq; - unsigned long flags; - int gen = folio_lru_gen(folio); - int type = folio_is_file_lru(folio); - int zone = folio_zonenum(folio); - struct lru_gen_folio *lrugen = &lruvec->lrugen; - - VM_WARN_ON_ONCE_FOLIO(gen != -1, folio); - - if (folio_test_unevictable(folio) || !lrugen->enabled) - return false; - /* - * There are three common cases for this page: - * 1. If it's hot, e.g., freshly faulted in or previously hot and - * migrated, add it to the youngest generation. - * 2. If it's cold but can't be evicted immediately, i.e., an anon page - * not in swapcache or a dirty page pending writeback, add it to the - * second oldest generation. - * 3. Everything else (clean, cold) is added to the oldest generation. - */ - if (folio_test_active(folio)) - seq = lrugen->max_seq; - else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) || - (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio)))) - seq = lrugen->min_seq[type] + 1; - else - seq = lrugen->min_seq[type]; - - gen = lru_gen_from_seq(seq); - flags = (gen + 1UL) << LRU_GEN_PGOFF; - /* see the comment on MIN_NR_GENS about PG_active */ - set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags); - - lru_gen_update_size(lruvec, folio, -1, gen); - /* for folio_rotate_reclaimable() */ - if (reclaiming) - list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]); - else - list_add(&folio->lru, &lrugen->folios[gen][type][zone]); - - return true; -} - +bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming); static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) { unsigned long flags; diff --git a/mm/vmscan.c b/mm/vmscan.c index 6d0cd2840cf0..edfe073b475e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3748,29 +3748,6 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) * the aging ******************************************************************************/ -/* promote pages accessed through page tables */ -static int folio_update_gen(struct folio *folio, int gen) -{ - unsigned long new_flags, old_flags = READ_ONCE(folio->flags); - - VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); - VM_WARN_ON_ONCE(!rcu_read_lock_held()); - - do { - /* lru_gen_del_folio() has isolated this page? */ - if (!(old_flags & LRU_GEN_MASK)) { - /* for shrink_folio_list() */ - new_flags = old_flags | BIT(PG_referenced); - continue; - } - - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); - new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; - } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - - return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; -} - /* protect pages accessed multiple times through file descriptors */ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) { @@ -3801,6 +3778,70 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai return new_gen; } +static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long pfn = pte_pfn(pte); + + VM_WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end); + + if (!pte_present(pte) || is_zero_pfn(pfn)) + return -1; + + if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) + return -1; + + if (WARN_ON_ONCE(!pfn_valid(pfn))) + return -1; + + return pfn; +} + +static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, + struct pglist_data *pgdat, bool can_swap) +{ + struct folio *folio; + + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return NULL; + + folio = pfn_folio(pfn); + if (folio_nid(folio) != pgdat->node_id) + return NULL; + + if (folio_memcg_rcu(folio) != memcg) + return NULL; + + /* file VMAs can contain anon pages from COW */ + if (!folio_is_file_lru(folio) && !can_swap) + return NULL; + + return folio; +} + +/* promote pages accessed through page tables */ +static int folio_update_gen(struct folio *folio, int gen) +{ + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); + VM_WARN_ON_ONCE(!rcu_read_lock_held()); + + do { + /* lru_gen_del_folio() has isolated this page? */ + if (!(old_flags & LRU_GEN_MASK)) { + /* for shrink_folio_list() */ + new_flags = old_flags | BIT(PG_referenced); + continue; + } + + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); + new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; +} + static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio, int old_gen, int new_gen) { @@ -3910,23 +3951,6 @@ static bool get_next_vma(unsigned long mask, unsigned long size, struct mm_walk return false; } -static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) -{ - unsigned long pfn = pte_pfn(pte); - - VM_WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end); - - if (!pte_present(pte) || is_zero_pfn(pfn)) - return -1; - - if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) - return -1; - - if (WARN_ON_ONCE(!pfn_valid(pfn))) - return -1; - - return pfn; -} #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned long addr) @@ -3948,29 +3972,6 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned } #endif -static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, - struct pglist_data *pgdat, bool can_swap) -{ - struct folio *folio; - - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) - return NULL; - - folio = pfn_folio(pfn); - if (folio_nid(folio) != pgdat->node_id) - return NULL; - - if (folio_memcg_rcu(folio) != memcg) - return NULL; - - /* file VMAs can contain anon pages from COW */ - if (!folio_is_file_lru(folio) && !can_swap) - return NULL; - - return folio; -} - static bool suitable_to_scan(int total, int young) { int n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8); @@ -5557,6 +5558,51 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * pgdat->kswapd_failures = 0; } +bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + unsigned long seq; + unsigned long flags; + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio); + struct lru_gen_folio *lrugen = &lruvec->lrugen; + + VM_WARN_ON_ONCE_FOLIO(gen != -1, folio); + + if (folio_test_unevictable(folio) || !lrugen->enabled) + return false; + /* + * There are three common cases for this page: + * 1. If it's hot, e.g., freshly faulted in or previously hot and + * migrated, add it to the youngest generation. + * 2. If it's cold but can't be evicted immediately, i.e., an anon page + * not in swapcache or a dirty page pending writeback, add it to the + * second oldest generation. + * 3. Everything else (clean, cold) is added to the oldest generation. + */ + if (folio_test_active(folio)) + seq = lrugen->max_seq; + else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) || + (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio)))) + seq = lrugen->min_seq[type] + 1; + else + seq = lrugen->min_seq[type]; + + gen = lru_gen_from_seq(seq); + flags = (gen + 1UL) << LRU_GEN_PGOFF; + /* see the comment on MIN_NR_GENS about PG_active */ + set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags); + + lru_gen_update_size(lruvec, folio, -1, gen); + /* for folio_rotate_reclaimable() */ + if (reclaiming) + list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + else + list_add(&folio->lru, &lrugen->folios[gen][type][zone]); + + return true; +} /****************************************************************************** * state change ******************************************************************************/ From patchwork Tue Jun 13 12:00:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13278669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0818C7EE29 for ; Tue, 13 Jun 2023 12:01:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31B056B0075; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CBEC8E0003; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16C608E0002; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 06D6A6B0075 for ; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BAB70C04B7 for ; Tue, 13 Jun 2023 12:01:14 +0000 (UTC) X-FDA: 80897584068.25.FDF6542 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf07.hostedemail.com (Postfix) with ESMTP id 99B9F4005D for ; Tue, 13 Jun 2023 12:01:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=nZ6YmCqA; spf=pass (imf07.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686657660; a=rsa-sha256; cv=none; b=SZO6QRWC7mucwiSasVo6GR90EbIEdobfMTdVpoR05C5iWzGfUtaTq473DSppdWVLqhwzv6 k58Q2Xu89d4qqn3OA29VKff8g40Mjj+6xC+PRrVdhqifE3x82a4RpHLEKaL6fG3jNo1eKD 9WZWKwbq/bjgh945jJ8mjd7H5XbD+1M= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=nZ6YmCqA; spf=pass (imf07.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686657660; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EtZwmgX6tLGpcyaG9SSuk4EKowCGTFn1dpTCAl2Vojk=; b=8JKF5db/X8GEoFXiQzq+wyryUGR7lqZMFWpw/TBH3VYHeZDeBPoQ4bCRU/YlAT2BecluPa Zisbql89OORo3mT3rEXYmMqLkRDqTPgaBpYSZ+ChsM4/RP6FMk8FebsH3UimQNT7Djn8Sa iKFFv0Qeu7aQ+cgqavNnow3CnqHUbug= Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35DBqYcb021229; Tue, 13 Jun 2023 12:00:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=EtZwmgX6tLGpcyaG9SSuk4EKowCGTFn1dpTCAl2Vojk=; b=nZ6YmCqAmbMw0mGsZU2Yilpfu/3fVV8BPJg0VEc1JGvidsANeRjiWZ3JQQHDV37WVHVy mTEiVWZze3/1mXx5LoDNu4lfhbtm121ndCKLtInaaa0pv5dXpNrZ2MjirQbgNpwjGczn K9VQ5E25jp5gIAIl4TYSGxHTidraWTpWUZDYC6WcxovBiKLBDNpxtGgTl3s1CitNjCGi Id6XeYIuPlxunYTqL0hnwUrFHv6txEGEtbN5xv+SPGvWiRDeuh2Wf1VitGjRSwUWqPEW qcFLi+pw3CiT0aOQGl2AZKK72e5+3WGr1oCYIKVUcFQeOlbH5Vu51V/LJ8GFcWMJHwSm +Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485x1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:58 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35DBtJew027755; Tue, 13 Jun 2023 12:00:58 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485wn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:58 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35D98WQP030258; Tue, 13 Jun 2023 12:00:57 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([9.208.129.113]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3r4gt4w0qs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:57 +0000 Received: from smtpav03.wdc07v.mail.ibm.com (smtpav03.wdc07v.mail.ibm.com [10.39.53.230]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35DC0uTd56951094 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 12:00:56 GMT Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EDE3A5805D; Tue, 13 Jun 2023 12:00:55 +0000 (GMT) Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 90DBB58066; Tue, 13 Jun 2023 12:00:53 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.55.172]) by smtpav03.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 12:00:53 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "T . J . Alumbaugh" , "Aneesh Kumar K.V" Subject: [PATCH 2/3] mm/lru_gen: lru_gen_look_around simplification Date: Tue, 13 Jun 2023 17:30:46 +0530 Message-Id: <20230613120047.149573-2-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> References: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 2qGw_7pNQhv9WZnL27yH-TQ1y-J-YSFL X-Proofpoint-GUID: fyID4UonZFw743Vrq88kolFqyGaN4hJm X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-13_04,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 priorityscore=1501 adultscore=0 phishscore=0 mlxlogscore=514 malwarescore=0 mlxscore=0 bulkscore=0 spamscore=0 suspectscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130102 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 99B9F4005D X-Stat-Signature: a3pupxw1pdheiphx4sz9t3yg68xzsjwp X-Rspam-User: X-HE-Tag: 1686657660-903244 X-HE-Meta: U2FsdGVkX1+IJEoMUX9RCTQcqLLCe57GxWevIhoqt8v22sG8Cq2x+FtXDifN6HnmPgRdGObxCe39WrB1qLlvo5UcOFPNoWOplUbiqPK5XoF83rr3K4pTCGOgV8CyXp3L1TS9Fyp4r+MAOh+lfSxD7SCAEkmU1Zi7eb6o2+waymnNTVePT04lUoX8dx55WkR7ZrIx4s6x0BrSXBdgjPC2VrJVImIOdszlxb/b8ExDwUqKWj/OPaX9qEhxt2HZ1lv4TrftlNVaL/8xI7fYU12W4KbBDmG1bkHOP4CgbSIAXomK2ngXX53mbEGRdV8Mmv9HUA+A8pAXBOeGHoKxb/tev9VBB0gXRgUsouXUgU1gwV/4ceCAes50fNekKdmj261uwIxNllyJg4wqgVKD7CKezXfG+zMe66GUQnEWXz8HvX+aJmq4yfIz22Y56+/HFjHOLHr1JQEC+ZWuzHXnek7hV4OgBhcGSUK4bV0eakAYJr/s9Hc+LXadC6wk/fVfn8n/uKOH9Jd78JfEHH6Tdb31PEkIwHsDRmdukYXLg+wsAZk6QA3pnWriH2UQ61p5KDOLdaZhnM7aTWQt2dJSOZI0jtW6pzx97gCFV9P4nC2/GPJnJWXuFr4O7tk3p/ee7UqCBQnrIA5Lg8MzRXnBgmFTc7w7Yd0/Bcs9WKcqARKd9PJiMw7dTDZvoeuLGARxAWHYh+aIHzMd3y/zXH3UMbDS0qOo3bEDhezeo5qWk0hcSpiwWt4be6dzTKhQTDlShDnNNIaEm5CKrCy+PwG2IurC7Ppj1rZWBcapi0/BmRrT0Qe9MGonw55yw7UkMt3SK4wN/2urIpU4fBtAitakCy69CTbKkxv/5UL0V78clKD4EBbU3iUHz/fvGLX8ZSMkwVDj+JbGPagS3PU7CIFPRQC+mmQ5YdK2VTlFVfd6tfQ3Mkw7k+wEHl7pzMPJMPJXHhpCLyoJDHFEvbzxdgfZhdp rRJfo/xB JroLVZrnH+XT1sPQzo3Daq0rVsfSQxGHIA0NEDoJLH9gh7O8IL2dDu+J8XwMuimOUW5VT7Jeht6NX8BMKbPISDLwthWowas922nfunISU1OKuAAFVzh9H7uswLAac7io6jvWhy+d7ocWDrdrobLo2qDIEsfftRVuKGqF8bawWnAycfiu7U0E8bTLP1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To store generation details in folio_flags we need lru_gen_mm_walk structure in which we batch the nr_pages update. A follow-up patch wants to avoid compiling all the lru_gen mm walk-related code on architectures that don't support it. Split out the look-around generation update by marking folio active into a separate helper which will be used in that case. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 57 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index edfe073b475e..f277beba556c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4619,6 +4619,39 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * rmap/PT walk feedback ******************************************************************************/ +static void __look_around_gen_update(struct folio *folio, int new_gen) +{ + int old_gen; + + old_gen = folio_lru_gen(folio); + if (old_gen < 0) + folio_set_referenced(folio); + else if (old_gen != new_gen) + folio_activate(folio); +} + +static inline bool current_reclaim_state_can_swap(void) +{ + if (current->reclaim_state) + return current->reclaim_state->mm_walk->can_swap; + return true; +} + +static void look_around_gen_update(struct folio *folio, int new_gen) +{ + int old_gen; + struct lru_gen_mm_walk *walk; + + walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; + if (walk) { + old_gen = folio_update_gen(folio, new_gen); + if (old_gen >= 0 && old_gen != new_gen) + update_batch_size(walk, folio, old_gen, new_gen); + return; + } + return __look_around_gen_update(folio, new_gen); +} + /* * This function exploits spatial locality when shrink_folio_list() walks the * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If @@ -4631,7 +4664,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) int i; unsigned long start; unsigned long end; - struct lru_gen_mm_walk *walk; int young = 0; pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; @@ -4640,7 +4672,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) struct pglist_data *pgdat = folio_pgdat(folio); struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); DEFINE_MAX_SEQ(lruvec); - int old_gen, new_gen = lru_gen_from_seq(max_seq); + int new_gen = lru_gen_from_seq(max_seq); lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); @@ -4648,9 +4680,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (spin_is_contended(pvmw->ptl)) return; - /* avoid taking the LRU lock under the PTL when possible */ - walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; - start = max(addr & PMD_MASK, pvmw->vma->vm_start); end = min(addr | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1; @@ -4683,7 +4712,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (!pte_young(pte[i])) continue; - folio = get_pfn_folio(pfn, memcg, pgdat, !walk || walk->can_swap); + folio = get_pfn_folio(pfn, memcg, pgdat, + current_reclaim_state_can_swap()); + if (!folio) continue; @@ -4697,19 +4728,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) !folio_test_swapcache(folio))) folio_mark_dirty(folio); - if (walk) { - old_gen = folio_update_gen(folio, new_gen); - if (old_gen >= 0 && old_gen != new_gen) - update_batch_size(walk, folio, old_gen, new_gen); - - continue; - } - - old_gen = folio_lru_gen(folio); - if (old_gen < 0) - folio_set_referenced(folio); - else if (old_gen != new_gen) - folio_activate(folio); + look_around_gen_update(folio, new_gen); } arch_leave_lazy_mmu_mode(); From patchwork Tue Jun 13 12:00:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13278671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D534C7EE2E for ; Tue, 13 Jun 2023 12:01:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABC396B007B; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6CFA8E0005; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E60C8E0003; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7C5226B007B for ; Tue, 13 Jun 2023 08:01:26 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 38507120499 for ; Tue, 13 Jun 2023 12:01:26 +0000 (UTC) X-FDA: 80897584572.04.21856B9 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf23.hostedemail.com (Postfix) with ESMTP id 113A914003F for ; Tue, 13 Jun 2023 12:01:03 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="ffXFpK9/"; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686657664; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n66WQDyiXSrXuj4/rlzufpSs1yap/rsndLr+ifqgiZc=; b=Dd+O+wlYUb6Jfh4P98dlDzaJ3QgiHfSZE+x2+dDaOyCJvBZek1mHNUU4dQZohYyzRIJjL2 9NJLxezK6ZdCUC++9Ji8JedQWWTVI09QzpMTwmhXY2ToBfN/rHpuJni1AceirHnOOXW6le bdMDDhjzWoFhq0qB+UxZ9KU217SIjGg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686657664; a=rsa-sha256; cv=none; b=DSNOGBfNzxhuqu9epYuPYJ6kYBimPFoPDTZL5QNkLVEvsQhdu8dANUC2OcKULMeD7FlI7/ b1d9De9fi1G6dNq9KUPuqCeIQa+6oZYBONQzABDNwI+eh5AS6B/8LyAhnR+c/fbuyvU+L9 qi2Br4VuryI9/8IrjVSLfiVc1fwMpKk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="ffXFpK9/"; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35DBqP7F021068; Tue, 13 Jun 2023 12:01:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=n66WQDyiXSrXuj4/rlzufpSs1yap/rsndLr+ifqgiZc=; b=ffXFpK9/6yjKKdv7blcX3BKiDAAeonm/i3HV9PxBwzEpMBDcDI4+YQUlKpKDffXPAXKD xdXQ6/OIwto5BZlA+ddcGXRJfbMJ1f3IV9hzUgArMHvNHxW5DXu9tbjb+2vxojFdbdUu uzPdwAkD2tIKMFSTNzKbWyCg+84uwndEgBZsqW19TPvb5GaVUK5g4BUEIwNouGjqVeja Q4yASFdIXaohDvg1WeK4s7vkzzL14TcSddO7TMSwHA+vs8TlV5/ot1buM/spLIUwEsj/ flVocH97cTZzi1cApcAfhiPiiGeA1I0MHl0EJSw3KyFB0gHuan9oWf/T943jbe3E/IF9 Jw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485yx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:01:02 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35DBwj0p005967; Tue, 13 Jun 2023 12:01:01 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485y7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:01:01 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35DA0qKB012072; Tue, 13 Jun 2023 12:01:00 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([9.208.129.118]) by ppma01dal.us.ibm.com (PPS) with ESMTPS id 3r4gt61dce-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:01:00 +0000 Received: from smtpav03.wdc07v.mail.ibm.com (smtpav03.wdc07v.mail.ibm.com [10.39.53.230]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35DC0xB556492304 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 12:00:59 GMT Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F0C5E5805D; Tue, 13 Jun 2023 12:00:58 +0000 (GMT) Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8D5CE58068; Tue, 13 Jun 2023 12:00:56 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.55.172]) by smtpav03.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 12:00:56 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "T . J . Alumbaugh" , "Aneesh Kumar K.V" Subject: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported Date: Tue, 13 Jun 2023 17:30:47 +0530 Message-Id: <20230613120047.149573-3-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> References: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 9DxxB5lLQ9em5fDUc3HQGKb4Rykxa5EU X-Proofpoint-GUID: XdftFgSUxaLQfUfyZuzU17Ncp8mnMCBU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-13_04,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 priorityscore=1501 adultscore=0 phishscore=0 mlxlogscore=853 malwarescore=0 mlxscore=0 bulkscore=0 spamscore=0 suspectscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130102 X-Rspamd-Queue-Id: 113A914003F X-Rspam-User: X-Stat-Signature: tdppbx8imw6s9f3abk7dttj8djyjfi97 X-Rspamd-Server: rspam03 X-HE-Tag: 1686657663-706344 X-HE-Meta: U2FsdGVkX1/dQQd3cBq3LBIx2x9WCCwIaWiKLgWHD0xm8Nmg0YACH7dwSF0324yqAe5M0i9iDMa0AJkrn2eUv6JGrANrYEpWB+C/YD/TD3k5dfEp5NV+Q7vb6SCJwFNUoPL3tGcS3z8uAigNpeL1oN/b8TqhutvTkUANA7Vg4ZC6SgbkXrWfv1pbhJQqId2qFU6iIUTWzEhhNZXi4LJhGzjlb98PL1bxm8bfM/phqYEbYA8eOVNg4oVt0ekJzwlcz74k+CXEGdDz17vS7kM4daw+cB65k9iJFh0RCGf4kxNeGCkZspGcz2gFBgTkmqIgPHPEhK1ikba4AHROPCuDiFrMyHwWPWpZNRC3UGBkNWpKrO67P1AuwXORWKTHoRZLw3rnl4ovqfq4XClfXekKJBuMEDO5HvxW7emuopJEwY2s7T3YTLVytwVKsnk74ro+HbCETy9/yB/BwYRjqzIIVmPDf528aoj+2BaHoUNNWm6BB6MFAkXurHG5ZhgzobtNopMw6kUW/ORMIHAx5XjDQfLse1YEoSRSEsERUwWMHUWt8wPFJATc9OXhkmbXeo47kYpuFD5Ep+/YKcVELgzLN73ZusIL7zFjTuDM/I1iOUXT0bTh9mXkbaWEeVbE7ZSBAiAPAhvS4cZxr8DYQQOJAiH1sW5xDmGbJs2HXPZZj5ESKHiw7NYMAFg0v/269Ky2CzxTpSOlTaD9MPgkN0hAhZkKiKiF2cu5+eSfQJgHvuk/1Hs32lc8xxDuLoFheKClJXJcKNNsUE6j+ucaZ+NqIy/NJSQoxoM1ZbONTXvgEG7ZLyi/A13LCbjPgvMNmZvK3zFsz11fwmfOAawrcK83gmiEJfqlfOdRGD7AT92HmEsXAio141jYOEf1xui/Qo5JMLiDKtepInz9Wms+D0V+pjSpBdIYpu/dScJHIGNnNkhAV6wR1Ol+YGeZpaBY4SX2QO1Rnr0qesN2jAdzP7S krbmQJz5 OwCwDNFYKh911t8kxmAPcv31/89dzYsgQzYm4bEThMyjl7989Jc7R4FmU9Ts7V4hrJGqwRlU1QezAVsn4sumeSjA02iId0OHSKS2IBh3LHaG8UehYxCRvM0XzzcriBPNzvNHxqvBOHd3WA+WhTbcQJDWqevCxFFaLcpvaNxawp2RCx2cNUPMJlDjoLQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture. No preformance change observed with mongodb ycsb test: Patch details Throughput(Ops/sec) without patch 93278 With patch 93400 Without patch: $ size mm/vmscan.o text data bss dec hex filename 112102 42721 40 154863 25cef mm/vmscan.o With patch $ size mm/vmscan.o text data bss dec hex filename 105430 41333 24 146787 23d63 mm/vmscan.o Signed-off-by: Aneesh Kumar K.V --- arch/Kconfig | 3 + arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/memcontrol.h | 2 +- include/linux/mm_types.h | 8 +-- include/linux/mmzone.h | 8 +++ include/linux/swap.h | 2 +- kernel/fork.c | 2 +- mm/memcontrol.c | 2 +- mm/vmscan.c | 128 +++++++++++++++++++++++++++++++++---- 10 files changed, 137 insertions(+), 20 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 205fd23e0cad..5cdd98731298 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1458,6 +1458,9 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config LRU_TASK_PAGE_AGING + bool + config ARCH_HAS_NONLEAF_PMD_YOUNG bool help diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b1201d25a8a4..e0994fb3504b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -225,6 +225,7 @@ config ARM64 select IRQ_DOMAIN select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN + select LRU_TASK_PAGE_AGING if LRU_GEN select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE select NEED_SG_DMA_LENGTH diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 53bab123a8ee..bde9e6f33b22 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -276,6 +276,7 @@ config X86 select HAVE_GENERIC_VDSO select HOTPLUG_SMT if SMP select IRQ_FORCED_THREADING + select LRU_TASK_PAGE_AGING if LRU_GEN select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK select NEED_SG_DMA_LENGTH diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 43d4ec8445d4..ea5d1d7bfb8b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -321,7 +321,7 @@ struct mem_cgroup { struct deferred_split deferred_split_queue; #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-memcg mm_struct list */ struct lru_gen_mm_list mm_list; #endif diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..f90a4860a792 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -786,7 +786,7 @@ struct mm_struct { */ unsigned long ksm_rmap_items; #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct { /* this mm_struct is on lru_gen_mm_list */ struct list_head list; @@ -801,7 +801,7 @@ struct mm_struct { struct mem_cgroup *memcg; #endif } lru_gen; -#endif /* CONFIG_LRU_GEN */ +#endif /* CONFIG_LRU_TASK_PAGE_AGING */ } __randomize_layout; /* @@ -830,7 +830,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) return (struct cpumask *)&mm->cpu_bitmap; } -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct lru_gen_mm_list { /* mm_struct list for page table walkers */ @@ -864,7 +864,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm) WRITE_ONCE(mm->lru_gen.bitmap, -1); } -#else /* !CONFIG_LRU_GEN */ +#else /* !CONFIG_LRU_TASK_PAGE_AGING */ static inline void lru_gen_add_mm(struct mm_struct *mm) { diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a4889c9d4055..b35698148d3c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -427,6 +427,7 @@ struct lru_gen_folio { #endif }; +#ifdef CONFIG_LRU_TASK_PAGE_AGING enum { MM_LEAF_TOTAL, /* total leaf entries */ MM_LEAF_OLD, /* old leaf entries */ @@ -469,6 +470,7 @@ struct lru_gen_mm_walk { bool can_swap; bool force_scan; }; +#endif void lru_gen_init_lruvec(struct lruvec *lruvec); void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); @@ -613,8 +615,12 @@ struct lruvec { #ifdef CONFIG_LRU_GEN /* evictable pages divided into generations */ struct lru_gen_folio lrugen; +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* to concurrently iterate lru_gen_mm_list */ struct lru_gen_mm_state mm_state; +#else + bool seq_update_progress; +#endif #endif #ifdef CONFIG_MEMCG struct pglist_data *pgdat; @@ -1354,8 +1360,10 @@ typedef struct pglist_data { unsigned long flags; #ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* kswap mm walk data */ struct lru_gen_mm_walk mm_walk; +#endif /* lru_gen_folio list */ struct lru_gen_memcg memcg_lru; #endif diff --git a/include/linux/swap.h b/include/linux/swap.h index 3c69cb653cb9..ce09b1e44275 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -155,7 +155,7 @@ union swap_header { struct reclaim_state { /* pages reclaimed outside of LRU-based reclaim */ unsigned long reclaimed; -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-thread mm walk data */ struct lru_gen_mm_walk *mm_walk; #endif diff --git a/kernel/fork.c b/kernel/fork.c index ed4e01daccaa..2c9e21e39f84 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2939,7 +2939,7 @@ pid_t kernel_clone(struct kernel_clone_args *args) get_task_struct(p); } - if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) { + if (IS_ENABLED(CONFIG_LRU_TASK_PAGE_AGING) && !(clone_flags & CLONE_VM)) { /* lock the task to synchronize with memcg migration */ task_lock(p); lru_gen_add_mm(p->mm); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 959d6a27e23d..d8fe30d880c6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6404,7 +6404,7 @@ static void mem_cgroup_move_task(void) } #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING static void mem_cgroup_attach(struct cgroup_taskset *tset) { struct task_struct *task; diff --git a/mm/vmscan.c b/mm/vmscan.c index f277beba556c..207e62d42888 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3304,6 +3304,7 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) get_nr_gens(lruvec, LRU_GEN_ANON) <= MAX_NR_GENS; } +#ifdef CONFIG_LRU_TASK_PAGE_AGING /****************************************************************************** * Bloom filters ******************************************************************************/ @@ -3650,6 +3651,7 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) return success; } +#endif /****************************************************************************** * PID controller @@ -3819,6 +3821,8 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, return folio; } +#ifdef CONFIG_LRU_TASK_PAGE_AGING + /* promote pages accessed through page tables */ static int folio_update_gen(struct folio *folio, int gen) { @@ -3882,6 +3886,16 @@ static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk } } +static void reset_current_reclaim_batch_size(struct lruvec *lruvec) +{ + struct lru_gen_mm_walk *walk; + + walk = current->reclaim_state->mm_walk; + if (walk && walk->batched) + return reset_batch_size(lruvec, walk); + +} + static int should_skip_vma(unsigned long start, unsigned long end, struct mm_walk *args) { struct address_space *mapping; @@ -4304,7 +4318,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ } while (err == -EAGAIN); } -static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) +static void *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) { struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; @@ -4335,6 +4349,23 @@ static void clear_mm_walk(void) if (!current_is_kswapd()) kfree(walk); } +#else + +static void reset_current_reclaim_batch_size(struct lruvec *lruvec) +{ + +} + +static inline void *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) +{ + return NULL; +} + +static inline void clear_mm_walk(void) +{ + +} +#endif static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { @@ -4468,11 +4499,15 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) /* make sure preceding modifications appear */ smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1); +#ifndef CONFIG_LRU_TASK_PAGE_AGING + lruvec->seq_update_progress = false; +#endif spin_unlock_irq(&lruvec->lru_lock); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, - struct scan_control *sc, bool can_swap, bool force_scan) + bool can_swap, bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -4498,7 +4533,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, goto done; } - walk = set_mm_walk(NULL, true); + walk = (struct lru_gen_mm_walk *)set_mm_walk(NULL, true); if (!walk) { success = iterate_mm_list_nowalk(lruvec, max_seq); goto done; @@ -4520,6 +4555,51 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return success; } +#else + +/* + * inc_max_seq can drop the lru_lock in between. So use a waitqueue seq_update_progress + * to allow concurrent access. + */ +bool __try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + bool success = false; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + + VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq)); + + /* see the comment in iterate_mm_list() */ + if (lruvec->seq_update_progress) + success = false; + else { + spin_lock_irq(&lruvec->lru_lock); + + if (max_seq != lrugen->max_seq) + goto done; + + if (lruvec->seq_update_progress) + goto done; + + success = true; + lruvec->seq_update_progress = true; +done: + spin_unlock_irq(&lruvec->lru_lock); + } + + if (success) + inc_max_seq(lruvec, can_swap, force_scan); + + return success; +} + +static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + return __try_to_inc_max_seq(lruvec, max_seq, can_swap, force_scan); +} +#endif + /****************************************************************************** * working set protection @@ -4630,6 +4710,7 @@ static void __look_around_gen_update(struct folio *folio, int new_gen) folio_activate(folio); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING static inline bool current_reclaim_state_can_swap(void) { if (current->reclaim_state) @@ -4651,6 +4732,18 @@ static void look_around_gen_update(struct folio *folio, int new_gen) } return __look_around_gen_update(folio, new_gen); } +#else + +static inline bool current_reclaim_state_can_swap(void) +{ + return true; +} + +static inline void look_around_gen_update(struct folio *folio, int new_gen) +{ + return __look_around_gen_update(folio, new_gen); +} +#endif /* * This function exploits spatial locality when shrink_folio_list() walks the @@ -4714,7 +4807,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) folio = get_pfn_folio(pfn, memcg, pgdat, current_reclaim_state_can_swap()); - if (!folio) continue; @@ -4734,9 +4826,11 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) arch_leave_lazy_mmu_mode(); mem_cgroup_unlock_pages(); +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* feedback from rmap walkers to page table walkers */ if (suitable_to_scan(i, young)) update_bloom_filter(lruvec, max_seq, pvmw->pmd); +#endif } /****************************************************************************** @@ -5156,7 +5250,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap struct folio *next; enum vm_event_item item; struct reclaim_stat stat; - struct lru_gen_mm_walk *walk; bool skip_retry = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -5211,9 +5304,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap move_folios_to_lru(lruvec, &list); - walk = current->reclaim_state->mm_walk; - if (walk && walk->batched) - reset_batch_size(lruvec, walk); + reset_current_reclaim_batch_size(lruvec); item = PGSTEAL_KSWAPD + reclaimer_offset(); if (!cgroup_reclaim(sc)) @@ -5321,7 +5412,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool return nr_to_scan; /* skip this lruvec as it's low on cold folios */ - return try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false) ? -1 : 0; + return try_to_inc_max_seq(lruvec, max_seq, can_swap, false) ? -1 : 0; } static unsigned long get_nr_to_reclaim(struct scan_control *sc) @@ -5929,6 +6020,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, seq_putc(m, '\n'); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING seq_puts(m, " "); for (i = 0; i < NR_MM_STATS; i++) { const char *s = " "; @@ -5945,6 +6037,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, seq_printf(m, " %10lu%c", n, s[i]); } seq_putc(m, '\n'); +#endif } /* see Documentation/admin-guide/mm/multigen_lru.rst for details */ @@ -6026,7 +6119,7 @@ static int run_aging(struct lruvec *lruvec, unsigned long seq, struct scan_contr if (!force_scan && min_seq[!can_swap] + MAX_NR_GENS - 1 <= max_seq) return -ERANGE; - try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, force_scan); + try_to_inc_max_seq(lruvec, max_seq, can_swap, force_scan); return 0; } @@ -6218,7 +6311,12 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) for_each_gen_type_zone(gen, type, zone) INIT_LIST_HEAD(&lrugen->folios[gen][type][zone]); +#ifdef CONFIG_LRU_TASK_PAGE_AGING lruvec->mm_state.seq = MIN_NR_GENS; +#else + lruvec->seq_update_progress = false; +#endif + } #ifdef CONFIG_MEMCG @@ -6237,16 +6335,20 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat) void lru_gen_init_memcg(struct mem_cgroup *memcg) { +#ifdef CONFIG_LRU_TASK_PAGE_AGING INIT_LIST_HEAD(&memcg->mm_list.fifo); spin_lock_init(&memcg->mm_list.lock); + +#endif } void lru_gen_exit_memcg(struct mem_cgroup *memcg) { - int i; int nid; +#ifdef CONFIG_LRU_TASK_PAGE_AGING VM_WARN_ON_ONCE(!list_empty(&memcg->mm_list.fifo)); +#endif for_each_node(nid) { struct lruvec *lruvec = get_lruvec(memcg, nid); @@ -6256,10 +6358,12 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg) lruvec->lrugen.list.next = LIST_POISON1; - for (i = 0; i < NR_BLOOM_FILTERS; i++) { +#ifdef CONFIG_LRU_TASK_PAGE_AGING + for (int i = 0; i < NR_BLOOM_FILTERS; i++) { bitmap_free(lruvec->mm_state.filters[i]); lruvec->mm_state.filters[i] = NULL; } +#endif } }