From patchwork Tue Jun 13 12:00:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13278669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0818C7EE29 for ; Tue, 13 Jun 2023 12:01:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31B056B0075; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CBEC8E0003; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16C608E0002; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 06D6A6B0075 for ; Tue, 13 Jun 2023 08:01:15 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BAB70C04B7 for ; Tue, 13 Jun 2023 12:01:14 +0000 (UTC) X-FDA: 80897584068.25.FDF6542 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf07.hostedemail.com (Postfix) with ESMTP id 99B9F4005D for ; Tue, 13 Jun 2023 12:01:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=nZ6YmCqA; spf=pass (imf07.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686657660; a=rsa-sha256; cv=none; b=SZO6QRWC7mucwiSasVo6GR90EbIEdobfMTdVpoR05C5iWzGfUtaTq473DSppdWVLqhwzv6 k58Q2Xu89d4qqn3OA29VKff8g40Mjj+6xC+PRrVdhqifE3x82a4RpHLEKaL6fG3jNo1eKD 9WZWKwbq/bjgh945jJ8mjd7H5XbD+1M= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=nZ6YmCqA; spf=pass (imf07.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686657660; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EtZwmgX6tLGpcyaG9SSuk4EKowCGTFn1dpTCAl2Vojk=; b=8JKF5db/X8GEoFXiQzq+wyryUGR7lqZMFWpw/TBH3VYHeZDeBPoQ4bCRU/YlAT2BecluPa Zisbql89OORo3mT3rEXYmMqLkRDqTPgaBpYSZ+ChsM4/RP6FMk8FebsH3UimQNT7Djn8Sa iKFFv0Qeu7aQ+cgqavNnow3CnqHUbug= Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35DBqYcb021229; Tue, 13 Jun 2023 12:00:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=EtZwmgX6tLGpcyaG9SSuk4EKowCGTFn1dpTCAl2Vojk=; b=nZ6YmCqAmbMw0mGsZU2Yilpfu/3fVV8BPJg0VEc1JGvidsANeRjiWZ3JQQHDV37WVHVy mTEiVWZze3/1mXx5LoDNu4lfhbtm121ndCKLtInaaa0pv5dXpNrZ2MjirQbgNpwjGczn K9VQ5E25jp5gIAIl4TYSGxHTidraWTpWUZDYC6WcxovBiKLBDNpxtGgTl3s1CitNjCGi Id6XeYIuPlxunYTqL0hnwUrFHv6txEGEtbN5xv+SPGvWiRDeuh2Wf1VitGjRSwUWqPEW qcFLi+pw3CiT0aOQGl2AZKK72e5+3WGr1oCYIKVUcFQeOlbH5Vu51V/LJ8GFcWMJHwSm +Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485x1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:58 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35DBtJew027755; Tue, 13 Jun 2023 12:00:58 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6qy485wn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:58 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35D98WQP030258; Tue, 13 Jun 2023 12:00:57 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([9.208.129.113]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3r4gt4w0qs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:00:57 +0000 Received: from smtpav03.wdc07v.mail.ibm.com (smtpav03.wdc07v.mail.ibm.com [10.39.53.230]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35DC0uTd56951094 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 12:00:56 GMT Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EDE3A5805D; Tue, 13 Jun 2023 12:00:55 +0000 (GMT) Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 90DBB58066; Tue, 13 Jun 2023 12:00:53 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.55.172]) by smtpav03.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 12:00:53 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "T . J . Alumbaugh" , "Aneesh Kumar K.V" Subject: [PATCH 2/3] mm/lru_gen: lru_gen_look_around simplification Date: Tue, 13 Jun 2023 17:30:46 +0530 Message-Id: <20230613120047.149573-2-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> References: <20230613120047.149573-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 2qGw_7pNQhv9WZnL27yH-TQ1y-J-YSFL X-Proofpoint-GUID: fyID4UonZFw743Vrq88kolFqyGaN4hJm X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-13_04,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 priorityscore=1501 adultscore=0 phishscore=0 mlxlogscore=514 malwarescore=0 mlxscore=0 bulkscore=0 spamscore=0 suspectscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130102 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 99B9F4005D X-Stat-Signature: a3pupxw1pdheiphx4sz9t3yg68xzsjwp X-Rspam-User: X-HE-Tag: 1686657660-903244 X-HE-Meta: U2FsdGVkX1+IJEoMUX9RCTQcqLLCe57GxWevIhoqt8v22sG8Cq2x+FtXDifN6HnmPgRdGObxCe39WrB1qLlvo5UcOFPNoWOplUbiqPK5XoF83rr3K4pTCGOgV8CyXp3L1TS9Fyp4r+MAOh+lfSxD7SCAEkmU1Zi7eb6o2+waymnNTVePT04lUoX8dx55WkR7ZrIx4s6x0BrSXBdgjPC2VrJVImIOdszlxb/b8ExDwUqKWj/OPaX9qEhxt2HZ1lv4TrftlNVaL/8xI7fYU12W4KbBDmG1bkHOP4CgbSIAXomK2ngXX53mbEGRdV8Mmv9HUA+A8pAXBOeGHoKxb/tev9VBB0gXRgUsouXUgU1gwV/4ceCAes50fNekKdmj261uwIxNllyJg4wqgVKD7CKezXfG+zMe66GUQnEWXz8HvX+aJmq4yfIz22Y56+/HFjHOLHr1JQEC+ZWuzHXnek7hV4OgBhcGSUK4bV0eakAYJr/s9Hc+LXadC6wk/fVfn8n/uKOH9Jd78JfEHH6Tdb31PEkIwHsDRmdukYXLg+wsAZk6QA3pnWriH2UQ61p5KDOLdaZhnM7aTWQt2dJSOZI0jtW6pzx97gCFV9P4nC2/GPJnJWXuFr4O7tk3p/ee7UqCBQnrIA5Lg8MzRXnBgmFTc7w7Yd0/Bcs9WKcqARKd9PJiMw7dTDZvoeuLGARxAWHYh+aIHzMd3y/zXH3UMbDS0qOo3bEDhezeo5qWk0hcSpiwWt4be6dzTKhQTDlShDnNNIaEm5CKrCy+PwG2IurC7Ppj1rZWBcapi0/BmRrT0Qe9MGonw55yw7UkMt3SK4wN/2urIpU4fBtAitakCy69CTbKkxv/5UL0V78clKD4EBbU3iUHz/fvGLX8ZSMkwVDj+JbGPagS3PU7CIFPRQC+mmQ5YdK2VTlFVfd6tfQ3Mkw7k+wEHl7pzMPJMPJXHhpCLyoJDHFEvbzxdgfZhdp rRJfo/xB JroLVZrnH+XT1sPQzo3Daq0rVsfSQxGHIA0NEDoJLH9gh7O8IL2dDu+J8XwMuimOUW5VT7Jeht6NX8BMKbPISDLwthWowas922nfunISU1OKuAAFVzh9H7uswLAac7io6jvWhy+d7ocWDrdrobLo2qDIEsfftRVuKGqF8bawWnAycfiu7U0E8bTLP1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To store generation details in folio_flags we need lru_gen_mm_walk structure in which we batch the nr_pages update. A follow-up patch wants to avoid compiling all the lru_gen mm walk-related code on architectures that don't support it. Split out the look-around generation update by marking folio active into a separate helper which will be used in that case. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 57 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index edfe073b475e..f277beba556c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4619,6 +4619,39 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * rmap/PT walk feedback ******************************************************************************/ +static void __look_around_gen_update(struct folio *folio, int new_gen) +{ + int old_gen; + + old_gen = folio_lru_gen(folio); + if (old_gen < 0) + folio_set_referenced(folio); + else if (old_gen != new_gen) + folio_activate(folio); +} + +static inline bool current_reclaim_state_can_swap(void) +{ + if (current->reclaim_state) + return current->reclaim_state->mm_walk->can_swap; + return true; +} + +static void look_around_gen_update(struct folio *folio, int new_gen) +{ + int old_gen; + struct lru_gen_mm_walk *walk; + + walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; + if (walk) { + old_gen = folio_update_gen(folio, new_gen); + if (old_gen >= 0 && old_gen != new_gen) + update_batch_size(walk, folio, old_gen, new_gen); + return; + } + return __look_around_gen_update(folio, new_gen); +} + /* * This function exploits spatial locality when shrink_folio_list() walks the * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If @@ -4631,7 +4664,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) int i; unsigned long start; unsigned long end; - struct lru_gen_mm_walk *walk; int young = 0; pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; @@ -4640,7 +4672,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) struct pglist_data *pgdat = folio_pgdat(folio); struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); DEFINE_MAX_SEQ(lruvec); - int old_gen, new_gen = lru_gen_from_seq(max_seq); + int new_gen = lru_gen_from_seq(max_seq); lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); @@ -4648,9 +4680,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (spin_is_contended(pvmw->ptl)) return; - /* avoid taking the LRU lock under the PTL when possible */ - walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; - start = max(addr & PMD_MASK, pvmw->vma->vm_start); end = min(addr | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1; @@ -4683,7 +4712,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) if (!pte_young(pte[i])) continue; - folio = get_pfn_folio(pfn, memcg, pgdat, !walk || walk->can_swap); + folio = get_pfn_folio(pfn, memcg, pgdat, + current_reclaim_state_can_swap()); + if (!folio) continue; @@ -4697,19 +4728,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) !folio_test_swapcache(folio))) folio_mark_dirty(folio); - if (walk) { - old_gen = folio_update_gen(folio, new_gen); - if (old_gen >= 0 && old_gen != new_gen) - update_batch_size(walk, folio, old_gen, new_gen); - - continue; - } - - old_gen = folio_lru_gen(folio); - if (old_gen < 0) - folio_set_referenced(folio); - else if (old_gen != new_gen) - folio_activate(folio); + look_around_gen_update(folio, new_gen); } arch_leave_lazy_mmu_mode();