From patchwork Thu Jul 6 06:20:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E67C7EB64D9 for ; Thu, 6 Jul 2023 07:04:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3FCEB8D0002; Thu, 6 Jul 2023 03:04:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AD318D0001; Thu, 6 Jul 2023 03:04:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29C1A8D0002; Thu, 6 Jul 2023 03:04:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1B3548D0001 for ; Thu, 6 Jul 2023 03:04:51 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D7E81404D2 for ; Thu, 6 Jul 2023 07:04:50 +0000 (UTC) X-FDA: 80980299540.22.055D374 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf10.hostedemail.com (Postfix) with ESMTP id 2E815C0013 for ; Thu, 6 Jul 2023 07:04:47 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=bu1f8eLU; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688627088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ORUTZlKjN0qMpiS0OXppYHa5tIArtT9ztqKk+Xsat/U=; b=6tX/xL7YDTC1Djhy5XWjR6Mi991SbffMFcATxxNVFzsDGDjDc0NGX6PuQfgLE35V4b6q7P VGa19CXi4t8QNJcW0WftjTDO9FPHxYlVkgwebBXeXVn+vzKG/0uQHToSPajDBOnaykO+it HvXdYNo8iMed/8fjnkjNRfo60EBWwcE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688627088; a=rsa-sha256; cv=none; b=puLpdZqAn3Oplq5nUhiSjM1mkKVeM+iKRsqOgAo5S4sPkbotm0FrYEjCVpaLhMHBqIBIat otoTwszuwjbZ66p5IO86l/mU9b8TXu3ccxAUN/wFCdqd0w81SHuQBr4LmPIeDK8wwa5B8k opQaXmzQvaZgGCAIOCLTF/9Qe2s7mCA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=bu1f8eLU; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3666qAvO031213; Thu, 6 Jul 2023 07:04:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=ORUTZlKjN0qMpiS0OXppYHa5tIArtT9ztqKk+Xsat/U=; b=bu1f8eLUcT9Sj+DJcMYEKZ5N+CQPEGC2Q+NmlQLA4C2hmC6FJBQDfjXAiTZ9evpF6A+E gznSxmy1oyLiRXq0PQuqyqiw5LJ97vtSkpe8f8zbCfsVkpgGPbkG9ogyqBM9apgHQg5t oQBAZ0l10FeKJqH6bXim2iCbl6O2U+uJDptGjkMlwvOaB/G0QC+oIoJsx0oA+W96VEM2 gpqWNnBHa/DW+U9IcanFBinMO8bl6No+E33gJA3f3nr8JLCEjekLOkgmDE1KboornwlY aR9xxk9Bx9/z14xHfsC5+13O3dtoMbouYykwMLTqHtKdTsajDd2nPp/3biqj7gwyuTOj Vg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnrqhrasf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 07:04:46 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3666qKu3031585; Thu, 6 Jul 2023 07:04:46 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnrqhranh-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 07:04:46 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 365KWKOw001784; Thu, 6 Jul 2023 06:21:02 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([9.208.129.120]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3rjbs57x1f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:02 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3666L1BO57671942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 06:21:01 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1071458059; Thu, 6 Jul 2023 06:21:01 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 253CA58058; Thu, 6 Jul 2023 06:20:59 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 06:20:58 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "Aneesh Kumar K.V" Subject: [PATCH v2 1/5] mm/mglru: Create a new helper iterate_mm_list_walk Date: Thu, 6 Jul 2023 11:50:40 +0530 Message-ID: <20230706062044.816068-2-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> References: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: OHH1ksSooPEyh82LjoWixJMCvgAsgvtC X-Proofpoint-ORIG-GUID: iKNf1M-S1aQDJT203zFYnru0eXnT2_nd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_04,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 mlxlogscore=999 phishscore=0 impostorscore=0 suspectscore=0 clxscore=1015 adultscore=0 lowpriorityscore=0 malwarescore=0 bulkscore=0 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060062 X-Rspamd-Queue-Id: 2E815C0013 X-Rspam-User: X-Stat-Signature: js35dghcpxd6tz9ris7fktcps1td9z8g X-Rspamd-Server: rspam03 X-HE-Tag: 1688627087-990594 X-HE-Meta: U2FsdGVkX18iOkJJj5FCSfP8yJ9c/AZiHmitr0TtkX2kbfOrS7R7cO+Q31DrHuyW9lc2KyyUnyy5IArFHzsOlgRHhc0UT887rRFWEAq/zIR2gZ35JGh2uedOUWuccgj7ZxChZRcacbUVPKPVz9+prJPmp6SVeREZuYq65JbZ4ZrRHD4HRR7Z9dB0LblcignQKVHX2S0q8GdZqdaKsXHSH9OPSi2m0pjt+5aDsN1VDxOJesTxuPd9FC2d4fuwIQ1DWuhEUELHy7qR55MxANncriSeFYNJd4fdIljKgb/oNhXikY58yYOVip9uixIYNqmZP/TGF7lR8dn+v8pOLRlQEKmFJ/P0aLLolRqWjmumyuRDa6gJIQhIkoH7hXV7Y6g/rpoaekOkWmVejUfB5GrCaCijF5LkHEq4/ujZ2MZMKGWYkqgRuqeSA0Hn5kAVLVZLknuzCT+0jdWYm2ic1drqKEG2WBRNx3jLNP7ZuADI/7OU7BeQDHcKEAKKs09oqQtyBQZXxMWaGHx/a76za+BWhiDsWVoieJRojHKmETISBQGjCqS53oKYDSJN+suqCrLjDvUgm77O8q50YtDl5ml0qUosZVz/RdcltrNOpBkWJxMUq93Si1amu+4fa8hHXRfYhzmwIk+FNeqJ3WSnThFxSN0wTVFa+AcO5kWd3Pp+R+/ELuE4HCRxbz9hNVcixVBs2QOz3hnu/zhTl01K9l6SrbA8QaYbcgmLvRn+bQtfwG/u7IVt/CmNo8hpl6DZCDgAhlddb9QKZRQ/4OYFGlhGkc6PIMAcfJ7J/bY7l1cPnZVoI/zKpswv6ZRlkzpY0ydC8Ub93DOgg11SD0U6UjDQUthCSoAv8BwjMgc2rfqQ0Nph6J30+e5bld84oGcJO5EEbm3Za3GLr+eTJolN4kC8WLlS0O17UQcQpIm9voJBnXB2Auuv/9ngKhMuCw2Iy44JPrOsAxCSly7Z/DVGVH3 2ZKQ8rQc rrL0t533fhcY1dcfSw2TtLoUeqs6IUNBZT61CNXWBLjSpfXg8aCWo74zznsi/Q1lG5ddn6sVmGhJMrmV1MW54bE+VQJw5GsKPTJbEPRIZ8pq9JLQbIUgJ+MMGfx24MEdyEpPU6A9VPAMSMIkUMmMY0C80hgI8E8bMf/d20SKHO6xfHUXaKe59i0gg0g0tZA1FeEpwD8hAqEa4ExE3eXUMOCcEn8UF/2ZO3UGGGM+ZhdN8g6TM4N/RHPX6Nlz3/a+0pPKeTlCvNarow/3aM+mbeEyldg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In a later patch we will not build this on ppc64 architecture. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 52 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index eb23bb1afc64..3b183f704d5d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4491,12 +4491,37 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) spin_unlock_irq(&lruvec->lru_lock); } +static bool iterate_mm_list_walk(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + bool success; + struct mm_struct *mm = NULL; + struct lru_gen_mm_walk *walk; + + walk = set_mm_walk(NULL, true); + if (!walk) { + success = iterate_mm_list_nowalk(lruvec, max_seq); + return success; + } + + walk->lruvec = lruvec; + walk->max_seq = max_seq; + walk->can_swap = can_swap; + walk->force_scan = force_scan; + + do { + success = iterate_mm_list(lruvec, walk, &mm); + if (mm) + walk_mm(lruvec, mm, walk); + } while (mm); + + return success; +} + static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, bool can_swap, bool force_scan) { bool success; - struct lru_gen_mm_walk *walk; - struct mm_struct *mm = NULL; struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq)); @@ -4506,34 +4531,17 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, success = false; goto done; } - /* * If the hardware doesn't automatically set the accessed bit, fallback * to lru_gen_look_around(), which only clears the accessed bit in a * handful of PTEs. Spreading the work out over a period of time usually * is less efficient, but it avoids bursty page faults. */ - if (!should_walk_mmu()) { - success = iterate_mm_list_nowalk(lruvec, max_seq); - goto done; - } - - walk = set_mm_walk(NULL, true); - if (!walk) { + if (!should_walk_mmu()) success = iterate_mm_list_nowalk(lruvec, max_seq); - goto done; - } - - walk->lruvec = lruvec; - walk->max_seq = max_seq; - walk->can_swap = can_swap; - walk->force_scan = force_scan; + else + success = iterate_mm_list_walk(lruvec, max_seq, can_swap, force_scan); - do { - success = iterate_mm_list(lruvec, walk, &mm); - if (mm) - walk_mm(lruvec, mm, walk); - } while (mm); done: if (success) inc_max_seq(lruvec, can_swap, force_scan); From patchwork Thu Jul 6 06:20:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A532EB64D9 for ; Thu, 6 Jul 2023 07:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 945448D0002; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D9118D0001; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 722C18D0002; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 59D4B8D0001 for ; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2438AA0213 for ; Thu, 6 Jul 2023 07:45:12 +0000 (UTC) X-FDA: 80980401264.24.205F83D Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf02.hostedemail.com (Postfix) with ESMTP id 8C5DC80013 for ; Thu, 6 Jul 2023 07:45:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=E7zR4qBb; spf=pass (imf02.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688629509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F8ycTdDdQYcTUA9W4ZMinDPt78Egmnbx9cZ0zDL/uk0=; b=TKPnNnZFSw5sO2/AdK3qPeio86KVuvQblvxDroE5RjaFlZCO+daHfZ3eLbH/5kTg3thlr8 oslVGHk8lNfikLv1R/fGKURb3+naZnSWaLnQTCR/TsQiL4K/WqKi0fIldJ4sBlcfHDjxXR cTVpdmN2Y3kk/n6xcwGaCKRMP1VpBJs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=E7zR4qBb; spf=pass (imf02.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688629509; a=rsa-sha256; cv=none; b=O088Gfd6F/yvg0qLofq/iTBMa7/InwXD3zSG2/+5mqfuPS7NYI/hK6ZYtpRhQs0V+ozfvb P3WEtTSWBVegfbJQfdghA6Y347ZuGMQW9+lSj7gxgpTnIiuTLhEXMrTTABeA9Fkvg6ljpB eWO8dq9rlYSp5ge9vVt4ceqSp6RKndY= Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3667gOn1008074; Thu, 6 Jul 2023 07:45:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=F8ycTdDdQYcTUA9W4ZMinDPt78Egmnbx9cZ0zDL/uk0=; b=E7zR4qBbKQQ1CTStQ5KFZ/4gsQNyJ7bLOGEaeuG9GA2dmhvrLhQbCvB2hX1Eko/UNDWh AOWeD9wD+4gHWRut7zT02NfO20k3uhFZChc1Yib1OaeWrHeZhEGL60pvIygO9YPWAVWt ygH2tvcRbHzUE+XDVubCjmkKWqN/T9flBe2KXKThvhzGpa1TpwYq/6nNNNUtrsjfxHDy 5HtZO2NudY3hfbkSQzgv+6JzaxKkP+N3fTyhAKe2uo29De1zDbdDdjK3fJE5gYAJomq0 UUMtN4Za487Dr0rpfen5np95Kn7yg6dLRjvh8RZcONz0qIUFPl9MutkFKn7d3JqaXZjj /Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnsev826q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 07:45:07 +0000 Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3667gega009373; Thu, 6 Jul 2023 07:45:06 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnsev8244-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 07:45:06 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 365IoHeP001861; Thu, 6 Jul 2023 06:21:04 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([9.208.130.101]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3rjbs57x1h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:04 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3666L3mR459354 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 06:21:03 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5F60158057; Thu, 6 Jul 2023 06:21:03 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A39FF58059; Thu, 6 Jul 2023 06:21:01 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 06:21:01 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "Aneesh Kumar K.V" Subject: [PATCH v2 2/5] mm/mglru: Move Bloom filter code around Date: Thu, 6 Jul 2023 11:50:41 +0530 Message-ID: <20230706062044.816068-3-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> References: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: J1aRR6Txd7H886ubl0dAyxFM-GyDAsR4 X-Proofpoint-ORIG-GUID: oFYcoHYmPZ5LnT0Cj4vcr7YSyuE9S5P3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_04,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 suspectscore=0 mlxlogscore=999 impostorscore=0 adultscore=0 spamscore=0 lowpriorityscore=0 phishscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060065 X-Rspamd-Queue-Id: 8C5DC80013 X-Rspam-User: X-Stat-Signature: r8jdwdpi5ab3sbbyactbx33e4mtmr5t5 X-Rspamd-Server: rspam01 X-HE-Tag: 1688629509-148181 X-HE-Meta: U2FsdGVkX1+D3NXonYhM/0Y0qB/ZnPqosFBi9tOroljVWRaUQkEyNXwdFF3SBA0nBWZ3ve3TNzFZQOMiZKqvEfbsgYr5JZPlMqz2OfBKYpsjDnVa+qu7ZApfqCTjRy/1L4PdA8124hfRGGgbrQGWuHK2TIHiCuyKjDLVPLvNmHEGlbH3DQOI6pWGqAknu6wt2C4wAL2CFStJLuKOufZrIhxaAXKRj1V4BQ4RYNJ0aXG8vbGlSC4TTdgwW5GDyr1WGdB1YstG3buFIvTZr8nZ+HzltPIgvvZoWllaobYmZhyQ2XTXs2agneUEQR0ZdZyC8mMOKDVTmEpzFCd7OWiANz2VtgbWlC4BOPQ2oUbzm0T6cC1mxD4T9Cw5eBf1xY39liiMU4goRS3s9M24Qn83bKEK+XKWvT8jdIS5wSSCjFOl1QCU0DcI4ZBg2TOPIHEC6IuApUEBS5cxcLdMdf3cEhHAG3bSZ/EB0rQ9LFL4K9k+AJ9N2mBCqVyu1F2pkKIUE/7TOCzlYSnIYc3UdE69ToHE24rKRJdK445OXJERzH1DraJQpA/juVnfQUfAeD2Pf5lvO/GnDPz4zl/8/ARLafjaWBnVzyG4uSln3SX205ehVULSxp40VOuSY/u7ij43KpxMNqKQuKUQJTKoJ2ce+tcsulMSIcGY8vklE2R4SBB0tt6FaFxUjEqOCnL956vqOMr9jiN/mLRNeiEPRxDIhsrLYpUTblma6/K/YTNcB4JsYNxaweJhfmGFgkUu+AKESOBENHtF0THXiWy+qxjE8Zw9FTNfq6lLzhpHKjYQYnPopTNgi32+kcNL5gswzyBgp0bxlvF8b0fDetcgKR+QP1ZNxiUxgOMLUob9DdDOgYx4V9iFeJ37Quo+aPtr/9DVwy+fmfM8QNYKcxzS8CCxe5Gvgf4AyROOlCRVXQNR6bdyJC686Ss9cnJViFRcs3g2SQX8i/fHRFbYwQ6vO/S PjGYVz9j yonsOriYAlky98XCfQdcWAV4Gz0C0Ieq3ZFMSeg/VSaNHOs8V/qHBF+FioR6hGsnbacLVkDeZKQVMzjKAfDTxuo4xytWd7Rwn4HCXOcU7ODCfM0tiHf+UhAjVc/T1/U2zyZHhjOnx9swiCXuTdwRMcNpMmm91Fw209McbkoqIBRxQ1XTpgVxCcjzCthvDWWh+blMkMnaomqV0RzwjXrwyZGlNgqb1I/tumSyPG9XHKqTljeE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows to avoid building this code on powerpc architecture. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 462 ++++++++++++++++++++++++++-------------------------- 1 file changed, 231 insertions(+), 231 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b183f704d5d..c5fbc3babcd8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3325,6 +3325,237 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) get_nr_gens(lruvec, LRU_GEN_ANON) <= MAX_NR_GENS; } +/****************************************************************************** + * PID controller + ******************************************************************************/ + +/* + * A feedback loop based on Proportional-Integral-Derivative (PID) controller. + * + * The P term is refaulted/(evicted+protected) from a tier in the generation + * currently being evicted; the I term is the exponential moving average of the + * P term over the generations previously evicted, using the smoothing factor + * 1/2; the D term isn't supported. + * + * The setpoint (SP) is always the first tier of one type; the process variable + * (PV) is either any tier of the other type or any other tier of the same + * type. + * + * The error is the difference between the SP and the PV; the correction is to + * turn off protection when SP>PV or turn on protection when SPlrugen; + int hist = lru_hist_from_seq(lrugen->min_seq[type]); + + pos->refaulted = lrugen->avg_refaulted[type][tier] + + atomic_long_read(&lrugen->refaulted[hist][type][tier]); + pos->total = lrugen->avg_total[type][tier] + + atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + pos->total += lrugen->protected[hist][type][tier - 1]; + pos->gain = gain; +} + +static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) +{ + int hist, tier; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + bool clear = carryover ? NR_HIST_GENS == 1 : NR_HIST_GENS > 1; + unsigned long seq = carryover ? lrugen->min_seq[type] : lrugen->max_seq + 1; + + lockdep_assert_held(&lruvec->lru_lock); + + if (!carryover && !clear) + return; + + hist = lru_hist_from_seq(seq); + + for (tier = 0; tier < MAX_NR_TIERS; tier++) { + if (carryover) { + unsigned long sum; + + sum = lrugen->avg_refaulted[type][tier] + + atomic_long_read(&lrugen->refaulted[hist][type][tier]); + WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); + + sum = lrugen->avg_total[type][tier] + + atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + sum += lrugen->protected[hist][type][tier - 1]; + WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); + } + + if (clear) { + atomic_long_set(&lrugen->refaulted[hist][type][tier], 0); + atomic_long_set(&lrugen->evicted[hist][type][tier], 0); + if (tier) + WRITE_ONCE(lrugen->protected[hist][type][tier - 1], 0); + } + } +} + +static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) +{ + /* + * Return true if the PV has a limited number of refaults or a lower + * refaulted/total than the SP. + */ + return pv->refaulted < MIN_LRU_BATCH || + pv->refaulted * (sp->total + MIN_LRU_BATCH) * sp->gain <= + (sp->refaulted + 1) * pv->total * pv->gain; +} + +/****************************************************************************** + * the aging + ******************************************************************************/ + +/* protect pages accessed multiple times through file descriptors */ +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + int type = folio_is_file_lru(folio); + struct lru_gen_folio *lrugen = &lruvec->lrugen; + int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); + + do { + new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + /* folio_update_gen() has promoted this page? */ + if (new_gen >= 0 && new_gen != old_gen) + return new_gen; + + new_gen = (old_gen + 1) % MAX_NR_GENS; + + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); + new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF; + /* for folio_end_writeback() */ + if (reclaiming) + new_flags |= BIT(PG_reclaim); + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + lru_gen_update_size(lruvec, folio, old_gen, new_gen); + + return new_gen; +} + +static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long pfn = pte_pfn(pte); + + VM_WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end); + + if (!pte_present(pte) || is_zero_pfn(pfn)) + return -1; + + if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) + return -1; + + if (WARN_ON_ONCE(!pfn_valid(pfn))) + return -1; + + return pfn; +} + +static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, + struct pglist_data *pgdat, bool can_swap) +{ + struct folio *folio; + + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return NULL; + + folio = pfn_folio(pfn); + if (folio_nid(folio) != pgdat->node_id) + return NULL; + + if (folio_memcg_rcu(folio) != memcg) + return NULL; + + /* file VMAs can contain anon pages from COW */ + if (!folio_is_file_lru(folio) && !can_swap) + return NULL; + + return folio; +} + +/* promote pages accessed through page tables */ +static int folio_update_gen(struct folio *folio, int gen) +{ + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); + VM_WARN_ON_ONCE(!rcu_read_lock_held()); + + do { + /* lru_gen_del_folio() has isolated this page? */ + if (!(old_flags & LRU_GEN_MASK)) { + /* for shrink_folio_list() */ + new_flags = old_flags | BIT(PG_referenced); + continue; + } + + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); + new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; +} + +static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio, + int old_gen, int new_gen) +{ + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio); + int delta = folio_nr_pages(folio); + + VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS); + VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS); + + walk->batched++; + + walk->nr_pages[old_gen][type][zone] -= delta; + walk->nr_pages[new_gen][type][zone] += delta; +} + +static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk) +{ + int gen, type, zone; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + + walk->batched = 0; + + for_each_gen_type_zone(gen, type, zone) { + enum lru_list lru = type * LRU_INACTIVE_FILE; + int delta = walk->nr_pages[gen][type][zone]; + + if (!delta) + continue; + + walk->nr_pages[gen][type][zone] = 0; + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + lru += LRU_ACTIVE; + __update_lru_size(lruvec, lru, zone, delta); + } +} + /****************************************************************************** * Bloom filters ******************************************************************************/ @@ -3672,237 +3903,6 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) return success; } -/****************************************************************************** - * PID controller - ******************************************************************************/ - -/* - * A feedback loop based on Proportional-Integral-Derivative (PID) controller. - * - * The P term is refaulted/(evicted+protected) from a tier in the generation - * currently being evicted; the I term is the exponential moving average of the - * P term over the generations previously evicted, using the smoothing factor - * 1/2; the D term isn't supported. - * - * The setpoint (SP) is always the first tier of one type; the process variable - * (PV) is either any tier of the other type or any other tier of the same - * type. - * - * The error is the difference between the SP and the PV; the correction is to - * turn off protection when SP>PV or turn on protection when SPlrugen; - int hist = lru_hist_from_seq(lrugen->min_seq[type]); - - pos->refaulted = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - pos->total += lrugen->protected[hist][type][tier - 1]; - pos->gain = gain; -} - -static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) -{ - int hist, tier; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - bool clear = carryover ? NR_HIST_GENS == 1 : NR_HIST_GENS > 1; - unsigned long seq = carryover ? lrugen->min_seq[type] : lrugen->max_seq + 1; - - lockdep_assert_held(&lruvec->lru_lock); - - if (!carryover && !clear) - return; - - hist = lru_hist_from_seq(seq); - - for (tier = 0; tier < MAX_NR_TIERS; tier++) { - if (carryover) { - unsigned long sum; - - sum = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); - - sum = lrugen->avg_total[type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - sum += lrugen->protected[hist][type][tier - 1]; - WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); - } - - if (clear) { - atomic_long_set(&lrugen->refaulted[hist][type][tier], 0); - atomic_long_set(&lrugen->evicted[hist][type][tier], 0); - if (tier) - WRITE_ONCE(lrugen->protected[hist][type][tier - 1], 0); - } - } -} - -static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) -{ - /* - * Return true if the PV has a limited number of refaults or a lower - * refaulted/total than the SP. - */ - return pv->refaulted < MIN_LRU_BATCH || - pv->refaulted * (sp->total + MIN_LRU_BATCH) * sp->gain <= - (sp->refaulted + 1) * pv->total * pv->gain; -} - -/****************************************************************************** - * the aging - ******************************************************************************/ - -/* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) -{ - int type = folio_is_file_lru(folio); - struct lru_gen_folio *lrugen = &lruvec->lrugen; - int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); - unsigned long new_flags, old_flags = READ_ONCE(folio->flags); - - VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); - - do { - new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; - /* folio_update_gen() has promoted this page? */ - if (new_gen >= 0 && new_gen != old_gen) - return new_gen; - - new_gen = (old_gen + 1) % MAX_NR_GENS; - - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); - new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF; - /* for folio_end_writeback() */ - if (reclaiming) - new_flags |= BIT(PG_reclaim); - } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - - lru_gen_update_size(lruvec, folio, old_gen, new_gen); - - return new_gen; -} - -static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) -{ - unsigned long pfn = pte_pfn(pte); - - VM_WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end); - - if (!pte_present(pte) || is_zero_pfn(pfn)) - return -1; - - if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) - return -1; - - if (WARN_ON_ONCE(!pfn_valid(pfn))) - return -1; - - return pfn; -} - -static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, - struct pglist_data *pgdat, bool can_swap) -{ - struct folio *folio; - - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) - return NULL; - - folio = pfn_folio(pfn); - if (folio_nid(folio) != pgdat->node_id) - return NULL; - - if (folio_memcg_rcu(folio) != memcg) - return NULL; - - /* file VMAs can contain anon pages from COW */ - if (!folio_is_file_lru(folio) && !can_swap) - return NULL; - - return folio; -} - -/* promote pages accessed through page tables */ -static int folio_update_gen(struct folio *folio, int gen) -{ - unsigned long new_flags, old_flags = READ_ONCE(folio->flags); - - VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); - VM_WARN_ON_ONCE(!rcu_read_lock_held()); - - do { - /* lru_gen_del_folio() has isolated this page? */ - if (!(old_flags & LRU_GEN_MASK)) { - /* for shrink_folio_list() */ - new_flags = old_flags | BIT(PG_referenced); - continue; - } - - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); - new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; - } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - - return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; -} - -static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio, - int old_gen, int new_gen) -{ - int type = folio_is_file_lru(folio); - int zone = folio_zonenum(folio); - int delta = folio_nr_pages(folio); - - VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS); - VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS); - - walk->batched++; - - walk->nr_pages[old_gen][type][zone] -= delta; - walk->nr_pages[new_gen][type][zone] += delta; -} - -static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk) -{ - int gen, type, zone; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - - walk->batched = 0; - - for_each_gen_type_zone(gen, type, zone) { - enum lru_list lru = type * LRU_INACTIVE_FILE; - int delta = walk->nr_pages[gen][type][zone]; - - if (!delta) - continue; - - walk->nr_pages[gen][type][zone] = 0; - WRITE_ONCE(lrugen->nr_pages[gen][type][zone], - lrugen->nr_pages[gen][type][zone] + delta); - - if (lru_gen_is_active(lruvec, gen)) - lru += LRU_ACTIVE; - __update_lru_size(lruvec, lru, zone, delta); - } -} - static int should_skip_vma(unsigned long start, unsigned long end, struct mm_walk *args) { struct address_space *mapping; From patchwork Thu Jul 6 06:20:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78FCAEB64DD for ; Thu, 6 Jul 2023 06:21:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 174BC8D0002; Thu, 6 Jul 2023 02:21:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1268F8D0001; Thu, 6 Jul 2023 02:21:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F138B8D0002; Thu, 6 Jul 2023 02:21:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E212F8D0001 for ; Thu, 6 Jul 2023 02:21:12 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BF42614021A for ; Thu, 6 Jul 2023 06:21:12 +0000 (UTC) X-FDA: 80980189584.15.666E2BD Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf16.hostedemail.com (Postfix) with ESMTP id 5E51C180007 for ; Thu, 6 Jul 2023 06:21:10 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Kh8zGare; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf16.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688624470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/5Njrz3reCCviyX3u70/+8GUNgIGGwoM6m0K6lhTJsI=; b=kCYrA3IB1GuLEhtR1hOaKa6CzSKA4qsHV7R+oDWtU+cCeYMl0YvoDUMOMDFJdY+pcqOk7P DRLPtVxPZJN3iTCB9l4OhcuRV6HaSgvSqmSN2CUWcLmTsTMEL75cgJFyMrJRR5p91Wm9er BmJkdIV5fbBF//CfvMn4W6yIiDvWk1g= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Kh8zGare; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf16.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688624470; a=rsa-sha256; cv=none; b=WaTCLwnaGoVpBBxMOizRoFqPZIWL3QwfIxEaeirJaoDlpeghqFPJv5+PevCeUBmaP4eF6V tJqVI/+Zxyx4Kw6Wven+xsqO9q369qGDBtMj4CjJn/Pt5nM0zXeuyUnZ9lxv7pbTzoLkau e4VjmXmjsp+XxVemYEh8kDoCeLExUlk= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3666H4kU026326; Thu, 6 Jul 2023 06:21:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=/5Njrz3reCCviyX3u70/+8GUNgIGGwoM6m0K6lhTJsI=; b=Kh8zGare4EVaUwaDGNAygO6JSGvAk917ESBrKmqMsmFB7/ZN2xBXNXi79MueOSyTOxvz 0YJ8j3MdEmMh2dXrY+RizAlUvghRGhn7iXl/jjts99icZDREitF00mZukPUW8vizhsQs x1hadnBoqQEqfJdlnY33UAy/+0YZxBAbVj4ZKTVhpd8Efp23IQiky9kAn00+mP+wQd7J s6BLRRGjcKFTownahjJtTekGE7fcpUzIp5tRqT9hrVArZYOFuZbxJZ1ozmYVKjR2DU1y XO/srrF7SIB0dqypynjCaPCL3zUigVsFuiFZtTQy0BS1SZK7jwYGv1or2BVnIyq2+fZ8 Ug== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnr7602mq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:08 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3666I2R1030053; Thu, 6 Jul 2023 06:21:08 GMT Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnr7602mj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:08 +0000 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3665uM8H016351; Thu, 6 Jul 2023 06:21:07 GMT Received: from smtprelay04.wdc07v.mail.ibm.com ([9.208.129.114]) by ppma02dal.us.ibm.com (PPS) with ESMTPS id 3rjbs6m7dg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:07 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay04.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3666L5Ir36766126 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 06:21:06 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A360B58057; Thu, 6 Jul 2023 06:21:05 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E0F855805D; Thu, 6 Jul 2023 06:21:03 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 06:21:03 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "Aneesh Kumar K.V" Subject: [PATCH v2 3/5] mm/mglru: Move code around to make future patch easy Date: Thu, 6 Jul 2023 11:50:42 +0530 Message-ID: <20230706062044.816068-4-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> References: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: uqn0r-hKGKVY8LhcOw1NG3LiAQy6HHKN X-Proofpoint-GUID: BAU39SsRs1uhqzXvGSqqV2O1rdtSPy_L X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_02,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 adultscore=0 suspectscore=0 mlxscore=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 phishscore=0 lowpriorityscore=0 clxscore=1015 mlxlogscore=996 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060052 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5E51C180007 X-Stat-Signature: sbocmwnkruh1gqxff4opttti5qiyrsrk X-HE-Tag: 1688624470-387619 X-HE-Meta: U2FsdGVkX1+9h2QoI65hInOzaX6LKGxrOcnL5II0wzuTSuzu7kywLBRkoLfhqXc+mAkP37ODpWcz+uR9ciJo70BlOcYvH0dkym5/SjqGlD/Po6MU4VXVVwFxxPsIJjsMTLL0eEu3tIIozW6FKGEbPYqHsamh/q6AtgB8Wep5LqEb4EpNHJU1x8duPOdmspGwfykvE/aV3o3gxyo0hzafbwaZieEgF4Ysh6AqYyG7XtuAJXvzkmS2dy1+P8xjph0F57qxvrvcYdU8mJjicolSRL2z+WHHxstnAObwglMygbAPsUr/1WHKEKu/wwnKseQh93wIufLHmDtLHedq4hyzvCbz7c4cpyBaMrw8YUyIvJd5sYmvR+sTA7/tiIpaBAb7FMWqlkWxd+84Xq4mfXE1tUBnrE2jxmfuNGrl1QrIuBsN9vgNSUjiASlYhwS3Vheyt/wkAl/64heQ1ZunKJr2g2dT73yPu+MAzGANkgZWUU8Z5QK/qZuth4LbOjmT7JKRTLvzCyWeE8kuLyHQTnDziTpP/RRHyC6oAD3rUDxBhewX3XS31bhYW9tPfYQGGK3lwsZuNcaoqRgikHsMcgNqJ5XeW+RCKW9GEQqNULUlq7C1nSPik6WT04L96byDUzhAphR29mAtspe+uXJ5DusYHS7JTlUfTSlFJYBUQGhwsgFzrxl+u5kCCwZm98hHmzvpnLNQ1qr/FkEkSkoN7MfSd2dGir5a8eInOBvj7nNf3xfk8IQpUZAHsi0PWKqxIcoDfIJigjpA6v4iKUzZ1TxEjmflSti17JWyz27Xxpy/10Nk4wJO3/RU+igYktrVnQvaHGbq/txCqjrowkuC0xh5R0BltpAA6S+pIRoFfDOvFeF8Jislxn3CTR55Mc1nLs4th9CaknOOhResidl6luN2YA5LPbT7EJVJ81BJW/mLJ2Qx2Xh1I70yrNeHe+BTrFX73fYsLFnXOqZjYaN9jLw kbBiijI6 DJ64ecNff/RXM0UYtQHBDIK9lacO4dxqSqtmWqIOWnaafmH09DLnkRbMyW6PV24wjp08Ub3LB0zSJfI+ZO+r/Wn3SHBdVkQa8GI2t5i6Pn8NxQUiDmhEoB6JkiYAja6hQrBncSK6hunBExWiHY2wIEVsXtQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 64 ++++++++++++++++++++++++++--------------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c5fbc3babcd8..a846a62df0ba 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3556,6 +3556,38 @@ static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk } } +static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) +{ + struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; + + if (pgdat && current_is_kswapd()) { + VM_WARN_ON_ONCE(walk); + + walk = &pgdat->mm_walk; + } else if (!walk && force_alloc) { + VM_WARN_ON_ONCE(current_is_kswapd()); + + walk = kzalloc(sizeof(*walk), __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); + } + + current->reclaim_state->mm_walk = walk; + + return walk; +} + +static void clear_mm_walk(void) +{ + struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; + + VM_WARN_ON_ONCE(walk && memchr_inv(walk->nr_pages, 0, sizeof(walk->nr_pages))); + VM_WARN_ON_ONCE(walk && memchr_inv(walk->mm_stats, 0, sizeof(walk->mm_stats))); + + current->reclaim_state->mm_walk = NULL; + + if (!current_is_kswapd()) + kfree(walk); +} + /****************************************************************************** * Bloom filters ******************************************************************************/ @@ -4324,38 +4356,6 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ } while (err == -EAGAIN); } -static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) -{ - struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; - - if (pgdat && current_is_kswapd()) { - VM_WARN_ON_ONCE(walk); - - walk = &pgdat->mm_walk; - } else if (!walk && force_alloc) { - VM_WARN_ON_ONCE(current_is_kswapd()); - - walk = kzalloc(sizeof(*walk), __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); - } - - current->reclaim_state->mm_walk = walk; - - return walk; -} - -static void clear_mm_walk(void) -{ - struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; - - VM_WARN_ON_ONCE(walk && memchr_inv(walk->nr_pages, 0, sizeof(walk->nr_pages))); - VM_WARN_ON_ONCE(walk && memchr_inv(walk->mm_stats, 0, sizeof(walk->mm_stats))); - - current->reclaim_state->mm_walk = NULL; - - if (!current_is_kswapd()) - kfree(walk); -} - static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { int zone; From patchwork Thu Jul 6 06:20:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30805EB64DC for ; Thu, 6 Jul 2023 06:21:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B39FF8D0005; Thu, 6 Jul 2023 02:21:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9AA38D0001; Thu, 6 Jul 2023 02:21:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 914898D0005; Thu, 6 Jul 2023 02:21:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8391F8D0001 for ; Thu, 6 Jul 2023 02:21:15 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 56E5EC0496 for ; Thu, 6 Jul 2023 06:21:15 +0000 (UTC) X-FDA: 80980189710.18.A31B207 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf14.hostedemail.com (Postfix) with ESMTP id E7A0F100018 for ; Thu, 6 Jul 2023 06:21:12 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=U2q6ivyM; spf=pass (imf14.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688624473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yr1a0s2KumhZ0CFkrU7sUoWndUDGgUO1gGa/UV3sVPQ=; b=lwXBELUQjJXBD1kCDWnXbgebatDNGuwpugcF7x6rPnp7+dlyxJcloXF3v6vKVx9c7AyF1k QZ9yZv5fV4CHED7EFpKcsbxKCb6BjJNqGynl88oMktdBdIaqyundCjKWE7hn4vUQqAd/HS wLXn0WLxlVLa5v4sIP2tfrcuoIIJjuw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=U2q6ivyM; spf=pass (imf14.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688624473; a=rsa-sha256; cv=none; b=2aQg/R2FV+YawDZHVT6P2GGrKQzHw2lI/Ho5LlG7E1oAWnpRFik2GNI7YitAVh0oVuVvQ9 MKNQSlTIwHqbPh/iLNTn3GX64FgdY+2MzG1usgi90voLLd4ETfeoFKQHmOBi5WhZNydZ45 ELyF4qhXFLSI+1g7wOFbRkezozhftHc= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3666H6Gw030031; Thu, 6 Jul 2023 06:21:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=yr1a0s2KumhZ0CFkrU7sUoWndUDGgUO1gGa/UV3sVPQ=; b=U2q6ivyMyuPCUMy2h4h7dbE7/+zm2UKDjQ5b9d/hjnKWUdlzFiy/YNaY0hto+1Dbwuvf HXleaZK8oKdYJeotQJ+a9GT321Ri3yjjYCaX0QjhliPufV2X8138GRUK45AT0fSWVbo2 PFXraZY999aKKlJKkDg61awVXdAh34t0cbraRfzm/RALnHoByyO1jt7pnKBcUR5MDwcr m70IKIDuofZOhE3hcZNiY+14gogrrgjWMS0d03clY4dMOe6ibLyNJ9NgSI9cFQwWE4yC bqjua/v7pylE5JN4jT0Bihukft46eu+gKdY5e3gyg0j9EY3TD/U3Dw1BPZMAU7gexbL3 SA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnr7702em-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:11 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3666HeF8032010; Thu, 6 Jul 2023 06:21:10 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnr7702e6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:10 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3663S0pu023006; Thu, 6 Jul 2023 06:21:09 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([9.208.129.116]) by ppma04wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5qe6s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:09 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3666L8Gr63439340 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 06:21:08 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1EEE758058; Thu, 6 Jul 2023 06:21:08 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 439A758061; Thu, 6 Jul 2023 06:21:06 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 06:21:05 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "Aneesh Kumar K.V" Subject: [PATCH v2 4/5] mm/mglru: move iterate_mm_list_walk Helper Date: Thu, 6 Jul 2023 11:50:43 +0530 Message-ID: <20230706062044.816068-5-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> References: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 07TClkUMhIei1Ay7t6zSQTdS9UF-K6Yc X-Proofpoint-ORIG-GUID: 7Hnd3S7y7YS0MZ6vaSPyUqtla9Ghov18 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_02,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 malwarescore=0 spamscore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 priorityscore=1501 clxscore=1015 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060052 X-Rspamd-Queue-Id: E7A0F100018 X-Rspam-User: X-Stat-Signature: 1efzmcmr7e4a17aqmb3ie7k9qm43yazz X-Rspamd-Server: rspam01 X-HE-Tag: 1688624472-67123 X-HE-Meta: U2FsdGVkX19INP/DoOjDFG0ldItwBj9fCj/idaymkSAiMPxoyBeA/NAyAPCGqOAxmvNs3iL7r/7//5hsBsja8wz23DvaVCqeDYKowd2j6yG7aH8mApjxuylFbk8vE/SrYlhgtGGpgV7oBQmkuaL33jrxs/JjZ4lQemg01sr+iIOVvfX/9Yk5PZUhtWfoDUpYBC/XNNzEKBIzmnZDA6rqLDjcAm2FyIRizB0PgnWBHKxwemEfzZMAV24lyAlhrqhQXl9dwqGtxPO2NqzXOycevVL00EX5biF3vRJbw4vGsD2jfBm9ig1bExxDXKdQsAZFrM2yvlvDGucOd7IclMGfbcn8Omf1+qy22/rHlpuKtJb6t1SH6KaH7Dlub/+odUNFbLvT1V/VOkVtkNkdn4sbviZpkkIrMj4CBUf43acn8ttyOQzEsY/D8Jvuk3UwbT14tHk/RV/nfV6+3KW/0TORVz9AP2IMsYLhI6eC1JQSLESBscEx6iWcKdsravdFAJzc4moK39OyUNij04PUgXMnKB2fBiE5hsUNCARZ5aLvu+yE0ySdNHAVxkL23NEZ2j8axe5ju6AA/yiD+pBK+leUR5IZ8gtBmffH63ggFL2VwdQWvPnrRW446BC6h5h/GiDGZhHNTWcqve5A8r3yjyCAA96DjPiMfLdwIwIbMDPY5dBn5DiTkDvP2pgSpp62eSBBSMozmyo7CYEXD7hYiyPQrHGefwepyZydgPY2d8ueuVhHuVLHt5r1NDeNzQh3Ae6PBEnQCbl1t7BTe5yjyynlMe5n5krXrKdA5ZmWgGQ8lGt//ZGu8CfiEqi9eFkH2YPpNy0jNlPVG5rfXpMmljqa88CUGNz92wmB/VtKySl27Dzkp/ePuWPVQDzScIdZBl8Gtxu0l88CP879j1gfXBhyOIVOr9ougG0KRSWlPIEjk3ea7j6RlhSMpboNDsL56f6TgbZFbnFBFYlQTOCpFhC fjUUI7Oc 6lv35CunWR5F1VMStR0FvNZJTKu58st1gy8bAh1Go1pKnWaJCHdY1fxDAQonCW6Zq7MMiS+jDWkwzNadKmZx6TZ3xlCgaE3RSthkCKm+l4RO1kmqebz0FL8m29/Cu25SYZjUluIkjw/KcsVtGdECbhPnoSVU1BZJ+UETsjBICw7Bp2ER5DqI+J1hlSA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helps to avoid building this code on powerpc. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 54 ++++++++++++++++++++++++++--------------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a846a62df0ba..0ea7a07990d3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4356,6 +4356,33 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ } while (err == -EAGAIN); } +static bool iterate_mm_list_walk(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + bool success; + struct mm_struct *mm = NULL; + struct lru_gen_mm_walk *walk; + + walk = set_mm_walk(NULL, true); + if (!walk) { + success = iterate_mm_list_nowalk(lruvec, max_seq); + return success; + } + + walk->lruvec = lruvec; + walk->max_seq = max_seq; + walk->can_swap = can_swap; + walk->force_scan = force_scan; + + do { + success = iterate_mm_list(lruvec, walk, &mm); + if (mm) + walk_mm(lruvec, mm, walk); + } while (mm); + + return success; +} + static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { int zone; @@ -4491,33 +4518,6 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) spin_unlock_irq(&lruvec->lru_lock); } -static bool iterate_mm_list_walk(struct lruvec *lruvec, unsigned long max_seq, - bool can_swap, bool force_scan) -{ - bool success; - struct mm_struct *mm = NULL; - struct lru_gen_mm_walk *walk; - - walk = set_mm_walk(NULL, true); - if (!walk) { - success = iterate_mm_list_nowalk(lruvec, max_seq); - return success; - } - - walk->lruvec = lruvec; - walk->max_seq = max_seq; - walk->can_swap = can_swap; - walk->force_scan = force_scan; - - do { - success = iterate_mm_list(lruvec, walk, &mm); - if (mm) - walk_mm(lruvec, mm, walk); - } while (mm); - - return success; -} - static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, bool can_swap, bool force_scan) { From patchwork Thu Jul 6 06:20:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFAEBEB64D9 for ; Thu, 6 Jul 2023 06:34:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F6AF8D0002; Thu, 6 Jul 2023 02:34:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A6988D0001; Thu, 6 Jul 2023 02:34:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 147038D0002; Thu, 6 Jul 2023 02:34:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 04DB08D0001 for ; Thu, 6 Jul 2023 02:34:45 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CA2BCA04C2 for ; Thu, 6 Jul 2023 06:34:44 +0000 (UTC) X-FDA: 80980223688.04.F8F693A Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf17.hostedemail.com (Postfix) with ESMTP id 7087E4001A for ; Thu, 6 Jul 2023 06:34:42 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="WvrgBd/l"; spf=pass (imf17.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688625282; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O4+jM1spGHwKV33TRjA5vsyhnQJCsg8YUFObqJ/ZjK8=; b=zhu1M4nxku/2iAfX4W7jkXJnhLWa3REyBIB/54LogAiFGPq9VRVj7u1efmSZy6GjmYfdQq QTFXSMKYJxxrUfBttvpHRQ4xubnvo2I+7aIgrlEPVr1C2M+3I2vTLX23imur7xGqZqDbg8 pxDaRw/R79RGWx2oLU2/ODsqZfRb/wM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="WvrgBd/l"; spf=pass (imf17.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688625282; a=rsa-sha256; cv=none; b=hSDmtrbHQLZ59op1bUwIqiEELwJQZN7XSqpqd6rPx9PuCXzsCTyepvNLfyMIgB4Dnnamq9 xwRozJI7RNPPen7U5GilYlG14bhNCDXY+akVYsdf4SzXWNFQSs2MEfwYJcAUvtECFD1MJM gucGui+N6t8mC/ni4MdpIYaSp3l+Vj0= Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3666H6mF005599; Thu, 6 Jul 2023 06:34:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=O4+jM1spGHwKV33TRjA5vsyhnQJCsg8YUFObqJ/ZjK8=; b=WvrgBd/l/8uJg7AR5Oh3Q8XTwyNKokUY7W7GvLC0PQQgQTXX0Za/34MPnIOPuKZhJvMd rw4KSKgmW4A0xJu8VxfQcGPg3Al2gMO0l6+aTcz9cCaf9ekN4lN0g6QDL8NYC8Y2lrKs BR1ldcVurutU9d6esAfNrugW99wBhEavi/gtS5JgR2/ZyarhvpcP82v/HF0MhruQPn86 b8KbewC3yWvV4m5F/0ijTD6RMuerWlqJ2C0Z0+pvuMdsO8ac4CIMR3mtq+P5PtFapJ+H FzwV4t1mR8YbjNyQ1GK2UklcNiOFK0i15emehSU2h/c7zayZgNK8HezgK67cvMFOPVmy XA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnr760eg5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:34:40 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3666IeiG010570; Thu, 6 Jul 2023 06:34:40 GMT Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnr760ec6-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:34:40 +0000 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3663ZObd014742; Thu, 6 Jul 2023 06:21:11 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([9.208.130.98]) by ppma02wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs67d6a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:11 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3666LAUm66191680 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 06:21:10 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7502D58058; Thu, 6 Jul 2023 06:21:10 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A524F58057; Thu, 6 Jul 2023 06:21:08 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 06:21:08 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "Aneesh Kumar K.V" Subject: [PATCH v2 5/5] mm/mglru: Don't build multi-gen LRU page table walk code on architecture not supported Date: Thu, 6 Jul 2023 11:50:44 +0530 Message-ID: <20230706062044.816068-6-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> References: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: e3MwpzB-0JAxJh8IEL0n1y7zN9nf_iY9 X-Proofpoint-GUID: sAu1eMb-MgyLFxfh6ONDSDyHGXayuuOx X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_03,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 lowpriorityscore=0 phishscore=0 priorityscore=1501 adultscore=0 mlxlogscore=887 malwarescore=0 bulkscore=0 spamscore=0 suspectscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060057 X-Rspamd-Queue-Id: 7087E4001A X-Rspam-User: X-Stat-Signature: 5hk4ww3reyqcsewnss8bojhkydebej3o X-Rspamd-Server: rspam01 X-HE-Tag: 1688625282-762355 X-HE-Meta: U2FsdGVkX194k8pbzZxs2n+wLjYeHaEk1kxue3LdZt9Ryy8oU64yd0Sl1e2zIALSitWltuIgSNLktLsjnbHaud2DwnFxHLoKybAut9oBGEbGgmXKZtxPWH/eNBLuNh7eDCXyqekiLk31pyugcEKwvtWEfc7KkUrNFch5yMWBQtYPY2h6o0FNpZAlKdBa89SqkORV9IdWKVBgPxpDtgXDEZHjbk3gpTUqqCuv4y8rZxfECT5CANRNrbFwQNiahpEtcXFfRoQbkC6r2s8t1+HNBMc0PJq/wlmSKvTP+pd0FqujGENb02+I1CPrzneLT8l1UWce9nZAAkGTOxiPfaa2PTNFMQj80JdlamWgyLg1Oso4GKfegkMwTvM31752F6Y8BN3lRwThNcDCa6YmfpktA08YQvzPX11dKmrshJS8WtXFmPu1xv9FhcUyxTVt6203GCfiVdvvFdBBmxYlSnZuzTGCgNBm1Md5kfZBJ8uaU98hQQOUIsYG1muy8pqdNWJEvdfkUM+UCnPorhrZpNCrwCEIDhqPQ/kfDu93BqSQLDgxPU70vFzqF+NXMIcssGkLrel10zz8+Wpx0Z8BxahtlrTOsH7nGPOVX7ND9TM+t5bxGxWPekPzsr5z8NZc4+c03IQ2Mmsjz66ErCJHcFnCs8IWI9WvK9R9ModfgjKWQ2gEVIxFgmhqhxd8GnpJunuB3sMN1wRhfptllYd4D2rPy9v0EkaMwIW3iTk6LCoaj/a5efT67Hrc62wG+dmtJoaTLnKIH2EHo8zhjjguHHG6xetxEBlX04x/glcAy5q4lQ1Co9E5qGWu3fAlUlaBBdUHsrUETMkUchI+C6rRO6OAsUa2evnckSxNEAyaRrbPo1FH+9nduardsFwmFZRjN8KkoZA1PUHsVtHSoHL8zjHVKaYt9acWuhTszXdlnhxvINRC1octAJo+fON9H/G+IR+m6NTsrkeXi83qyulvB0+ funbeiM3 8lTmSFPBrxDdNZccl6ah6AEgS+kzYatR6IK8hymL2dE/pQpvrKZa2L/lE6RHzMiXd+TS05YN/yL5iAS0n4OKf/Uym9LqyFtaSlDH1S9Nhnrk45BpkmaiPlPKH6Se0kWKoqgbNIPZaUXGTnqU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use a page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture. This avoid calling lru_gen related code (lru_gen_add/remove/migrate_mm) in fork/exit/context switch Also we don't build different components like Bloom filter and all the page table walk code (walk_mm and related code) on not supported architecture with this change. No preformance change observed with mongodb ycsb test: Patch details Throughput(Ops/sec) without patch 91252 With patch 91488 Without patch: $ size mm/vmscan.o text data bss dec hex filename 116016 36857 40 152913 25551 mm/vmscan.o With patch $ size mm/vmscan.o text data bss dec hex filename 112864 36437 40 149341 2475d mm/vmscan.o Signed-off-by: Aneesh Kumar K.V --- arch/Kconfig | 3 +++ arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/memcontrol.h | 2 +- include/linux/mm_types.h | 10 ++++----- include/linux/mmzone.h | 12 +++++++++- kernel/fork.c | 2 +- mm/memcontrol.c | 2 +- mm/vmscan.c | 45 ++++++++++++++++++++++++++++++++++++++ 9 files changed, 69 insertions(+), 9 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index aff2746c8af2..ec8662e2f3cb 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1471,6 +1471,9 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config LRU_TASK_PAGE_AGING + bool + config ARCH_HAS_NONLEAF_PMD_YOUNG bool help diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7856c3a3e35a..d6b5d1647baa 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -233,6 +233,7 @@ config ARM64 select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN select LOCK_MM_AND_FIND_VMA + select LRU_TASK_PAGE_AGING if LRU_GEN select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE select NEED_SG_DMA_LENGTH diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7422db409770..940d86a0a566 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -281,6 +281,7 @@ config X86 select HOTPLUG_SPLIT_STARTUP if SMP && X86_32 select IRQ_FORCED_THREADING select LOCK_MM_AND_FIND_VMA + select LRU_TASK_PAGE_AGING if LRU_GEN select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK select NEED_SG_DMA_LENGTH diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0ab426a5696b..5ddc1abe95ae 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -321,7 +321,7 @@ struct mem_cgroup { struct deferred_split deferred_split_queue; #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-memcg mm_struct list */ struct lru_gen_mm_list mm_list; #endif diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index de10fc797c8e..9089762aa8e2 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -793,7 +793,7 @@ struct mm_struct { */ unsigned long ksm_rmap_items; #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct { /* this mm_struct is on lru_gen_mm_list */ struct list_head list; @@ -808,7 +808,7 @@ struct mm_struct { struct mem_cgroup *memcg; #endif } lru_gen; -#endif /* CONFIG_LRU_GEN */ +#endif /* CONFIG_LRU_TASK_PAGE_AGING */ } __randomize_layout; /* @@ -837,7 +837,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) return (struct cpumask *)&mm->cpu_bitmap; } -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct lru_gen_mm_list { /* mm_struct list for page table walkers */ @@ -871,7 +871,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm) WRITE_ONCE(mm->lru_gen.bitmap, -1); } -#else /* !CONFIG_LRU_GEN */ +#else /* !CONFIG_LRU_TASK_PAGE_AGING */ static inline void lru_gen_add_mm(struct mm_struct *mm) { @@ -895,7 +895,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm) { } -#endif /* CONFIG_LRU_GEN */ +#endif /* CONFIG_LRU_TASK_PAGE_AGING */ struct vma_iterator { struct ma_state mas; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5e50b78d58ea..5300696d7c2c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -461,6 +461,7 @@ enum { struct lru_gen_mm_state { /* set to max_seq after each iteration */ unsigned long seq; +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* where the current iteration continues after */ struct list_head *head; /* where the last iteration ended before */ @@ -469,6 +470,11 @@ struct lru_gen_mm_state { unsigned long *filters[NR_BLOOM_FILTERS]; /* the mm stats for debugging */ unsigned long stats[NR_HIST_GENS][NR_MM_STATS]; +#else + /* protect the seq update above */ + /* May be we can use lruvec->lock? */ + spinlock_t lock; +#endif }; struct lru_gen_mm_walk { @@ -546,9 +552,13 @@ struct lru_gen_memcg { }; void lru_gen_init_pgdat(struct pglist_data *pgdat); - +#ifdef CONFIG_LRU_TASK_PAGE_AGING void lru_gen_init_memcg(struct mem_cgroup *memcg); void lru_gen_exit_memcg(struct mem_cgroup *memcg); +#else +static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) {} +static inline void lru_gen_exit_memcg(struct mem_cgroup *memcg) {} +#endif void lru_gen_online_memcg(struct mem_cgroup *memcg); void lru_gen_offline_memcg(struct mem_cgroup *memcg); void lru_gen_release_memcg(struct mem_cgroup *memcg); diff --git a/kernel/fork.c b/kernel/fork.c index b85814e614a5..c7e8f65a72c8 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2932,7 +2932,7 @@ pid_t kernel_clone(struct kernel_clone_args *args) get_task_struct(p); } - if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) { + if (IS_ENABLED(CONFIG_LRU_TASK_PAGE_AGING) && !(clone_flags & CLONE_VM)) { /* lock the task to synchronize with memcg migration */ task_lock(p); lru_gen_add_mm(p->mm); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 92898e99e8a5..cdcf1b6baf3e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6357,7 +6357,7 @@ static void mem_cgroup_move_task(void) } #endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING static void mem_cgroup_attach(struct cgroup_taskset *tset) { struct task_struct *task; diff --git a/mm/vmscan.c b/mm/vmscan.c index 0ea7a07990d3..3c9f24d8a4a6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3244,10 +3244,17 @@ DEFINE_STATIC_KEY_ARRAY_FALSE(lru_gen_caps, NR_LRU_GEN_CAPS); #define get_cap(cap) static_branch_unlikely(&lru_gen_caps[cap]) #endif +#ifdef CONFIG_LRU_TASK_PAGE_AGING static bool should_walk_mmu(void) { return arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK); } +#else +static bool should_walk_mmu(void) +{ + return false; +} +#endif static bool should_clear_pmd_young(void) { @@ -3588,6 +3595,8 @@ static void clear_mm_walk(void) kfree(walk); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING + /****************************************************************************** * Bloom filters ******************************************************************************/ @@ -4382,6 +4391,33 @@ static bool iterate_mm_list_walk(struct lruvec *lruvec, unsigned long max_seq, return success; } +#else + +static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) +{ + bool success = false; + struct lru_gen_mm_state *mm_state = &lruvec->mm_state; + + spin_lock(&mm_state->lock); + + VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq); + + if (max_seq > mm_state->seq) { + WRITE_ONCE(mm_state->seq, mm_state->seq + 1); + success = true; + } + + spin_unlock(&mm_state->lock); + + return success; +} + +static bool iterate_mm_list_walk(struct lruvec *lruvec, unsigned long max_seq, + bool can_swap, bool force_scan) +{ + return false; +} +#endif static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { @@ -4744,9 +4780,11 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) arch_leave_lazy_mmu_mode(); mem_cgroup_unlock_pages(); +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* feedback from rmap walkers to page table walkers */ if (suitable_to_scan(i, young)) update_bloom_filter(lruvec, max_seq, pvmw->pmd); +#endif } /****************************************************************************** @@ -5896,6 +5934,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, seq_putc(m, '\n'); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING seq_puts(m, " "); for (i = 0; i < NR_MM_STATS; i++) { const char *s = " "; @@ -5912,6 +5951,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, seq_printf(m, " %10lu%c", n, s[i]); } seq_putc(m, '\n'); +#endif } /* see Documentation/admin-guide/mm/multigen_lru.rst for details */ @@ -6186,6 +6226,9 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) INIT_LIST_HEAD(&lrugen->folios[gen][type][zone]); lruvec->mm_state.seq = MIN_NR_GENS; +#ifndef CONFIG_LRU_TASK_PAGE_AGING + spin_lock_init(&lruvec->mm_state.lock); +#endif } #ifdef CONFIG_MEMCG @@ -6202,6 +6245,7 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat) } } +#ifdef CONFIG_LRU_TASK_PAGE_AGING void lru_gen_init_memcg(struct mem_cgroup *memcg) { INIT_LIST_HEAD(&memcg->mm_list.fifo); @@ -6229,6 +6273,7 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg) } } } +#endif #endif /* CONFIG_MEMCG */