From patchwork Wed Oct 18 08:20:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huan Yang X-Patchwork-Id: 13426606 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3283C41513 for ; Wed, 18 Oct 2023 08:22:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 378DA8D0143; Wed, 18 Oct 2023 04:22:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 327A98D0016; Wed, 18 Oct 2023 04:22:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A28C8D0143; Wed, 18 Oct 2023 04:22:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0BDD68D0016 for ; Wed, 18 Oct 2023 04:22:14 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BAD76141299 for ; Wed, 18 Oct 2023 08:22:13 +0000 (UTC) X-FDA: 81357889746.09.3F29AA0 Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2137.outbound.protection.outlook.com [40.107.215.137]) by imf27.hostedemail.com (Postfix) with ESMTP id BDC1140008 for ; Wed, 18 Oct 2023 08:22:10 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=laGlQ26D; spf=pass (imf27.hostedemail.com: domain of link@vivo.com designates 40.107.215.137 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697617331; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y/eT6KVcewOpLb+WlLYKCO09cMKS2vKtu5gikp+mGh4=; b=KlqS1EIG8i+IPe6V+RmTuMXxU1xZS7dLO1Z8vu3dF4Agmm1N3RSK0nWcVwL7nzKs96ZU1V 6TQedj2UYxVxgCJG37i8/zc52N0eieqaKpY0xngeu5yU7Ug8Z4CZjz3ARS/KJT7l2ZNWwd dazLR0o8hX8+2XVdgtqycTSfVto8up8= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1697617331; a=rsa-sha256; cv=pass; b=VXKmf61cdqzv5fiuyZxsvHyVKxr22eoiIyGXyLQuQt8FtTlfk0rZRKMPfOkeal95GnAAFK Vz0BlHKZKj7FDJJ8nKz7LvZzZM0KKsEeDQyDwfVwSmDvWABl6qmGs7voTfmr2StMxtJdZ0 JBm2HHnuxQZh9zEakIriPwUO2yp1nXo= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=laGlQ26D; spf=pass (imf27.hostedemail.com: domain of link@vivo.com designates 40.107.215.137 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aZBj635fOfB3pvM3VEC6KgU22Nepzixrvhqb8A3RL79ZvXV0G9l1WsS9a8r2VR7PJO+essHhfH81x4dgKjneJ2lYZuryuOZeJMYKupPg+xEkCgFtSa7g1OBG9QvtzR6A+3ueGLLtj/+gm+HveEO9wVxe8ZDuUDBlHLFgXOh/NxoFMZyvMR1IyPDf/0muu0V0FOAHRHfwlWshs6CvZ6pvfMVmzDBcg+jJSyyLuF6a5Ro4PtzAydlNjWR45KX1v4M7rSRhb3vN4WK1PFVPuIMHAh9wcDcMU6E9XNNUJ1nIdI/05pZNq+0O51GdlYCgUZzMmMueuiJ6hzC1LGRu5PB/jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y/eT6KVcewOpLb+WlLYKCO09cMKS2vKtu5gikp+mGh4=; b=bYK+7Y3dUAYtljh1FeyJHbk+sq2EK4rRE4GILFPPNIHErDhESASMSAp2FUWsAm8SC0DzkfC+aBfwLFGME/Wd4lI2cIeeW9YA+ejCiHOUT+e11FNCRb7sT48mfryiODqcOeO8KXhxSajMfc93R4XcCTn/aHgyE2uSm0mohGxQbvTtf4nWaN5OIaNZK+LBNFNGsu9ftYdQuC3fO8bGx2G7j12aChf6njQDxgKjsAKPo9fUWmMsz2c8d3v5iQn59dnAj/rGvDqP9CjxTU3LHagK1Wh3koGKI+SHSk+hT07XEL3FhGqEnWYsZfljikrlVWn2LFM83+C53ExdLGxD3lmzEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y/eT6KVcewOpLb+WlLYKCO09cMKS2vKtu5gikp+mGh4=; b=laGlQ26Db/3Oo5Q6rbxOh0QBdnQzY20bHviAclUVHZzVPRfGbudg0GLr4YhGyNjuG+ZuxpR+/dNvPW7DScsNDEtkZ1+txNEzRyhZWYnMzeBEs5ldkkC4BKXWvwUdKeix556+2+rJVLS9/yMg0R/gQIYGkFroHjh4R9zJXzhPQxyAkm0q4ihErFZWkYsmBxOOoEnObPCa9UuyGekuY+7iyukqd6zVQ4WI/PK94LzfAM7+XdEQXKA3I2Mtl9fuigy0xBuLBLeNL6hPzFdxsCW4FGzXAhY/UEeP2PwIsfB97zNoYPDTwWWOX0TLBCS4P6wiPcnngn9zsOX8SlChvQ76Bw== Received: from PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) by PSAPR06MB4391.apcprd06.prod.outlook.com (2603:1096:301:82::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.22; Wed, 18 Oct 2023 08:22:02 +0000 Received: from PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::40ac:5701:4617:f503]) by PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::40ac:5701:4617:f503%4]) with mapi id 15.20.6907.021; Wed, 18 Oct 2023 08:22:02 +0000 From: Huan Yang To: Yu Zhao , Steven Rostedt , Masami Hiramatsu , Andrew Morton , Suren Baghdasaryan , Huan Yang , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org Cc: opensource.kernel@vivo.com Subject: [PATCH 1/2] tracing: mm: multigen-lru: fix mglru trace Date: Wed, 18 Oct 2023 16:20:59 +0800 Message-Id: <20231018082104.3918770-2-link@vivo.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018082104.3918770-1-link@vivo.com> References: <20231018082104.3918770-1-link@vivo.com> X-ClientProxiedBy: SG2PR04CA0185.apcprd04.prod.outlook.com (2603:1096:4:14::23) To PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PUZPR06MB5676:EE_|PSAPR06MB4391:EE_ X-MS-Office365-Filtering-Correlation-Id: 4bee04ef-c9e2-4559-d701-08dbcfb34e78 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: o+3FWqGIOzyLZVowgOupzkXO8oAJTlvnqz5bWKMj8Kg9qBCCIn9nri7dYvSMLwPLEmb3VXMk7xNDfnnLhtIN11Amf6/O0dQGcprcv3FiaC+UQbwHEzTHEQFlk9icw362J4Ne57g1mJ9MGqSiQ+EmgRYbbp5vb/m+Gl0T+CddAypBMgBF75/LpMMAAT7dKlZ1GNZVNB+jGORAoir9RiQUGNGULtkwgCIsxmjw5futbd5eG6Xe9RRDWKYbvOfhCUjV8Qz+ruTIgYYXsRrjl9n/t6IL2CYKXARBbcDWewHdK7BRvyFyyZiwUvTClTWPtKcMYPsjBDjrgIhQhT4briucqPDWJERpzprUEa+gQs4cdU/GSbDB5K8MhioC4XhR5WxJU6+4GvdE31xUd1FXVOf5SGo655RVLVOfjx0CcL8zSaSt0ixvRyfAueWHRkqYINwMUP/0tZ8rtNHwF51VlL31WY9FVyEDRHnh15F55IAMJa9g4wJkPbztvtjgnBinvZcXYTtcGfk6ZNjiB477y4eIQUQbE331aH6uRj1XDkgfQOb3dQ9hm80PSt895ah9taFV39bqgFyCYIMYKRuk+e9mxR3wquhb6QiS4s1wewSOQOwHFpt2UERmeNioKFy44Czo9KQcP/QLhw+z7LgF7pq6nA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PUZPR06MB5676.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(136003)(366004)(396003)(346002)(39850400004)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(7416002)(4326008)(8676002)(8936002)(5660300002)(2906002)(86362001)(36756003)(41300700001)(38100700002)(6506007)(83380400001)(6666004)(52116002)(6512007)(6486002)(107886003)(316002)(478600001)(1076003)(26005)(921005)(110136005)(38350700005)(66476007)(2616005)(66946007)(66556008);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TdHSSxkHxMNJr/kQEBhZL5/r02/l0KTt7Xtv0guRMyGaYBvEhJSv7xUDtbJplMwYUNYyUKHZUEK+4sluBapXAKE5DiKk06h2drfXLAvheXUUJqYNG74twByapol7v9A5LfwWtTtNqCB21gXUQBxfqnVZustVzR0wNHRdSAGB1hUVLOKSwixjnENNyCknfI6mjQGxJ9MtGJdRbp/d9Qz2K4sXQo0Em/AFFFD2L0q5TGcTY0cYYVWcxlwVvrtWGLuRtbEU+an5Xc/IB6kcReV4rRq4q001gum8YLQnG1o1yA9+emTFUGT+8cY7y4l8k8C/rd5kJH1sc03WZ9WgKb2gAqZY68Z7cdsH0aucogLHkoX3TF1epEG5GdBGqZI1qlTAUEuajIx19Yz/VCGNAmAp6rDMIL4fAR6EE6ELUcN2xIZ8xkvRFAQzijyuCZlb8iXhn2Q5LoTEteiGNvQB+Rxi4Kyh6UiqWf56ZKcnXcE8HX3+qnSKDJPI45mhUeh+//itmJDUQxAWKJbmCAbmkWa6wHjtwjbHc/r7EvbuqJDkJxdxR2A9GUirGZmxQ2X9QALthRYyLvBrr2mXc+ozq2jae83JsgpEHrsNQ1fdSXhuxEoi9ebUw1UPyLiqusJYuEV8LZGwnDh9C0OUJUmL+1+K/nmjt+K3C6YV3IXmQOv9cq3lByyhDDPD3ljfkZXGiJ7jLha/6JwrbaDmZGdHGE00ejYCACQP3L2/eWm+l0QyP3UCRtQO0On4L2GAvB/gdfo5s8re6OJddDARRj5xraVxjXQGD4MThMVju5RpkKJGDTocIH/Ew4q1JzwUo/CZYs/22/QB6x9Iu7GLrT4OwS+BR6DFR9iGjx8q1U4HI+/pjnOGBUMgcVz0t1ApIeIX6HS/ZJZmttF4TZcOFe3sjcMcaQNaKr86+vdq9Vsmf+cyX3WhftspZCJtFp3yod7qgy35o2cCN88jcZVaUiX7F2XcJYC2nljFppkd6FdzIbutcYRU0VuqJODdV1xXRx1ZRMOlCdWedMDS8c7vydW8SZDcwsitHnj2wnIehIerdXrdaoH652MUIpByyNQZjc7jhhdZEyeiYXW1JK+njHrhUd9HWK/4i0HeGCOQCxzQmEQf4PypBewb6t7mRsr6pg9ONiFk3ZlXp+903XugciYbZq6ZlWHp/oeTg3AE5mqP5TF9Z7h9hx8RCb7GATVuKwKP+05Rk78XX89DBDsyOvnBEdWSVjJ8eIsECvBBXKUsPG9uAy5AZZdvjLRWevJp9XKpueGt4l0L50w5EaNaBPZfwJdSbI/ecTVnB4jTKQUaxVkBOta86fCFWwK+GfMUCZJUeu9Agv8frzHz8ROyarJjUtlr1rqN1RdQb+hDbslWXHuDd/93FGqVM5njihyzP5d4jn1Up4SKD5RmzoHNpK9TUS2KFwk4yK77+8soqbZoGk3jAq9KNVkriluvCyPbJReTzAo42hOv/+0sCjMHa3KbnWZ1bwnzmEp1tVezbWcT5fp/jcNlADCMcQhZHuOuHrGiOIth4gUmenrahiRozAW9N8ezaha2HoOtNScDrDEh1N5hwc7rNBqziWOtrBtJR6u4sysL X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4bee04ef-c9e2-4559-d701-08dbcfb34e78 X-MS-Exchange-CrossTenant-AuthSource: PUZPR06MB5676.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 08:22:02.6948 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u/VOtR9oXkT+G4/qSHOzkBvjHozfJmpIXojLB3fT2UVNgTUjzao2ujf/7vYGkiULOU1MmkYA919eAS5GH9WU3A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSAPR06MB4391 X-Rspamd-Queue-Id: BDC1140008 X-Rspam-User: X-Stat-Signature: 9mak5tdxb4x7zbkqruhpr8rgzaiwbn5r X-Rspamd-Server: rspam03 X-HE-Tag: 1697617330-853344 X-HE-Meta: U2FsdGVkX18RERD9iX1ejdM8WhyxW8euihCa3hka+gfmozB6y7cfH+5a60WzikeCiExA+CD/Iw5W327Ti/UjORgzMciZzsVAZ5X6cPJqVG1b3RutG+M/8HtihS+B+HDZI36lveb3oyzT5aI1UP0E8p5pHU9Tn7HK0KbmXrUAoYf9/15VWRmx7hkfxwlZd8UAZSkOqJCjXXO4xIqmdO7OZW+Gq7R+ubRDkdYeMxXy+DXkRuny2+uu7fgGPK4vPVfA0u1Z7aYb42vG4ymjSkghKWtAG38OO0UfSf9VRJAPEyeJxgOfDvNI3cLoV7iv0cD7G0I/bCt43gySs3cCejpb4OzmKIPQ/6iS49+yzkkbbAKvVUO+XNdLn6XqQmTV452dSo9vd/uGY110yO63iq2qcr0S/cAx/MjmSWo+GoQWD2K1ndDsCznRAK+V0xlPb3FW6jogHfKKvuL6YMcbS4gq1yIAPEp2t24kZBNkEaMlWbRfXXGXEV9Zb6TyUs/RWE02MgrtSR1iZyNm3GNRggU4q+cbN39ChBFC52F3OdR92tyYZGR+Q7Ct9dsxU6gtZuwbnOzi8MZyDUaXIYDR6eKpnpzGG1qVVRuhFthTA56UbNJpwIzmjzqBFN2AsVmuQBipewjPWKFFRou4Fsc3Dj1qOW3kPO4m9X3BbV5orLiAKOH/DdVBOnmSPGlAPdPVYhG8FbMDSa3H5PDcvc8M5tYssMAli9X+P03I9eqBrxzrrOhg56SDivap+SyPL/yTkMGrGhamTvAiOdC8O59ET3p5OiqgvI1asXFbUXJ4lmdMFmfvm8YJeZFLG+MRLEK5gNXh1A0p+vCejjIyfjrB4lQpCKmWR8htjmwuaHK9sWz97B1TsU5uf1X0ooLXAEo5vCQNittow1av2gci3sTKu/E/bOmnyNYli080/ClAQkuOIYwugVeOBlQX6hWCwhKnNqMXNb3YJtp6yu30X3r9EMk i41GMALW SsqnBBGQBZWG60ETESUIWIrEFb5Pq1EZDEBNJYMUoiezoth3C5mKPkMjv7r321erNOgOMrbj2/AQgmtlAlbYtRo99YI6MhLRnHXWVElXE0jUomX9fS286uX5l6LOnG4awXf8mTCnNIfgj6p7m2WZpr+Vf0PcOokLPGrob2/axB4B0UGiQwxJmZZ4+8TKCXo/4Nr0gfCJye8Iy68j/zaIgHZ0ikOwSOUTfHA0Sq47Wr6LzbST+qf6kK9wncFqHS4mZVHY246sSdwo5tCXj5/XqfiqOgoXzAucI2eVPPk/O+dz+DlDwV5bmnkqpB7w/BWuCRUu3l2YQ7WYsOJqBd8682NLo8w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000053, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch add two reclaim stat: nr_promote: nr_pages shrink before promote by folio_update_gen. nr_demote: nr_pages NUMA demotion passed. And then, use correct nr_scanned which evict_folios passed into trace_mm_vmscan_lru_shrink_inactive. Mistake info like this: ``` kswapd0-89 [000] 64.887613: mm_vmscan_lru_shrink_inactive: nid=0 nr_scanned=64 nr_reclaimed=0 nr_dirty=0 nr_writeback=0 nr_congested=0 nr_immediate=0 nr_activate_anon=0 nr_activate_file=0 nr_ref_keep=0 nr_unmap_fail=0 priority=4 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC ``` Correct info like this: ``` <...>-9041 [006] 43.738481: mm_vmscan_lru_shrink_inactive: nid=0 nr_scanned=13 nr_reclaimed=0 nr_dirty=0 nr_writeback=0 nr_congested=0 nr_immediate=0 nr_activate_anon=9 nr_activate_file=0 nr_ref_keep=0 nr_unmap_fail=0 nr_promote=4 nr_demote=0 priority=12 flags=RECLAIM_WB_ANON|RECLAIM_WB_ASYNC ``` Signed-off-by: Huan Yang --- include/linux/vmstat.h | 2 ++ include/trace/events/vmscan.h | 8 +++++++- mm/vmscan.c | 26 +++++++++++++++++++++----- 3 files changed, 30 insertions(+), 6 deletions(-) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index fed855bae6d8..ac2dd9168780 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -32,6 +32,8 @@ struct reclaim_stat { unsigned nr_ref_keep; unsigned nr_unmap_fail; unsigned nr_lazyfree_fail; + unsigned nr_promote; + unsigned nr_demote; }; enum writeback_stat_item { diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h index 1a488c30afa5..9b403824a293 100644 --- a/include/trace/events/vmscan.h +++ b/include/trace/events/vmscan.h @@ -366,6 +366,8 @@ TRACE_EVENT(mm_vmscan_lru_shrink_inactive, __field(unsigned int, nr_activate1) __field(unsigned long, nr_ref_keep) __field(unsigned long, nr_unmap_fail) + __field(unsigned long, nr_promote) + __field(unsigned long, nr_demote) __field(int, priority) __field(int, reclaim_flags) ), @@ -382,17 +384,21 @@ TRACE_EVENT(mm_vmscan_lru_shrink_inactive, __entry->nr_activate1 = stat->nr_activate[1]; __entry->nr_ref_keep = stat->nr_ref_keep; __entry->nr_unmap_fail = stat->nr_unmap_fail; + __entry->nr_promote = stat->nr_promote; + __entry->nr_demote = stat->nr_demote; __entry->priority = priority; __entry->reclaim_flags = trace_reclaim_flags(file); ), - TP_printk("nid=%d nr_scanned=%ld nr_reclaimed=%ld nr_dirty=%ld nr_writeback=%ld nr_congested=%ld nr_immediate=%ld nr_activate_anon=%d nr_activate_file=%d nr_ref_keep=%ld nr_unmap_fail=%ld priority=%d flags=%s", + TP_printk("nid=%d nr_scanned=%ld nr_reclaimed=%ld nr_dirty=%ld nr_writeback=%ld nr_congested=%ld nr_immediate=%ld nr_activate_anon=%d" + " nr_activate_file=%d nr_ref_keep=%ld nr_unmap_fail=%ld nr_promote=%ld nr_demote=%ld priority=%d flags=%s", __entry->nid, __entry->nr_scanned, __entry->nr_reclaimed, __entry->nr_dirty, __entry->nr_writeback, __entry->nr_congested, __entry->nr_immediate, __entry->nr_activate0, __entry->nr_activate1, __entry->nr_ref_keep, __entry->nr_unmap_fail, + __entry->nr_promote, __entry->nr_demote, __entry->priority, show_reclaim_flags(__entry->reclaim_flags)) ); diff --git a/mm/vmscan.c b/mm/vmscan.c index 2cc0cb41fb32..21099b9f21e0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1063,8 +1063,10 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, /* folio_update_gen() tried to promote this page? */ if (lru_gen_enabled() && !ignore_references && - folio_mapped(folio) && folio_test_referenced(folio)) + folio_mapped(folio) && folio_test_referenced(folio)) { + stat->nr_promote += nr_pages; goto keep_locked; + } /* * The number of dirty pages determines if a node is marked @@ -1193,6 +1195,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, (thp_migration_supported() || !folio_test_large(folio))) { list_add(&folio->lru, &demote_folios); folio_unlock(folio); + stat->nr_demote += nr_pages; continue; } @@ -4495,6 +4498,8 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap int type; int scanned; int reclaimed; + unsigned long nr_taken = sc->nr_scanned; + unsigned int total_reclaimed = 0; LIST_HEAD(list); LIST_HEAD(clean); struct folio *folio; @@ -4521,10 +4526,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap return scanned; retry: reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false); - sc->nr_reclaimed += reclaimed; - trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, - scanned, reclaimed, &stat, sc->priority, - type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); + total_reclaimed += reclaimed; list_for_each_entry_safe_reverse(folio, next, &list, lru) { if (!folio_evictable(folio)) { @@ -4582,6 +4584,20 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap goto retry; } + /** + * MGLRU scan_folios return nr_scanned no only contains + * isolated folios. To get actually touched folios in + * shrink_folios_list, we can record last nr_scanned which + * sc saved, and sc nr_scanned will update for each folios + * which we touched. New count sub last can get right nr_taken + */ + nr_taken = sc->nr_scanned - nr_taken; + + sc->nr_reclaimed += total_reclaimed; + trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_taken, + total_reclaimed, &stat, + sc->priority, type); + return scanned; } From patchwork Wed Oct 18 08:21:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huan Yang X-Patchwork-Id: 13426607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A0F1CDB482 for ; Wed, 18 Oct 2023 08:22:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B19DF8D0144; Wed, 18 Oct 2023 04:22:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC8738D0016; Wed, 18 Oct 2023 04:22:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96A1E8D0144; Wed, 18 Oct 2023 04:22:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 83C9F8D0016 for ; Wed, 18 Oct 2023 04:22:17 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 55D0412017A for ; Wed, 18 Oct 2023 08:22:17 +0000 (UTC) X-FDA: 81357889914.13.D804635 Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2137.outbound.protection.outlook.com [40.107.215.137]) by imf27.hostedemail.com (Postfix) with ESMTP id 576874000D for ; Wed, 18 Oct 2023 08:22:13 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=MjIb9KPM; spf=pass (imf27.hostedemail.com: domain of link@vivo.com designates 40.107.215.137 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697617334; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EWIhVH8kG9NLSbfhfkLvuBEveomuvEk4IFEU8DuIXio=; b=57E8S8aDABk/9A0BZPBV5/M6jjPqffcZoKcHiuMxJN+eq0Fi/HeSRLG2f732du6wkU50gb o/iMhRhWX67aO64ZM/5rffM7CjpbFUhC2MqNv86mCWbKPzUZ0hu3fFfrAFMCm5Rywf0OGq Ccn5RTn3QVRTfUxuS5+gQubBHte6F0w= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1697617334; a=rsa-sha256; cv=pass; b=zPZH74damUWv7eFi0rfG4yeZUMGSSXm1I5yqObSGroKmrN6OjTkk3NBQmk0Cs2sTbchfOO gUJqFQ7Kj/tB64hCBXO0D0Jpcc/FJL4cX1s4lIcqQvagcGul2bCwW3ufkIQodoiaADNuZW WkN5RtmpymhsSiwx3tvNQcHMU+Q8y5Y= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=MjIb9KPM; spf=pass (imf27.hostedemail.com: domain of link@vivo.com designates 40.107.215.137 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dJNjtDbq5MY8+/8k53ArckBxifmZ3KLitaHfWuxF7Nz3cOU+QjGYAk5m0wFCitNS3M8GSfOezR3YzGbwOpzQzsMczdWV21Mc5KCgIuOFPAFMVwM+IeR7I1E+IBnWz/TKijbCAuSWc9gUV1XYkaV4W6EvVdmWCpMpRGE1W0vbS+t3FCwXT5MGBCwzAaO5MLGgHt4bvnqPT37uX5t9SB1aNgQwFim5wyqYCr9Raq5uUgbCAk371n7CH3T3U1+MyxgKrfwnUcvOmtyeQVaRA9EWuJzsO8lX+WcuI6QKDZQi///G/wsm0jJux2R221KkZMxL5G9FcWGcmTwDSrIhkmERbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EWIhVH8kG9NLSbfhfkLvuBEveomuvEk4IFEU8DuIXio=; b=HoBI4C5m4aE6SBXmEpXbVG+5nDYOYPQoDTMVFYO9jWmNd0rXgqDco8BCrTOybW0rOvDHaP7CmGgqSJ6rOFQj0HFDVjSvLCi91zpUUi3x6E4zVA6CrHCVLpx44nAlmqaXxBMaMRdJ3Wwezad2D7zEMSekGiaXUPLyJf98KIloCfObu3Eno8gs7OQyJFM2TGr6mS+yzgUvPAdcJ1yr2jfrAZNwfXvrHc364iXERIIAB5Ow89FzCtSP39pJqQBgZq2lRI9d2ht8cSs/QuBZInuD4EuBmx2GzUxozcIwNsufpXPtP4rus+ZqcBXIAujpN7pMDKB4D1Ozp7tFeUj0GKEZtg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EWIhVH8kG9NLSbfhfkLvuBEveomuvEk4IFEU8DuIXio=; b=MjIb9KPMWPUYy+bMJuJpjnkuL420kv/3z3pEzhwh8SAjHMqqKmt/y+Tjb8LgoNT6QlWDIKhTuaAwVfKiTP2vr8yZATvWYUio4lA1R8dgTspcSIXPariaag6SvQrqXub8w/rwOK8q99B69S2+ULL+ymD64ZtkliXS14n3F5J1vi87A8joyQSemM9Pzydh2IrQJGGgYgZkYMb7dtctvJH3boF34We9FPgBRbbkAHp7UHuzpgRk28AVu2p5JyRrU9dnV2XhypmwEpPZNQ+OAxF/6l5yDV+hX4dJlLo0BHTB8qU6cPsz2seR4LxaHDtc1zNGTHMcwubLm51R1I/CidfgLQ== Received: from PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) by PSAPR06MB4391.apcprd06.prod.outlook.com (2603:1096:301:82::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.22; Wed, 18 Oct 2023 08:22:08 +0000 Received: from PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::40ac:5701:4617:f503]) by PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::40ac:5701:4617:f503%4]) with mapi id 15.20.6907.021; Wed, 18 Oct 2023 08:22:08 +0000 From: Huan Yang To: Yu Zhao , Steven Rostedt , Masami Hiramatsu , Andrew Morton , Huan Yang , Suren Baghdasaryan , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org Cc: opensource.kernel@vivo.com Subject: [PATCH 2/2] mm: multi-gen lru: fix stat count Date: Wed, 18 Oct 2023 16:21:00 +0800 Message-Id: <20231018082104.3918770-3-link@vivo.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018082104.3918770-1-link@vivo.com> References: <20231018082104.3918770-1-link@vivo.com> X-ClientProxiedBy: SG2PR04CA0185.apcprd04.prod.outlook.com (2603:1096:4:14::23) To PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PUZPR06MB5676:EE_|PSAPR06MB4391:EE_ X-MS-Office365-Filtering-Correlation-Id: 9545bd03-848c-4418-0106-08dbcfb351cc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jUXaTwskgInwOhmnvEmDReDAfqDVqzr1IwWmWfcoWsvlRGiNX4DdFKXyC0cUpqsDU1/L7wAWXGXxSWaoKes/+WY8NKHpK38lOUkf6H7vWwxuJxcxa5+Sq7tM1lcI2SWHrtTLtw7g+88YXsYpdDAkLnAMRAUxfPopSL/6FK3LTCayh3Je90ngSWQabyYR0ViikbfA294kwBAPR1BOC5NDeKjt+21sXewM+5Evl10Lrq+VZx5GcFqvB00cnV42EPtlyTK7hl+chsVZ0yVVYJ7PgOf5xGcs+bW2bqoeSM8u1mavoNI/PhHng88meKc0/9230F37vj1OAyJ5qgAYsgpmwuDC78dT4sdZ2xZX5erfFe+ibcHsHPA3jj5W6gSWIU7aZt6gkmnNJkn9XUge2xpC6lzlRPb/UkUBUR4v/v6EcMq+GvMc2cseGS8ITq3Kj3IWoCDzWiRTr3HPmW+kvhrFK7yIRDw8GeZOsi12N1lZLtk5zoclxO6TTX4e6MExk9okVudUHfRgEK20WvtkHlSPF1ZUolv5JnQ0wXxM44tQf1ou11k7lGP6fzNApK+R78WUz4niOhOiJL0YQ7526uwKD7TrjipdDkxdA79no2KRhF81PVI7so0jEFGTmu4wLLuBikc4qjppQ1GQEILok2/gYg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PUZPR06MB5676.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(136003)(366004)(396003)(346002)(39850400004)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(7416002)(4326008)(8676002)(8936002)(5660300002)(2906002)(86362001)(36756003)(41300700001)(38100700002)(6506007)(83380400001)(6666004)(52116002)(6512007)(6486002)(107886003)(316002)(478600001)(1076003)(26005)(921005)(110136005)(38350700005)(66476007)(2616005)(66946007)(66556008);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1mnGyRBonP7nfJzGGd0Um/FvTMR58vdZaxRHqIIFzgsilsKyy5clAdGYSP1dhvgZwe4sCfCVXTq8JZ27b+qjTHILP5JFbK3s5ZV4qggD0iS/AHRNWwQSW+FxnzmogV3PxCcXzGiI5KjTknsvMDYNP5pMjT/E1waJx1+fTtHV2oNNpGsLOuQk1RGLsldoyweJoLFLTF/ofHzyGxLzZpkfVhT1AGLciZWiybkUkiqXGqVinDB80iaVQQ/1dwznJFQq8A4IlJgie3sEmAkMMsANTT/9xGAvdYezSZ114BofuBXwaul0yOkSMZfBX9z29b/8tSPjdnqfXjB0Xxtm9gw6wRePyopGHnZNNr9tcxCCojrBI5aGdEamZbLCCuWKuDO8bEtXNfFlwEG3OqUEARRYX2jGwvr1hoj5ceX9zohH3JIeFQDOD1N2c+YkZZUYrBUgkTvIZqlukvhjEJmrrsc/mTOmvz8m9LLPwQ/VzkC6wHUX0EBN9W26wWtDewB/Mq6RVOHe2I2KtyfhYBNvGVKhpKsyLY+8cJUYMkLaaAIWiCK2GqREghKJEBhn0gMGLVa1RocUoAtsHm2IxN6DTJ9mvZWi/GyGe3NUdAiABohIRWF3zn/Igfgi0yHAL1msVdm3CB5P7kmXlsUj3AFP/oFzzx3vRa7zf03owoIXsrUf8kHBca+T7WJ9L/YR101sb7P/wA6XV1SpufKFKpa5tINUWV804zq1kwQHjbR0M/7TND8Swmi3e22mQYR/oLcnMln8suqut0kP0JkX2BsEs1xNWPsz4BIqi0rkBGMsBl2tAGdTFKnIoybqFCOVJF2ZddBbLcikk6vvV3n6nT68lDQL1gMCi/5ErV/J5KTt9hVihk06Rd6FjxHootOd5Beh2zyGIZA7vVzIUCR/lE3NPjJLZ4ooQ0LTICL5y526mtyQA/MTaQn7CWFF7O9I+5uepj8CdPT7X6wEn7aQ7L/FB1C4Fg2tFVIDpHZR6yA1vzkFywmKONvwRzoSBqflx/PAvSm1MDGsZjtCGOTWWhmwzXxz4fDBJc1nmqRwltsM5mdGbkFTH83EeblFu8Qpi8ESswdFs7f1WHkzYVSmo5YKc8JkyS70WtJ31EOFUuxQzpw3IYW3yMRYXLh2BSfe4fw0v6ChAkJl1LEfEntQI7aCMbeHa+fqfeJuHvKzeShQXqmy9PgrkLh+QZYugvEjS4npEwYqXcvNoGpPsDFvLHnLa6dSxXmkEpB/O7ZxqVCih79zJxqMYTonxev19RPF6VFuiL5nMVPZjsyd+SGWNx5bhaiIqG7yN0gbNJZMeF4P/oCzDaoQY1mB3kb229llzSewyzoHsbTs6JQYDuFY0cPzqEaXJOpgcdVyjeYLaiiN03x8rGHh2BkjnU5RBwJ4n4eK/iPbB1AgdJTDnqrjAJSt7y38Wunn141Gi9JClhM1Mrn06uhg449iq75GqZLrvWtjLOFXfujGBxsubClRIqYMAJgyzKsRMu46o+8m/z+WyFIV9uIS8MtXpFdZEZr0Q6haNoJi8WwrRA0f7kg4zFeZfdw1FattIfA41MX+k1rH7eZA03YoG96EKSYTaSkxP9Fm06/9 X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9545bd03-848c-4418-0106-08dbcfb351cc X-MS-Exchange-CrossTenant-AuthSource: PUZPR06MB5676.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 08:22:08.2941 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: humrRbJhtzRz7mHw3ImA04YdvnTUCHIFeVefqEAMtiz6oeqaw/URu+bmPYOWCWlYnOX1btJyeZPTSBYDOvXzJg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSAPR06MB4391 X-Rspamd-Queue-Id: 576874000D X-Rspam-User: X-Stat-Signature: exunbz6cr6184758hpzdn1hzho1hs4c8 X-Rspamd-Server: rspam03 X-HE-Tag: 1697617333-544531 X-HE-Meta: U2FsdGVkX19136g5LIGpCyNJ3sTEM2+Vwcjq80QhlYj2L8Qu0yBOE6uorNut/q1uCvXGh0ZV3bHzwRZPenN9KNsQn5+5WVWXKjB2An5a7WAuTlEB86pwWBeKRUKu5DMPvzVK+SEMUQSgOuIiHY+WF3NHQOBfrUvYxk+Dhow0IS9LzqdSPFcnmu69iaqArmc6lZ6U6/70jP9pnS1XL3YS3dgdFPlaIo/BbOYwUycLcLPt4wRISjfUYSDod70OYTNYsoZPPrdi3eQG4BCxQ19fhIfvEL8+uow7y7JlBakLk7xH+Yt8IyQShh2CTNWiScgW2fCyKxXH0bD8Aa+gKksiR4//hHfsIWtFnAvLd8DpCqRFFmmjqxZ2mgWzLeZq1g09MhyguWW0XEMW0VZUkAsn9ZJ7fM89uP3vrKv32arQ3Hp2bc4UZk/0RtlXaLdbwENo47ArlTVI4OW/3cFg+fmZRjo05fxx3JgmdE59B2eyiBaqlQmrB0eYvC6rIYKGF4tXQsjjZFLGl2LlviesDGVsicWLZZSqqTKk1cQYoAYdYT0FPdYBNayp0hd/WeCwFAosEaTB3UsAZJ23GYSOOxu7gKsms+xQQIDEkjUEKxUDJYVTcHvZr3fG344IT115C01GpHAW8W13i1gIqQQVS2h8RpDW4M7lCSyZTRJqKmbIMqS5Ot0eM8vtmxPeQmqTtj3osKlrs+NG4ALiJYuO7URUfX/a0jOOsCc99ihtRG/bIy6KrvU7AqTwwusvxkX2d6EgytxhSGQWvYI2DUv/HhRhP7wHjV4dJjaPamwdG+rDDWR48EcWb2YxIz0rxQtvJ7gij3ZiInz5QctO0B+ec94tZpqq6eP75WhlLOxQjyFHBJM0w3VElns4ELnNDDglR/9TxtPVWNxiA9pYpUlKVMvBSaKaH5snIz5YsHWZVgxHNpec7ZSKURc6iyYh6WP2+Faenoc7uGriHEY1DbesELK uRo05jni eW7bSEesgXyo0trerFwHvMQrfiUrRd5m0kXisXaAgtLFBhcRDr1kc3Zr8JHv4BxMveiz2A59Ej8dlPTpGu9i14tqRGNxqhWrZFsP1COpAGnssOK+KT2D10CtHDpdrD+CGdK4Q0WD/ovQZyzfgirDiwi2kL3Mxk/grWVYVsotlrx2CqINMxX4DzD5QHiceODH5MtnK1Zlvb+nmjSvk+uPb1FYARaQOvyJBQReNHE9evKbOVtQM78AScZV/2w2ZvSDXGOzR1vhyXpFskHFJVF5YhuINHv8ZIcdy1j/wQU9797G2qofWqWegvBeYI5TtIStU798clAEnM6f3l8UA+bwQQv9CQQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000440, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For multi-gen lru reclaim in evict_folios, like shrink_inactive_list, gather folios which isolate to reclaim, and invoke shirnk_folio_list. But, when complete shrink, it not gather shrink reclaim stat into sc, we can't get info like nr_dirty\congested in reclaim, and then control writeback, dirty number and mark as LRUVEC_CONGESTED, or just bpf trace shrink and get correct sc stat. This patch fix this by simple copy code from shrink_inactive_list when end of shrink list. Signed-off-by: Huan Yang --- mm/vmscan.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/mm/vmscan.c b/mm/vmscan.c index 21099b9f21e0..88d1d586aea5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4593,6 +4593,41 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap */ nr_taken = sc->nr_scanned - nr_taken; + /* + * If dirty folios are scanned that are not queued for IO, it + * implies that flushers are not doing their job. This can + * happen when memory pressure pushes dirty folios to the end of + * the LRU before the dirty limits are breached and the dirty + * data has expired. It can also happen when the proportion of + * dirty folios grows not through writes but through memory + * pressure reclaiming all the clean cache. And in some cases, + * the flushers simply cannot keep up with the allocation + * rate. Nudge the flusher threads in case they are asleep. + */ + if (unlikely(stat.nr_unqueued_dirty == nr_taken)) { + wakeup_flusher_threads(WB_REASON_VMSCAN); + /* + * For cgroupv1 dirty throttling is achieved by waking up + * the kernel flusher here and later waiting on folios + * which are in writeback to finish (see shrink_folio_list()). + * + * Flusher may not be able to issue writeback quickly + * enough for cgroupv1 writeback throttling to work + * on a large system. + */ + if (!writeback_throttling_sane(sc)) + reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); + } + + sc->nr.dirty += stat.nr_dirty; + sc->nr.congested += stat.nr_congested; + sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; + sc->nr.writeback += stat.nr_writeback; + sc->nr.immediate += stat.nr_immediate; + sc->nr.taken += nr_taken; + if (type) + sc->nr.file_taken += nr_taken; + sc->nr_reclaimed += total_reclaimed; trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_taken, total_reclaimed, &stat,