From patchwork Fri Mar 7 17:39:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C17FC28B23 for ; Fri, 7 Mar 2025 17:43:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A451A280003; Fri, 7 Mar 2025 12:43:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CC86280001; Fri, 7 Mar 2025 12:43:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FA74280003; Fri, 7 Mar 2025 12:43:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 58210280001 for ; Fri, 7 Mar 2025 12:43:32 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 191EDA9577 for ; Fri, 7 Mar 2025 17:43:33 +0000 (UTC) X-FDA: 83195477106.11.D800A03 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2076.outbound.protection.outlook.com [40.107.220.76]) by imf10.hostedemail.com (Postfix) with ESMTP id 304B8C000D for ; Fri, 7 Mar 2025 17:43:29 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="Sy/tMsa1"; spf=pass (imf10.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.76 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369410; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0eykWjQF1YXBPjxsUA90+cJ6vRNVY7h+NYjEjvdbdz0=; b=VkzKDhyiDrGpU7qDXESdpuLFvYevvV7ndMLIQGjzK7fu3kVcN22GNOfrZs+8sMWNmgwtoD yByD+uPqE3eC8cDKnYAO/GJ8c6Ou7L2lvQI1NT1gXdkOshIbiWU4V0j9c3gCQtcXom4hjh xEDAdAvCnVfIsM+UMZU/x9KB/lIY6Rs= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="Sy/tMsa1"; spf=pass (imf10.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.76 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369410; a=rsa-sha256; cv=pass; b=Pi613Xq2tFkyekPFoOZOwXQsg8UHMhTNthb4xz+V/d26GQZtGTunKMtpOhHLgAsQ1qNc9F DQn9gD69TyWm7NamLcBzrs9Pw0iz0/uKODTg3lXT45fyoSab0YXhy6huzINqpXS1+gtVAV wFSbOd/4Hkga9oyCH1QD+hWX0l/v3/Y= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=a6Kim043lkXQRPUwJ0vr8MZ28fbFB6pt5AP4K8I1nlg5THBwLvuYM79bqT4aUOyh2mUMVjcYuQzJ39MSvsiVB30bbMbLh8exoglvW3JZXh2pJrAO931EAJilMPWuRgc9luHjsYfvcQGUi2fS6UmxlTCZYOidlHbwn/Rqbtj7F0aaoY/FrtbAz44A7DWkyZUEFDABwz2BCqCXOqFZm0+RZ6fsiZE6cw/h2odSaSJ4JrzIY43YkMrXrnbu6OxMo+YyFhUXnZGyR8PyJXA5Xbry0r3+YFz3oSiVvljCG8fGHR4B+BsNFhVRNUfeQfLW7AvTME8EvxcuIb2K5inzW3oCCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0eykWjQF1YXBPjxsUA90+cJ6vRNVY7h+NYjEjvdbdz0=; b=ZdmeF1Q/peMAhEiwMNzxRtU4p7MfyVAywP3NV6PPzKDkf1j5+xKVCj67Yq+c7S6Xjy5SnXwbmYsDZ838+At1jlbkJBteyKPY/VWqqfhksJGVsdoS/U5kaLtoNAfeRCA4fEEpD9JnapXb2ezfm/JnDJNojLlUU+dtavqBMq48p/f/dOxKWwBGsKn16I94BcG5YX4xIjgV7TKwosXn/aeIp28z7t3Wcm4dPmSoGDdkZhgo9/NwWEXNx1x9jILt7bUxWP5kNgM60mL+FrMa1mPe2EtIOcJkDh4rGQ6sfUUaMCuj6gK0PcLWCZowtM1fVdbuRTXT9yME7604s2moWAlb6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0eykWjQF1YXBPjxsUA90+cJ6vRNVY7h+NYjEjvdbdz0=; b=Sy/tMsa14egndzRhHFq1X3sd8oC2GYO7laVBV/x3Gbvsu/sNdPNp0XUdyDIc02hOJKfsb9KRR2dhDuZlGyP8V++673APVRG3ZBmJr2Rk9XabIm7v4WYdT4zFlYqAL8xK8OeqXStZDl53kod5Nroy4+MCDdOViQIcgXPJuGb2XTUY5yHS5wbAcZ5l46mo9l1CZnKNuW+1V7bx8OxkQ///t9jNexOzkgCIJw4R+mUTENcRsztuIVW2X7Ed6NnW2BJd3AfLwYnVfQxBQQmrmUTBt3QPNCjcARWHSLg5+5mkgBsi/CEdGHmPtOKlPQ9Y+g4g3VL7h2o6lz/Z3wqfFCOK4g== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:09 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:08 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 1/8] xarray: add xas_try_split() to split a multi-index entry Date: Fri, 7 Mar 2025 12:39:54 -0500 Message-ID: <20250307174001.242794-2-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BL0PR02CA0136.namprd02.prod.outlook.com (2603:10b6:208:35::41) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: a7ef5c3a-8798-464c-a42f-08dd5d9f1ad1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: 7JGlpFc5IzAuwhuhnrixdaRFeaK3Xr9t0MthJaHakuELd/olCmWnlX9HSR0pblCMx1HH0G5yOw9IuiVpeOvo3kJ6cUB37poOpQMUbnEWhLLJP4HYe9pKDZpuU9/7x8sn2r/BMnkmpVGCmflT7MzvJ/+csy1Xk3GKBwETsGYB3mlr/G3iTL3RpSa49THmNRiYyVA+RneHsLyz5v2jYpwsQ3W93Q9H2j/qxYxJKV6np76DDNXa47u1x4jlmKBP3niDOPuzoljgnCmspjd6xfFKIOvk1AMNmjpRHEVCPiuvuhkZlAguXnB20hX0SidI191cizdGP+T6a3Ui0CaThosiTtBoqujj0mTA4ioH0A0sSnlICarUIbKG5V3khzUyxjk/juIPMSdP9F0Znu9xZPolnQvdWeV8fiv0joqmU+t9l3XVsnzWXv0Wrk5QaaNIYH1deCL64UmqLZ9kLIyfynAw6t+shMEOXIahbkH3T15p1I3k1nDgqcDIg2aKDRRLEDYwmnke9S0UxsSEXpZ85XJo7dD1RrjIc2R9Nfsn2s6dJbeXtdvIXic8w5Bp2mL7BA4onhoWze3pkCoGRnDG9eULTB6+E2BWnnsVK+ubDmuGsTwhpYo63k6IH7+B/gpFgi9w90aObL7isPN6CHfx5oP0ecM40fTb9YsUxdas2PvzXqD5QMPAskUahNXG/R7QTBzTnZ1hx1W3y9pa95zF5G1RLHBTA0Ijb/zWDB6Tqa5hEOU278r7SuSImAaO4uxI5p7/rtUzmNN9FKjMTd4QJPrt8VT1EoSBlL5fe916QFPUvhQpOD5HNh8k4sYaGKJvWSsf537WSGtHIIGTkIk3hJrbtbm2/ZNLrmIcd71PK/wO0JWzN3wwHNceCfMnLJl8dn5OM35f2dr1ylIOipPjH4L3U4GHcoqApnOtCP3qrH/x+aY90iGq9WQ0tLluXxkV4amXnfkIfVtoU0AhAqwcI5rNtK6t1u+U9ccgdZ/WdTfzOe36upZze5ofLaHsV5qan59jpQvapOm7GSf5L4v2o25y2r7VoMDWtRCvzClEiGWF3yrZwf/AQXY6ie/n3tILaRVK29a9SuasWEf04oUs1/lDPcNE4B2eWToNNdZeXskslWsONutkzkrAilzoY6eTIMAsWCKYNHwOQG5VR8sy3lzlj1Cxn48kGLddjUmRS1f+01xHieLYWR4dtck0xCqy7lgV1eAntjx32qwXm+b1oT3Qi1nTv2jcuxKhOpmvUNdZ9Q+NhtMbPmlpomYF2wZPzM1t6wRpbqULVoYkHIRgiYwYYl96p3TbpFvkv50foBDddRUX+m8hsyOXL+3T3HxY7g+8w/seDW0K9OguiKAcaJGzsxXa7w7ruNztxHmbXBEe89jklD53J1YPQQYAWCbqFgki X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HzTMtRSbTFPNpmT5WN6WuMkv2YT2QTV09qiSaEkaPZH/MSJmRx4PHSQ6lKUloJZA3+rkUIy5UDYfeaeDonLBJCuwc+/ygUzfaN1RDNe9wDMH3zdkWORfQ7nRyaejg6z8henbYbMf08n3iWO+ITDaAQbb5Lt/Y3oWT0x20cNeXKKlpfB3MPtWRlrFd2n2KPMv8UxCExuGmkeWT5X7y6AFCK2Dp6qd2IJq/sDOZsgazOf+eIuS3iTYefrD6zgmlcub4/L8d7u7XZgq6TMjn/mjQQYCiz7A8udBiXoA2q0WjnUN9cpQ9Jm6LVloGRm6+CMu5QxXuGiQ/+ro+d82jw8v2by7nxkBRhuig9mwJygioQcRBSNKC7Sh0Y5nhDefc+5TqLcwIGvMI48aacAh52qcA9+ToQPLI+e4DaR/oLrUNDYzKxIoxuEymJBn9lnvII3odSUEAT7wg7s/+gjyX+tyjA2mZr7oVmcqPDqvpHw50pI8KWMiS7G9YptFialLUySCuMMAGCARxLprYuWZVS2ODDFoOHwYr5EXjTpC5i3Da2QjGLTZc2d8C5sMBwStu2h5MH3fWgSWlTDTwyGA35uw0i6AVfcPYWEu67zeUgtrkld7OdnMYXkm2EBQ9fGPCYP+ShUqYEUge2x10uRXRhj8cu7o4FUVas8yoYJc/NI9PaRpUDoI8zMo6h25FrmVZBIBbHKbRAiwBz/wNlPMCTqdPGtVr95gHKnNREqoUDWb2dcttzSEYWrNh49JLW9f04E7/Ircx/nJSk8cOBUbEVWfhah3VihgIphdwJ1DwmhIc75g6n1aydaO7SJqrbGtF/mYC/Ek28Ft7eQQSTBXDOZqeybmxnbIOv5vQPqAxIbllkl05mVaT7oEA9+xxpgzQDtwHE1Vj/Zuub8+lOaWJ9qMdw8G5LYZ8B/WOWKg6OM9+GJeJLTU+BlHgbtSE8EgjL+TdkBB/GJJeB86WtSrzcUxaHuP7oHhVbDgndclFrLcn3dVDeulmu1KlK+audp3XO44VirrwLeFiSiFpbCF2NBmqLuRFqNuY0cuBe6obrzsk/61vX5MDpRvmCGbjPU0Wj11JIjGkEtZlpHu4Tdhav/H5jj2AMUOjh8DLGMDAjE9xiewSOdY3BgZvAXbVfX4kbbBPDUR+izD17Udd2FvgUItiKaLVslUm7lBROlhzS6ZyMLTDCq6bKG3UPLG4ueRX4Cm4+PUCsgfLFs/+THKNPyWRIRgG3YjX9MBCxsdjzzoFxWnREousiyDec5OZV5pheFxtFck+lXPkswtaUVnWMfSEIcuXxNzGQ3NjiXSYqPXW77LNU+YM34IGSjZnGRjnxTl5E9LUXRVAJGDa4Hxpvc2BCJwC1wFl/WV5R/p1YMYFqYDZvmd3+p9xp9FXkwT+Oo4cCJ2oKni32o5rjDLpneyp65kpKc5H56LwbW05oOjjVhA9KPgJXonD7rDqalKA/NWhek46n+oadNySqkyiPjaXIx/mpEsN3+WHQv4BHpFPjUjOSVwk4DD9knXJMAjfyPMg9zN79HZn+zS1DOnKfzfAmPyheGgfpJVvy/xFsIBLpzqBR4LGc7Tu8EqidqmMDgl X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a7ef5c3a-8798-464c-a42f-08dd5d9f1ad1 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:08.8861 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Yys3Bu5h+OdrIlHU/rmJbgnOWfwmEEqxv2pvo8T0zEUmmH5GB8lsEv07FN5djD6P X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Stat-Signature: mqqc4mf53buxqh8hyu7718isei8atwgp X-Rspamd-Queue-Id: 304B8C000D X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1741369409-858836 X-HE-Meta: U2FsdGVkX18pUqzduce1xqj9DzQREdIxa5689RIpmgC8k0EyVgTsQjumw09n7em/dbVeT7l3cya6oXEXaSconglDQQ9rZZVnhUY33gzIH/fvHdsa2ilacfs4TzHRMKFeTbYB6px0w/xyZTahIFjMFJiw3jQk9RzemWNwRzb1o+LVP05epCsaae3ikMAYzeV95ceukLUkIaUvpd089ULCR4IYC/iS6eDQZ/V/pJzVlJkLMnLtf7vqOF8kpri7/upzLqNKep3Ai3PRsHK0M+ate1nf4ZkalHD9il9YNCm9pVCEIfWSN7cw+DP/n+ZGijLKVVamUv4lOhq1f7PVRIqzSAWzopceuGPDVBGRz9uI57mEuK1v8G834MRt0RgRG1iRSLpjEOWOME0vvZOwo7gbXMyL+gEBd4EfHyYFzOoXWggV3DeO/h3ZoloQhpI8hBXMYIRqT2qzVUwNWrwokEygEiFfPcCYB5SPFrP5F5fUU1QJw/l1cDI4LcHjfMszzEz5sy/p2fAEHDj+v8yuImhAIpCYpUhWYqHb5kRjc6maoIUuerD0dhfz3bOivs0lQHzS+UgI61ttDFsMNkKfc52MHjKSMDQjfHzk1o/L9h/efedXRXXMQzLUlqz93fSoN0DGkjYQe8hZ8r6ou4PVWGMBrzFQXCVGEjo84b0sM+xvfhRUvkuE6dAqBCg/z5eYBJ3UQuHCJ75oFoDD2TQ/VK55caomD6w9OJVdDIXRsvCdxA0iT7RT0qYXMZaDkH3oK+jfxj1f7cJualhQnFKp0QQh5fehism/35l3ml3UjdybFOSTlNivhM7gFdB9ZGdNr8KIgfGy6Fjh3vq1M+kScZlXL/DfDuChmv1zbkg/7bunSZx0yOhH4ehqo3jC2coBhhi2DMwWzcd6iM+7Gkmyr1R9YES66X3ccql684X3GbkOIAXaYQ69096i6xWpHuCrrKaZT/frJiiOQVlf/6dH/Lw ylz1yAqj ftNz9yMRAI2VaDonEfBp5dSJ7Ip6XtOg29ge6PTQilQMO2/Ew/y8MDvTkl7sftMNsFq93f65VUeWuzKN0xb4xzlPrY+NJ71wm8gLj30phBSsuZl0GpajOFFdK47lcm2meWA0clEBqbHU/qkZ2OfcqNseQIjwcreDk0bPXHLH35OouYhNfWSzEKIBgFv4R4k4bEkTekf2wsAjhNzacHgABen0jMUEhcUkIvaQvNi8fEv/3RlIyZE2MJMfeKB682+mocFVjhM5hdmljAjaqBfqb/INbzd63euneJZKssGfB0sQjDxoVoswuKVIISwqIc9zk/l+na43eeB8D+5Yi36q8icMWnpSGDdFs4sRiU9Gzii6hwnb1+2DGzfNy2+IwVDkm9TkNs+i4haY32KMf55QUc3iZad7YjR5cftUAO/ioUYiqaWJqs7EqZrTsQCkWNk1E3qjHE+MurpSsFbV/vzEmeSBVzNQYd3kfgoieWsnpWYulptSWbj4GG3IYHcf3tArgucM73CSyQ4LiZRSqQCtTx1t/R54cQtyvJ8az5T7IEl/KPMHAZnKGhd31X2x3WdK2XS8+XtNAvsUdGfQUzipL0FSL3pHwwTXxBVOK+4Jxb4ap3+rvYjFNDeeY2nYB7WLlBtxdoOXTOW7KeTLDEP89W8TVGBKw4xJckHNSfWIQ9UJLJQsa/eI5wB3tlh6T5eZsyCJ/1+UVhHtqDPGBcf7CmxfphpOw2/iQF9ZE6y1caYoGfyn/+bXt2wSKaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A preparation patch for non-uniform folio split, which always split a folio into half iteratively, and minimal xarray entry split. Currently, xas_split_alloc() and xas_split() always split all slots from a multi-index entry. They cost the same number of xa_node as the to-be-split slots. For example, to split an order-9 entry, which takes 2^(9-6)=8 slots, assuming XA_CHUNK_SHIFT is 6 (!CONFIG_BASE_SMALL), 8 xa_node are needed. Instead xas_try_split() is intended to be used iteratively to split the order-9 entry into 2 order-8 entries, then split one order-8 entry, based on the given index, to 2 order-7 entries, ..., and split one order-1 entry to 2 order-0 entries. When splitting the order-6 entry and a new xa_node is needed, xas_try_split() will try to allocate one if possible. As a result, xas_try_split() would only need 1 xa_node instead of 8. When a new xa_node is needed during the split, xas_try_split() can try to allocate one but no more. -ENOMEM will be return if a node cannot be allocated. -EINVAL will be return if a sibling node is split or cascade split happens, where two or more new nodes are needed, and these are not supported by xas_try_split(). xas_split_alloc() and xas_split() split an order-9 to order-0: --------------------------------- | | | | | | | | | | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | | | | | | | | | | --------------------------------- | | | | ------- --- --- ------- | | ... | | V V V V ----------- ----------- ----------- ----------- | xa_node | | xa_node | ... | xa_node | | xa_node | ----------- ----------- ----------- ----------- xas_try_split() splits an order-9 to order-0: --------------------------------- | | | | | | | | | | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | | | | | | | | | | --------------------------------- | | V ----------- | xa_node | ----------- Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Miaohe Lin Cc: Matthew Wilcox Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Zi Yan Cc: Kairui Song --- Documentation/core-api/xarray.rst | 14 +++- include/linux/xarray.h | 6 ++ lib/test_xarray.c | 52 ++++++++++++ lib/xarray.c | 132 +++++++++++++++++++++++++++--- tools/testing/radix-tree/Makefile | 1 + 5 files changed, 192 insertions(+), 13 deletions(-) diff --git a/Documentation/core-api/xarray.rst b/Documentation/core-api/xarray.rst index f6a3eef4fe7f..c6c91cbd0c3c 100644 --- a/Documentation/core-api/xarray.rst +++ b/Documentation/core-api/xarray.rst @@ -489,7 +489,19 @@ Storing ``NULL`` into any index of a multi-index entry will set the entry at every index to ``NULL`` and dissolve the tie. A multi-index entry can be split into entries occupying smaller ranges by calling xas_split_alloc() without the xa_lock held, followed by taking the lock -and calling xas_split(). +and calling xas_split() or calling xas_try_split() with xa_lock. The +difference between xas_split_alloc()+xas_split() and xas_try_alloc() is +that xas_split_alloc() + xas_split() split the entry from the original +order to the new order in one shot uniformly, whereas xas_try_split() +iteratively splits the entry containing the index non-uniformly. +For example, to split an order-9 entry, which takes 2^(9-6)=8 slots, +assuming ``XA_CHUNK_SHIFT`` is 6, xas_split_alloc() + xas_split() need +8 xa_node. xas_try_split() splits the order-9 entry into +2 order-8 entries, then split one order-8 entry, based on the given index, +to 2 order-7 entries, ..., and split one order-1 entry to 2 order-0 entries. +When splitting the order-6 entry and a new xa_node is needed, xas_try_split() +will try to allocate one if possible. As a result, xas_try_split() would only +need 1 xa_node instead of 8. Functions and structures ======================== diff --git a/include/linux/xarray.h b/include/linux/xarray.h index 0b618ec04115..4010195201c9 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1555,6 +1555,7 @@ int xa_get_order(struct xarray *, unsigned long index); int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, gfp_t); +void xas_try_split(struct xa_state *xas, void *entry, unsigned int order); #else static inline int xa_get_order(struct xarray *xa, unsigned long index) { @@ -1576,6 +1577,11 @@ static inline void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, gfp_t gfp) { } + +static inline void xas_try_split(struct xa_state *xas, void *entry, + unsigned int order) +{ +} #endif /** diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 0e865bab4a10..080a39d22e73 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1858,6 +1858,54 @@ static void check_split_1(struct xarray *xa, unsigned long index, xa_destroy(xa); } +static void check_split_2(struct xarray *xa, unsigned long index, + unsigned int order, unsigned int new_order) +{ + XA_STATE_ORDER(xas, xa, index, new_order); + unsigned int i, found; + void *entry; + + xa_store_order(xa, index, order, xa, GFP_KERNEL); + xa_set_mark(xa, index, XA_MARK_1); + + /* allocate a node for xas_try_split() */ + xas_set_err(&xas, -ENOMEM); + XA_BUG_ON(xa, !xas_nomem(&xas, GFP_KERNEL)); + + xas_lock(&xas); + xas_try_split(&xas, xa, order); + if (((new_order / XA_CHUNK_SHIFT) < (order / XA_CHUNK_SHIFT)) && + new_order < order - 1) { + XA_BUG_ON(xa, !xas_error(&xas) || xas_error(&xas) != -EINVAL); + xas_unlock(&xas); + goto out; + } + for (i = 0; i < (1 << order); i += (1 << new_order)) + __xa_store(xa, index + i, xa_mk_index(index + i), 0); + xas_unlock(&xas); + + for (i = 0; i < (1 << order); i++) { + unsigned int val = index + (i & ~((1 << new_order) - 1)); + XA_BUG_ON(xa, xa_load(xa, index + i) != xa_mk_index(val)); + } + + xa_set_mark(xa, index, XA_MARK_0); + XA_BUG_ON(xa, !xa_get_mark(xa, index, XA_MARK_0)); + + xas_set_order(&xas, index, 0); + found = 0; + rcu_read_lock(); + xas_for_each_marked(&xas, entry, ULONG_MAX, XA_MARK_1) { + found++; + XA_BUG_ON(xa, xa_is_internal(entry)); + } + rcu_read_unlock(); + XA_BUG_ON(xa, found != 1 << (order - new_order)); +out: + xas_destroy(&xas); + xa_destroy(xa); +} + static noinline void check_split(struct xarray *xa) { unsigned int order, new_order; @@ -1869,6 +1917,10 @@ static noinline void check_split(struct xarray *xa) check_split_1(xa, 0, order, new_order); check_split_1(xa, 1UL << order, order, new_order); check_split_1(xa, 3UL << order, order, new_order); + + check_split_2(xa, 0, order, new_order); + check_split_2(xa, 1UL << order, order, new_order); + check_split_2(xa, 3UL << order, order, new_order); } } } diff --git a/lib/xarray.c b/lib/xarray.c index 116e9286c64e..3bae48558e21 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -278,6 +278,7 @@ void xas_destroy(struct xa_state *xas) xas->xa_alloc = node = next; } } +EXPORT_SYMBOL_GPL(xas_destroy); /** * xas_nomem() - Allocate memory if needed. @@ -1007,6 +1008,26 @@ static void node_set_marks(struct xa_node *node, unsigned int offset, } } +static void __xas_init_node_for_split(struct xa_state *xas, + struct xa_node *node, void *entry) +{ + unsigned int i; + void *sibling = NULL; + unsigned int mask = xas->xa_sibs; + + if (!node) + return; + node->array = xas->xa; + for (i = 0; i < XA_CHUNK_SIZE; i++) { + if ((i & mask) == 0) { + RCU_INIT_POINTER(node->slots[i], entry); + sibling = xa_mk_sibling(i); + } else { + RCU_INIT_POINTER(node->slots[i], sibling); + } + } +} + /** * xas_split_alloc() - Allocate memory for splitting an entry. * @xas: XArray operation state. @@ -1025,7 +1046,6 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, gfp_t gfp) { unsigned int sibs = (1 << (order % XA_CHUNK_SHIFT)) - 1; - unsigned int mask = xas->xa_sibs; /* XXX: no support for splitting really large entries yet */ if (WARN_ON(xas->xa_shift + 2 * XA_CHUNK_SHIFT <= order)) @@ -1034,22 +1054,13 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, return; do { - unsigned int i; - void *sibling = NULL; struct xa_node *node; node = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!node) goto nomem; - node->array = xas->xa; - for (i = 0; i < XA_CHUNK_SIZE; i++) { - if ((i & mask) == 0) { - RCU_INIT_POINTER(node->slots[i], entry); - sibling = xa_mk_sibling(i); - } else { - RCU_INIT_POINTER(node->slots[i], sibling); - } - } + + __xas_init_node_for_split(xas, node, entry); RCU_INIT_POINTER(node->parent, xas->xa_alloc); xas->xa_alloc = node; } while (sibs-- > 0); @@ -1122,6 +1133,103 @@ void xas_split(struct xa_state *xas, void *entry, unsigned int order) xas_update(xas, node); } EXPORT_SYMBOL_GPL(xas_split); + +/** + * xas_try_split() - Try to split a multi-index entry. + * @xas: XArray operation state. + * @entry: New entry to store in the array. + * @order: Current entry order. + * + * The size of the new entries is set in @xas. The value in @entry is + * copied to all the replacement entries. If and only if one new xa_node is + * needed, the function will use GFP_NOWAIT to get one if xas->xa_alloc is + * NULL. If more new xa_node are needed, the function gives EINVAL error. + * + * Context: Any context. The caller should hold the xa_lock. + */ +void xas_try_split(struct xa_state *xas, void *entry, unsigned int order) +{ + unsigned int sibs = (1 << (order % XA_CHUNK_SHIFT)) - 1; + unsigned int offset, marks; + struct xa_node *node; + void *curr = xas_load(xas); + int values = 0; + gfp_t gfp = GFP_NOWAIT; + + node = xas->xa_node; + if (xas_top(node)) + return; + + if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) + gfp |= __GFP_ACCOUNT; + + marks = node_get_marks(node, xas->xa_offset); + + offset = xas->xa_offset + sibs; + + if (xas->xa_shift < node->shift) { + struct xa_node *child = xas->xa_alloc; + unsigned int expected_sibs = + (1 << ((order - 1) % XA_CHUNK_SHIFT)) - 1; + + /* + * No support for splitting sibling entries + * (horizontally) or cascade split (vertically), which + * requires two or more new xa_nodes. + * Since if one xa_node allocation fails, + * it is hard to free the prior allocations. + */ + if (sibs || xas->xa_sibs != expected_sibs) { + xas_destroy(xas); + xas_set_err(xas, -EINVAL); + return; + } + + if (!child) { + child = kmem_cache_alloc_lru(radix_tree_node_cachep, + xas->xa_lru, gfp); + if (!child) { + xas_destroy(xas); + xas_set_err(xas, -ENOMEM); + return; + } + RCU_INIT_POINTER(child->parent, xas->xa_alloc); + } + __xas_init_node_for_split(xas, child, entry); + + xas->xa_alloc = rcu_dereference_raw(child->parent); + child->shift = node->shift - XA_CHUNK_SHIFT; + child->offset = offset; + child->count = XA_CHUNK_SIZE; + child->nr_values = xa_is_value(entry) ? + XA_CHUNK_SIZE : 0; + RCU_INIT_POINTER(child->parent, node); + node_set_marks(node, offset, child, xas->xa_sibs, + marks); + rcu_assign_pointer(node->slots[offset], + xa_mk_node(child)); + if (xa_is_value(curr)) + values--; + xas_update(xas, child); + + } else { + do { + unsigned int canon = offset - xas->xa_sibs; + + node_set_marks(node, canon, NULL, 0, marks); + rcu_assign_pointer(node->slots[canon], entry); + while (offset > canon) + rcu_assign_pointer(node->slots[offset--], + xa_mk_sibling(canon)); + values += (xa_is_value(entry) - xa_is_value(curr)) * + (xas->xa_sibs + 1); + } while (offset-- > xas->xa_offset); + } + + node->nr_values += values; + xas_update(xas, node); +} +EXPORT_SYMBOL_GPL(xas_try_split); #endif /** diff --git a/tools/testing/radix-tree/Makefile b/tools/testing/radix-tree/Makefile index 8b3591a51e1f..b2a6660bbd92 100644 --- a/tools/testing/radix-tree/Makefile +++ b/tools/testing/radix-tree/Makefile @@ -14,6 +14,7 @@ include ../shared/shared.mk main: $(OFILES) +xarray.o: ../../../lib/test_xarray.c idr-test.o: ../../../lib/test_ida.c idr-test: idr-test.o $(CORE_OFILES) From patchwork Fri Mar 7 17:39:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53298C28B23 for ; Fri, 7 Mar 2025 17:40:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C2216B0083; Fri, 7 Mar 2025 12:40:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 44A6E6B0085; Fri, 7 Mar 2025 12:40:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2780A6B0088; Fri, 7 Mar 2025 12:40:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F40B06B0083 for ; Fri, 7 Mar 2025 12:40:21 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 071C21A0566 for ; Fri, 7 Mar 2025 17:40:22 +0000 (UTC) X-FDA: 83195469084.15.C87E14D Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2068.outbound.protection.outlook.com [40.107.236.68]) by imf23.hostedemail.com (Postfix) with ESMTP id 4660A140009 for ; Fri, 7 Mar 2025 17:40:19 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=sfsY76Wr; spf=pass (imf23.hostedemail.com: domain of ziy@nvidia.com designates 40.107.236.68 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CBuTLWbsJir6CHZpkVKu+RnRUxfxtCLRhi+FYL+5qAE=; b=kQZor/aTgCdHv+T0/47afa+73XmZBog3+fCj+dukU/V1LdRzS5cUANBuGts7u07Py58trE lVl2c6RIONhIZSH/Qy6fyKisVvcE8X0un/fuhp0Vud36T5NuDJP3ARppM+9yAAQXPkmYCK nov+DGjtZNOcUMZbjNvvV5Bw/lzgqBw= ARC-Authentication-Results: i=2; imf23.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=sfsY76Wr; spf=pass (imf23.hostedemail.com: domain of ziy@nvidia.com designates 40.107.236.68 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369219; a=rsa-sha256; cv=pass; b=JZoGk9nZTIaW+B/ApTVxRBg1zjY951x5Ik3Y2M9psK+m+k2U6uoE2ymlTqyYi6wbt/sLRD LpyoqswFYpWTeJzl4l8iOrjq6YMq0qRa00tL9VDLVOcZgf8sn+mqQMMGoO/1ROZNBT120H R06VByfpolKR+yPAzM8hsZfblQ6+S04= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PKCHtgYAwoeMW/EN0whmrqP/95Tq6t8NFpyVOd6kwlugzya1VQcbeZzlszrkufCm+g4bG7BkkZ8mBEM4qZ3663iNRR8X8dMXoLureTmJyYTz4v5P4yWlaYmaanoS2Pb6DXADuPlyIppnPcUY1XuN4rLpYLR57nMTLHC3MLc+lwXbUt0QJt1vIlLxUiJeo52qJp+DHDPgFRjiVZz5vIaylvelDEgsSwB5PVaZ4oyvPk5yZnUrJS1e19GxnKDnhjevjPWMShG7qISMMbscEjznUUvm7TT2nZRYkvHwRLpPOz1W4HjMmUW99QTqgq+8YgCS8bUvDnKOQSvFHNiHCwmAHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CBuTLWbsJir6CHZpkVKu+RnRUxfxtCLRhi+FYL+5qAE=; b=WMfkVzSB0VCUAZjxPYRAa4aUKaBE4B/8fCHRzCgYUXIo9hB6cTUmuyMzd3YAicAXDuHOpKyROOtkYHVubHlSeFzAfP7GEE0fHXQaxovK1IiYII9jMzNpeGSGWpo10E6knNp3sziPFeMb5Z1DjLga4YR3CPtquB6nOBK5N5v+0XGwSzn63aFo3Io5YkcV5dIYY4p4rYF+XqYazhOq3F2x3PvVw3gk8RTbyXIRhlU4/UuIdsw32w4jvXoEYk+dAMgLT/wbRNEH/hVxpTHFY30uTtPNDZC615BGee7XJeQ9+SK8Hi7kxwmcmIt+ItaIrCwMSpXrwQDoHJZbVZSwD6Px+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CBuTLWbsJir6CHZpkVKu+RnRUxfxtCLRhi+FYL+5qAE=; b=sfsY76WrZZCBNiN7WuPVsPeclhmpyKkHesoRdrTpS8t9x7e904qcjEvuRnvvgJoXOpdyKf5XvdvPn35kwcsc8mfHS0wKs9rq3bHKIADufQLMj0jheP2V/tXKpfXucE/b5pm2awpHDpIWZEuNrXt9d73meR+ZQDP4yD7Q2/9zxuunIznOkssVGNyd/SYUcEEuj1TkhFehQ0fGCS7/bSBNrI7jflUdGxO+i3MfEmIG6ZIY6di8x1FErjoMB3nDuuC12eN98AAuiZWVIIvHuDWQIfcYHB8l4+EVMmfjAiUGjrO7xuz5AfM7S2PRny2VpBWp2ACtxo4arbVGk9WJIBSRDg== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:10 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:10 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 2/8] mm/huge_memory: add two new (not yet used) functions for folio_split() Date: Fri, 7 Mar 2025 12:39:55 -0500 Message-ID: <20250307174001.242794-3-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BL0PR02CA0119.namprd02.prod.outlook.com (2603:10b6:208:35::24) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: 4f84b7bd-8d07-4afa-73de-08dd5d9f1bb6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: mn/zLZeJNII7c95ZGyaIxnP1fjYJqN6w7WT1t2jv2gwqvZuprUKioOx6zVhq/ujAAfQww8pqdyYDAoHdecfd1/mOgEjuzSVtpsHZVwqVRo6pD9yG1JqKUfizjKkdSsOguDFTSWZ5PkCfP5wgfxGElTgVuJH0cfYB8oSJzIZLqZeWK5WYKJgegcfqKQV+fMhRXKvUWYhxr0w5f5U0+iowOkatPbUV923ZoD4KFTQOARUbztnYEt0RSVBbFZyox9DwEfAyCZd94ddxYgj6e/OPBBdnhZTdztV/g544CgStqp9bv6w3Hj/TCYuZD/9LnUtPdDfDC6stJG05itcuMLI32UK/8vtSIzYk1sNIDow+RZfhw+sp//ixCzMnMRJjlaOuscqEl1GQxhJp7JWrGLom8l0j7Lpx0Yu8fb+VnX0fY+mgj0HsRXDnk5z63jFXCzXVOH7nZ+0X8iPwVWaZ8kEaeBUFwLKt7Tg8WErsjPE1TKv1WDh5PcT2S2jWTzxl3erI/Tvk0w8BgnaI26IjDcgeBVD8feo+Ii1hB4Xn6XMqsTWjPIpUgzs+dYZtc5/o/UHfBM9J2TOYFAHvxdmGcbAuY6khnmvaGhSpTNUAEoOY1xEuqaTYD9o3p6X0uezA75M977gJPWvhAc8p8ac1N4Rb6/Hjcr8voDMbXZkgBSppsq2XfknZZoAjKdm0AvKBwDYfi/yCYlISyc/JbusHbOCLUQ/BWsJeKK/dKojYAqOJULJxo0+OCVPrBIMziNmhyR1z8Bm0G/hopsnyDUoKGzwSa+dM9oBcLlu+7DfsrHci6m3XaC/ZwbEOh13GcEldu2qBTDv81slZshuXMRtkxvJH06wmlCUnZOAeABxMxXjzWSwk35uRgTyW7TAOMUC6pP/mFEvJql5133h5bwaXA6lM5c5xa+QWRLFI/KlWzyoF9Fh8C+Bevtr9tC49fa8oDcrR1cZQBybG7ILKrOVW3Xq5CmcEoLvZ4RRJzi6EARK8NEiMo1gv1SYAvTwhboahYrk+GOARnceKL9fAJKW3IoolPn9DUXfxYUTGs3J7B2g5VR4wVdtfdgKBsfOvh91EXQLfw47jQIbXsewxzUhxpT4D/RbDNS/cY8onveszBY9WRiAJbHiGin6U+YQmQJ7K9+WfiyUixB22K08fud6dxMIPdKNBEh+YgVYJu6OtIw5bTCvIt/DosYffmbaGJ5KWmdJF7NjKT+i/EThx6/uAHKzMC1KR+AcKlReKnR9qXZp6xKVyT0MJr0RahGpU9ly/QmE7OUIa95yCfY/year02J12s5MjJTIF+3LWy65Dc5HjFCTEGg3ZufCtG+oMXSAAvA6X4/EUb7wXXa1pm9RljM0113HMqHmr0yHIc6ynVd4dfB+X2rwt8DBj3xzSd2Y0Mr5P X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FSydk9Av3+mKBmtPWJGJj8U0pEMN8Oey2ddozv4Ny/CN40SmTvMfuWuZg2UY6/SgX9m3jRzapmKV8bZlxM8ZilUzGRigCgX6B4TWkbtiPTERAypQ8mi6gS/vp5BA6gDynIpxXPvTB8GY3J9PKdQerrUO5l4xxKkfAIYFru5OyG8BGi3E/gHkfh72T7wio7vwUEzfgv1azVlgqEaHZ8Uz+PW33ttoYOm4WwlG1590sNERqIVkIUyQB2gi5/6muwWz1w2epga8jYl7FXYvF1zxRV4LHNF6zIYuKk86dEXbbON4FVUuGv50GaFB0hJfx/FfnsYolsSGMS5eIxg12nAAfMnOMFg6AXsnyjewdxAauZuwxSWr2ZaYDDhxSPGOlZa4sOrMGien72XEHZ1WDQxzLn7WVMvCZ4VZeaWkMo43O4p469GhqbMF6SxJ52gRZzL78dtrCiLXmHk4nkurT1ARZTLb2XBhuLI9DuEfTWOZAxNpsiNFsRMekcxclginc2uKqyVWhoEPlq8jtd03i3K9Id4lAUPJpFI7D+6BC3po4pBasLyB1exgG7Jb0pXQQDjUZVPxZgbfD56h71bhc24KyuLQ96lgECoQPqbFhdViYi1dOulFB4dcgrsWYYBH7rWcIsLSTnLQtvdJS7b2NJosCar8RddJUy9R9baKt+FLkf8PkxRzhnrPxRpsnjjT+1sbSXAFmgmNoSIuW1AheaD4sczOt94J0+3LNMWoQEOBNRGkdjvMT0ZEz3UJwdXnszSu18haHpOpLmgMm9ij7NbgxdJZLEz38Lo1jWVpMKeHB/JMlZFb5dMwJBks79Op1b48XTa5IuZzPhMZ5tyCz8Yk+O6a/v3xNnNu0BVs4n4VsVWMlmFUyTrqIe4kw6xDV+lj6Uteto9lwOsTiwZZiFNfeHPUZ85ySo2aBgFouAggdlEoP0woiQ+2gK0c3hX8MRZhGjmhmO1LmExhAtK0J4NCJu51sOXlUOb3oZvMBw4rniOT/bPIDUvw/0n9Aa+U8wgNHlKpAYI7xPoGKIKZvvDuEUmvJ9EvkVFmFu40lAUSuJOtAqGKTkcQuJ+fpOhFiuvxwf31IrEFoxOr8wpBEcIkQdci633lKkkTEx4KP1tOIKnraC8nau2H2vRdoPGF1WTGub71onsRNhuXHAOLd9waKcmjS0+X4Lh8Ujvx9K6v3Ps/VfJkYhwSEfW0uQmu1lfHHkHiiWlXGIvZ82kPglyzmUH2SYSWtq73/Mi1CeKLca7/yyQIC/y5L4sSr5ROkTVvvCFNqCmNgWe2IVhikrbBujWQVffB4smverzMLkL4VjQBU+SGsl15Pemewe3byCnfCFcngloLtPVhR5y4bvIwUMKKaTx/sUbVPgxl0gjb/QZfIq8d9snbZS3abtIqUO8qkzsAuaVoWWpYNXIUef6S3EzAuyzDehRLnE1P9oUzXBigwYD7pTMkfxRqqZdqpdF5F9HLDkmqxzLVie38VZyzvh2v0Qldcca+1BH/XZQIygiOFvvjrxy9yROC1Ozwvj0UY+pIaKdw76X5Qt6aOnUwzyy7m1IBHG7FMiO7F4+H4uc= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f84b7bd-8d07-4afa-73de-08dd5d9f1bb6 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:10.3860 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: q9uiLgBFnEqLI9CMm6MIntMXMmJLqa+uhsef7rC1ip5b/nC/HlVW+apOJgmS7fEN X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4660A140009 X-Stat-Signature: ge5jeskaexdom8u4oni6cq4iy7x1iz4h X-HE-Tag: 1741369219-229020 X-HE-Meta: U2FsdGVkX1+R1F5jTFiu0hXWkARzmyK2+YIGotBM0tEDBDz4C7sHb7YviWeB1CWT73mf5BDC7X0AMmiGp+mL5WpkkGIiiWuU/XYF2t06VuPZG8Hy7xdArDUn7iyBMbnua2RdRr8AF0KwlueWB8g1gO86AmOfjZ7V3uTEEJMTrjIpN2rEiTlYeOKAHFGqCnuUuv6OZeWQ824A18747K00fy9P+q90qHNgfgSGys6AOQ/z0JXgb96TPwVbTp2PR2R34/PEPitWnrhlqb2toGLDpTSz2C3O2UDKoftu2HQ/sfVYDLmBTs1ZU+whOUq1LoiOLYhOxwU9d1TtN2HjQYXUsDeFtPL50JhQMq8lihOPSoiyCyv2QMpEkOLqple1Kkn3+rhiwH7ch4h8u+Yzo1jgXl1sD88U5ixb8ncstZqqI9pGcjcqa4MCP4Ipd2PrfFzA6ouQuSce4hysxmv+nY4zFYgdi5Kt15Kc4BU6b7+J/qEGRxUu/ebXj3geA2+IEbWh2jMo/6FW2P55j2qYxS+bWMJV8FkoueeJIZpbwy+1eK35ZabfbU4iy3iedpVBVkAgBblWsEi3TKi4dIwFdfaW3YE0gChwrCJsv7IZCBHwDZq1nM6XW+Ge5/rEX66Tv658Ca5UAqlzjfv0yNUqdvs/JDy/RT1ogVfz/yruSLx/I7Tn6AB4cOnO56vwuMgmQnugoolYhuOVJu0F72efWezqh/GSFX778cs92WC3bBOsBMGMAtCFJ5W5cERKfcvIFAnc0bDI/+JjmIKD+vn0DIJeqaprxGb08FR+fSmHz1DtQaXYswWNLbRcVbfn4txnMJPXKokXLupVr80ShE56LPzcZxWPP4bExlRXjJ0mOqY9rM50V9NThH+NPwm89bQ1ia6LtpJWvxk2lhCVgOajc5wvN2ECXC6LQGrdvRAdu5/TiRv2IMhfRIx3cNah+HOQBiERKRtF69vUorwwOFUki2m odWATk/I XIuWEgM/p0PhnWJCmc5rpnvJQP0tIqPms5vXZYNXX61dNH97I1cqMW9asK3uPFYl3Mx/wEIIeIq9utZiK8klcH4t/UM3kigFMI6mq8IzpnozOv4BVXa+EJ+y2ahadF5UrCg7RflvzAkc9di+bOmOoI+EKC2lVJxGt5w2swwEhix86IJWGmxopUBMbeFCUtviDgYxsImqifGdiSznSMiFH2yZUFmAmFO5UxHqd2m9G8OSARBWlK9IbjH+N3qbcO5NqzhSCTIcLM8jJMbhSx11P6RymxQWuRUZPXXzCrUhMXlhXBdzZfEdf6CTBIEPqICfhnLywweCREpFjau0afZyuudQfzXRrT0pI9y2JWKshicyHlH85w2bQPK995na1YErWV3SkFrid3ccHqPIMjoDDtTAT8houqHC/vFy0eNIkPFSv0/K1b1Yne9Fhi6oYoaannRjJ7msId+5qkkS1uz6PWApuETfguIxGMhZWNJsvyOdjnNSv5bAK9Rp6ZLTrt0zjqWIhk0/7mR0YAJ5YzQ6q5yZN2kWHGSonlB7XURokmx3ewz7rm/uUJDclPh0cF3joEa2pDAZCCKJyHGdyYZhG/w0w0n0xKnWUj6zfPnRtO/EjkdOSPDUwAoyx73mS1jfLqPKsqMaEPc372bZwKSxtj/7QLjTdOeb5fNISpo3IE+K1MUqPz2YySLHZj5eNJwrL4uMU+IFoHbmfotgb7DfACNk6KQnZamB+bURnlZhGm0b421RDfyJx2smlPA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation patch, both added functions are not used yet. The added __split_unmapped_folio() is able to split a folio with its mapping removed in two manners: 1) uniform split (the existing way), and 2) buddy allocator like (or non-uniform) split. The added __split_folio_to_order() can split a folio into any lower order. For uniform split, __split_unmapped_folio() calls it once to split the given folio to the new order. For buddy allocator like (non-uniform) split, __split_unmapped_folio() calls it (folio_order - new_order) times and each time splits the folio containing the given page to one lower order. Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song Reported-by: Hugh Dickins Signed-off-by: Zi Yan Reported-by: Hugh Dickins Signed-off-by: Zi Yan --- mm/huge_memory.c | 348 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 347 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3c5d01aecac8..c10ee77189bd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3265,7 +3265,6 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags) static void lru_add_page_tail(struct folio *folio, struct page *tail, struct lruvec *lruvec, struct list_head *list) { - VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); VM_BUG_ON_FOLIO(PageLRU(tail), folio); lockdep_assert_held(&lruvec->lru_lock); @@ -3517,6 +3516,353 @@ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins) caller_pins; } +/* + * It splits @folio into @new_order folios and copies the @folio metadata to + * all the resulting folios. + */ +static void __split_folio_to_order(struct folio *folio, int old_order, + int new_order) +{ + long new_nr_pages = 1 << new_order; + long nr_pages = 1 << old_order; + long index; + + /* + * Skip the first new_nr_pages, since the new folio from them have all + * the flags from the original folio. + */ + for (index = new_nr_pages; index < nr_pages; index += new_nr_pages) { + struct page *head = &folio->page; + struct page *new_head = head + index; + + /* + * Careful: new_folio is not a "real" folio before we cleared PageTail. + * Don't pass it around before clear_compound_head(). + */ + struct folio *new_folio = (struct folio *)new_head; + + VM_BUG_ON_PAGE(atomic_read(&new_head->_mapcount) != -1, new_head); + + /* + * Clone page flags before unfreezing refcount. + * + * After successful get_page_unless_zero() might follow flags change, + * for example lock_page() which set PG_waiters. + * + * Note that for mapped sub-pages of an anonymous THP, + * PG_anon_exclusive has been cleared in unmap_folio() and is stored in + * the migration entry instead from where remap_page() will restore it. + * We can still have PG_anon_exclusive set on effectively unmapped and + * unreferenced sub-pages of an anonymous THP: we can simply drop + * PG_anon_exclusive (-> PG_mappedtodisk) for these here. + */ + new_head->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; + new_head->flags |= (head->flags & + ((1L << PG_referenced) | + (1L << PG_swapbacked) | + (1L << PG_swapcache) | + (1L << PG_mlocked) | + (1L << PG_uptodate) | + (1L << PG_active) | + (1L << PG_workingset) | + (1L << PG_locked) | + (1L << PG_unevictable) | +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + (1L << PG_arch_2) | +#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_3 + (1L << PG_arch_3) | +#endif + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_REFS_MASK)); + + /* ->mapping in first and second tail page is replaced by other uses */ + VM_BUG_ON_PAGE(new_nr_pages > 2 && new_head->mapping != TAIL_MAPPING, + new_head); + new_head->mapping = head->mapping; + new_head->index = head->index + index; + + /* + * page->private should not be set in tail pages. Fix up and warn once + * if private is unexpectedly set. + */ + if (unlikely(new_head->private)) { + VM_WARN_ON_ONCE_PAGE(true, new_head); + new_head->private = 0; + } + + if (folio_test_swapcache(folio)) + new_folio->swap.val = folio->swap.val + index; + + /* Page flags must be visible before we make the page non-compound. */ + smp_wmb(); + + /* + * Clear PageTail before unfreezing page refcount. + * + * After successful get_page_unless_zero() might follow put_page() + * which needs correct compound_head(). + */ + clear_compound_head(new_head); + if (new_order) { + prep_compound_page(new_head, new_order); + folio_set_large_rmappable(new_folio); + } + + if (folio_test_young(folio)) + folio_set_young(new_folio); + if (folio_test_idle(folio)) + folio_set_idle(new_folio); + + folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio)); + } + + if (new_order) + folio_set_order(folio, new_order); + else + ClearPageCompound(&folio->page); +} + +/* + * It splits an unmapped @folio to lower order smaller folios in two ways. + * @folio: the to-be-split folio + * @new_order: the smallest order of the after split folios (since buddy + * allocator like split generates folios with orders from @folio's + * order - 1 to new_order). + * @split_at: in buddy allocator like split, the folio containing @split_at + * will be split until its order becomes @new_order. + * @lock_at: the folio containing @lock_at is left locked for caller. + * @list: the after split folios will be added to @list if it is not NULL, + * otherwise to LRU lists. + * @end: the end of the file @folio maps to. -1 if @folio is anonymous memory. + * @xas: xa_state pointing to folio->mapping->i_pages and locked by caller + * @mapping: @folio->mapping + * @uniform_split: if the split is uniform or not (buddy allocator like split) + * + * + * 1. uniform split: the given @folio into multiple @new_order small folios, + * where all small folios have the same order. This is done when + * uniform_split is true. + * 2. buddy allocator like (non-uniform) split: the given @folio is split into + * half and one of the half (containing the given page) is split into half + * until the given @page's order becomes @new_order. This is done when + * uniform_split is false. + * + * The high level flow for these two methods are: + * 1. uniform split: a single __split_folio_to_order() is called to split the + * @folio into @new_order, then we traverse all the resulting folios one by + * one in PFN ascending order and perform stats, unfreeze, adding to list, + * and file mapping index operations. + * 2. non-uniform split: in general, folio_order - @new_order calls to + * __split_folio_to_order() are made in a for loop to split the @folio + * to one lower order at a time. The resulting small folios are processed + * like what is done during the traversal in 1, except the one containing + * @page, which is split in next for loop. + * + * After splitting, the caller's folio reference will be transferred to the + * folio containing @page. The other folios may be freed if they are not mapped. + * + * In terms of locking, after splitting, + * 1. uniform split leaves @page (or the folio contains it) locked; + * 2. buddy allocator like (non-uniform) split leaves @folio locked. + * + * + * For !uniform_split, when -ENOMEM is returned, the original folio might be + * split. The caller needs to check the input folio. + */ +static int __split_unmapped_folio(struct folio *folio, int new_order, + struct page *split_at, struct page *lock_at, + struct list_head *list, pgoff_t end, + struct xa_state *xas, struct address_space *mapping, + bool uniform_split) +{ + struct lruvec *lruvec; + struct address_space *swap_cache = NULL; + struct folio *origin_folio = folio; + struct folio *next_folio = folio_next(folio); + struct folio *new_folio; + struct folio *next; + int order = folio_order(folio); + int split_order; + int start_order = uniform_split ? new_order : order - 1; + int nr_dropped = 0; + int ret = 0; + bool stop_split = false; + + if (folio_test_swapcache(folio)) { + VM_BUG_ON(mapping); + + /* a swapcache folio can only be uniformly split to order-0 */ + if (!uniform_split || new_order != 0) + return -EINVAL; + + swap_cache = swap_address_space(folio->swap); + xa_lock(&swap_cache->i_pages); + } + + if (folio_test_anon(folio)) + mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); + + /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ + lruvec = folio_lruvec_lock(folio); + + folio_clear_has_hwpoisoned(folio); + + /* + * split to new_order one order at a time. For uniform split, + * folio is split to new_order directly. + */ + for (split_order = start_order; + split_order >= new_order && !stop_split; + split_order--) { + int old_order = folio_order(folio); + struct folio *release; + struct folio *end_folio = folio_next(folio); + + /* order-1 anonymous folio is not supported */ + if (folio_test_anon(folio) && split_order == 1) + continue; + if (uniform_split && split_order != new_order) + continue; + + if (mapping) { + /* + * uniform split has xas_split_alloc() called before + * irq is disabled to allocate enough memory, whereas + * non-uniform split can handle ENOMEM. + */ + if (uniform_split) + xas_split(xas, folio, old_order); + else { + xas_set_order(xas, folio->index, split_order); + xas_try_split(xas, folio, old_order); + if (xas_error(xas)) { + ret = xas_error(xas); + stop_split = true; + goto after_split; + } + } + } + + /* + * Reset any memcg data overlay in the tail pages. + * folio_nr_pages() is unreliable until prep_compound_page() + * was called again. + */ +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 0; +#endif + + + /* complete memcg works before add pages to LRU */ + split_page_memcg(&folio->page, old_order, split_order); + split_page_owner(&folio->page, old_order, split_order); + pgalloc_tag_split(folio, old_order, split_order); + + __split_folio_to_order(folio, old_order, split_order); + +after_split: + /* + * Iterate through after-split folios and perform related + * operations. But in buddy allocator like split, the folio + * containing the specified page is skipped until its order + * is new_order, since the folio will be worked on in next + * iteration. + */ + for (release = folio; release != end_folio; release = next) { + next = folio_next(release); + /* + * for buddy allocator like split, the folio containing + * page will be split next and should not be released, + * until the folio's order is new_order or stop_split + * is set to true by the above xas_split() failure. + */ + if (release == page_folio(split_at)) { + folio = release; + if (split_order != new_order && !stop_split) + continue; + } + if (folio_test_anon(release)) { + mod_mthp_stat(folio_order(release), + MTHP_STAT_NR_ANON, 1); + } + + /* + * Unfreeze refcount first. Additional reference from + * page cache. + */ + folio_ref_unfreeze(release, + 1 + ((!folio_test_anon(origin_folio) || + folio_test_swapcache(origin_folio)) ? + folio_nr_pages(release) : 0)); + + if (release == origin_folio) + continue; + + lru_add_page_tail(origin_folio, &release->page, + lruvec, list); + + /* Some pages can be beyond EOF: drop them from cache */ + if (release->index >= end) { + if (shmem_mapping(mapping)) + nr_dropped += folio_nr_pages(release); + else if (folio_test_clear_dirty(release)) + folio_account_cleaned(release, + inode_to_wb(mapping->host)); + __filemap_remove_folio(release, NULL); + folio_put(release); + } else if (mapping) { + __xa_store(&mapping->i_pages, + release->index, release, 0); + } else if (swap_cache) { + __xa_store(&swap_cache->i_pages, + swap_cache_index(release->swap), + release, 0); + } + } + } + + unlock_page_lruvec(lruvec); + + if (swap_cache) + xa_unlock(&swap_cache->i_pages); + if (mapping) + xa_unlock(&mapping->i_pages); + + /* Caller disabled irqs, so they are still disabled here */ + local_irq_enable(); + + if (nr_dropped) + shmem_uncharge(mapping->host, nr_dropped); + + remap_page(origin_folio, 1 << order, + folio_test_anon(origin_folio) ? + RMP_USE_SHARED_ZEROPAGE : 0); + + /* + * At this point, folio should contain the specified page. + * For uniform split, it is left for caller to unlock. + * For buddy allocator like split, the first after-split folio is left + * for caller to unlock. + */ + for (new_folio = origin_folio; new_folio != next_folio; new_folio = next) { + next = folio_next(new_folio); + if (new_folio == page_folio(lock_at)) + continue; + + folio_unlock(new_folio); + /* + * Subpages may be freed if there wasn't any mapping + * like if add_to_swap() is running on a lru page that + * had its mapping zapped. And freeing these pages + * requires taking the lru_lock so we do the put_page + * of the tail pages after the split is complete. + */ + free_page_and_swap_cache(&new_folio->page); + } + return ret; +} + /* * This function splits a large folio into smaller folios of order @new_order. * @page can point to any page of the large folio to split. The split operation From patchwork Fri Mar 7 17:39:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0EC6C28B23 for ; Fri, 7 Mar 2025 17:41:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 900F56B0092; Fri, 7 Mar 2025 12:41:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 889096B0093; Fri, 7 Mar 2025 12:41:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B5496B0095; Fri, 7 Mar 2025 12:41:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 450B86B0092 for ; Fri, 7 Mar 2025 12:41:49 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BFD6D560A5 for ; Fri, 7 Mar 2025 17:41:49 +0000 (UTC) X-FDA: 83195472738.27.5F7B963 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2075.outbound.protection.outlook.com [40.107.236.75]) by imf11.hostedemail.com (Postfix) with ESMTP id F2A7340010 for ; Fri, 7 Mar 2025 17:41:46 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=a3Xma+5t; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf11.hostedemail.com: domain of ziy@nvidia.com designates 40.107.236.75 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1YAH75U2IxjioX4gzhQCg62wiGb6vowPt2F3W3wP32I=; b=PeACGsYU7yfr3+U9EvVBaMaNwGn9EC3XDLfwLNNV96+v+ldF7gYNdWDpAiRhaWlI2mHcPz hvSPUmfUsnjF1pHxa+9RkNnnvQ3b6vSQM/R9akK0cUc+jGtAv5IO5+ffvyvs55imEqjge+ s10NBXgrxmsfKFFufIs4mkPtPqYE68Y= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=a3Xma+5t; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf11.hostedemail.com: domain of ziy@nvidia.com designates 40.107.236.75 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369307; a=rsa-sha256; cv=pass; b=WbBj/aUbQvn+KLJrmOt/W1FFddxcno9wviU5toUf2KXdue7CjBtxwWg4FLRoqYAu95Q641 F5lG54ebvedTtqwlCtySHqp0okw+gfQ12Zcpm/TYUou/h5jBVDSgQgrvlqzdsjBL8zJZ20 a2c9N/H9tslLcM5RwS7QogE1dMB6oyo= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JxrEZqQn+lIt3no29wsYL4AhAmjfnVuc1JlK8z8tgwmHxEPLHGNAy+GG7xTiXhkwJj+5SgQk9mZxDgnD3kyp+Zj4LIoYY6pw400iRl4IZJvaW1X4Bp5Jf6m0ousQBegqdDe39US0odCCVWMVR3sV2D4gsk2rJS0w+IzIRoHPiB/ASC8mBZBXz3HSTzfJ9v6MqhbGg8p51VIUbLTIaG+SqKiscCupQOFpLuXIJzACiYb7DFuy6NbPQZYsySzF1g+lnc0a4SGQMXrTFW2Fe/qtC3YBee4+4iEz2c5V87feqixtuSxgY2sZpamfRtNnNqsc6pGKQMiq9oCq9EtSKz9TYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1YAH75U2IxjioX4gzhQCg62wiGb6vowPt2F3W3wP32I=; b=nBZvUa/B+qemIWGkgSgbsXcxVvWiwqHnT+arHJWGy023bCWte6uGq108oxxjQjCbr0Vya/yZTuySGxmOKat6fVhweAMZP6vArgQeB+XcBMW459f9sx21mwBeeTAQNaPf3HW+X2NFmvobbh0SGepia+UlgNrLeYq08JSxAb5WLpRkXe3K+SkWTSOlDWOhdOrmzQyDBW9kR7n3KvhOavcR/1+gABLh7DL3EMC4F/JFvNaZxGAZmF2LtTFyuF3zPaerf6bvOdHVjNr8K+/qOJwAM2AOl8/j5fROWcVxGGB0toLDkY6MBbRD0ZHAVijvLxmN5U/bD2cVnp72CUV9mbmmEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1YAH75U2IxjioX4gzhQCg62wiGb6vowPt2F3W3wP32I=; b=a3Xma+5t/phmPozdwB39gPnd5TQheHcwAouOANF+h9sv+x0vPeIhZeyEVgEzM5QHoafrBLiBQpdJhJoVtbAXVYo0MCAY4bhJDsDAVp7l+8ne8OjGCwbpL4KQVM50FWaUdzZNnKEoSxUlMlOGBTMM5nesVB4t0GXLJDDLDddgO8Ko7lSwBlpThMaK5OG2nes8N0Vd7Xm5UE/srpCaIzcoTfjrXhK4adVgWMChXTnzPUREno0/jk07GZlqgkmeN19eafE9z2mTQ1Qptkrdo2kiaoFh0RcAG5reO7ZD7p395JAbr0mXFMsIEd4Igd2ZDFk9IYgGPmtUEwWMQOYBs7qDGg== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:11 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:11 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 3/8] mm/huge_memory: move folio split common code to __folio_split() Date: Fri, 7 Mar 2025 12:39:56 -0500 Message-ID: <20250307174001.242794-4-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BL0PR02CA0121.namprd02.prod.outlook.com (2603:10b6:208:35::26) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: cb2f5eac-ead8-4ee6-98f6-08dd5d9f1c77 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: RB74sPCA4mz/ktuzseFzFDZzPAKmE4mCiwmaMAdynVr7hZ5P6nNTASeQjWRrSerBXyoCBK+7eW/v0tCOvyxHliQ88FgJ8BUPnxCQqo9s/E5cperTc7E965uWPmpcdgb24btxAl12gUT8GvW0RJvDk1O0fdHcf435kFxUVvIQvHjGqu1kbki6za858UluZ7hAbExL3zbvffs5qeoKT5tkRkcZ1j5T1su6KKF3BWyn9R/Spi0yFctuQV0lZD+8da++CgaIkVtICFI1+rXQZcoLVLUGByIt17U4kvUxViiCv2Ig0jFCB8T2znILVsd0fnOlvPvq1JuEp8UEsST4WGk8233nCJjtwNRwPb/cQEOUqvyMPU5rs4uXN9JfAoaaM8bZgECfJC9lI413bObOLBsjR19+XjkLaYgPWFCNo1XpXe/+x8N1XXFbcfX64FvjyJoBBSPB78ELW50rr3o4crrSl+KBnJRln/CG5GHAf1T3E+vxEAM+qu6rSPbS9YcZwo06ygmGKWeLDpKVE//IIEIIzFu/bs4J9hE43H9pAa+lbxPilyPh+Cfuvufdp4ukzWPQci3f2aQbkAl9HoEnxiI2F2m3VLC6lQ2u/uCyLJtx/93uZuBz5F2XnO+dZgPE8RsFkC1z2ARvz3PXEfaii7w06wueiD9aMXFtK2VuYOZb/sdD6x+cS5iHBpiVckfdWo4vrTflBijlg/VxVk8uMzow54DV3X5T992Gx70xUgr53q+MB3MdafWe77w1wwe0qYNGmctQ7D4RIKiccLE73RXHgjVnDN0yXpnt5l7SHsnmOuXtOhNPe8CwRNzEFWdw+UGN6PIHy/pbIo6ShYjdGyAG3ShrMO02NQavocjK3w3T6hOCFXtCcYZhZknqdb5fqFI+xldA7y2BDupHn2YDvXQLdiFZYeDPazSG7EieYhIsblsvzk291r+30VdqVnBXbDd6jIlbqkxFVvMYZEVMQdsSZZYnus3J2fKERVvZC2/BMDkr3xtbB13Vwrp4zQumM6RD02vuO/h5UR9ysOlpXBSFk17ZOlYu27lAryO2VXixIvDe6pve9GOGVNhMs+jpwEIO7gBLAmCY5j+KlQri5K4T0ohHvLFIb3fC4MoVMRagp5P7tcYIHzReTmKzGyNNCv5snpCH8lbS+B/E5MF70uU1eyBOkGBHs5uszIqrn7vlSGfl1hJQlhIYMcmL7HHQCsUxJmsBRmKxQVB35vcJ+Q2j506/iMmwE05uLQDcjsxitKegVO6RHMWzoUATFUI3y+qzXq5Jhimcu7/ZgDqotFwBoP6VHMjgObWfPPXQLaLW5aIQEYNmSAu2P0EbIu6YCzsiESn12A0O1R6ugpgjYDRpFPQkMTgZ1WzMM20Nw2yPfl/ruQSA965wa9+X7uHCiYpt X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: p8QkKvqInpeUq4+oe5Q1vIPveV9cJS86x7tnmqHE+gvq4tE0pbCmOhsK+DKgAb1fyamA+QkgXl9DtUZ5Or1CoDWJdLmCz84/4r2acP49WZEkIYz6jR+Q63QmsCYpyjpsBUOvggWoV2lNoDBshfw4d1ldsywgH1Fim7CEkGyfKPvylyfRWfcHjRCzQfQuK5hTQbiDA9yIjIcO8ZeKTtsLQJRe5sweqOzWcySn2crT2HOzVAzh8u0f2Lyts2hkjytQy9B9CmpgP1Sc6J7ncLqyiYlrHaXz9/9A7XyRpiMbbvQBu9Y0prIbPt9rcGey5LwA9pwHK+lsXuhEVBPwDZoJl6UwHpXWWhf8DmoTHuthmsRx329nNdOEPpH+BRzCrtkmeBC89Xjo834BRECabRNbs88Lgu+cUobraVR6Ocv+FRbLUJd5Bhhb4bwu9f/xZ94Mgd0JZiOGyaJJuKE1vCYmBvxzRs/hpEJN0N9cmIxjd6XnSD0/pajFVq+v7Q0WSVzx2S5HGwXRYQ6w6SW8ua6jho6D40YhBRwGKgzGc4Ih5Ki6OZvrekw0858PNkgQhIOGubFaOoTGT6W5LjTLTW4hu/hLILRy0b0Mk7WLR5b51dCYQE5TbsOQO4/LRiSSzkvs6OPJTFKPVv2ePevx/6ApqQTRtrkME0of1/ZhGjKa3l4OLqulJJ8z8IbjAoXC2OG+gq1ylCEikVmXQniuzmaS+mCZfXvbyIDzPwIj5hgqjJrUg/ovnj7tHsVxJDSCTZ6Cbvz/lCZBybXYIyOLD60z/4W90MfpTaZT3qgM43lVzOHihwnS/A8o46h7zj0JbGlRwzwxnGAXT/gYIzlirYRhC9zh77jVjiddxzEpdBpNR9kxqhCJNL2A3EWgAH+RbNZhs304KYxA0MM8JQmn2wOs1zv6rU+qx/spIRBYnJfabpz+6Zj3cziPAFqXJpOgMxn6JFouAlosHuQxlGQeXmjIG+dv1dyhamRPPWz5Clw9FGFWQz+yOB+nxCOWJOWxMJnIqSDJzzKprnm5IPWaQX2/o8k1oYEcP7BThSNNXqq/pNTZS9OlOFiHrRZ6w5u+nxKr1Fidy2M2rsFI7SSc+qb6FdazeK+p1lTM5ftDRNwtLx57lw3mZFegkdA/lZ9KpJYOi9wEwfXqJiD7Ixb7p/QtXokUG1xgPLUxlkj3yFFj0iazr5VOUmy+LUFZTw/zXI76KMmOFmYF/LwxjxEdGXTWW6P3/e96Q6ifkFpeqqVHjt5ubM4JiVpLto4URy5CTO7IeGYSUJSmZ8NH2C+rLGBFrZHOTC3rLDUJfD6ErTgQvTbeO2qzpLF3wO/RQgYR0tAQ+FZsTshYn01glkZavOZVIYpSKwGhfLqRmEIGkGqygrSzUmcAkYfX7jhTx8vxik8VYGjMbn95n+l90ubVR7BwPLdB/PTshooyGfhg8YhGrw52kKrTAJiJ3fbKMNy3rTVmh4deXLS0qv0SB0E/u4Sfauqv7Jdo1vZxs9DOXtlUr7MehdU7GH5vuGVRWlett7+HcR818+A8CIQ1I9RTNOT7Si36jbdF9WYqo4/id2Xb414= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: cb2f5eac-ead8-4ee6-98f6-08dd5d9f1c77 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:11.6861 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5hoik0q7OdqE8yX2NdH2PJKXebYK+Ry6oFLMpyHGtIfVU8/cZlpFh1tljOpNz/Zn X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F2A7340010 X-Rspam-User: X-Stat-Signature: 39rgtescw1jw1hxsry75ouhum5jmxf63 X-HE-Tag: 1741369306-210410 X-HE-Meta: U2FsdGVkX19EMu5G3nLHEgk/nnij8pbq/ug2iXtXCsVT2i8qrEZ9eJYojNYpaclDzlW2c85msE9UDhosY9YNtZF29FSShXvGz7PlnyrZ9wx1yGQWMAHX0caPMJY14af0inwy38OYofSHzPa6q/sC0SFJDUr39RrUSzxBlruKjTrBm369S3cFl8nywU3dKNOLO2ZR18S8hEdG/rHLB+hYNFqYdsM2SIXwF8Fv0fT7p/D+Vt0m+86jr7Zx1eiSgnE4MaD3Q3NICBFGYl+U0aDYO7q1ZkWYqxDacbrvrloq7gjUOcs4pFRximVndbQvWqXR9JV/KLcE37a3Mep6dQpXWE/VHOzI2LbPPlXZEe7z+FluLvhaSzbMlkb5pBeBGFLbQQj03W52qn3oHfNr5LA4grsaVRfLHL2WSNpFwyla/A/PENa0RRBhWBntFhtWvn1JMlxuAXN2d7rtPTgkEVm4EOXlz5yXhZSETlJj+Y6hwEPkLZ8mpTj/n1WqsFsvprDnoM/7wgE9llParvjVaNMTKsQylfXAcnY9/PQG8ha0J2UDYt9F9s2D7ku5UoDn5ZBW/5594V/U6WIzHFslTtwtGufzqmqLbuWIuJL4pNm0Z3/nkwqUqRR2XONHnlg+FGyTWd5i7wwtNhnfmj7IjxrTKw6onz3SlDAnyLy6jp5JvpqcuQM2KsWpLpIOJe9C5xBXZs/zOrRKyJtZS48igrvCAJmXSaOmnLidmYmCUQnC3hJeuQOLL1pvcak27zS4oT1DMnLf9vdgTe2nHqMzGo2uuoTKg37jvkMyTGKhTJPARHlm8mrj2lAZYEAUmuir/IaXikGkBinJzBw1b8BKgG09S8/Ew2hO7N1MNlHgs71vR/95E7AMA4eWJoeRb3RfjJtXcvKnwmAZxy+HDLY5zqFIEc+QUrMayQ8Ek6nFVbdiHdo00unibiTaxR6O+bckZWAXGhqnpF+PO0GZbEGuZrO DbiLGKUD BmwGrTjrqJH32rcceCxW09YvVi4fGf8dpvicfgf/Z+9HUqCjcsq9ZezDDST/GIcunQeWDOcsPJRV2x1puZ5PpgQpnw41cDfjCQ4YeiYHvUi6JVoKDz48+/oTkzTCtjLt185bWfR3Q5mGIWzJqgNUKKXEHOfB2cZIQ9Ar5cGVGVA7VzT978NcNWju+WVJMGBo8sFOMesv/9mqeFUZJXwgOb2H3lRqiGq1IUTsORmFtQGVJQHP+PyVT944FJroRbyw+YpOrq1mxRoXJmJcoEsWtOb+e6zINUmq3gBVTfuMyCbmvn0MJI/8S3ttN1qEHXMWRP8+ZzYrIh5c3io80huIdg5UoRHCcf8+D+4Ya2dA/MiUElcw5s8/2mMHpbqeUDa2ySxc9DKZ1XYKygMc9HtNcptu5cQuGjASpvNhomRp+KeMVTDFUQ7fE7zg/3seIMisUcGzMrfCw7DgFX2Caz2qe86a0VS48IKc2zUoBOFzED8EK9bIoSSONTknKNTuNeKaVMyhWmLO3CYe81TaQhIBu1uc/1bmkf8vSxeQpB7ckIG9xdaqs+8xzm+cSJACRwEsH9FYc23YMPckrEryR4Ik2ytuTvbSLmQU5IF+IQdpQdb6DqeSnDtTii1HD9w/5VybqP0y7Zf+rU7gl5+CEiZuyIyDg0XqNTmfi3wNq4Kue2VS4RfoqDlffE2g5qhBgOTO8vlX9MkUGrRG0uy3y1MvZHTu8V+ihEfPTijwGd2BZGXGhfU3lsSRxV9npgA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation patch for folio_split(). In the upcoming patch folio_split() will share folio unmapping and remapping code with split_huge_page_to_list_to_order(), so move the code to a common function __folio_split() first. Add a TODO for splitting large shmem folio in swap cache. Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song --- mm/huge_memory.c | 112 ++++++++++++++++++++++++++--------------------- 1 file changed, 62 insertions(+), 50 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c10ee77189bd..871c260163f1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3863,57 +3863,9 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, return ret; } -/* - * This function splits a large folio into smaller folios of order @new_order. - * @page can point to any page of the large folio to split. The split operation - * does not change the position of @page. - * - * Prerequisites: - * - * 1) The caller must hold a reference on the @page's owning folio, also known - * as the large folio. - * - * 2) The large folio must be locked. - * - * 3) The folio must not be pinned. Any unexpected folio references, including - * GUP pins, will result in the folio not getting split; instead, the caller - * will receive an -EAGAIN. - * - * 4) @new_order > 1, usually. Splitting to order-1 anonymous folios is not - * supported for non-file-backed folios, because folio->_deferred_list, which - * is used by partially mapped folios, is stored in subpage 2, but an order-1 - * folio only has subpages 0 and 1. File-backed order-1 folios are supported, - * since they do not use _deferred_list. - * - * After splitting, the caller's folio reference will be transferred to @page, - * resulting in a raised refcount of @page after this call. The other pages may - * be freed if they are not mapped. - * - * If @list is null, tail pages will be added to LRU list, otherwise, to @list. - * - * Pages in @new_order will inherit the mapping, flags, and so on from the - * huge page. - * - * Returns 0 if the huge page was split successfully. - * - * Returns -EAGAIN if the folio has unexpected reference (e.g., GUP) or if - * the folio was concurrently removed from the page cache. - * - * Returns -EBUSY when trying to split the huge zeropage, if the folio is - * under writeback, if fs-specific folio metadata cannot currently be - * released, or if some unexpected race happened (e.g., anon VMA disappeared, - * truncation). - * - * Callers should ensure that the order respects the address space mapping - * min-order if one is set for non-anonymous folios. - * - * Returns -EINVAL when trying to split to an order that is incompatible - * with the folio. Splitting to order 0 is compatible with all folios. - */ -int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, - unsigned int new_order) +static int __folio_split(struct folio *folio, unsigned int new_order, + struct page *page, struct list_head *list) { - struct folio *folio = page_folio(page); struct deferred_split *ds_queue = get_deferred_split_queue(folio); /* reset xarray order to new order after split */ XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); @@ -3989,6 +3941,11 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, mapping = folio->mapping; /* Truncated ? */ + /* + * TODO: add support for large shmem folio in swap cache. + * When shmem is in swap cache, mapping is NULL and + * folio_test_swapcache() is true. + */ if (!mapping) { ret = -EBUSY; goto out; @@ -4123,6 +4080,61 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, return ret; } +/* + * This function splits a large folio into smaller folios of order @new_order. + * @page can point to any page of the large folio to split. The split operation + * does not change the position of @page. + * + * Prerequisites: + * + * 1) The caller must hold a reference on the @page's owning folio, also known + * as the large folio. + * + * 2) The large folio must be locked. + * + * 3) The folio must not be pinned. Any unexpected folio references, including + * GUP pins, will result in the folio not getting split; instead, the caller + * will receive an -EAGAIN. + * + * 4) @new_order > 1, usually. Splitting to order-1 anonymous folios is not + * supported for non-file-backed folios, because folio->_deferred_list, which + * is used by partially mapped folios, is stored in subpage 2, but an order-1 + * folio only has subpages 0 and 1. File-backed order-1 folios are supported, + * since they do not use _deferred_list. + * + * After splitting, the caller's folio reference will be transferred to @page, + * resulting in a raised refcount of @page after this call. The other pages may + * be freed if they are not mapped. + * + * If @list is null, tail pages will be added to LRU list, otherwise, to @list. + * + * Pages in @new_order will inherit the mapping, flags, and so on from the + * huge page. + * + * Returns 0 if the huge page was split successfully. + * + * Returns -EAGAIN if the folio has unexpected reference (e.g., GUP) or if + * the folio was concurrently removed from the page cache. + * + * Returns -EBUSY when trying to split the huge zeropage, if the folio is + * under writeback, if fs-specific folio metadata cannot currently be + * released, or if some unexpected race happened (e.g., anon VMA disappeared, + * truncation). + * + * Callers should ensure that the order respects the address space mapping + * min-order if one is set for non-anonymous folios. + * + * Returns -EINVAL when trying to split to an order that is incompatible + * with the folio. Splitting to order 0 is compatible with all folios. + */ +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) +{ + struct folio *folio = page_folio(page); + + return __folio_split(folio, new_order, page, list); +} + int min_order_for_split(struct folio *folio) { if (folio_test_anon(folio)) From patchwork Fri Mar 7 17:39:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E683C28B23 for ; Fri, 7 Mar 2025 17:40:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54A226B0088; Fri, 7 Mar 2025 12:40:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D21E6B0089; Fri, 7 Mar 2025 12:40:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FDB96B008A; Fri, 7 Mar 2025 12:40:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 09E4B6B0088 for ; Fri, 7 Mar 2025 12:40:29 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1C341120E95 for ; Fri, 7 Mar 2025 17:40:29 +0000 (UTC) X-FDA: 83195469378.28.641B0E7 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2087.outbound.protection.outlook.com [40.107.220.87]) by imf24.hostedemail.com (Postfix) with ESMTP id 0CEA918000E for ; Fri, 7 Mar 2025 17:40:25 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=KnP9aI11; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf24.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.87 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vyxwjhNAe16++5qZINdjjuM142tYqzuQZ3aY+eriH5s=; b=BmPY/Qx+V3kUd48gX7eweG5dVjBT+ZHsRcMS90YPmIWOMXsUfCRXblb2VhO3rvyvAqsvkZ tZHb8Fi3r9DSCrRRpIxVzOoBwtsLRAo1jI9yreev7j82Vq7q3A0Hz4KBHtmKQw0yOM17Uk NRXYSBIyDb/GhYKNX9b2TKISGNARbbI= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=KnP9aI11; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf24.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.87 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369226; a=rsa-sha256; cv=pass; b=p9uodTdWJE1F68Ah52Bei2fffo+ybPfKtNWYvm6P0Avoz3yFDIZv0EpS2GqOirA7m/wYs2 /BBTbn8UpBXyi2q3Cl1HJIAsQsG009YjiXhRzFWvW5WXqwCPAIkS21d3RfAsJwS3t0Uyjz kvSxofYDtIrknrCkRcMDtuFRFgifciM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bmQyDPT3LQm/9KrJAvVOpCgaBllJl7yMn6m+fa77CKotpYh224dQ71TnQJhmDyuDecyIfvMhcqZlY351dI+5uNL2DfFqUvQCdpXQUe9lbQsODneIaqzTm5fB0e8wORoSMeqfV5oOZCKDAkykvqtJfsRiSLQe84H5Cave8U9u6E4G63VcwVUTOcR4Uqolw2bs46fHPBS0QqV0drUx+31dZJDJUqqxe825+ww0s3BE9OfVyxxpHjH3stY6gLuTTYaM8nuJAGi2bT5muqV6fdU0WzDL+H2le1UBb96fq1G4gQXgDF47mTrv0kZKSIyGzUYCfP0awG9k5dfKTZi31RYQBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vyxwjhNAe16++5qZINdjjuM142tYqzuQZ3aY+eriH5s=; b=kK0Rsvg3iMPAZ662JxFofo8mQ7K70uId9k1aTiWCG6w9Wce/XFHDCakmwbjPY5W0GacfuW5BBeOCm4u21yDfGM5boVv5Fj+0dO9jvfNWJgN40oknjNkZJGd27MzZjZItij9uhr0I/GhhNZdPrVA92H+k7BdI9MU3tpBBFLo2jRI0TdZdGfax0J/cFBB20qCgmytsvLHT+ZPOKbgzpyMKhGtVnAk6JpVoaqxGMj5tuzXtA78NvZdyCLRqfwyMD3n27mlsvoeOGlD1ZFTF6IDmMeTJEbbP5+Sh3maRiGd7XdvN8Efm53fVK8dePmwHaQh/PwF+G9lkhnnDGXENILkKag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vyxwjhNAe16++5qZINdjjuM142tYqzuQZ3aY+eriH5s=; b=KnP9aI11ddoBKj3nOIuDI+qeM/89uXcXMXxSmEHPDNzn3E0rtR2LfYJjjxe0DYeDz+SnQ8FEk858OhipXEVymrtOJRycwgXt0wcSlGif5kl2KSyJgJLlhxVkDyXh2CMQD4Ut7BS5ByMeYpyzS/wJw6w6bJIg/kApCdxqd1+hJCeRF2Ara4mzll0c+hgx+Vus+WILJCpZZNUurNpOWoS/HGQTPJQYpZyvLiJHF7E1TksNiuzjIW/iP+AMTdrMYDdzIvlIEQdiTno7deui1dBmmGIqXwMKWFJHee9nkaErXARCxIe7RdyAUDYZwC/axb3muy5bfLjtYaGS5UOA6RVc8w== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:13 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:13 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 4/8] mm/huge_memory: add buddy allocator like (non-uniform) folio_split() Date: Fri, 7 Mar 2025 12:39:57 -0500 Message-ID: <20250307174001.242794-5-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BL0PR02CA0140.namprd02.prod.outlook.com (2603:10b6:208:35::45) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: ea8ae57d-d34e-442c-848c-08dd5d9f1d56 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: ZQb5mWKJ6O0sAPOcuUZfQ9mxi+u+K8fNTRyNSvf0uZbq/56Bs9hZusIXnCNWwElxyB6R/u7oVgBrOfceFYMo0Hhdo2uq0yyDeFzftWsMYak1BE60fFJd45aEFjyeJsNqcGgnFlj6zeGdaq8ZJmsBVir+QHFHPiljcoIkQ05+iTfH8jiiUF+RLHG0Vz8tq6py2g0lwALeROYAEoEgoFKpm87PaxxbxE5kGhRPqsmgG/9Nog46VS4CZvx/9S2ZQMrPuaD0Q/5HnTjGahnUlxL4F8mVk0pZefJ4nCUA3E5Htix/eZX3yY8SLuGKSfED5HJu9k4cOZj3nCfd4TnjoYas56qxFHj9bF/bkBVU/JW/aO3frfpJF3xWDs9Rdzxub1A/iiS1lQ8AWu7yEPOwxtoyUNe1+jahQHOGLNwoVU2/y3cObhqiGfK6rxD99K3/AQnm4jDilVDCP5m53ykB5Z+kk3KiwGR16TP6O1SJn7tkOjPeI46rZfGN1AK4GooDHkcW72x7Rry1Dsg6VcKvCdQQ9HCPPSzcjRmZpvFUSu9vrfZzbifaDNHAaCeBQZnvpx1qAkAQsKUqN1RfUQXmrlB41VnK8Bif566KshtdLlhTvNzzSoN55Jz3MAitgTFiZRUHNXXSxNyaHjDmsb+sDpIgDhx+B5W1Nwgo9oUZ0CBgxy86lcPVSgT8B34TtevnaeW4hFFjwSV+CvO1Pq8ZPqnfz9WEPE3jAl53D+n4MtSbjNdMXL6IsPIwXyJnViUIPuIXEh+HeEcTzxm5FBcFNn7qrEPpjQFiFiMeL5sRaBmnYXhehJGL1kfEEAQRufAM3Ik4dNDZIGsXhIwQrHhUBjvvNA9p8vZDdvpN1TEQMu0f7s0xptFDttS5Tcb6GZ06EN5lHubZ7O+HFpn1BbBfy6lF08eRkxLj12vtQUl9bEi4S3w+41FvSFbB4GsHh9Ql2kE1mCZLgogFHsGRn5SfwpUziUuI0LmxDnuFUb6WNdTqCm79HSFfQHtjXLJIG1GA9KjyizwXEmGBq1RjWDtoDTUdLcQpWwaKrp3WzxbzzWmAfsf9lBRKaMqM8sTt2bpO2kCVGxmvp0mZNo0AFu05E9r3bl1/P+yTexVn7cP4augRsIZUtEEZnqO2pHiAnr224nR3lUmw31oSqKTAKRm0O1VRTMGRiAw+Jr6GmBRE1wctigu32tf4DOUSMjK7BetmHU1hCMXxBm5agwL5Sl95j6STz7Ep/9DQHsm0G0290Q/pZNfpzd0ngr+ZWiKsZgyrjD6YYPSuKtjYq0zYB4qZeFVaHjtcyVHIfSvcudYdVZPTj2JAoin2lQt5djChIKjbrKqVxHah46eRi8r97GHt9kmT0LPmWQzWgWwGj+BPhT5xdQFPOAPhjHMAt33CeN6xgh0s X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: r4VhafHT6sCbbV8qKWs8O76MEEHcnzWXTI5lZ8v9NNHy0MLk3Chz5fAh5rIrpDEiZ01LRrhf0kNvzXL8Uj4XdOHPVf433ZDxzwQsiuT33WwOZft+H6yOZacaoyv4Iyfxy4pC50HWogU8ZghkmqqobVfchOhLyqDf2miPdEmY7XzYwy5OVskHd7iSaVRenXcQ44Swp6W8BpUUlYSXkOsLEZ04XzeRW47IrBjvWOQ/b3FPPMAxgxV6p+J34NH8+7/G2MwEMEuzbkdETlEwMsxN0DljETwFrKqLsjuRrPVMVjhYkaaveRqgtzZfemvlUygi/m4mqlCzrPYQXHpC+ooMYcEdtUeo2rxa/F1hifGh0bbj7AM3RwTPH7HX8NPUaqzmtPbbDnOslp/3DEjOJxKzymZEE6FutUrjNFC7RokGlxTESL4Zw+OSTQ9yk1M1eJOR4OfVnXU7/Xl1X+6jTLGG3G8tMnaGWh9pXrRAB5rFCeE6hcxIhT5dJE6f4U1IQcFY7NgZ/X1ltb7qXP1QEocw4JVcgzwbEFTx/KmqsEQTibC2RLbfm01q8vCh7Jx+1F39THE7aQ/2T2B+/wLGkCPTot3x733W5dThlMHIhyNGSvlN2fVKBh2yGbK0v3or6tHkC++52VwISnRvXDw1uiNj7SBTp84zkekkzrzqh4p0WZkL27k2f6XO1JHF9zPLjsbSjjKRN/TAtcygU+m9ubT7FyulS38f50QIZb4hcRoX0n/9SG97X06VYVCkvb4sxXq3h8bgN9I9zUhVGL/YHuHuYfNpQGj7pT7xE18NblKxrXKtgpxsnmy7KaJDzPakFxIvpd9tccM8nUqJxycEcwSVwEjYF7PIkoA3YxWFcarKcjSI5N0P0cb1QP5KL3cCjIEG04Jf6eoE5n6lRQO/4wX1LNqVQ0F17tL3FfA+uFcyDtg1rMcY0zLaJWloLS7reDcwODeLis1k+NcL8kOwWNCx5YQBUrL53RMyWY/aO+n7DA7auoDz8HAXk7ETqZhhbq/pGimaMhvfIvTUnI8PqC9pkKupreq+7d5IbkK18uTzunHzEB7FX9ck6uEJSwvIHRsIRvfBnFqBCbUbOa75GD7Q+Hv8GcbWLbrmUccoltiqBMVyVLL8+SlQ1L6GnoHGZrhaWXGpEI6DWcQjUcOgPlo3WPDjsaCeSF5B6Q3FvppKfKaLF18HZdQxOxhK93Lyutzioip6MGm+WNYd+uuRlMs62bT9Q9ju3hkcf3LPfzEHso+Vxn4VrxhMevlMidhDDPIV6kroeOtmzBDECzB4oqDHQNTO4nzbc9ZbRL1KGxY6GMm2BEHAc9T+OI9fPQdexxt0bxQmjc3wjwNSNraDnWBbUSTbh47/TDGvko2zDsQns0B/46S+DWrOyuhnqK4Ig9cilV5DgJPGeQ2pIjEOdr177xYGJ6RN7KJBfZjCkqitN9voRYwRT6rdfdMqr/5I9mCTb63WUTlGPiT0cAwdbYQSjsRG1YTXanLywmqoYPL24xQiqOlks2PdDRzw89BJ8oBlPYAcGRs+CJ8t2WsBGFugY49xxV5mOsl7NxbwX9KPeis= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ea8ae57d-d34e-442c-848c-08dd5d9f1d56 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:13.1924 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JSvtka+1DE+TNO2ec5Qa4JtwRQbBr4tQGcvi1XxqrGGur+jtIekMMhvbBDuB745d X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Stat-Signature: s5nbzugtmygkyzt5hwcxedohi93bjdih X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 0CEA918000E X-Rspam-User: X-HE-Tag: 1741369225-475434 X-HE-Meta: U2FsdGVkX1/Xr+mVeXcRAtqwrx7ishrXQ2BBwJH20W2KITdVwfZhc5XPT/IHIuddMfMH0p5uu/AHWutZivULo4fnQa251lyHE2euUO2XqwTWcZNfDl7YiSBQhqeWS82QM0TpO1gsd+AhtQ1gR40rAivu1xC/ZAM9Vt5+eNSKmVqrV9aiz6nrcKZMkoAcqecEaQAFKKwZeTNL+1pNq0QpG/jxK0PoXd9cyPQuthdMgxeBbcuBRfvHpPVOp3w8ZZNfqW/E7i2Nl+TePI/vqVTyegvbHgtzxjbnSO0EpYNWlT+o0699yRi/Tl+HhYsMlHjGrEQT+8vpp538LVSyWQKZNyrMAsOEQOQ1ZfqNJ5YzQpoVXZkeC0xX31zyFVdoaP4u5GsiiYGaqgHlPUDhvo2uCQdUIbZSjMHyroJpx8fq6C8vNJlwm6a2S+n3C64tJeLZmh4CiRCHERwk+Zvl4hhS4NyCqFVeKKz+N6hyXmJeflG6I92wYMk5c05K+LsOgDQufV6Y1otdoK5sitLVB8d0fNLdeWutigZNy8dxrJeYq8siGTcCbgh6cZQX8HKuYwSd/PpHQlodCu2jNOd+ermDmceIx+6dfeqL8pWP/W6v6FTZI9GCsBTNRFFbPfv4a6kFkIDsc9lAWZ4iyA0aRptAiu1BHW/E7wJaUnekiVUvjRQikLfKfztw+kH8DXmEwPrgMmJL/f8EJXaALiZwqL7JtojJXh8SyZ9iBMhQR1oBW+a28ZsFX+vrhaVYv1btmKPQDAbTl0D0CIMg2k0jAskqK74wLBcEx+Dtfec8gLItpXfg8YcizHfjJDWbh5RfgD7x0P6v83+WXrjrZ147BPomvDLtVvFwXIW3lJzaLilaYusdgThfPBwl/qyKukwHDBeb8tKqfe6x8SpG4uOrZuGPBuvo4UE0Bk7c2vClbv4A9bgSuzt/Uj++ZD5VQkGwh+i8DGdYF9Dnkapf6hTq4Vt OSc+QKVb KbLwOpik7DeHwYIfEZDZlvsKxac0OSbdLeEAxc5KSZVBhMXtJcYXQZ0I+tGQnw0tq6ooHkGzHkfzBzBLuMTCZUoc2l5sbzSd2QTLHB6fQFjwwESRZwejeXIauoZ994ZAK3WIoaDjzfDUfoPrEUKCMS7I9T0ykOHq/ybCi2DNZFUXiJou1qhCxGXTZ02JFyojUY8IGVok+PtbpcKXAWUYe9tneB/Oad0y35w3S/RvWAKerm0fRv5QpmRyaLZRNW5lTTOgC3hiljDb8w4ay9tRSJGckOUDcJxFx6GX43KTUVIrzsvAKH70clFwtAPlW2oe2a1TPi6n21QLOzZC8Gx+xPYCjfDUweLRZLpUmyQ1JIHFWoP3vxAOYC0H41IJq/YdCvf2kHpB4WSNFJvHNRHIv5L564as878CEkBRsEch7cYtfU4Be76jboUOKUZnSzs+LpJjGHu4j2q3iI4OtHJVQjwJ8IsWQ7Q88TDdlvXnHbKZM1qB+w75aJ2xRMTarvVJYnxWjl2fCHM2AfjDUbfPlkEQEmWlxww1fI/KOGmxSsRC7cho2vr3AkxueJW5Do4AhnZNre/Itj3ZmLc3ejiS68cjAHanbv998ok/8zIPe7DOAgtUuM4eDM8nozOYhx/iJ1YRmN/mj3X4Kvb8e47jpGbxRCOGJwgMskz484QQuTX3eiNZXt66UeAAgvGwoqHhbkYTWnLTd2lv4IWcunXjlqgXDiUrG7bYab97IAtScHRRoNuLHycNPofQR7g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_split() splits a large folio in the same way as buddy allocator splits a large free page for allocation. The purpose is to minimize the number of folios after the split. For example, if user wants to free the 3rd subpage in a order-9 folio, folio_split() will split the order-9 folio as: O-0, O-0, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-8 if it is anon, since anon folio does not support order-1 yet. ----------------------------------------------------------------- | | | | | | | | | |O-0|O-0|O-0|O-0| O-2 |...| O-7 | O-8 | | | | | | | | | | ----------------------------------------------------------------- O-1, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-9 if it is pagecache --------------------------------------------------------------- | | | | | | | | | O-1 |O-0|O-0| O-2 |...| O-7 | O-8 | | | | | | | | | --------------------------------------------------------------- It generates fewer folios (i.e., 11 or 10) than existing page split approach, which splits the order-9 to 512 order-0 folios. It also reduces the number of new xa_node needed during a pagecache folio split from 8 to 1, potentially decreasing the folio split failure rate due to memory constraints. folio_split() and existing split_huge_page_to_list_to_order() share the folio unmapping and remapping code in __folio_split() and the common backend split code in __split_unmapped_folio() using uniform_split variable to distinguish their operations. uniform_split_supported() and non_uniform_split_supported() are added to factor out check code and will be used outside __folio_split() in the following commit. Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song --- mm/huge_memory.c | 170 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 128 insertions(+), 42 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 871c260163f1..3e05e62fdccb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3863,12 +3863,85 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, return ret; } +static bool non_uniform_split_supported(struct folio *folio, unsigned int new_order, + bool warns) +{ + if (folio_test_anon(folio)) { + /* order-1 is not supported for anonymous THP. */ + VM_WARN_ONCE(warns && new_order == 1, + "Cannot split to order-1 folio"); + return new_order != 1; + } else if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && + !mapping_large_folio_support(folio->mapping)) { + /* + * No split if the file system does not support large folio. + * Note that we might still have THPs in such mappings due to + * CONFIG_READ_ONLY_THP_FOR_FS. But in that case, the mapping + * does not actually support large folios properly. + */ + VM_WARN_ONCE(warns, + "Cannot split file folio to non-0 order"); + return false; + } + + /* Only swapping a whole PMD-mapped folio is supported */ + if (folio_test_swapcache(folio)) { + VM_WARN_ONCE(warns, + "Cannot split swapcache folio to non-0 order"); + return false; + } + + return true; +} + +/* See comments in non_uniform_split_supported() */ +static bool uniform_split_supported(struct folio *folio, unsigned int new_order, + bool warns) +{ + if (folio_test_anon(folio)) { + VM_WARN_ONCE(warns && new_order == 1, + "Cannot split to order-1 folio"); + return new_order != 1; + } else if (new_order) { + if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && + !mapping_large_folio_support(folio->mapping)) { + VM_WARN_ONCE(warns, + "Cannot split file folio to non-0 order"); + return false; + } + } + + if (new_order && folio_test_swapcache(folio)) { + VM_WARN_ONCE(warns, + "Cannot split swapcache folio to non-0 order"); + return false; + } + + return true; +} + +/* + * __folio_split: split a folio at @split_at to a @new_order folio + * @folio: folio to split + * @new_order: the order of the new folio + * @split_at: a page within the new folio + * @lock_at: a page within @folio to be left locked to caller + * @list: after-split folios will be put on it if non NULL + * @uniform_split: perform uniform split or not (non-uniform split) + * + * It calls __split_unmapped_folio() to perform uniform and non-uniform split. + * It is in charge of checking whether the split is supported or not and + * preparing @folio for __split_unmapped_folio(). + * + * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might be + * split but not to @new_order, the caller needs to check) + */ static int __folio_split(struct folio *folio, unsigned int new_order, - struct page *page, struct list_head *list) + struct page *split_at, struct page *lock_at, + struct list_head *list, bool uniform_split) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); - /* reset xarray order to new order after split */ - XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); + XA_STATE(xas, &folio->mapping->i_pages, folio->index); bool is_anon = folio_test_anon(folio); struct address_space *mapping = NULL; struct anon_vma *anon_vma = NULL; @@ -3880,32 +3953,17 @@ static int __folio_split(struct folio *folio, unsigned int new_order, VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (folio != page_folio(split_at) || folio != page_folio(lock_at)) + return -EINVAL; + if (new_order >= folio_order(folio)) return -EINVAL; - if (is_anon) { - /* order-1 is not supported for anonymous THP. */ - if (new_order == 1) { - VM_WARN_ONCE(1, "Cannot split to order-1 folio"); - return -EINVAL; - } - } else if (new_order) { - /* - * No split if the file system does not support large folio. - * Note that we might still have THPs in such mappings due to - * CONFIG_READ_ONLY_THP_FOR_FS. But in that case, the mapping - * does not actually support large folios properly. - */ - if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && - !mapping_large_folio_support(folio->mapping)) { - VM_WARN_ONCE(1, - "Cannot split file folio to non-0 order"); - return -EINVAL; - } - } + if (uniform_split && !uniform_split_supported(folio, new_order, true)) + return -EINVAL; - /* Only swapping a whole PMD-mapped folio is supported */ - if (folio_test_swapcache(folio) && new_order) + if (!uniform_split && + !non_uniform_split_supported(folio, new_order, true)) return -EINVAL; is_hzp = is_huge_zero_folio(folio); @@ -3967,21 +4025,24 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto out; } - xas_split_alloc(&xas, folio, folio_order(folio), gfp); - if (xas_error(&xas)) { - ret = xas_error(&xas); - goto out; + if (uniform_split) { + xas_set_order(&xas, folio->index, new_order); + xas_split_alloc(&xas, folio, folio_order(folio), gfp); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out; + } } anon_vma = NULL; i_mmap_lock_read(mapping); /* - *__split_huge_page() may need to trim off pages beyond EOF: - * but on 32-bit, i_size_read() takes an irq-unsafe seqlock, - * which cannot be nested inside the page tree lock. So note - * end now: i_size itself may be changed at any moment, but - * folio lock is good enough to serialize the trimming. + *__split_unmapped_folio() may need to trim off pages beyond + * EOF: but on 32-bit, i_size_read() takes an irq-unsafe + * seqlock, which cannot be nested inside the page tree lock. + * So note end now: i_size itself may be changed at any moment, + * but folio lock is good enough to serialize the trimming. */ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE); if (shmem_mapping(mapping)) @@ -4035,7 +4096,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order, if (mapping) { int nr = folio_nr_pages(folio); - xas_split(&xas, folio, folio_order(folio)); if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { @@ -4049,12 +4109,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } } - if (is_anon) { - mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); - mod_mthp_stat(new_order, MTHP_STAT_NR_ANON, 1 << (order - new_order)); - } - __split_huge_page(page, list, end, new_order); - ret = 0; + ret = __split_unmapped_folio(folio, new_order, + split_at, lock_at, list, end, &xas, mapping, + uniform_split); } else { spin_unlock(&ds_queue->split_queue_lock); fail: @@ -4132,7 +4189,36 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, { struct folio *folio = page_folio(page); - return __folio_split(folio, new_order, page, list); + return __folio_split(folio, new_order, &folio->page, page, list, true); +} + +/* + * folio_split: split a folio at @split_at to a @new_order folio + * @folio: folio to split + * @new_order: the order of the new folio + * @split_at: a page within the new folio + * + * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might be + * split but not to @new_order, the caller needs to check) + * + * It has the same prerequisites and returns as + * split_huge_page_to_list_to_order(). + * + * Split a folio at @split_at to a new_order folio, leave the + * remaining subpages of the original folio as large as possible. For example, + * in the case of splitting an order-9 folio at its third order-3 subpages to + * an order-3 folio, there are 2^(9-3)=64 order-3 subpages in the order-9 folio. + * After the split, there will be a group of folios with different orders and + * the new folio containing @split_at is marked in bracket: + * [order-4, {order-3}, order-3, order-5, order-6, order-7, order-8]. + * + * After split, folio is left locked for caller. + */ +static int folio_split(struct folio *folio, unsigned int new_order, + struct page *split_at, struct list_head *list) +{ + return __folio_split(folio, new_order, split_at, &folio->page, list, + false); } int min_order_for_split(struct folio *folio) From patchwork Fri Mar 7 17:39:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E40BAC28B23 for ; Fri, 7 Mar 2025 17:43:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FE91280002; Fri, 7 Mar 2025 12:43:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AF0A280001; Fri, 7 Mar 2025 12:43:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62918280002; Fri, 7 Mar 2025 12:43:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3CC34280001 for ; Fri, 7 Mar 2025 12:43:18 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id ED06EB7DDC for ; Fri, 7 Mar 2025 17:43:18 +0000 (UTC) X-FDA: 83195476476.29.D7E2957 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2087.outbound.protection.outlook.com [40.107.223.87]) by imf29.hostedemail.com (Postfix) with ESMTP id 238D812000E for ; Fri, 7 Mar 2025 17:43:14 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=BKSGt6P0; spf=pass (imf29.hostedemail.com: domain of ziy@nvidia.com designates 40.107.223.87 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369396; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YLKTs1iCIsIrZWDcg4jplK2n3gitTWDTC5hh13uSgm8=; b=aFfPWj9+EaT2cKEj8/mRi9GoOmmf7TcCQuw6qsjoxr5IzzhnUyIuCwWjaYGvlTCxP0n46X 4BcRHlqNxjxur0G/qqqE7SK6VuhTQbx7/g6vgoqGNk1uOOfZ0BzYTHgDVvqoXWN7WI/pmn 4nRaz9gJ5cYlA6xCDqyRKUMOWzopOe4= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=BKSGt6P0; spf=pass (imf29.hostedemail.com: domain of ziy@nvidia.com designates 40.107.223.87 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369396; a=rsa-sha256; cv=pass; b=60NH3FlPzE36HE3H8bDrwmvpSZW5m9mFTjcBObTA8p5M3w9Cxw9n+hqGqvwey1uJSgmXAi MexP7BVzJHNh6xd2MBxaEt7zgrecLrIGinN+RN3oNSf7jPjbYG3p2q7UYjwEJz/0vkpyRO rMchQf9MJy/BnA1/G47kT2TQIQs2DhM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QpEP95e5z+L003b4we555aJBN4V+WVGpj1omJoResylGjija/HZ/TndvUEbl+an0DphILt6g8mhrKVhVWkHcu8UiuJ1ke5/U0kJnXuHIS57UtF+8nq5OmHasDUJscHFQ2Cw6LLF/Y3JUkbpVDES8pTt80kq2KxUZQF96rj3Vkx8a2uNgX/wUnfEMGaYSuchAMyp2LAqbaWHbdwJzwOMrxiKaGdlZZ2XoRi2Fo36K5Nv3M/ORWYEP27gee7pfJ+hHnPsIius2tu6vY7hJxwgdvi0pB94DQzylVTFS3kYzlOqEFsFuslVkI+6+DAi3q51ZnOcGfGUaDTwEFX2UIzq49w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YLKTs1iCIsIrZWDcg4jplK2n3gitTWDTC5hh13uSgm8=; b=PY24TeKfcKLIx5R2zQ685xDJqjqQ8vvClWR54xFHgeilLsoLEImQtSZc8h7x5onPF6xpfQOGWKPhDZiPgiN2wQsNiGkIl09nvz16MAW7D2d66jJ+LltYZVKGQXGP9fNKAcqcfa4QC0fH/FSlD9YfcF1bCUmArZdPjdMYbDoJW+ROQ6nBRvsOuU55jQOBR0rOjIjCXXaR6dEtCRR4MqD3bJnLanoWGwxtqZXrW9xrYNR4bzl+Y7aZmwv5zvpDLSTwuWSX6A9P6N20RbyJJmQSk0CEFcY4PL4xW3+aS+/9A9LW9LZjZPaeSNhjejlodBgmAgRVs0V7cgns2obRS1eVIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YLKTs1iCIsIrZWDcg4jplK2n3gitTWDTC5hh13uSgm8=; b=BKSGt6P0KJiq0PRePOhIarnQJOm/epk49luC7YovfM23iaSDaUEY5yf3HbbdFu1JTw+gRrjHMxgJZS3/D/IoAIQ5BP/W6RagKQJEh5jRDtJjE5zRX1M8Mk46KUqGbTO7XyGbuL6x629bY9USykrM0imnduZRIgGti5UEYLw02nLW9xzoHa+TMlFAENEyOb9BTpnEMePLma5aBpo2OjhauAUNb7Cin9mcNxhKXjwgc7M5vbv0voAaLlfXnsyJ+jM9kEk40CoOW7Sk0kSf6cgJmlDo+Wv5/lnW1XygaFx9CBwLyDF5EIvO53Cuafgk13S1XGDyMgsXH9EdXsy38G19qA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:14 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:14 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 5/8] mm/huge_memory: remove the old, unused __split_huge_page() Date: Fri, 7 Mar 2025 12:39:58 -0500 Message-ID: <20250307174001.242794-6-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BN9PR03CA0090.namprd03.prod.outlook.com (2603:10b6:408:fc::35) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: 1b0d24c2-36c3-4f4a-fb49-08dd5d9f1e55 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: 7dNS6wnaeyVLEN+8lDt7Y1y9Pv9t6ohlJjXtMic17wl6TbYCAKz/FKGBparewf73fj3UzmVLHlqUC0pM+/CJx4He1N9fRg0G7TxgU66XbHbtJMeb/ZI5f38lcTDgMRpph0cmkpW540Zo3Gkd8rBOQkd5D1/xpKriXvSg6i74zl/Zi2cJyij+uPeQ/XvnrLbSEiRJDSjXP6X69QFaS0XHdYX/lPeMQ91R0nE/zoVDCFKdwFeP+TVwtbmVFZ7o5CFgMgiIs3PKeXSWpkbbyANuebgVwmYnF6JIf5p567T2wN8pr0T6OGU2hL15+GTyUYOPIFKneUfaMxPc0X4pPu9ghOC9V3uaJl1vMzTw4gsKU40hkhBy7SRMts4ByQCkMOe702nkLeyCkZG81fyqZl5ziZolFyH4sexec+UCkkSYJavr0hCIYjfKYDK9b4UJbQG+Fq31u1BDhvV5i2Kjk5KI4LZWCQBAw5oS2qLaJUzdOg5Nhh+JIJHGOJX+QFRtOVBnrKe2Uop/ei/+zwEEKQfAZGJ/QMU2E6Caw5L7S0fNGk3wbDrk3pQueh+vwC0QXe3/liccBP3JzowYnb6jnG1iMRCTr0XxGzWM34HQ8fw1IGsmdPIqbrZ3r0oAk95p77xHyyn1hvrIuXCRg4tZfmIan2kVqN3PWkPOVM7zQjyj8iMIYjImb86qNjFKotsnwBVsiPG9scoc5faEn0FpuU+FOpcSBoJSLRtvKzJ/c3+fwUsRw2suvQk2UGDyUf9oYEgKy8ZLIr8K25zzHNOZA4SHAZU+J0C5ELXV3wJv5w8K8Dmguhfszo6VY3/45FrTcnqz1nJoYeunGXlWG59vS9oJ2pXrBi0GzYnBmBvvGLHpAn9bOV9sRPkLkZUQ526UD03bteBIzjIZ/S6K2ggYTcOcyEX6U1n1STyzoedW4446O6aF2jPaVQIDxmqIZ4AkmMV+x1cScdOt48elmXBXZliNn++OtkuxKgvNfKR2b7VX0tzu2SYJRwySVuTM11pWo7PljjNK/faXNDIb6i3IioZa4TX66cwVEKomBW8Xr/6YjJWmEd58TUcsbpSj35Uw2StxLzocnm8ndP6C9Uimm7WzWQVy+u9sKjs4SJbzZom4mI+imrw9xaie9PzN+a9BBiY5jkzpYi7A9aYGVQWPUp4MJHDIS0NQtdUiJPPC6sptyvX/7XuEEr2iy3QOtiPcUgWqlOCCBusA8JOSTHi+V6vOER6tfUZQcrR3kPt4Tb7uTm8x4shfx07fOzJihrFviM9eyE0RUPRktGASgdO6V9c6ItkZqEKWDtthzeUtTpa4ZKzwbfUsUsO4Kwp3dPyf4xyOMVJztFDvoY1V4lEkLZ3awOwQskVLSQpwrTV34CZmOsVrxbRRJnWaCQKrmZnG2Zb6 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9Cb+vYkv9A4M56bdX1KAmZn9CSy1OmDHZNeRvMBNFf+3znUzAk/2hFQGjqvBfyS7r7WSTqpn2QBivIbsuQ+nwmlTd+GNXhnP/xDVvehSVz+k/IJzKxxg2bNC/18bvjyUrfmd9/c5WVdEaC7xLiUV1hAKLKYiWSAOEFd3dM4w4mauImMhRf//hJ+iNEup8yVNBIsUUoAG5Zy+QqcAVIGDlVDijafp3n2UBIOsWHIQ3xATZwm1UMRAvO+5gle/0wXpx5ZgwOk8Vk4r+OOM4NXOCVlr1IkQLt84KWuqA6E0QRYOJ0+TC7PdX8LWXzdjEp3Yuye++A6OpXgiTbZA7wnfJbr18pTt1UeAReGeW616leYR3/TLVBZn4aBTOM6QR0uDQhKxlTnXOYJpN1RXOuzFgYKihOyvFJ81Z2Px7eOlNVw/eQoO4jSEDbegZcbhjWyS4oCMe+Y+v5ZYkbx0mhEbuLvCX3oQrYcReT9+Hk7ACIjEitj/4Dg3oG2f4ZzL6JhxhzXGKYDHOu4+oGQH9ejqQNWDdSh6hDEZhHJOZQcXCN20h1zoHN6UiWgBsnQU0B4xlKoFs0CHTKKg3aIAQ+4QjE/H7VnwRwjozrMblB4owtdv0ydqPJ5NE19KUQwGYaaRE3jLnXr7AsEnXpeXII4te5ttmRnQ6iSe/PZ28FddNjn8mmmHm1pGWsx7SAO+fBjjuzqmkE6USQrpeWTEJ1tnRpTChJfWPe0WiY0xny5l/DzXHsrucOzNJvVl+ecc1mFjEA3EtyouIQly4BIxD4f5G8KsebyW2ouhCa1UD2HxiGhfATH+00RuISglwPWnk1tY0UqONT2CIBSlAEXkh0yofNNls15FlGRJTydDPcbHBpmg9lL9b2sPK/J2dnEHXxugN7497vm/0zPBpCt8gpF2/MvgHA3PhV2M8EfAs3cJ3aPegt+O/JH50kZQ0eHnWRZeLSJw5biwN20657n23Ax3Yl1ZPZD88lNPTtVhx8GdNi+5hBAY12+ovK5Z3K5S1rmkBJZ3/kudwsfWuEZ6s3tJrFK05A+VIEI9bCRW8cUab95BVAZm/FI+yvN1kiF9keO8AZ+F50TQFVBDS2o2+TatF00LWXun06HRv3EmX6d41877K5FkfMUH3vKo/d5M03sIKT6/W1PDUKbcOs4yr8kIfas7vbJJecjBY0VA6+d87/acx9mYvxmRsaZnQHh27q7sL+0WqIHXpyE3UwYHLXpkL1HS7uh+Ri35KeFKxKIJlfrygAjy4BXxRL19FlDdSxOON0QRyzvk/0o0joXaHMfGhejXxvTZSjH6WjbhldVUaqjoKZ3Taf/eSsoljiIAZ0dUTdpOQF9atoQ4enZjmJVvYaUClhf4Nv9PWoFLbijTu7dQr2wAvfgBYkVZB9U02vUt+mb/QYp9KnDj2XLsGMVirGLpyWDBtuJDfa2u4AS9UJ7d6czl8z0Di1z46ZBL5rYAUlNBNHNxB678nkZCSLRgIrzYWRz2mnMm1MmcSaiQw9Xewhe9wI/1vPKcczUxEkCdMD+w8rzK/Okc/z0zg/KW8pNdGHqqy4zbPGPfbT7sC8w= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1b0d24c2-36c3-4f4a-fb49-08dd5d9f1e55 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:14.7980 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zTfnPpzrj+iZMQEZGuOkdhmpc0CbgPsbgZTG/41JX58sOvJpMXU1I16eK8ThqFz8 X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Rspam-User: X-Stat-Signature: b4meawydip3mxospyfa7k6mgtobi83o8 X-Rspamd-Queue-Id: 238D812000E X-Rspamd-Server: rspam07 X-HE-Tag: 1741369394-663619 X-HE-Meta: U2FsdGVkX1/wjsf2tGExlCI+r+FSlW4KGn3Zj7OavuYKmW/bB8X1VvKXwGm6uuctRpZJ/IsFnI9PUYtZPID6D3Uzzs6NCk/tqzOa+ecARyBDz5oAV0G0XrcLOlkwoBPidrqaa+QsRRKuYd1rPbDuns3mmThXIBsYHe01aA87JtKg77n9FVmOvN1d7pXjjyb6cmMPQ2qh8Vsxvl9Om23iRnJqmlwI2/yQmO7rt0llEb2nQS4KjPmSsGMv32rxkTcHepSXyl8HIC0dxVydhk5ndGr1m5igZ0LV5YDrXV0x+7gC/gCNA0z/IRyOV54TURAwi2GmYUL56M5ujPTg1TAbs/tZkX+fun1BacWAuQq6AVNYela9y2cKQdoa+9kXwQMjz6dBvqYWdTcmKMKI3dS/4oMwhgPjFP6s91lf3oCcS6b2uogyo3H0v6q0BCIMuvjfaUigNNCtgzVGMtzEpP+CIQinheiSX4iANqrF/wAAs8OzK38nYi64Sm0saQ4jW41N88oJtEX3bBBgIP/TcmNGS2yrN2pHm2PK816QSUSnJtUOV4o0GsLYvHwjo/0USVNbwiwyFCi0aLQ/M2nNDzhqxLnXMkASK+XH+ceGs0rLuW03icwnOB1cfEdlXKXtZP/ZSPCnx3/UBszBOFSZkwFRGl5rPO7inTZ4MDCbtZYf0RPdlKuBRR7SRgVk8PrKYhAKo6xwppmZMizBWpOak3fp9afDCA6LVe91+ztBA78DFln3IehvCJefOh/g6OaZr/8+mpqXhX5BCQms1r3OHP5mmllBT78QlsW5s+e+g2lEIPGsamU5/lh1Hadxlu8RVvc/hhKSwdMRhCW2EqK9cGSQBLoIdo6a8vito5ZFXETe3J+5TiCZKpN2mmLuBGBs3wzcULMnYKY9UIHvOD6XlpLond55CTvW29ACR67i+L4pvrW7fMPoPFIs1gy43s1Qtvzn+NKOxmY6o9cOeKQ7fR5 JDUZSAJv Q/O2Oi4pPB4+MeY8ngtB3uZ986iHctEj8bKfsMb1PQI03TjDwcFm7SLTPf0GTTBKutMPGg99CDGRtT3XhtJXjCPUlq6DlaEn9G6kKKAfRBsiPfvADQKm3bC/hnEtrIgq0YW+5FePzCx17JbGZ1BfbB5717icjdKeIuVuxbs8St7SSgQp4kfw2FNUq+wZBAL2Kp1C1he+QP6Q/plthJkXVw/EtFDePsuctUvXkw9IdidhXDoPd0JYm8o9KZ/lrd+XlGXhwtdtF/HnlOv984I+Osjdg1QQMiztpGBjDQw6AXWzgqKFjvsIotPwGcZLdFWn1My+YdSWYv+QL96m231REumFxksTgNBS7m+FzPAevVm4twDfEGT23d2nhy/C8MH/UyXPycaqp9Uj1sz4CFx5uu/y7zmKy+V7xL7BZH3yCEFPWN1pzhxfkV6Jy/PdmsujZ7uG1W2/Ihnf3CECQXFx6/GP8/kT7IZlJq3lq+u9tClJvGUQbfd07zqok/EB/w4EtlkyJPmq5wYQQoSaiXkfTbWjDz/YMpS9qAI1VIEiFZRX+1GLdowPN83UwDV2WrclOUeurYwl510vBOLe5WNPnsTA1gBFhHy6fx+mqKRSZAIdiKcRGCguLJm+DiTGzME3p+VU7XrDbQU46E0VxQAivKOZoTUOZ0n2gFF4+VJVwz0vF0bfOqhkt0iq8uucSdUnr2306Ibd+jhKZKH6xGR24DjN1EZ852j5UFaAvvY6w9P+iQzLe88rDxf9tbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now split_huge_page_to_list_to_order() uses the new backend split code in __split_unmapped_folio(), the old __split_huge_page() and __split_huge_page_tail() can be removed. Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song --- mm/huge_memory.c | 215 ----------------------------------------------- 1 file changed, 215 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3e05e62fdccb..6cc97d592797 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3284,221 +3284,6 @@ static void lru_add_page_tail(struct folio *folio, struct page *tail, } } -static void __split_huge_page_tail(struct folio *folio, int tail, - struct lruvec *lruvec, struct list_head *list, - unsigned int new_order) -{ - struct page *head = &folio->page; - struct page *page_tail = head + tail; - /* - * Careful: new_folio is not a "real" folio before we cleared PageTail. - * Don't pass it around before clear_compound_head(). - */ - struct folio *new_folio = (struct folio *)page_tail; - - VM_BUG_ON_PAGE(atomic_read(&page_tail->_mapcount) != -1, page_tail); - - /* - * Clone page flags before unfreezing refcount. - * - * After successful get_page_unless_zero() might follow flags change, - * for example lock_page() which set PG_waiters. - * - * Note that for mapped sub-pages of an anonymous THP, - * PG_anon_exclusive has been cleared in unmap_folio() and is stored in - * the migration entry instead from where remap_page() will restore it. - * We can still have PG_anon_exclusive set on effectively unmapped and - * unreferenced sub-pages of an anonymous THP: we can simply drop - * PG_anon_exclusive (-> PG_mappedtodisk) for these here. - */ - page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; - page_tail->flags |= (head->flags & - ((1L << PG_referenced) | - (1L << PG_swapbacked) | - (1L << PG_swapcache) | - (1L << PG_mlocked) | - (1L << PG_uptodate) | - (1L << PG_active) | - (1L << PG_workingset) | - (1L << PG_locked) | - (1L << PG_unevictable) | -#ifdef CONFIG_ARCH_USES_PG_ARCH_2 - (1L << PG_arch_2) | -#endif -#ifdef CONFIG_ARCH_USES_PG_ARCH_3 - (1L << PG_arch_3) | -#endif - (1L << PG_dirty) | - LRU_GEN_MASK | LRU_REFS_MASK)); - - /* ->mapping in first and second tail page is replaced by other uses */ - VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING, - page_tail); - new_folio->mapping = folio->mapping; - new_folio->index = folio->index + tail; - - /* - * page->private should not be set in tail pages. Fix up and warn once - * if private is unexpectedly set. - */ - if (unlikely(page_tail->private)) { - VM_WARN_ON_ONCE_PAGE(true, page_tail); - page_tail->private = 0; - } - if (folio_test_swapcache(folio)) - new_folio->swap.val = folio->swap.val + tail; - - /* Page flags must be visible before we make the page non-compound. */ - smp_wmb(); - - /* - * Clear PageTail before unfreezing page refcount. - * - * After successful get_page_unless_zero() might follow put_page() - * which needs correct compound_head(). - */ - clear_compound_head(page_tail); - if (new_order) { - prep_compound_page(page_tail, new_order); - folio_set_large_rmappable(new_folio); - } - - /* Finally unfreeze refcount. Additional reference from page cache. */ - page_ref_unfreeze(page_tail, - 1 + ((!folio_test_anon(folio) || folio_test_swapcache(folio)) ? - folio_nr_pages(new_folio) : 0)); - - if (folio_test_young(folio)) - folio_set_young(new_folio); - if (folio_test_idle(folio)) - folio_set_idle(new_folio); - - folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio)); - - /* - * always add to the tail because some iterators expect new - * pages to show after the currently processed elements - e.g. - * migrate_pages - */ - lru_add_page_tail(folio, page_tail, lruvec, list); -} - -static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end, unsigned int new_order) -{ - struct folio *folio = page_folio(page); - struct page *head = &folio->page; - struct lruvec *lruvec; - struct address_space *swap_cache = NULL; - unsigned long offset = 0; - int i, nr_dropped = 0; - unsigned int new_nr = 1 << new_order; - int order = folio_order(folio); - unsigned int nr = 1 << order; - - /* - * Reset any memcg data overlay in the tail pages. folio_nr_pages() - * is unreliable after this point. - */ -#ifdef NR_PAGES_IN_LARGE_FOLIO - folio->_nr_pages = 0; -#endif - - /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order, new_order); - - if (folio_test_anon(folio) && folio_test_swapcache(folio)) { - offset = swap_cache_index(folio->swap); - swap_cache = swap_address_space(folio->swap); - xa_lock(&swap_cache->i_pages); - } - - /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - lruvec = folio_lruvec_lock(folio); - - folio_clear_has_hwpoisoned(folio); - - for (i = nr - new_nr; i >= new_nr; i -= new_nr) { - struct folio *tail; - __split_huge_page_tail(folio, i, lruvec, list, new_order); - tail = page_folio(head + i); - /* Some pages can be beyond EOF: drop them from page cache */ - if (tail->index >= end) { - if (shmem_mapping(folio->mapping)) - nr_dropped += new_nr; - else if (folio_test_clear_dirty(tail)) - folio_account_cleaned(tail, - inode_to_wb(folio->mapping->host)); - __filemap_remove_folio(tail, NULL); - folio_put(tail); - } else if (!folio_test_anon(folio)) { - __xa_store(&folio->mapping->i_pages, tail->index, - tail, 0); - } else if (swap_cache) { - __xa_store(&swap_cache->i_pages, offset + i, - tail, 0); - } - } - - if (!new_order) - ClearPageCompound(head); - else { - struct folio *new_folio = (struct folio *)head; - - folio_set_order(new_folio, new_order); - } - unlock_page_lruvec(lruvec); - /* Caller disabled irqs, so they are still disabled here */ - - split_page_owner(head, order, new_order); - pgalloc_tag_split(folio, order, new_order); - - /* See comment in __split_huge_page_tail() */ - if (folio_test_anon(folio)) { - /* Additional pin to swap cache */ - if (folio_test_swapcache(folio)) { - folio_ref_add(folio, 1 + new_nr); - xa_unlock(&swap_cache->i_pages); - } else { - folio_ref_inc(folio); - } - } else { - /* Additional pin to page cache */ - folio_ref_add(folio, 1 + new_nr); - xa_unlock(&folio->mapping->i_pages); - } - local_irq_enable(); - - if (nr_dropped) - shmem_uncharge(folio->mapping->host, nr_dropped); - remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0); - - /* - * set page to its compound_head when split to non order-0 pages, so - * we can skip unlocking it below, since PG_locked is transferred to - * the compound_head of the page and the caller will unlock it. - */ - if (new_order) - page = compound_head(page); - - for (i = 0; i < nr; i += new_nr) { - struct page *subpage = head + i; - struct folio *new_folio = page_folio(subpage); - if (subpage == page) - continue; - folio_unlock(new_folio); - - /* - * Subpages may be freed if there wasn't any mapping - * like if add_to_swap() is running on a lru page that - * had its mapping zapped. And freeing these pages - * requires taking the lru_lock so we do the put_page - * of the tail pages after the split is complete. - */ - free_page_and_swap_cache(subpage); - } -} - /* Racy check whether the huge page can be split */ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins) { From patchwork Fri Mar 7 17:39:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D8D0C19F32 for ; Fri, 7 Mar 2025 17:40:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77B936B008A; Fri, 7 Mar 2025 12:40:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DFB56B008C; Fri, 7 Mar 2025 12:40:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50B7E6B0092; Fri, 7 Mar 2025 12:40:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2D9FD6B008A for ; Fri, 7 Mar 2025 12:40:36 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3C0135626E for ; Fri, 7 Mar 2025 17:40:36 +0000 (UTC) X-FDA: 83195469672.07.95308CF Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2062.outbound.protection.outlook.com [40.107.220.62]) by imf14.hostedemail.com (Postfix) with ESMTP id 5783B100006 for ; Fri, 7 Mar 2025 17:40:33 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=Ex8VJ9xi; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf14.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.62 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369233; a=rsa-sha256; cv=pass; b=rp9BQD7mahQYwBXM7W5clV9NJtQw5RzspOxpg3/IUfMQZNQRb5To2JG+g5S94gct9WuNT5 N895AFBZ7JS8SbQhrcNDKsOgYDWE/sRxEyYNpGG/JU3xmDAk5G+RLdpFea1SeYQat/j9js ictHREU+GevB6gZO1Rplb/3tItQaUBQ= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=Ex8VJ9xi; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf14.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.62 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZgrkPhISa1K9KKRxthtvg4n6DwyTM7a7N/h+qLHQZpA=; b=liqdsW5zObWqUBPXAMIcZO11Z6fVwG36Ghw0jQ9tgwsalO4z2vbw7PKjjzSyPrYGCmtC5K sKfiONiieW8R0EP6bj/2Li8YS717kMDmpYyXJCe7JtkOuHxc5++n/Ga2cNVOwPk17Wh6P7 qSyQn09KY7p4HdbYWNR6nbxvqjzLekc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vssDaFEdF8cq2Nbz+xc46Jm6SyqO3X58nR2rNqWjoEoHzRLHj/FWS1v4L/inuKDEYTDuTufoHRRPh4ilzlt/k1Bf906ktdNt3kYrvp379WP0tKUwVVv6o7QzLBDtGdv7+Yr6f/cILBzE8RE6+h2Hizm9/d4+qXjjyX3wxDjPrpMCrice8gYU7IM5MHjbuP7eIetRyvoNt88/F/cBRkhGTVwHBOXrkVGQvEOMb+4rZFW889hW6jRu0h16pmwszQALE+PnEQ7mLPMG0K+OhgoGvpToFCwGGMGXptJduXaLT6jcz9UcYqB79wZKIPCD2ofIrGW0bA2FlWIg1TyWShcNmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZgrkPhISa1K9KKRxthtvg4n6DwyTM7a7N/h+qLHQZpA=; b=DCuncu7XtU71ep/hjYrfj4+yITHz0W3QWv086x7HZLks9HARJ5GFpUJJfSLLyzhqFyh1xCqngxnETV4AxDT48FQDtjidKybXGIQ1FAzOhDQ0y3NJaRRmRT+BQQl56ZmwTQySiX2aPR0/6n0lcSNHRZhTw2CYq+D3wTsy04wD0p5ff17qwBSuGeLFNGxBbrduYAcz307nIUM/+JMJT7BUHbF4291EsnPcAly6d3qCxDWx05aVTPRp0yYTRvuevmVsDP0oIJXDvMTNjOGBav68joPZd2s0uQxwIeVlrac8mvYvStsQeh2JygOHmdlTF1brEKe7zTIZ8nLpTCoLoFxFvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZgrkPhISa1K9KKRxthtvg4n6DwyTM7a7N/h+qLHQZpA=; b=Ex8VJ9xio9rZcFvxwgFFCBu3zRd5ByHYRPB9P2favvEwIERGPU44fc5uXLgmB54swIi1z0/87r5D8tg1G99QQli2xZpuLMn2Uesft4s774c8evVu+3KDFg8jdXFSmm1c9UE5Hz5WtN01gcDS/Z1qhIpeyassJ1NKlV5VgkVJrztGv6uOTu6pQu/DnrloS1AS/zmil3S5nI57yTmByxrDqy6A3/UHwi0Ui/5/TZPy+XpR0EVVYtFPEHoxkEe/6e+Anc39wtWse+n7iEyFNKa44VIJz5LycBZso+0hBfbcI3kLEe8pd6BjEOWcRJWTqyNUhbAFkKswUyZC2cMN9RidgA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:16 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:16 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 6/8] mm/huge_memory: add folio_split() to debugfs testing interface Date: Fri, 7 Mar 2025 12:39:59 -0500 Message-ID: <20250307174001.242794-7-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BN9PR03CA0083.namprd03.prod.outlook.com (2603:10b6:408:fc::28) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: 2c38b2aa-87a7-4c25-8960-08dd5d9f1f2b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: 9BZvG6fv2I0H1Ly8uKfQlrtSxH96CIYKxXRQ/EQHfkI3iCXZBZ96J8z/IQJTVflCjlKnOg64jUkRvWClkyTEzYrTeB/rPbpNOJw8/rmUW2ScRGp83xJ2t3vJbTxoMzyxs4lBzbJk9kiE1RxVynVdWA9N8InrXPh7/Y9rOb5V8MSr8CuAkKo+xuGjk+4Z8tL5yNkXB3mnMzj36Oy82qSKJFe10yc+2d58vAdMzORlyUA9RXfedAi38X2gCqJ4vhx20O5VQVBnmtT3T0yn/FnBzai0hFkiZCxEX3tEH59z3RZUmHkKeuvsL6GiZUX8lTLwtZYQN8Oo6YMYkqA4tbQl3xx7qsxQmCvgFI2DxXd9xQ4xANTaEY7pscdeRuUfhYNWesrvt/P2uFOPi4BKumKB7+9l4itOuNg7/tpOgfsS7A0AsNArjwUMgyhXA26ElXprvvOX09Tck0smxFy+xm1tuBlvOB6SuUBmIBuTJc9BXJWpSIqrYEQohiqaxM6NFUlBiSTNzJt/PzUUZHBTOJNx1o6+rdFVADQiL/36wJNwrCqUxisNDz8ZyNRM4nme5ffxGvp2VlgukHmBIvg2Hr21l/T+flhod1h9IKXRHQOGtMOWTidHyDWc4Mrn8wTqqKTp6WGTA5l1Tc8rfWtazHrchBt6kTHgGqvRNgdxV59fmMBUNpsU2WWQ46eXix+idGTtkfXbNpJXrTIYdWem52BIKBYDhgvcl28GX3fYXtyiJVoTujCX7scWI/+mw+Joe7/od7E1RWe8sRIO3fJqY6DotlKNR5ZwWOADNuZCjAXTpvqf6iBrWKVd/PO/iY5tgCVVx9/y1CQJLNpywUOS1CEA+Eo/PpP/AwavGqpcRMWr2k5xN3PT38pcLLkrr25YVSOZqvs1DLgzp7u2ZEkqnuJDv9LOQ+kNU5//r9+dQhV9wbvfBRJDjRZv+Q6EFIA0sy3z0cihkIkPs4lMPPzMtyqZrNtcM9X3IcnQotPtXOJDiQaa2R4vZvEiA2rwicO0klfaRdXUcapE/IwUdLq4aqoQkmLVKLGLxbQim+Ofd2MaIgu/jXk45SAdzBzG0Ntfti+C8X9QPy/C50Ha88wNuXWCrqiazyyVDkv39ZV9BneeDe1pu1g8tXiMAnBn1Em2j96schB2xqLx+qCD8rCbF/dRpUHlmfCsdz7uGqbRRkWvT0DQ2lIWKGSYtKTpQ+BV7eRJOeHuHgPvbuGd4m5M8iBgfHLezxd6X4+OQBjV13f7SiWT1+jkm7MKkgaR/iRGXrolE8g9sK0JdHr0HPTrDIG751MYN+mh/CCBv0/hDuKGMRrbXqossHc25RGLnws58jD+Y8fmhyVey7LS/IBGs/DW7g1/xcxoICejueXFBR97JtZhn2qWxTOTy0ZHNK1i6htN X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NZ83g5cOkAxqvG4o5EHWcN6lVrk65HoWw80TgsohhUETCQklAzt65tIrMTdM9aWxsC6zI/6qMu2Utx/rQcqsqwMhySAteNNaux8QD9/+GXIh3sItY8emQ2QWUldU9XWD1STJ1kYrHv/vy808+RWU1yYv6ge+T6a9RRAJSOQx/LDS9kM79G4CObo0svZ/Mw8u2rkNGEDlmkMvy+HDCwynqgxqZgaKTjjkmkHTFUKhvgquHr2I7pjXQiP0eBCCkZuabFABwrTAJzU15ANfBUc/njkAbPkQN+H3q5473aeL6DisMztkfqjl8NtL4F1QygF6ywHvSq8X5dXXjG2prG6xzQeQsA4pWEeRkTclk7kty+OTJ1kkYb5jyoifpqmBAa6evCm+RFbh/vfd68vi/m3PPMV7VQTdwCVGepu6qMT8m5/0MtQkn4F8QFlUBi5OTEdQC1FuzFvynCcS85NPFmLp1V3rkxTaAhXy74N4wl8QoUw3AM/KIdzw9wpUsHcNRVXUf/xPhxyJ1NVIa2nnNMn7Q3KfYlrCKPcSzeKGY1bqeRLNm+OHmfNQVT9W0YU0L6jdS4TkRpbCFkRVFKzt63G2CVlh2vjvHtgCOuxtNWZ91r/zPrhDZBnsVss/82IfeLcvW1yMGY0Zhn4WcMZKRXiNLPryBtdWPCjq+1XxsIMy1nLVrSZeywt+pfcjlbHBKbuTjYBmEnFIiIojAzw3ChC1c0KWR0+aTqplb5gBvKlHuhhmUEqnFe59YUVOLmNeyXL3YWzyGvvN6B7XgBGRYfI7hYdcmgU62WCyzFYk7eEkqY3MdwjuiI06mICbgjdqHbRGiVBrBfNpZ7r7h+bAYeoTN9sLbjn0YhZWfVcz7VDZn9g09LTogM6sjm3ox1DcOgpRFF9lgDoM8xIqR/m20Tx+qno6nLizwyI3U+F7SSKbyY4jOY5v5N6qJyLdqFNrgLHfOEPYLAD3b6tGqDZFD0+nAxpbHlHe5ndYzn9eeGBTen/mOvDDbm6/lnAMuhBLfT2xmTyF0ab3XjIsXhaGVrKOBHKhjsJ9zCI1TeY++Dy4LekMY0h3Qj3ZRmxirAq6FMJr9I2UDxBRE+GrHpQybFbnTBjSLIccjuOCAR4TN+Ik6hGOeG2a6cViA2gvP6GoFkUlPFFv72UoTmt/IP9QEl+0Kj659oHWle+V5pDgZPo9iXMEiyVC27eGKvZ2ibLFWXADGgAtLHPIHA3HBv3gx5ZgJ7N2fgADCSEVQewn35Ozq3lfytEw4X8J93fGw1PAeqkCVg88ErogTkiGZyOh60pFkApLm1h2Bu6m9f3Q9zMBgqjqkql7EwRlAJbLPPa/x98BKV0oV172C8n+0W03LekzDpLn+rW/OtCkhqbdfhNyGr+EOtLcW+7wsFvzeoic1cyHgkSiZi8obj20z5+3YqJWQwBEDv54Q8VdllZJsEfK06zlpmAjW3Va40fhd9N5A2LgkD4oeJ6UBYsNqBYfMHJP+Bm8yk49+ABr3bz9YUl6W3yD9lyipfHKhq4gh9QiuBfHNexqLN7gM1791yXQhcZxJkXTUpqvR6SRCR70JE7bzMOcZHJK6yJ8WInG5g4qdy64 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2c38b2aa-87a7-4c25-8960-08dd5d9f1f2b X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:16.1870 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8rWF7XfKpjp3ysjCk59kVWJQlD7Cc7TGRtXpfl0pVJUF4Er0irGitb6FJkmIUPAh X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5783B100006 X-Stat-Signature: yp5mxkre8w117fqt5fhbbjrk3yat6ras X-HE-Tag: 1741369233-104399 X-HE-Meta: U2FsdGVkX1/cfU29xWyLyLGN7NkCv1F12uXCbCdAGMBQYeoQwUuRKyOEIboQQ2ttGGYgy+AH2xNdh8tp74oA1EMoFNJQRB8csKNjPGScCgIQ9TV+vNZDJ+wuY8nb3P3J5UeA9umsx2U7NneSBzHiqn0/DQIjOLM1Nge7a+dwyhYGDxoXsOdcqZ5dYL3EiGeaWFc4k/824iRl1KzF6ihdFFCsNbvRIyin41M0jeVKEGcO/S82JgdZLaRWwUB2lMiKLqMwelanoWjgUKI7GIWNxhxqC7JiB6hn/h+ZHokcfanliKL/0zdhs+27OBhZFbIal9qTEfwsGJb7hlEx5gbN9ft3m+cLy9PiP29CiL2qH2CuwooQsi71o5hLJHeAYHmUjBY3D7MrZEmFFH/ipCQh7r8+sJuK8xEmexpFF+8k1UimV92trdDW0vb4zqOnnIRGNVJYTMGTBvP1xWTQw8XYyi1b8C/n45jZPMSNBJHR/moLohA6WWIrbjSwTSqRkO0SdRj9j7st+ALgc1O22gL30SpWvSY47Y18tZjGjWwkJpVdRL6OCkAQVpX9xh9jCnKGGURzYmekx4qaORrt1Tetd94flVspcG3X8GxXgzrdwY8fKc9bLZza4IqO3qit076sFtejAwzoo/+yFwKQlWqom6VjoKp2Gh588roqSTo0iYOHf0KqbioSVNdjo0QKmi50qdMnMWORfj9MKrRdcqzCKUoqgZzXd69Z82w6gNbtAUxma9XPOZNCZx7ai+0vLx4UA6epHpW3vVpcACKzAER7NMn/LUbsalfdkxKxmv7zUjsWutvZympsBPkP4E9pJU2tkS/EJvmwxs9hS0YG1VfUJgPA3DED/Ipr4Oh9OOFojXjFxv5FMOONZt5NTgGUUvfCAw2D6YdR4rFitaX0GCU78C8Yzb7Ao/E9aB0ScqtyUhnkm5EU7TR38T2u1j1RsS2BsOUU69PxmC7seSEGPFd LDTqdpZE G4S2G+998RS0JUntKfccoEqAnOLFNU6l2ija30wsP2Wy2Q4ArrhUZhYebSAjPnEHJFUHgQ5jFHAnW5dKdM2cFjgkRi+hn7xcaLeNuPcHbVjHfLAoYHR3C7hcZXEIYg8HHGByNHcvPv1AYuNz7PbKIYzgcTAI0oy27+WoI7EeagFC1WvuzY5Fb4Aq76MmW/VV7+okuAbOLivj+wQ0rjYbpJ6JrvLyKpvQoewDQz4muAx0l95jRtowrPO4moO5Rre+d7i4JNWW5nJVJC38onyZ+3KmtNZKE5aW1ajY1BDhO0G3WSmS97Yv749bCSd1+Rdn1zxvnsoRZ5HX+hqg1ZBmRAFXmuVdbQj4KEKcHIso8MKys3ZvLOVy+A1rcOzB6LVHJKZxO8oURd61Qg6cjNH82QxD3MjYEpoXGKuFfd4AZ8jk/r/QDO2/tg5HsMM41qrqW+mxHXIckbVAhi9HRdYacLwerdQRKztkI/CJuMBZbya7Sc5dq7z3UEB/ufTToLhAvrshuN38Q989SohtjLjLnrb1Pi2NUBTTjPI+2tUpb5UiAWcGzRkdUb8kKvhGHGHmCORhKvxFjJZxx7BK6qH1eccQvwzuTzQ1LUs6CCC30vu8ymVBMLE8bSTnA/IqFPaqNQnUhQSkwyCx+l9JyuasaSZkMP8c9FiqLXpepsL0g3VK14LqS5lQQ3h9svahZcBB/eimg1NsxiSXW4NWEyO+iCa9RuHewzUFrfF2GhB5CsV42bCNjlDlNZoIk1g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This allows to test folio_split() by specifying an additional in folio page offset parameter to split_huge_page debugfs interface. Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song --- mm/huge_memory.c | 47 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6cc97d592797..d44b6d4b7f32 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4325,7 +4325,8 @@ static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *vma) } static int split_huge_pages_pid(int pid, unsigned long vaddr_start, - unsigned long vaddr_end, unsigned int new_order) + unsigned long vaddr_end, unsigned int new_order, + long in_folio_offset) { int ret = 0; struct task_struct *task; @@ -4409,8 +4410,16 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, if (!folio_test_anon(folio) && folio->mapping != mapping) goto unlock; - if (!split_folio_to_order(folio, target_order)) - split++; + if (in_folio_offset < 0 || + in_folio_offset >= folio_nr_pages(folio)) { + if (!split_folio_to_order(folio, target_order)) + split++; + } else { + struct page *split_at = folio_page(folio, + in_folio_offset); + if (!folio_split(folio, target_order, split_at, NULL)) + split++; + } unlock: @@ -4433,7 +4442,8 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, } static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, - pgoff_t off_end, unsigned int new_order) + pgoff_t off_end, unsigned int new_order, + long in_folio_offset) { struct filename *file; struct file *candidate; @@ -4482,8 +4492,15 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, if (folio->mapping != mapping) goto unlock; - if (!split_folio_to_order(folio, target_order)) - split++; + if (in_folio_offset < 0 || in_folio_offset >= nr_pages) { + if (!split_folio_to_order(folio, target_order)) + split++; + } else { + struct page *split_at = folio_page(folio, + in_folio_offset); + if (!folio_split(folio, target_order, split_at, NULL)) + split++; + } unlock: folio_unlock(folio); @@ -4516,6 +4533,7 @@ static ssize_t split_huge_pages_write(struct file *file, const char __user *buf, int pid; unsigned long vaddr_start, vaddr_end; unsigned int new_order = 0; + long in_folio_offset = -1; ret = mutex_lock_interruptible(&split_debug_mutex); if (ret) @@ -4544,30 +4562,33 @@ static ssize_t split_huge_pages_write(struct file *file, const char __user *buf, goto out; } - ret = sscanf(tok_buf, "0x%lx,0x%lx,%d", &off_start, - &off_end, &new_order); - if (ret != 2 && ret != 3) { + ret = sscanf(tok_buf, "0x%lx,0x%lx,%d,%ld", &off_start, &off_end, + &new_order, &in_folio_offset); + if (ret != 2 && ret != 3 && ret != 4) { ret = -EINVAL; goto out; } - ret = split_huge_pages_in_file(file_path, off_start, off_end, new_order); + ret = split_huge_pages_in_file(file_path, off_start, off_end, + new_order, in_folio_offset); if (!ret) ret = input_len; goto out; } - ret = sscanf(input_buf, "%d,0x%lx,0x%lx,%d", &pid, &vaddr_start, &vaddr_end, &new_order); + ret = sscanf(input_buf, "%d,0x%lx,0x%lx,%d,%ld", &pid, &vaddr_start, + &vaddr_end, &new_order, &in_folio_offset); if (ret == 1 && pid == 1) { split_huge_pages_all(); ret = strlen(input_buf); goto out; - } else if (ret != 3 && ret != 4) { + } else if (ret != 3 && ret != 4 && ret != 5) { ret = -EINVAL; goto out; } - ret = split_huge_pages_pid(pid, vaddr_start, vaddr_end, new_order); + ret = split_huge_pages_pid(pid, vaddr_start, vaddr_end, new_order, + in_folio_offset); if (!ret) ret = strlen(input_buf); out: From patchwork Fri Mar 7 17:40:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BB59C19F32 for ; Fri, 7 Mar 2025 17:44:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CB26280005; Fri, 7 Mar 2025 12:44:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77902280001; Fri, 7 Mar 2025 12:44:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CD1F280005; Fri, 7 Mar 2025 12:44:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3A377280001 for ; Fri, 7 Mar 2025 12:44:46 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0BB73C0E6E for ; Fri, 7 Mar 2025 17:44:47 +0000 (UTC) X-FDA: 83195480214.30.6069325 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2056.outbound.protection.outlook.com [40.107.223.56]) by imf17.hostedemail.com (Postfix) with ESMTP id 2BD6440006 for ; Fri, 7 Mar 2025 17:44:43 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=XjyP7AEy; spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.223.56 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369484; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6O6gpyeI6VrnpSWocOraOLupk9fA1jVzX24L3jF2o38=; b=fvjDNqrkqgU2T31zqdexY9crHINsNSiUhO7ZU/jhQPKfY6Jmnqg8wQxzc+tSpv9KVQ9b/S 6sxGLVGc0GpuPzaPY8IFUjRteajxi8KbO69A0u0I964A+Bx3l969D3QOlM7lVsCnhywfXs aymWZFDXP8EoBjvh745Svz3YPZ7YLAo= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=XjyP7AEy; spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.223.56 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369484; a=rsa-sha256; cv=pass; b=AaCl80hns26X05DHNUX8XWxMQjtTM4n5USK6p2hjmEeJdM0KZboFsvb7A3a1JDLrPo7VD3 PHka8LwO7DsHId4HxhT8TiiVISSTQH0WfeTP4C5cpxgcmuMwvZzjQBy0DVaEKMVmCGi62t rqCo4SycMgloDEryw+1MxFIT4YG0BIk= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MM3o527WkgmJENLMFx/8bAnGAowMQdBRC7EurgepFuGCplQSY75GdQJKoLghPT87UD9GJpFKhmqcjVo/KDFYPhEC/DBO+G0/9XBvb/mZMtYIKyryLY9D7ps8klyFg2B7hAuxcLEfZtprqyw8pTqI1mhbCqI482Lshh4hTRZnDQ56Nu9ZVIFoWsxZK7fjVS689w6PnSFN5tzVWIW1MnsAPifT8FCBDGvvIOqV7fixLkQkR3sgbsur+5QlzpcafYD7YcfI7Zc4ptkUspzYTdS5Xbdd21aUDGr1X7+zZUrChBy+JjL8300QglG1maQFUtFBa+dtaGr+odMgGxc6Veekmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6O6gpyeI6VrnpSWocOraOLupk9fA1jVzX24L3jF2o38=; b=yEQtsOPmhTYGaVTcLFs9SbJ5GcNWWoeZEWw9kYHty1ydgLmTTAxuEo49+qj3wlw5jalJZFm2jLK8wgDQCWqdrDOTFp036M2cUTJ+VR4X2hAp8WM7T3pqha1ZTH7DLeUKmOsLRv5V15LmUbZD3YSnr14OBUGI3W5q8LgQhBA47Ln3C7syZWGeEEiwxYaYbny2sffKQvBDBD3JGUVR2K3C9a6/1H7vIwiUloCtw9QDVJXRZiVM+aJS4TPKdDbUcocbVWhPock5YGbOin2DCutCV1jCm5a/D+HKrIGHjU31bBLpxTEztnGr+0RfugUQFvEIWpx2sKZStmY95EKzJeZRYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6O6gpyeI6VrnpSWocOraOLupk9fA1jVzX24L3jF2o38=; b=XjyP7AEyq5o27oa/8IzLwphWUr3qwip6e6aLNQVGFXYfEcxXCMfmTr/ts+6qOCPh5lJ8k1c+aiTvqNJvLdAKCYPCm/yLB84pRiMRRkUnH5+7q2Gnt5pWheqFrJ/k1x0WQVdsTDyFEl6XpnS0o+Z4XYxxFIZNnj5V7G6xUDr0BHr9o3ovBWbbfVRchlsNCB87U00vBIf2eYo1QHJtYqaq6zfbOWO+5ICo9Hwq1M2G2LM7gMDnoUr/YVFaYpYxP+A8w6OavVDEAw7l1+l2fTaJT9fyfLusPMmWiwhqz4FVuOYO3cQk/TyYkSjX+P3mJFwuRkxwyyJC+ekSTFrp4K99CQ== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:17 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:17 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 7/8] mm/truncate: use folio_split() in truncate operation Date: Fri, 7 Mar 2025 12:40:00 -0500 Message-ID: <20250307174001.242794-8-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0131.namprd03.prod.outlook.com (2603:10b6:208:32e::16) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: 47863168-e15b-4c6c-8039-08dd5d9f201a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: RBKuPU+xsE/xSDuQYo8WPIiANepTIwg1Ot20OAb/3ND5CUgpHnyeyUy8fGYQQ9quUWGgBkjdSbuD48dyUR5UsExdYuV3800O3QqPllK6mNgYsMlY835U3XTIh632zD9eUUJKP68id7FZsgRW8mFN7I4ZL4ltETuD6TzIDckNovQhL2GVSWSUruW+Zl/LqF/p7X+/XO5th1YxYDVr2OqLTqnMEvPzWp6GrSt5Sv4i1blxvSC1005kpFJwfSK0J8ofkZJe2tYNX6GZP6jg52I6SXERIchjLxKe7HfGF+KC1A6lNbcdLefyCOEKrwUoDaxsXZIoM2/3cM+OPPg6CHO0xf+mpobweQTsYVdvy5Hz0bUQtC3+oU0syVQyKRFlg1N8mcAMCBM7gTm91bGk58iPZ6O7Drfqh/So5krNwXzUy59HhsGrHQOptCCnTzmHuqrnF7XghkNPkJmNyhIS/Hdq7sC7K5UARLWkwJZSdh4NjH/92s3piscG4/o2iatWqEW0oiGdGGlX78cFYQN7xHdqy3p5ACVW/253g3Qpl7IOMeBZHlSR4lZc1uwXD74zA+isgRegQO7wNptBTJYObw/XcFNQTTvsJQkxEpeAi2Flile7mEKBmWSB7SUx7FNZklq42JK12xR1CF+ARiVPiK1Ai8T3NOJMtRRx1F6tkREHKvWLXNrwwQOC7PRM94InAVskeMCHpnKZL8EG2W0x5W0tnIcE0ZKzi20WTABbtMD8sq9zU9bgQa43fdPNobK6hb16IWf8h9jPU+FCTc1OmhezNY58AD6ODadNZnQXs+TRh5Ivwa3Usk3fTJJfTy0HrHT69ASrGKEvGEUbLz4k85BdqsQhrnYvr+DxHbHzgCfH8b2EejOhqWjlrmdf1qsSSExw/EGyy9PGXrRLRq48dUt4h7tCoDPqkOlfXCSs50paXC/SyLTU2P1qeX8iuhxOKJtygC8X8KAP1cnM8U0IC5qFJINdukVQxH/mSb4ps18WYoVMV/v9eznz+8Mgfrw/rBfpknL/29LFPZLHN3oQGEDss/vUqniiMnbs6D+ECNAkAEG0+W/IokO95rgY6FVIzkq7z+WNe2P/VCMNxHN2yVoSrk1vIKkyMP2EWxRD1kQoip6hTPpOI3xme6Q+xfIVK6x1KRpyC92dig4hW0Zgeh/UUm6v5iSi+DSbAGULpjkFHWopLUV7fuh63aI8wx2v7AjGzOSAdpc3oMUUTjM/iC2+VEXl3gpnT0c3C24JSuyZtKzgekMHUWNM2CjD6Ejz5QNfbSN8zuuYd1NOV6z5DGznRjCz/mkGX1yvPGQNz9gnsjzknubEKmqYKXsNRHwDmHhhGwyJj8cQwcbwf98ruLBouQ244UXs18fnu0dHkPIxCSqHVwPX3XptyG6KSYd+b0VS X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WDO7nzWu40KlzhuZHLZOfYunBTP2G/xS5S0ussQNqwzmLxJFBCnsnNu+Y5CMMHQ64W1hnmL7hstcrbZJtA2352uUyR6VTpeHbJuCji6iOuU7AnEruUnCm9YhbGzQwTkZfwyxlJDnhqJ8YXADmfq7ItynsY7KxTW4W7ldVhP/eRaZS2B6oX8Qr0gjYzPwr11BePCuyO71jrErEtWCglvYroSlj5MMpNDjwpUtyn3RqMS9JtWDN1S554hYBYDNJFED2TtdsUjdaMkFkptS25hzftFZ6PTJsGR352GXRT3W5HBHtvATj6/t6jldspIpk/x6D2Nq1i8uDXFHZBOD1tMbJdiob97+J2WDIlPiqFDu9NKhBEZWVs0F00H8uylHu8uFOhRVhvdCFzzUHzhtv39pwIDCK/YlATCxgYs0y4P4hhZW3N5aZ99j1itUsiFAGNwNRYFqv8kUmLO4HpjAaPQovhjVcsdl8pgoYg3tZBiQbgeAIsTDhY48LMJpVuA0nk26JMDz3y/axSvyF7wvcjiXZmczPYvz4M4+pfv7WMw9CBF3vfGWWZKVvq8iJaHGSqeiaCGO8Bht59wssPRSbx/wevP6OO3FkeWwr0l9cHQQPlwyXLcx15rimInkcieLnOcaoUTE62o97HTgsFrutxrKhOUwdaSOWkAn43PEXMhd3AC07R7zFihtsubRk8ErIcjip93AhiUDtjzY6t2JI+Njlt7Owl3fKNU/UmJ4YX1OdbLF5NytIUPSE1dWef/U+ZUSDqRVWhnqdcz3CGf68evQbujKYctLO0r8CEljHfa14t9vr9rW/H+Sa/yiOty7fPXcnwgyHIG5sGTOzv0HWjxOP6z+bRL3O8QLdRRZMLJDxmli5PwnzxdYFY7f7bpxITlCotxOe0tGEqVShRfS+BZ4thQ3GrBkdXn/K5GHXeolWgjheAbYJmrr08r9bmXcFnY4zIZdLOvokDC92pW3rQaAKGESzuu7wjiMBqrPE5wr12AQKtpdutNzZJg8tHx0mrjWho3tqDVB/i/f/wWVUgj+2pBqJXQldh622EsvU4ZyLyVkwkVrqSinNRLDU8br16bh9X2whlLSryOYU2Y44KL4f8Optp27CQrKYh1dlWVNkZ/ZZ/6gCXuCpEcXQeYm6ApcZnO2wpBMYvGfXmrgYyal9jz7g6tDNihoe5HpAYl9pwIYNwwwtSM8wh5sqrZK3hXRxhwSLh3wDZxSRGoSXINrlHnxg9ARG1JcljryFVe69SbshPqtpfS2RHmnflkhxBpZiJTdt0NKkdGwEbhzYjq0uXsraPLFmZuCqi4ope7ocbxAGmKxSYlVToOpQAYWuafXJ0oI0xCt5XlhMPWp3rTuz8BisvSxx+BIXMJCqo+c4/ZJtI8hTDgkGpqpbtzB36y7TLJfs5VfxbEpTvlIDguqxJLZARqaj95U9buChE6zqYxEy2HyZKoVs8IlrQQMxqQEaIdMM4K9qcuQpAny/NPoeWEz+KVqXcH5jgUZ6jSfdELXfEGeqapLf0+NyCrf9u1m5Q9G6Wj1exQTVWR5hh8uJuwyEW5vKWJpCGFxBM8OUEMpTXHQL29CklKHmyIB7f2n X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 47863168-e15b-4c6c-8039-08dd5d9f201a X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:17.7718 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TSYSPo7n9A/GQdZgPR/0GdmE8DPAQ/ClF+L25PjquYTMEKZBasmMwUAr8QZefZXh X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Rspam-User: X-Stat-Signature: zjj4k6x78ku1ygs5oiyfx3odosketf5k X-Rspamd-Queue-Id: 2BD6440006 X-Rspamd-Server: rspam07 X-HE-Tag: 1741369483-94184 X-HE-Meta: U2FsdGVkX1/mrtu2ro1LdGiTGusa7JiVgmUTniHWt4NhFtatvDtMi3eWgStq5hzvVPrRdV9sgLjhiBwzSIrOFlNTPH/JpnouHrDVsZD8F2xnaw9N/UlT3raFyHE5OUQq85NAglGLc7Or2mu6R5UhUnvI5CL4EygHzY01ADJpC+WEE3UEZh2Xi5dlnqVw5FAxrE99cpTaqcT/9PbigDse5zNl5gtcXHE17xTXhuTRaT4eLnLHFfmtACtG5uJRIR/2zD4gWLBFkW0fcx7QBKThg1G46z8zyzUQD65BwuHo0YGKWG0w1gLKaZPiGD0YAt6YpDt6HXD+PtIKffZHwLMXQ0VXC8+Yz/boyVV6rTmcKvdlbtMfNjPJeZGcEJQGg0EVrdXMGTOGLChmXs7ibSJbqGiCNlF4OOYSpX5R5TBR2mIgTZx0c4EuT/1ybqA4mh0iMqvEUBn+0oZ+E2SAkytluQCKTMeaZrYg1Cry+Bzqusu5/Jsp7MSCsMnKdgE4PuzEvQqmrsNI5TSfUf+qHAwAU50q3N+9COrrBTGtvPXVk+SSLRPEkbtGafpF+GPc/xmWP2E1k191pctxCH0QN8kB7lbuK0HU2PSGJlQ9l3nMg4hl3/i39WCAzzVAH4Dg9CmqFI6DR2gl/6Thj7Vga8mhfAyX2PZd6euTywaUalJOXcN71snDEbRTQ+s7SBoq0mnMJ9maZybODg5JvJA/wEV8d/bSt0EsXUM8/Jikh5dXhYlgHtJkla+KswbfYMvUisK9a5gY4WeLLkOFBh+/Gyu028G2sDmhCbJu6Jib+6RO0Lu1V25DePWJ6ADiI8ym/Ljmx8UjVEQBQ0DO+zjy4jmWsQI0VFjr5wYzScV+shAm5RjCUrcMAO7dmUK0+MD159U7rtEJPjt4EX/lq+DQhHsG4MyGdWYa42naa/FpG33PAntiTus3HJD63fwyPGm/FiCxXeSsi4AwfsUgZbq71Y7 h2zOdZBf nbAjuGrysOr1+HPNE0NdZmzqHVgspv4/tcpOg3C2vSou8nTCa9M8meZdRfoEmMHpM+FUdIXRmntGMUxBmRm9S+Q9XuqK3AUX5P8FhJYpFCq18HNM0EMifRqLRBYjo8CTfzr2uLwS0SMynyTPs6THJveDCI2l1Inwp7c6NiXxY1jVlI4JfF8gby+yE0bfAEO+nsBHEuOnfQa/TjjkEel4A+8YUfULubGomFuLppIRK3KtxE+KEXWGbP4TLNIFSelZ84J9A6kvaIcMjqeR8UHCQ0ozlOTxJkAoZT4+0B1/b7st8ePchMeWFvKRi4M1r54R+xU5AvEr5zjCruIQRCrG6eLEBMLPV5mTJjGbE8KKYb/eNjFGeoYH1Gx8AeBCLAG5MEaex1gweW3d7Kpovans5tgH6RwCBQDjck0iDAxyRs9QDtPW7EPPai08RRiprcuR0CuPoNHBR2dhUQw4JCM+xOAPzwwKhhAKCLyF61mMNEMjoNsHytJrgn8hnWuS3wZLP3BbjMqBgwFSZHuUx6ktB95F0YrEYHXGpxr1fahUX5CSkvna+gq6P7RLS3dnFHoZ+pkBJRQvcpAXATn/KS3582gusdzv35NlA4hzJ8ZckHQFmjbtZfHB1LV9LJb3o42YvOw8IrIN2bK0cprEXJpL6B8cnPyIvFBcfvBnG71hAMtgCBtoMRHkKj/cEtDORP2RXqxChr5zaiNp3SjsMQ/OEAfozBTVpkGVIC9dk65nZ/k6WrS/5SxVng4l7NzrtQa8D2fjif61S0T4svPh9dvP08uJ40vnBFJrRt015ukkxkOxRc7NPPKw4FojHtctsT37Ri6rSzhway/Lkw20= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of splitting the large folio uniformly during truncation, try to use buddy allocator like folio_split() at the start and the end of a truncation range to minimize the number of resulting folios if it is supported. try_folio_split() is introduced to use folio_split() if supported and it falls back to uniform split otherwise. For example, to truncate a order-4 folio [0, 1, 2, 3, 4, 5, ..., 15] between [3, 10] (inclusive), folio_split() splits the folio at 3 to [0,1], [2], [3], [4..7], [8..15] and [3], [4..7] can be dropped and [8..15] is kept with zeros in [8..10], then another folio_split() is done at 10, so [8..10] can be dropped. One possible optimization is to make folio_split() to split a folio based on a given range, like [3..10] above. But that complicates folio_split(), so it will be investigated when necessary. Link: https://lkml.kernel.org/r/20250226210032.2044041-8-ziy@nvidia.com Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song Signed-off-by: Andrew Morton --- include/linux/huge_mm.h | 36 ++++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 6 +++--- mm/truncate.c | 37 ++++++++++++++++++++++++++++++++++++- 3 files changed, 75 insertions(+), 4 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e57e811cfd3c..e893d546a49f 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -345,6 +345,36 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, unsigned int new_order); int min_order_for_split(struct folio *folio); int split_folio_to_list(struct folio *folio, struct list_head *list); +bool uniform_split_supported(struct folio *folio, unsigned int new_order, + bool warns); +bool non_uniform_split_supported(struct folio *folio, unsigned int new_order, + bool warns); +int folio_split(struct folio *folio, unsigned int new_order, struct page *page, + struct list_head *list); +/* + * try_folio_split - try to split a @folio at @page using non uniform split. + * @folio: folio to be split + * @page: split to order-0 at the given page + * @list: store the after-split folios + * + * Try to split a @folio at @page using non uniform split to order-0, if + * non uniform split is not supported, fall back to uniform split. + * + * Return: 0: split is successful, otherwise split failed. + */ +static inline int try_folio_split(struct folio *folio, struct page *page, + struct list_head *list) +{ + int ret = min_order_for_split(folio); + + if (ret < 0) + return ret; + + if (!non_uniform_split_supported(folio, 0, false)) + return split_huge_page_to_list_to_order(&folio->page, list, + ret); + return folio_split(folio, ret, page, list); +} static inline int split_huge_page(struct page *page) { struct folio *folio = page_folio(page); @@ -537,6 +567,12 @@ static inline int split_folio_to_list(struct folio *folio, struct list_head *lis return 0; } +static inline int try_folio_split(struct folio *folio, struct page *page, + struct list_head *list) +{ + return 0; +} + static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d44b6d4b7f32..8a42150298de 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3648,7 +3648,7 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, return ret; } -static bool non_uniform_split_supported(struct folio *folio, unsigned int new_order, +bool non_uniform_split_supported(struct folio *folio, unsigned int new_order, bool warns) { if (folio_test_anon(folio)) { @@ -3680,7 +3680,7 @@ static bool non_uniform_split_supported(struct folio *folio, unsigned int new_or } /* See comments in non_uniform_split_supported() */ -static bool uniform_split_supported(struct folio *folio, unsigned int new_order, +bool uniform_split_supported(struct folio *folio, unsigned int new_order, bool warns) { if (folio_test_anon(folio)) { @@ -3999,7 +3999,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, * * After split, folio is left locked for caller. */ -static int folio_split(struct folio *folio, unsigned int new_order, +int folio_split(struct folio *folio, unsigned int new_order, struct page *split_at, struct list_head *list) { return __folio_split(folio, new_order, split_at, &folio->page, list, diff --git a/mm/truncate.c b/mm/truncate.c index 0395e578d946..0790b6227512 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -192,6 +192,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) { loff_t pos = folio_pos(folio); unsigned int offset, length; + struct page *split_at, *split_at2; if (pos < start) offset = start - pos; @@ -221,8 +222,42 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) folio_invalidate(folio, offset, length); if (!folio_test_large(folio)) return true; - if (split_folio(folio) == 0) + + split_at = folio_page(folio, PAGE_ALIGN_DOWN(offset) / PAGE_SIZE); + split_at2 = folio_page(folio, + PAGE_ALIGN_DOWN(offset + length) / PAGE_SIZE); + + if (!try_folio_split(folio, split_at, NULL)) { + /* + * try to split at offset + length to make sure folios within + * the range can be dropped, especially to avoid memory waste + * for shmem truncate + */ + struct folio *folio2 = page_folio(split_at2); + + if (!folio_try_get(folio2)) + goto no_split; + + if (!folio_test_large(folio2)) + goto out; + + if (!folio_trylock(folio2)) + goto out; + + /* + * make sure folio2 is large and does not change its mapping. + * Its split result does not matter here. + */ + if (folio_test_large(folio2) && + folio2->mapping == folio->mapping) + try_folio_split(folio2, split_at2, NULL); + + folio_unlock(folio2); +out: + folio_put(folio2); +no_split: return true; + } if (folio_test_dirty(folio)) return false; truncate_inode_folio(folio->mapping, folio); From patchwork Fri Mar 7 17:40:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 14006835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF180C19F32 for ; Fri, 7 Mar 2025 17:42:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BE956B0095; Fri, 7 Mar 2025 12:42:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 947656B0096; Fri, 7 Mar 2025 12:42:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7757B6B0098; Fri, 7 Mar 2025 12:42:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 55AAD6B0095 for ; Fri, 7 Mar 2025 12:42:03 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D301D80EAD for ; Fri, 7 Mar 2025 17:42:03 +0000 (UTC) X-FDA: 83195473326.13.E0C9DF0 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2040.outbound.protection.outlook.com [40.107.220.40]) by imf25.hostedemail.com (Postfix) with ESMTP id 01075A0017 for ; Fri, 7 Mar 2025 17:42:00 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=QQqJuUuD; spf=pass (imf25.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.40 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741369321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EU/UsANrLIuJHJQZyVRl3KO3Nn3lqvBBpTugjGoZsIE=; b=i8msGjDZ5lMrXHWY8EU7xYt3bcSSao8bIUmDGPfdYcrv5o0NXUnv8fSs+Y8xbvsfBO0xuw L+OVZKN6XrQvDJmw8huurtX2f2VZcanaaWW5KL190GzxGzLKqUUuI+1dehHxHOVh1FbdE4 kxffBBp58Z/db5v65XeBRmODYEPZWVE= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=QQqJuUuD; spf=pass (imf25.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.40 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741369321; a=rsa-sha256; cv=pass; b=gjs687hEddHALqI3yM5h8kyaowHWdbhlREkn/4oSrPBycls8CG2XNGX4gCP1jr28zMOszL Rz9sXQRYs0jFra/H1xCX8pjxzwApJnbOS8A494EeNOIBhoOI5fQu98c6UzDKS5W5bltVBo odrNrWKRfv/A8Pty9M7fwYUjwcSQndk= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=im/YBZmNMhWo93cPNyxZAmMq9KIc710mpX+gK9blfKbPn53j6PCP0VvZGuk7u97nHU/11AwLtPtvzUs8oqOv/ffzT7hVZ62bfrrfgtJ6S1qHk2eihwUfRvFl3pzeEDyiTStD59sTGPih4WozXYOsMzKMgbEkRL3329cfOLdFprKSrr8/9uzb8soWuSQ9gkZJA91kHf7lGb90RtgGejLtMrLq62m9Vd6CdOpgzCGVGaPWi4PWCeahyl/pkvDq2Iie4cCqMVButrvjUEtZhefvael9ERHMjuc5dHGoU22FG9MgvAcQWRlBvZdJUwuVy3mvWrn/OtQRiD1DDNA+We2nnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EU/UsANrLIuJHJQZyVRl3KO3Nn3lqvBBpTugjGoZsIE=; b=qDEbC9MF/p0srvPjRW5XGgncOSl7880FJ7jWV4GxWgBUjR4yBG/Uy1a5fpTyuuJFY88tXlg14n33R/t2O4DpLZbiwnFmgf7L6pG8GcHYWDxDJYS7snqO0wpRu+mcL0b9u/OJrPwuod+sDDLwPJ4l2Fudvy0wSxgONEpw7b/xFWpZ3c3CAUuD/a8Wdf1aMBu+sYzivE7jkGr64UaSXF9lBHqwBUbXg7Qp6kfoVUU5h1ARq9Unvvoonx4hUKQ63GxJjFZHND6ul/9SnKtw4ycaWUuZrSZhRHPBwoW4lKZ+T9GgV/iJsVojDhdJFy8Sj1cU2WhR+tEsmAJtIvtdA/DFFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EU/UsANrLIuJHJQZyVRl3KO3Nn3lqvBBpTugjGoZsIE=; b=QQqJuUuDaAc60gak+anhdN539mruKXcuW+bdlx8519u3zOL1onw6o1cLmpmYsR9eBQA8OwMYKjYoSv2jfhl77DDudcADy3T5MMNIDo9HYvPF+/TRZTw3EQRURaXm8Vf1y2FcdLO44HmC5g4NeNdoKNg0WOVoZ+Hn6bMPO7fG1Mth1PuwjJoFB6YjyZuqD2UsW6zrzSqpv4qod+9MNCOQXi9O+g6zPqVusLauE6x+vsPdEMDe8JAtQ8y/U3lbsyKCBl58pAa4qPPjpKzSJOlz++//ucRMhbjkYGQvuKBAz6qovlzMJo6nUTTYFrUHR0tkOqsgQcVcwmchnmuoa1kNQA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 17:40:19 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%5]) with mapi id 15.20.8511.020; Fri, 7 Mar 2025 17:40:19 +0000 From: Zi Yan To: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , "Kirill A . Shutemov" , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , Baolin Wang , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Zi Yan , Kairui Song Subject: [PATCH v10 8/8] selftests/mm: add tests for folio_split(), buddy allocator like split Date: Fri, 7 Mar 2025 12:40:01 -0500 Message-ID: <20250307174001.242794-9-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250307174001.242794-1-ziy@nvidia.com> References: <20250307174001.242794-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0127.namprd03.prod.outlook.com (2603:10b6:208:32e::12) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: 4b1ec0c4-8235-476d-7c8a-08dd5d9f2118 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: gBsVQo0vJBwP6WM85h9zTOVMLG+dB9y4Q9ABNWmZL9MmK+NTxKoQWU3D0vhoEDw6oNuX5pcgA8h8M8HHDQx76WSXy6YS69NowB8kKpZ8y8ZpXBnaqyKzsfzFPk/Wjip6WJKkOy3wMoXbkpUn786P3Hw93lN8hb1d4WwxZODtQaZDjv5tiTU67oKaJepaZRiS0hPybwraYj9P9Aycv5vxB6kZI69Ph+u1nGr40tGkaF9sW75YXSdxETRKd/8S19uSOoe56pP4L7gJej8SlEmdWi5NohyeG22f5Nvo9UcyuCP4DPg6gSxan4zak7XXkXyGyyAUXgno360g4fr0nmVliG+TrJIrJZ1LG4p8On4FjIeU80Up8v9q2Uh7fK00snDirVGUmiiiIJVJTa/gZxPCHLCY9OGaKJsI2gQ6peqopLCGxYnsd/2YSjLd5GKJiWzdhHO/IRJuFh/P5aJAsf0TklJuDyvxHxbV6hUmS17r2uv4BnOcUsrrThtlyaXPJcMIycGLP7pWEkOgk2o38iKpN7blmJ+8fMIouZDWYGeqfENu0+vTkcD4YJjOxY9+JpPAKlB/9oHGLFCTV5cYS2c/oZ93y9d22xAvLCqGhElYHvNp+OwdokVGt4RNqhXAObUUxSHA+lMzwWyRXlm0qW+7EoC8WyrhM79E4KLphYYPos7s9OVao5YwVITQ/RYadss9ra43hi0K4SjVMRS1FGtwYiuQR3BX49gqZd3APVQVnBMxihOi84Uv/TBZ73zF6mSScfcN4yC+nkgUAmiCPm6nrtIjbXa0t1FwsFyk9JLJblpjTdP14qvcD7GFqiNGv0bghC/4hwIwGLpvLI1B2obAg8vQI0AARJIh/gAt8R368/IhUyi29qjs/V1cH/wEZ+b9icCCmi8o5+SKD5UfPibVc5r5srfMtVobKq0VTu0orwc7VUnU5y3UE3O7Vrk3iJVZqm7wQDfG8AFbo/gZttRkwtvhax1MH83gv9S5cYvMmkwJziorAIJ0SmCtEu6gWKWDrWkBRnSi/onjL21uMqa4bIIEPG7anLf/D8njN1VzpmP67TF11Qry3x8X6qBLNPfbq8JNMx1CBOLHXLy0ELsx5r0zJbOsjp2GfVYgooYosP/h5B873ORusg6OB2p4EF0IHUro3NwDFzO5rWccidOdF6oV3JRY4d8DX0gpms2OT4H71x5w+/4NLZsUMRuK1MP6H8AGnfke2OrUqBijwkyoKVZbdFNMOiQaySfveBjzFc0X+0b1W/ekTl3S49uY+0axegsiYsPJH87KVasF+pD/W1r0L+pXpQvthvC+Gi3bV0AtPMePVKMfAdZKoXBkGraL2LgoWb1WcIZ0sHRyo1xjV/X1DBF74h9NaAb2DiH4sEfEaPXM75BoYNL0L+QfgnwZ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mr04GzhqoY+C1koUb0QHWSny4KwHQ9uVVlqLOXuXr6eyS7zwDt6020H4XUY2NFaBhYuyBN5oa/lXTWwT1LMW4CDOrm3gORKJNYc97L02LOrJDqcVpmoeTp3r2if/bQSQMKjpD/1ky6WUG5TJgh5q/r2TCDX203aueW8m3LAoxdjpPsTEoZ+Z2bFXqh+XxuA2E4edQPZOyP+3O4/wGElmlCS9IlIHfJ0nhTDJRhkdmiUSw3n6wOhkrIa19jfHriBLlLMDcYLKZdc5n7nbZnPbxkuGPKXbK22e9bk1DWS5UIoxBTz9BiUOb4SVH39TSWuS/dIT+PF3L4YWRH74mJbTmNi4Fxatoqq2b6bvyfQJ/AjFRsTnPpL5QEKfScKWtQh0y76sqC2SzxpzE07maimQNVQ1yWvmSoBDSqbLa0EFl3drHxLMfPBDoSMXePiyZpGHDkUOw1QkIbGu/OY3XIWmCDR5ggKdoKUONIwS9V8rgcEbVkaDz4xisqDqmFkFhhSTaenbVG0o9IEXjIR2x59XnFDZRDAnTGKtzdF7CZRT/kdZISagcwTMcLzVYwV6MOOMPueFfPEfMFgZW2Q5Ey9KyONlHteiIRDHyKoGWRaQzie7AOQJfz4ZgH5Yenje0NtsDJ+F770MnZMf6ZrzivaXUdQOqNvo8adA181ceVdyltkItely7f5xsEUczjsnLhIIqJXY3WeI6rdUWyEt38VLkLuPZxbkr3J9Ur0GAa8jxty5bxvwlbjeItplYIasqfLAB4IBwiBUQewaZuA+XwcvkVpwAuDj5vlFZhb0FvlFWS+Yb9sF6MTRw5UvvypY5VyoULoOxswLhLQEe6AB7WQbJqwO303/oQNed+I60gmjWBGInaqON5bYQD3RgeaM9tqT1YiC9MUhAvTLQHSQFZC9x2wo8yF6cvwT6LAlvvQwzDEEG99+2bl2PyJJrTQcMUHEoZEL6zm3oYsWZM3t8LdA3uR3p9WQeI4aXa7mafnOvm1gFztWsB2zmOB5pp7xy8dPoWUOCdzURA99gStr8lch/Q74n/YhbTAQXQ7lsmFz1kb1Dc2tjcULRl1Z6zIqmSWTyEsP8KlK/Np7EFFCvtVx8zFQFSmfeDdk8RyTXxed9f1jcHqr1sIEbhfe9wJVGv/Z1Ah2QeUeEkRJrmcM/qQT69FOd6YqDgEKqGkd68tFJIzaqFdcQ8aUYJEEOHsN85UJawxqQYyCl6WzLdxOmdWkH2h8YWN74Pfhxs5/aKVSVq504dUl02+16AFah6t3L1SS0tu2JNbvfstO8SfjfO24Ee19+FAJivE/AAKRW7diPNF8wtNC1qzH5amKMLoADgpCJBbINXoMlTW64zGyKKZWIKDgI9NRpDgA6toP3u00JdUHKt6nxio6yZnKQuHU6gp4gYBrsb5JGRZmB5v4UaGRktUlDaH3Ms5LcSFhfhLtEq5fTtmNrOZc+ikbJJEmc6JGlth+YlAkj8b04zEPhRLICHHtWcZd+Fq4uCLyQ1MxVEArTCNyWx/DEexXI+XA6qKdLYXq3ZQNExcTNh+Xr1ZzOxvBEIK7n9xoirWeFcZmBoDEHgfh7xzOle9wN35l8L8W X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4b1ec0c4-8235-476d-7c8a-08dd5d9f2118 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 17:40:19.4359 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: W+2Ndelv22zYigcxYPvj5DilNrWhQy4CAnU8GMT1MGsFeFkyYovsKdKAPX9hA3zk X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 01075A0017 X-Stat-Signature: 7nqszbbbgg5grcujciof863xtjf6ix8w X-HE-Tag: 1741369320-59976 X-HE-Meta: U2FsdGVkX1+4N1400t285SU3ptP3YE+yd6FAVl6Y3U8dnO6W0qujvDwCzJBuALZZPXYwRI7X+ToGFFIpR6TyfDJOMbi+IHOOl2tYIAU39MrH+f7j74HnZf9NWGkJbIs8Wcb4NLk4Y3yjwYvRA5pRUZvRz9bLCVuBKkceB0oxXWNwZlBo1iTL5inDpBmNilz46ZJMf/QdwERPnr0dRW5U5KbiTtZlgTj9+k2no/EoV5MTn1bV9PK+GqqUSgbIgutiu/YQx6vPqPfw0t9odM5srEicMJSglIcgZC30QYQapjqhrk53yRhfHcu9UBcKpk2Ko4CPem/gFSxIpR5EHGsl2tdV7CPSkyEQp8z+rl1qcSwUlSuW6GDD7G/iwLM15nYbPkLY4QeLK8DAkG8DNAe9M7kMmrGETIUO6zUpiCWAJf1vHq3KHubrgSL+uIaLMBrf6w66L95Ny+JmKbP7VRgyem7W5z6A4zzWx547LYoGwft1lvA4fjV8SPJuNVGkNDzFWS9riJliP2JgUjTiS1vIFRb64d+sS4OyIoUDFLyImOdMPyUi56LMNpaFcEWjGXPt/ECRSeuUyc9cfBR7I0o4oSZ0aJrRn7sy2HqQW38knDxRuwhbz2M10YSUrVANFvY5ZcZqctpNQLacSOwPWqCGIWVOOlQB5M9zsRebdRt7jsj84RtiiB9H7ryyKKdAFYiTd7tc2oZnB7vZC7DS9SmXByAFCUt8QOSznKDjPu+aKjW7u3GqxoARfqS1tZ1eVKcxwfQhEhvgLWA6LiJb7y2KpmvZjnNXm8KFPs6ORu+ExTZEzjoo5fvk1oD8FU1hxTk1AMV3hpGGZKTjx0kewlJKqadmncefXh0bBd3/wGeGNWK7L4k9NKUX1JENC6rlCOVfHCY9z9Gv5TDtfu/cl6q2uauOheJ1b9kLYgTdisbN+qtuqsGfdYmUiDQy0g/GaTTBZuPr65ICB3dyefPv0Ll Cz8YFAPc MGIGNVyKzLWX4OjW5FFNtc4oVs3meGENMotuL5QzhZ/8v7tFlkwge29x87hrQxW3xoKqNruW2PSV8JgLXiaVoKC6+8UeQsILCWkWF481pmwE+GVEWaqCn/BnHDJMhoO5oZRlUlc+1f07+XTvz8xp/SLUT0iEWLjgfrTifn1LnRvTH7pQsVACZ6KTY6VefAFQRAxxM2mE1Hqce/DH1llJS+gvj2domNm1bgnnbqGRjKNHSdesJO70ZkUOgechTC6erIlpEmmsgISEiyNFaSv/9i5RinIlBhxrcwH1uAVemxfrZ0KW9Vgh472u0QM7P0QUxokGGPiePeb2yjs4QANZ6CKewH5E+E61uLNWUyRYxcrn4r9/ovhTRk2jmmmEwWC3HS369u0uGMObp9gXG5pPbA5ki5tY/iwx2PDVZgeSQCyN5/94iPwoDVuBDJcOBe1RD88YC/NdBPrZrBYRgeTrAEkvknn4Nnw5Pf3DH8VEoYAD7Iapsb5cxMaglCC6fQC4tatl9zIEml4PY1Ujw8f7F+q5UEgcSWx7QOzkp80JRS6mnHzdhVWYHntSgxUYUtChP0anniO1TiK5/rid2VbfgrRlvR4a4wanQoL5kXNcj+rw2jMZ4gNZhLcmbKaEn7rjseTqlhCgq5GqvcGCqLor9jZl7ovI+t7JfJqznXg3dDoLsHRFx2kjPSI5nRdZ5JblNUVwEnZRhqVlEYly3t4p3JTV9uXEwJwsKttLo//YtKbCQJK3gT6jr8HfEiA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It splits page cache folios to orders from 0 to 8 at different in-folio offset. Signed-off-by: Zi Yan Cc: Baolin Wang Cc: David Hildenbrand Cc: Hugh Dickins Cc: John Hubbard Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: Matthew Wilcox Cc: Miaohe Lin Cc: Ryan Roberts Cc: Yang Shi Cc: Yu Zhao Cc: Kairui Song --- .../selftests/mm/split_huge_page_test.c | 34 +++++++++++++++---- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index e0304046b1a0..719c5e2a6624 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -456,7 +457,8 @@ int create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, int *fd, return -1; } -void split_thp_in_pagecache_to_order(size_t fd_size, int order, const char *fs_loc) +void split_thp_in_pagecache_to_order_at(size_t fd_size, const char *fs_loc, + int order, int offset) { int fd; char *addr; @@ -474,7 +476,12 @@ void split_thp_in_pagecache_to_order(size_t fd_size, int order, const char *fs_l return; err = 0; - write_debugfs(PID_FMT, getpid(), (uint64_t)addr, (uint64_t)addr + fd_size, order); + if (offset == -1) + write_debugfs(PID_FMT, getpid(), (uint64_t)addr, + (uint64_t)addr + fd_size, order); + else + write_debugfs(PID_FMT, getpid(), (uint64_t)addr, + (uint64_t)addr + fd_size, order, offset); for (i = 0; i < fd_size; i++) if (*(addr + i) != (char)i) { @@ -493,9 +500,15 @@ void split_thp_in_pagecache_to_order(size_t fd_size, int order, const char *fs_l munmap(addr, fd_size); close(fd); unlink(testfile); - if (err) - ksft_exit_fail_msg("Split PMD-mapped pagecache folio to order %d failed\n", order); - ksft_test_result_pass("Split PMD-mapped pagecache folio to order %d passed\n", order); + if (offset == -1) { + if (err) + ksft_exit_fail_msg("Split PMD-mapped pagecache folio to order %d failed\n", order); + ksft_test_result_pass("Split PMD-mapped pagecache folio to order %d passed\n", order); + } else { + if (err) + ksft_exit_fail_msg("Split PMD-mapped pagecache folio to order %d at in-folio offset %d failed\n", order, offset); + ksft_test_result_pass("Split PMD-mapped pagecache folio to order %d at in-folio offset %d passed\n", order, offset); + } } int main(int argc, char **argv) @@ -506,6 +519,7 @@ int main(int argc, char **argv) char fs_loc_template[] = "/tmp/thp_fs_XXXXXX"; const char *fs_loc; bool created_tmp; + int offset; ksft_print_header(); @@ -517,7 +531,7 @@ int main(int argc, char **argv) if (argc > 1) optional_xfs_path = argv[1]; - ksft_set_plan(1+8+1+9+9); + ksft_set_plan(1+8+1+9+9+8*4+2); pagesize = getpagesize(); pageshift = ffs(pagesize) - 1; @@ -540,7 +554,13 @@ int main(int argc, char **argv) created_tmp = prepare_thp_fs(optional_xfs_path, fs_loc_template, &fs_loc); for (i = 8; i >= 0; i--) - split_thp_in_pagecache_to_order(fd_size, i, fs_loc); + split_thp_in_pagecache_to_order_at(fd_size, fs_loc, i, -1); + + for (i = 0; i < 9; i++) + for (offset = 0; + offset < pmd_pagesize / pagesize; + offset += MAX(pmd_pagesize / pagesize / 4, 1 << i)) + split_thp_in_pagecache_to_order_at(fd_size, fs_loc, i, offset); cleanup_thp_fs(fs_loc, created_tmp); ksft_finished();