From patchwork Fri Nov 1 15:03:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13859517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13E80E6F069 for ; Fri, 1 Nov 2024 15:06:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AF526B009E; Fri, 1 Nov 2024 11:06:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95EBA6B009F; Fri, 1 Nov 2024 11:06:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D8CF6B00A0; Fri, 1 Nov 2024 11:06:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5C6136B009E for ; Fri, 1 Nov 2024 11:06:04 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 13AEA1C5580 for ; Fri, 1 Nov 2024 15:06:04 +0000 (UTC) X-FDA: 82737850608.24.776AA4D Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2076.outbound.protection.outlook.com [40.107.92.76]) by imf17.hostedemail.com (Postfix) with ESMTP id 1E5EE40033 for ; Fri, 1 Nov 2024 15:05:38 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=Eo2gG95e; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.76 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1730473504; a=rsa-sha256; cv=pass; b=bukAYCrl//sZ11hJE6pmHk5r08KvVgHk6Dtdn0Dd/xPjXwILytSDe2mI5c1gK/6CrlNdAM UgIZYlndKczpUp3a0SlDVrratpEJrOc0hKPSpepumevFplYFmVOPVOrdPO7kL0G26l9lon raIaBf5agTzw/6G3KsowCHHh/NffISY= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=Eo2gG95e; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.76 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730473504; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wKioLr1HycFarfrZhAg1ByCjDsmUJhCDoEV444XxKIE=; b=ZbUHV3LO/Hm7KdbVQaUpUQ6IBBCXv51336haMlRgUy7mUSV3w4VQhw03uxrbXyGH5SZsMy 8mnbAG9NYq6k5aRuRvVYOyaVobjWIv4uXJ3pYFHTVSvkdxoGl4UikiZZTokZNOPLersraX d5N9QpMbIF5xcKCg0T+R7BvMkmGARkA= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=i+3dZDJYJTsFXi9IjFNOva9oVaYsqWUklTS8atRzfxKEtDPWplFaK1j11CjkgkkYitLDkj4rU1w6O10iIKxuPlQvJVWMP3O704zJbv6QsHkxR1yzcsJZbwCQNR+xves89uMMA64LbyGxjraPKWS1PQm9f6FQ77NkgRY0FQQu9y6rjUHx6Vmw/Bm4tLcuuR1RrMCmwDMhVig2c4qPUcTLFrApw79unqTXdApYhrD+lTkkIH5CKt4ZmykjcF3TpPEzURleodkBFs/OFCyS6fnrq3ytNiYpkEJUaBNaJOrJk5/oXGYyy3re/L4FvrTMyuHKEzWNARKFRJavT4KiwioHIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wKioLr1HycFarfrZhAg1ByCjDsmUJhCDoEV444XxKIE=; b=MZRlOAbGcbwGp9du39Pm12CdH2fmLgkQs97foLeWcHrTiCj8rXj72exvHlwTb7UkgZ9fm/BzsKkJs53+rTzeeVUmnW6Yf1V2hjr30HImlkiLJ6SlA/mc2JkkcGY73Ga0YOAWr8pguOBzVJWJr3TPzpHo/58UWH4qBivwDpuPZwvaMkfC2mU56tb0iI2B8ox5PzITx8NcWD43+S0hCQkkJjRUIi3o8G58GWiv4q6dx20x2VZLhdzNbtg9E1YZnCrQE6wWUzP3hoioNO/sZIY4BgGi/YZx7fD5voqiBHHSMOpFN21BAI/nhbxEO9RPbMjQFUrxU0YGka75Spx215a4kA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wKioLr1HycFarfrZhAg1ByCjDsmUJhCDoEV444XxKIE=; b=Eo2gG95e23JftwobdxP9NxR0XvbDR7F191JuyHUjKUdZIDNjf3MZFK0fZYLKjFL3X04/Qz7gJBUmcTJ0Png1kR2mYwNfC1Mi153c0KgmiOXYNm/FkhrXDkFYm+fIzbrCz5SpsnoYSA3dx38aqHNatgI+qzao5DerHfi0pyIpLKuyDkIbBcsyCTktjll2Z3PsVI4BMUTXWpr3oqBfK+1PRqs3NNEPP99B8naB1HTxkBXYjnDyGgyXbvwydf15Vq5pZzWZvM2V1k1QeBopfLI6DQHCzfLQf8buuhDdcgr2UyG95/WxXMF8iTTMz2SR84nh7XHWkDH8hpJWGsBh6Nl5bA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.20; Fri, 1 Nov 2024 15:04:14 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%7]) with mapi id 15.20.8114.015; Fri, 1 Nov 2024 15:04:14 +0000 From: Zi Yan To: linux-mm@kvack.org, "Kirill A . Shutemov" , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , Hugh Dickins , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v2 3/6] mm/huge_memory: add buddy allocator like folio_split() Date: Fri, 1 Nov 2024 11:03:54 -0400 Message-ID: <20241101150357.1752726-4-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241101150357.1752726-1-ziy@nvidia.com> References: <20241101150357.1752726-1-ziy@nvidia.com> X-ClientProxiedBy: BN9PR03CA0104.namprd03.prod.outlook.com (2603:10b6:408:fd::19) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: 3f806450-3353-41e3-a9b0-08dcfa867195 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: 2xLtUKd59diCwbYnvUE1rCj9K5pkd6OHA+BwvnMsSwf0rPYwzuVQT584elsL/WnyE4Wqtj1A/Wvs9Lgb/UPD0p+1eQ7G1ZCPuc/Gelgrm/l2lGhzH55sQOrfEU5A8kBQOTpSymrx+Dxh6RW7OpQk5bithpI66SZ7XO0+k/srY+T2Pzibgbi7QVUHfSEy+Xpe9XPHKVOBNfsEhbmAOz3FE1DYgQcJHO9snUlwCzPqQX6JhV/ycdDDV7jZ0jROpres4ya7JEFIWgXRl9GKA0THAZesSAg3ncoeNTU57i3hBwauuo/0+/RSSX/oZwPYokIk86sK+/KI7QmXrdkQBYB9v90WUs0nGpS1I/6aSGs+HpR1T/15DyNY3dFmj8npqYljZOnzW3lpLRH6ocwrEKGXPXmNbm13J2MHl3ywRgkq6HSqRbZbJDijonh4NGFmyLB75h07GXYGjbXcPVm61x39k83iopeiqF8KJ1/Xk1Hx9oQwHud5OZxtcmeUQYQiM0K8KTYTnlO01Cqq2DsdCG6cuwPOShXcT1byDGNe+yfWBVvDkH9yPJ4AieZI+8hW5Zw1YfL1aXUKRyyWDg+KOROJPtbwpCPIZyPnXQwAFHo7c7g5VCvSJt3N8G5h7WiqyOTRUaEOhLJfu5jFOeLe2oOaDGCr8fXf3LitmeOvlim3b7H0bejY3yLmAJIC6BbUUjJ515TsVjeEa5MKaYBGjf5m0uARLFhKkPn9ZplcM88bu/57vVnSrGOjaLelTLZcmNyZf/xKZR0LO4wRg4N2FyS52pryN+kQyD3dIp4UwY5JAJqyPgIFax7UIYk2vAI01/GweQeL5h2IoHsweUVSr5rExoSQk1vkAkBviJ7ti4ZuesrxZ6g5L9We64kfXmdCR8BLlvRTXYKdB6VHl30AOjoHSs12aMoqOu8QY0vF+H4oMPSDz3sl+bdWWwxKOn/pBHQ2woTpfHV9a6bgBgfMndd8Dny/me/AJ0gFilHdVuZ7ziSKkKkXuYIkLGF7QP0fi5KkS5cJDc90fGqfdIw+eURdBmRHDNCFIrllcl+R6AjaTolQEjeg1aGbzgbq3klO1DmCezujTy680vaoJu6y/qB5At46AJRHDViTZvXr2HAmOdnp0RiRBM76U8GZJRUzlhU+NxOkFZfmTBdbs/XJVvnla9YZwiDSCyMD3pz4i56mNccISekyRsRBp5OZXTiTHZaHjQaAFyKFZXyg6zh1Hb/6b/5Bv9EQGwGNPvazFI8M+JMqLSRDr5mvB3H6MQSaUOnK5hD0Q9ocQiipx4A4cd24fswboepYIo3FVcjllelGsvXjvXcAp0hHAPYN5cgF045h X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +uYK1geha0o2VRJocO2907wiRjG6hNi1tdzSGb4a+k7SWA8qRq0BNj0+IJ0xG48smVVbH7rlo1ZS0N48jqD7GbHPz5xOebEr8A0KQTolUET6eRMNO34ooVB+2/0yrx3/s+Q4fze+Blsu9KmHVuysLa8q/I3aZDrSox+0pF0glfc8G9qlEjjXv+ZzHK72kTr+602CrAV5LStC20lcq3n67l4xffjcBH2xyEQjaCbNS3JzthbVqIWI6fWGGOSHgp7ciE/kMBN+vQpHCqdOR1PSM4dD64Stlc4r6hee0ZnxDKcqHTuC540+ZrOfaEQHPRwcbHgds+w7W0Xsp1+wwpSHjPeeAtFLOPTOgsd76axwp3HEfTQyTNHSEu1/Gdy+rn6D2q3J1Qxq87149PASUOoUUJsxOR7UETNjXfjFAGnN+20W+xOPiQnOBIEQADoj8a/AJDpwT/uiQRw2k20VF46Zn/8TXGpVM8NvY4DYfR/8IskLuUmhT16QxNp+Qz0LhaPIVjlufhhrtbmdaRQwf9t3anT/nxOmyen2/lYtkKLmsevpwA9NoTvCIAqSsSrVSODKcK3GGkLC7pDxTMF7wCvIta8RyHtPJ27kfemY2/bOJ9Dy/7Qf5eLfMXy+VVFEbofhIalzf6ezEYxKDX1miYX315uGZxP1x9QdiCgtMWqH0IEbBulgXPvKqCsrz9oN4cw9ZeVAjt/k7fdT9aSC6Z27QrZbvF9cQlJSn6richBMalHhtm/JOpZrsNThy/MVA/L1TACCFYwc6UN4kz+wqwTqpLQ3L1VBGNCeletXn4sDK74IPnzAjLhfdVzqXYrwytITOHFHreBnKEs9N5OGT6/LthCdVkjv6IRYrlevkf07fOdh9iOF8eSIAyCO4znxvYxKf8VKy9fVcHaVkVSA1ZL7tEib9ankGruJV4WnO4QX4IMLqVSLw2jPp1DJO0gSV1QAru59W9l/IcdWWcf/ZL2a0h+yalV/xtoE4qOaL7v3qk3m5Wya+C0ohPw6BjH55eM4Q+MriahsoRP2uaBOluAIjYlHbQamMGxfU/tA/wcZJ4hrdWW3t/mNM8hxH1+b155Y3xoG1ctN0CfSTAebLbyJLIR8Nn5Yy4o05c7HK6sB0LVMRpjmH/Ldz5bCNVKQRY6QAAj8FJEYhbxoBIse0XXaDERqT6LUID5r5IC9SCTm/28DWW7QYAC0Y9vPftIr+f2ULLV7mdHdyEWNldjknPsOjrOe6F1uyYmvsI+t+00hwbeKfnwidhpJ2FNhAmBcWcE1Lb1J0/JgfGG7WhtuIdF/Go856HVrwA1qrjRPujOyDj2XIHvPzTjwZuRCn33IXqI7d7B1CE7lK7H911GWPO6LHZ1bdzCddu4dc3axwP/1kwYR8zJSLQpEYSpGdVEFdlunFAj8lIB8koesZDmBAdK9PzW5P4fvk34FRm0Za77qVBquJ3cpHBIKgxmddn5csQgZTef9eaFoFyiXuRYCK1x51OvzNCtsJkvCRJSeuIlVIQCn6TcCAAkhQz+oHq6Lod/akUN0SFpLB7T+TJzLqiUSFDw80pXH1grj3VHxLIx8kkl/Yt6UBhxTNT5vepWOb0Iv X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3f806450-3353-41e3-a9b0-08dcfa867195 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2024 15:04:11.9369 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gWhB/mhHh1S3cKk54Ffm4G7e0O59KhunT/Sq8f1zmTpfByb/sskKmI2gVP/xkEal X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 X-Rspam-User: X-Rspamd-Queue-Id: 1E5EE40033 X-Rspamd-Server: rspam11 X-Stat-Signature: o4hm3br8tnwogyw7ncdn1tud8ojmbyuy X-HE-Tag: 1730473538-127116 X-HE-Meta: U2FsdGVkX18Yczq1j6SksBIM8tHjOV4p2sudviwdAFfpGBbDrW1f+oe13ctWdobuor4hqVD2d1WIMr832u+XVWJJsK3Z2Y1aN+060bMejKWDbFViZ9S+RIcxxopLJn15P6nkadmfSsRm7m5IzfOh8q/E9is06wkvNrvTQ3P6z4aLh8Angc6YG2l5HxFXNlkxHm5R6ldgXQvvjUaUAJPOqGnZFC23AuA0F+b2FmiMGxbA1OzlpQCr7oCAJmS5vBedno/lt2zLOcXE6P5TvgYoHbN/ktWSxrW/jvNtfA4xTevbcD4OaD/qOz5JnCPYONg6W1hBb+NH5hufos8ZcCKZ1Iqh1ZmJazqQCkrIvXg1d4NvacTe6OWaaeQgfKJ8NHhdQO6r99lXhf+C8ioSAViXaLdpCWfOfaXUl+zI7GIHTegbmbna6Cs2Llj+UMqCpBgZFWY1iy9ZY1oZASrDKp3uWfuo8mwSZF6XnKyO4M8yg/Vkn6HfYZKUhYZZyZtLZ2ceeJd+L+JHrxti0eQpUPOeQAHOvFxs5BwxZ0tsfNvEvuTkeOPMITruX5rl0ynt01h3JUR347JOnobzSL/VPJWJEtWeeDXtpbZy44jopqObd4OfoIZPt9/LGSoU4kC8BSOIGyNgt4x1bXPDc/woqNQVrcXMQxdbWZCZrfdARrMQwoUTGVrYKDxrymsucSjT4C6p4nHyLx4dQ07C6KLIlgT4iVuFos17DIK4WH1vy7vpa0othaxQw1xcDSig+LxjZ5Don3MLx8Elqmsf3/oH5XEcC8lcn11EQ6bpTAy747mah9kb1+C81t7mfk6jv+p1zUp3MwV5O61YnIU8DoXOeZRtuciBfghOPR0HIVaxMU06wdIpbS/U5I/ujtwSrKGMpAZLRbQdHT5N9TNRq0w4TXAteXHnrwkwocT3EGqfLqGaJfG+ABIL4BZKmCViG8cUNZIWKRcSVzORGY340BGXOgP nDc81wnW MmuUDVCeUARaI9KDC2Bt9hEW2ihraMqQYZM23im7MPJZPVjMh9FClVrQH5agpEuLzIKK03boWskrZ0V/bpjofUtGC/bIa1AWknZR4tjesoRCJkF1QKVLro9bcmHnbeSkVdAgaB3t092iBToXb1Vy2zz5tx57Ul0llZ2YNqSO5vbSYHr/fL5yhrfjmWML1P7bQ3jJyNdgcY2SZPcj9WZon8xzdz7JnbxxGcud9lmLodO6shRGJXVcsWZhUlkyTHSYhbTfDeVKakkOVGJ1gP6ZOOgfuF2Py+oCtHbIM7/dzRWtabuqLQ20dq3kOR6Dx5k4iUPB3Fs6baxDehVAQAPnBi8tV0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_split() splits a large folio in the same way as buddy allocator splits a large free page for allocation. The purpose is to minimize the number of folios after the split. For example, if user wants to free the 3rd subpage in a order-9 folio, folio_split() will split the order-9 folio as: O-0, O-0, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-8 if it is anon O-1, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-9 if it is pagecache Since anon folio does not support order-1 yet. It generates fewer folios than existing page split approach, which splits the order-9 to 512 order-0 folios. folio_split() and existing split_huge_page_to_list_to_order() share the folio unmapping and remapping code in __folio_split() and the common backend split code in __folio_split_without_mapping() using uniform_split variable to distinguish their operations. Signed-off-by: Zi Yan --- mm/huge_memory.c | 56 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 63ca870ca3fb..4f227d246ac5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3706,11 +3706,10 @@ static int __folio_split_without_mapping(struct folio *folio, int new_order, } static int __folio_split(struct folio *folio, unsigned int new_order, - struct page *page, struct list_head *list) + struct page *page, struct list_head *list, bool uniform_split) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); - /* reset xarray order to new order after split */ - XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); + XA_STATE(xas, &folio->mapping->i_pages, folio->index); bool is_anon = folio_test_anon(folio); struct address_space *mapping = NULL; struct anon_vma *anon_vma = NULL; @@ -3731,9 +3730,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order, VM_WARN_ONCE(1, "Cannot split to order-1 folio"); return -EINVAL; } - } else if (new_order) { + } else { /* Split shmem folio to non-zero order not supported */ - if (shmem_mapping(folio->mapping)) { + if ((!uniform_split || new_order) && + shmem_mapping(folio->mapping)) { VM_WARN_ONCE(1, "Cannot split shmem folio to non-0 order"); return -EINVAL; @@ -3744,7 +3744,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, * CONFIG_READ_ONLY_THP_FOR_FS. But in that case, the mapping * does not actually support large folios properly. */ - if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && + if (new_order && IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !mapping_large_folio_support(folio->mapping)) { VM_WARN_ONCE(1, "Cannot split file folio to non-0 order"); @@ -3753,7 +3753,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } /* Only swapping a whole PMD-mapped folio is supported */ - if (folio_test_swapcache(folio) && new_order) + if (folio_test_swapcache(folio) && (!uniform_split || new_order)) return -EINVAL; is_hzp = is_huge_zero_folio(folio); @@ -3810,10 +3810,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto out; } - xas_split_alloc(&xas, folio, folio_order(folio), gfp); - if (xas_error(&xas)) { - ret = xas_error(&xas); - goto out; + if (uniform_split) { + xas_set_order(&xas, folio->index, new_order); + xas_split_alloc(&xas, folio, folio_order(folio), gfp); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out; + } } anon_vma = NULL; @@ -3878,7 +3881,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order, if (mapping) { int nr = folio_nr_pages(folio); - xas_split(&xas, folio, folio_order(folio)); if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { @@ -3896,8 +3898,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order, mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); mod_mthp_stat(new_order, MTHP_STAT_NR_ANON, 1 << (order - new_order)); } - __split_huge_page(page, list, end, new_order); - ret = 0; + ret = __folio_split_without_mapping(page_folio(page), new_order, + page, list, end, &xas, mapping, uniform_split); } else { spin_unlock(&ds_queue->split_queue_lock); fail: @@ -3975,7 +3977,31 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, { struct folio *folio = page_folio(page); - return __folio_split(folio, new_order, page, list); + return __folio_split(folio, new_order, page, list, true); +} + +/* + * folio_split: split a folio at offset_in_new_order to a new_order folio + * @folio: folio to split + * @new_order: the order of the new folio + * @page: a page within the new folio + * + * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might be + * split but not to @new_order, the caller needs to check) + * + * Split a folio at offset_in_new_order to a new_order folio, leave the + * remaining subpages of the original folio as large as possible. For example, + * split an order-9 folio at its third order-3 subpages to an order-3 folio. + * There are 2^6=64 order-3 subpages in an order-9 folio and the result will be + * a set of folios with different order and the new folio is in bracket: + * [order-4, {order-3}, order-3, order-5, order-6, order-7, order-8]. + * + * After split, folio is left locked for caller. + */ +int folio_split(struct folio *folio, unsigned int new_order, + struct page *page, struct list_head *list) +{ + return __folio_split(folio, new_order, page, list, false); } int min_order_for_split(struct folio *folio)